How Much of the Web Does ChatGPT Actually Use?

OpenAI published exact figures only for GPT-3. Its paper says the Common Crawl portion was 45TB of compressed plain text before filtering and 570GB after, roughly 400 billion tokens. For GPT-6 Astra, OpenAI's public training summary gives only a bracket: text of "more than 10 trillion tokens", the top option on the EU template.

The 570GB figure is the Common Crawl part alone, not the whole GPT-3 corpus. The paper's mix looks like this:

• Common Crawl, filtered: 410 billion tokens, 60% of the training mix, seen 0.44 times.
• WebText2: 19 billion tokens, 22% of the mix, seen 2.9 times.
• Books1 and Books2: 12 and 55 billion tokens, 8% each.
• Wikipedia: 3 billion tokens, 3% of the mix, seen 3.4 times.

Those weights were chosen, not proportional to size: higher-quality sources were sampled more often. Wikipedia, the smallest dataset, was read 3.4 times while the crawl was read less than once.

The tables total about 499 billion tokens, so Astra's bracket is more than 20 times larger on paper. Treat that as a rough comparison, because the two documents define "size" differently.

Astra's summary also confirms the training mix is not only the open web. It lists Common Crawl, data accessed through partnerships, synthetic data generated with earlier OpenAI models and, subject to privacy settings, interactions from products such as ChatGPT. For GPTBot crawling it gives a collection period of approximately March 2013 to May 2026, and says the crawler is designed to respect robots.txt.

Common Crawl describes its own dataset as a sample. Its FAQ says it does not generally archive entire websites, only "a randomly selected subset" of each. Its September 2026 crawl holds 2.17 billion pages, 361.4 TiB uncompressed, from 40.4 million hosts and 33.2 million registered domains.

That is one monthly snapshot of a rotating sample, not a cumulative copy of the web.

We found no authoritative count of the whole web, so we do not quote a "percent of the web". Any page that does is guessing. What is documented: Common Crawl is a rotating sample, OpenAI filters it, and the filtered result is one input among several.

CCBot, Common Crawl's crawler, checks robots.txt before fetching a page, per the same FAQ. The FAQ does not say a block removes crawls already collected; it points to a separate opt-out registry for that.

OpenAI publishes no fixed number, and it varies by search depth. In a 2026 audit of 712 politics, health and environment questions put to four AI search engines, ChatGPT with web search cited 14.7 sources per response on average. That was the highest of the four, ahead of Copilot (8.4), Perplexity (7.9) and Gemini (7.3).

The count moves with the topic. ChatGPT cited roughly 20 sources per response on environment questions, 12 on health and 10 on politics. Citations are also a floor on pages read, not a ceiling.

Search depth changes the picture. OpenAI's web search guide says deep research "often" taps hundreds of sources and can run for several minutes, while plain non-reasoning search returns the top results with no planning step.

The audit found a long tail. The 25 most-cited domains made up 29.9% of ChatGPT's citations, and 82.8% of the domains it cited appeared twice or fewer times. A page does not need to be famous to be read. It needs to be one of the few that fit that particular question.

Limits: the audit covered three topics, drove each engine's web interface, and counts citations, not every page fetched.

OpenAI documents the mechanics, not the ranking recipe. The model decides whether to search, sends queries to a search tool and, in agentic mode, reads results and decides whether to keep going. Which pages win beyond that is observed behavior, not published policy.

Be wary of anyone quoting exact ranking weights for ChatGPT. None are public. What you can verify is simpler:

• Your page has to be fetchable by OpenAI's search crawler: per its crawler documentation, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers.
• Our advice, not a documented rule: lead each page with a direct answer, because retrieval matches pages to a specific question.
• Our stance: one strong page per question beats five thin ones, for Google and for AI retrieval.

For the selection logic in detail, read how ChatGPT chooses its sources. For the retrieval mechanism underneath, see what RAG is.

Few brands do, and publishers often do. In AEOeye's crawler access study (October 2026, 384 sites), 44 sites (11.5%) blocked CCBot, 35 (9.1%) blocked GPTBot and 16 (4.2%) blocked OAI-SearchBot, which OpenAI's crawler documentation ties to ChatGPT search.

The split by group is the story:

Our reading: the open crawl may lean away from publishers and toward brands, who mostly leave the door open. Being in the pool is not the same as being remembered, but a block guarantees you are not in it.

Limits: it is a convenience sample of established sites, fetched once on one day, main host only, and robots.txt is a request, not enforcement. See the study hub and the per-site CSV.

Spend your effort on the slice you can still change. Keep crawlers allowed, make each key fact easy to fetch and quote, and check whether ChatGPT actually cites you. You cannot enter a model that has already finished training.

1. Decide on training and search separately. They are controlled by different crawlers. We allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended on aeoeye.com, because our business is being found. Our case against blanket blocking is in why blocking GPTBot is usually a mistake.
2. Generate a clean robots.txt with the AI Robots.txt Generator, then confirm in your CDN or firewall logs that the bots are not challenged. In our study 60 of 375 homepages (16%) challenged our self-identifying crawler.
3. Lead each page with a direct answer to one question. Retrieval reads pages that match the question, not pages that match the topic.
4. Test, do not assume. Ask ChatGPT your buyers' questions and see who it names. AEOeye's free audit asks ChatGPT three unbranded buyer questions live and shows the full answers.