Does ChatGPT Use My Website? Training, Search and Live Fetch

ChatGPT reaches your site through three user agents: GPTBot for training, OAI-SearchBot for the search index and ChatGPT-User for visits a person triggers. OpenAI's crawler documentation lists each separately and says each setting is independent of the others.

• GPTBot is automatic and feeds model training.
• OAI-SearchBot is automatic and feeds the index behind ChatGPT's search features.
• ChatGPT-User is not automatic. It runs when someone's action in ChatGPT or a Custom GPT calls for a page.

A fourth agent, OAI-AdsBot, only visits pages submitted as ads and is not used for training. The table compares the three that matter for visibility. The study column counts sites whose robots.txt disallows that bot from the homepage.

It may have, if your pages were public and GPTBot was not disallowed. OpenAI says its models learn from publicly available web content, and that site owners can disallow GPTBot in robots.txt, the plain-text file at a domain's root that tells crawlers where they may go (definition).

Three facts frame the decision:

• OpenAI says its models do not store or retain copies of the data they train on; they adjust parameters to reflect patterns.
• OpenAI says it does not intentionally gather content from sources known to sit behind paywalls.
• For ChatGPT Atlas, OpenAI says it respects the GPTBot signal for content acquired through users' interactions.

Training knowledge cannot link to you. A link in an answer comes from search, not from training. That is our reading of the documents, not an OpenAI statement.

Our position: blocking GPTBot is a defensible choice for publishers and licensing-first sites, and we hold that most brands gain little by doing it. The reasoning, and the real exceptions, are in Should I block GPTBot?. Neither OpenAI nor we can measure what training exclusion costs a brand.

Only if OAI-SearchBot can reach it. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Allowing it is the single documented step toward appearing.

What OpenAI's publisher FAQ and search help add:

• Any public website can appear in ChatGPT search.
• Allow OAI-SearchBot in robots.txt, and make sure your host or CDN allows OpenAI's published searchbot IP addresses.
• Robots.txt changes can take about 24 hours to reach OpenAI's search systems.
• If you allow both bots, OpenAI may use one crawl for both purposes.
• ChatGPT adds utm_source=chatgpt.com to referral links, so you can see the visits in analytics.
• Ranking uses several factors, and placement is never guaranteed.

The CDN point is easy to miss. In our study, 24 of 88 e-commerce and consumer homepages (27.3%) challenged or blocked our self-identified research crawler. That was our crawler, not OpenAI's, but the same bot protection can quietly keep a store out of answers.

Search providers are a separate route. Our page on whether ChatGPT uses Bing covers what OpenAI and Microsoft each confirm.

Yes, when a person triggers it. OpenAI says ChatGPT-User visits pages for certain user actions in ChatGPT and Custom GPTs, is not used to crawl the web automatically, and is not used to decide whether content appears in search. Because a user initiates the visit, OpenAI also says robots.txt rules may not apply.

That has two consequences:

• Search opt-outs belong on OAI-SearchBot. OpenAI tells site owners to use that token for search opt-outs and automatic crawling.
• A robots.txt line is a request. RFC 9309 states that its rules are not a form of access authorization. If you must keep user-triggered fetchers out, use firewall or login controls. That is our inference.

In our study, 20 of 384 sites (5.2%) disallow ChatGPT-User, 14 of them publishers. Given OpenAI's wording, treat those lines as requests rather than guarantees.

Search your access logs for the three user-agent tokens, then check the source IP against OpenAI's published lists. A user-agent string alone proves nothing, because anyone can send it.

OpenAI documents these strings: GPTBot/1.4, OAI-SearchBot/1.4 and ChatGPT-User/1.0, with version numbers that may change. The search and training crawlers may add a "robots.txt" marker to the string when fetching your robots.txt, which separates those requests when your logs omit paths.

A first pass on a standard access log:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User" access.log | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head -20

Then compare each IP with the ranges OpenAI publishes: gptbot.json, searchbot.json and chatgpt-user.json. The dependency-free log analyzer in our crawler directory keeps the claimed token, the category and the IP check as three separate fields. It ships with synthetic sample data only.

A crawl is not proof of indexing or citation. It shows access, nothing more.

Write one robots.txt group per user agent. Each group is independent, and crawlers match groups by product token without regard to case, per RFC 9309. Pick the stance first, then the lines.

Visible in search, kept out of training (19 of 384 sites in our study):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Out of ChatGPT search as well (16 of 384, mostly publishers): add User-agent: OAI-SearchBot with Disallow: /. You give up citations, and you may still appear as a bare link.

Fully open: allow all three, or leave them unmentioned. aeoeye.com allows every crawler except private app paths, and names GPTBot in an allow group, because our business is being found.

Three traps to check:

• A catch-all User-agent: * with Disallow: / blocks every bot you did not name separately. Seven sites in our study have one.
• OpenAI says a blocked page's link and title can still surface, which it describes for ChatGPT Atlas. Its fix is a noindex meta tag, and the crawler must be allowed to read the page to see it.
• ChatGPT-User may ignore robots.txt, as covered above.

Our AI robots.txt generator writes these groups for you.

Fetch your live robots.txt, test each token against it, check your CDN, then ask ChatGPT a question your page should answer. That takes about thirty minutes.

1. Read the file. Open yourdomain.com/robots.txt. Look for groups naming each token and for any catch-all disallow.
2. Test, don't eyeball. Run the rules through a real parser. The directory's 24-case RFC 9309 test suite is a good way to check a parser first.
3. Check the edge. Confirm your CDN or firewall allows OpenAI's published IP ranges and does not challenge OAI-SearchBot.
4. Run the live test. In ChatGPT with search on, ask a question your page answers and see whether it is cited. Then paste your URL and ask for a summary, which exercises the ChatGPT-User path.
5. Check analytics. Filter referrals for utm_source=chatgpt.com.

If you would rather see whether ChatGPT names your brand for unbranded buyer questions, AEOeye's free audit asks three on ChatGPT and shows the answers.