AI Crawler List: 22 User Agents and What 384 Sites Block
AI crawlers do one of three jobs: collect training data, build a search index, or fetch a page because a user asked. Major operators use a separate robots.txt token per job, so you can block one without touching the others. In AEOeye's crawler access study (October 2026), 1 of 193 SaaS sites blocked any AI retrieval bot, against 27 of 38 news publishers.
Vendor facts below come from each operator's own documentation, checked on 3 October 2026; treat those pages as the authority. This page replaces four earlier AEOeye pages: a crawler list, a robots.txt test suite, a log analyzer and a statistics roundup.
Which AI crawlers should you know?
This table lists 22 tokens from eight operators, each tied to one job, and operators control each job separately. OpenAI puts it plainly: allow OAI-SearchBot to appear in search while disallowing GPTBot to stay out of training.
| Token | Operator | Job | What the operator says |
|---|---|---|---|
GPTBot | OpenAI | Training | Signals content should not train foundation models |
OAI-SearchBot | OpenAI | Search | Opted-out sites are not shown in ChatGPT search answers |
ChatGPT-User | OpenAI | User-triggered | Robots.txt rules "may not apply" |
ClaudeBot | Anthropic | Training | Future materials excluded from training datasets |
Claude-SearchBot | Anthropic | Search | Not indexed for search; may reduce visibility in Claude results |
Claude-User | Anthropic | User-triggered | Claude cannot retrieve your page for a user's question |
PerplexityBot | Perplexity | Search | Surfaces sites in Perplexity search; not used for model training |
Perplexity-User | Perplexity | User-triggered | Generally ignores robots.txt, since a user requested the fetch |
Googlebot | Search | Affects Google Search and its features | |
Google-Extended | Training and grounding control | Controls Gemini training and grounding use; no effect on Search inclusion | |
GoogleOther, Google-CloudVertexBot | R&D; Vertex AI agents | Generic R&D crawler; Vertex AI crawls requested by site owners | |
Google-Agent | User-triggered | Acts on user request; such fetchers generally ignore robots.txt | |
Applebot | Apple | Search | Powers Spotlight, Siri and Safari search; may also give AI answers context |
Applebot-Extended | Apple | Training control | Opts content out of training Apple foundation models |
Meta-ExternalAgent | Meta | Training or indexing | Crawls for AI-model training and product indexing |
Meta-WebIndexer | Meta | Search | Allowing it helps Meta AI cite and link your content |
Meta-ExternalFetcher | Meta | User-triggered | May bypass robots.txt |
Amazonbot | Amazon | Product improvement | May be used to train Amazon AI models |
Amzn-SearchBot | Amazon | Search | Eligible for Alexa search; not used for training |
Amzn-User | Amazon | User-triggered | May not follow all robots.txt directives |
CCBot | Common Crawl | Open dataset | Stops collection into the open Common Crawl repository |
Three traps account for most misconfigurations:
- Google-Extended is not a crawler. It has no user-agent string; Google crawls with existing agents and reads the token as a control. For AI Overviews and AI Mode, Google says a page only needs to be indexed and snippet-eligible, so Googlebot and snippet controls decide.
- Many user-triggered fetchers sit outside robots.txt. OpenAI, Perplexity, Google and Meta say theirs may or do ignore it, and Amazon says Amzn-User may not follow all directives. Only Anthropic documents Claude-User as controllable. A firewall rule stops the rest, but it also blocks the user who asked about you.
- Unnamed bots inherit. Apple says Applebot follows Googlebot's rules when robots.txt does not mention Applebot. Amazon says Amzn-SearchBot follows the rules given to other search bots.
What do 384 well-known sites actually allow?
Brands almost never block AI search bots; publishers often do. On 3 October 2026 a self-identifying research crawler fetched robots.txt, /llms.txt and the homepage of 391 well-known sites. Seven were unreachable, leaving 384 usable. Download the CSV and JSON.
"Blocks" means the bot's effective rules disallow the homepage /. A partial block such as Disallow: /checkout does not count.
| Sites that block… | SaaS (n=193) | E-commerce (n=90) | News publishers (n=38) | All (n=384) |
|---|---|---|---|---|
| GPTBot | 4 (2.1%) | 5 (5.6%) | 20 (52.6%) | 35 (9.1%) |
| OAI-SearchBot | 1 (0.5%) | 1 (1.1%) | 11 (28.9%) | 16 (4.2%) |
| PerplexityBot | 1 (0.5%) | 4 (4.4%) | 26 (68.4%) | 35 (9.1%) |
| ClaudeBot | 3 (1.6%) | 5 (5.6%) | 30 (78.9%) | 44 (11.5%) |
| At least one training bot | 12 (6.2%) | 9 (10%) | 32 (84.2%) | 64 (16.7%) |
| At least one retrieval bot | 1 (0.5%) | 4 (4.4%) | 27 (71.1%) | 37 (9.6%) |
None of 43 AI-visibility vendors blocked any tracked bot, and 5 of 20 UGC and review sites blocked a retrieval bot. Bytespider, left out of the table because we could not verify ByteDance documentation, was the most-blocked token (56 of 384). What stands out:
- Publishers are closing the door on sources AI leans on. 9 of 38 block OAI-SearchBot, Claude-SearchBot and PerplexityBot together. Reddit's robots.txt is two lines (
User-agent: *,Disallow: /), making it the only site in our sample that blocks Googlebot. - The "split" setup is rare. Only 19 of 384 sites (4.9%) block GPTBot yet allow OAI-SearchBot.
- Missing robots.txt is common among stores. 30 of 384 sites (7.8%) have none, including 15 of 90 e-commerce sites. Per RFC 9309, a 4xx response lets crawlers access anything, but a 5xx means they must assume complete disallow. A broken robots.txt is worse than none.
- Bot protection quietly blocks. 60 of 375 homepages (16%) blocked or challenged our declared crawler, including 24 of 88 e-commerce sites (27.3%). A firewall that cannot tell a retrieval bot from a scraper can keep a store out of answers while robots.txt says yes.
- Content Signals are a SaaS habit. 36 of 384 sites (9.4%) carry Cloudflare Content Signals lines, 33 of them SaaS; 16 set
ai-train=no. - llms.txt spread faster than evidence. 173 of 330 checkable sites (52.4%) serve a valid one, but no major engine documents using it for answers; see what llms.txt is.
How does this compare with larger studies?
Larger datasets point the same way: training dominates crawling, and publishers restrict most. Cloudflare's June 2026 report put AI training at 52% of crawler requests on its network, up from 22% in spring 2025. An academic study found 60.0% of reputable news sites disallow at least one AI crawler versus 9.1% of misinformation sites. Our 84.2% publisher figure covers a smaller list, so do not compare them directly.
Should you block AI crawlers?
If you want AI engines to name your brand, do not block search or user-triggered bots; training bots are a business decision. For brands, blocking search bots is almost always self-harm. For publishers with licensable archives, blocking is a negotiating position. aeoeye.com allows all of these bots.
By bot type:
- Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot): allow. Blocking them drops you from that engine's answers.
- User-triggered fetchers: leave alone. Robots.txt often does not govern them, and a firewall block fails your prospects.
- Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot): your call. Anthropic says a ClaudeBot block covers future materials, so it is not retroactive.
By business type:
- SaaS, stores, local and service businesses: allow everything. Only 4 of 193 SaaS sites in our study block GPTBot.
- Paywalled publishers and licensing-first sites: block training tokens, keep search tokens open, and negotiate.
- Worried about server load: rate-limit. Anthropic supports
Crawl-delay; Apple and Amazon say they do not.
Do not enforce a block by IP: Anthropic warns it can stop its bots reading your robots.txt, making opt-outs unreliable. For the full argument, read why blocking GPTBot is usually a mistake, how to block AI crawlers and ClaudeBot explained.
What robots.txt rules should you use?
Per RFC 9309, a crawler obeys the group matching its token, and the * group applies only when no specific group matches. Start from the default that fits you.
Brand default (allow all AI bots, protect private paths):
User-agent: *
Disallow: /api/
Disallow: /account/
Sitemap: https://example.com/sitemap.xml
Publisher split (opt out of training, stay in search):
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Disallow: /api/
Three pitfalls:
- A named group replaces
*. AddUser-agent: GPTBotwith one rule and GPTBot ignores your*rules. Repeat the rules you still want. - Rules live per host. Anthropic asks for rules on every subdomain.
- Changes lag. OpenAI and Perplexity each say up to about 24 hours.
Check the file visitors actually get, not your repo's copy: Cloudflare's managed robots.txt, for one, prepends its own rules. To draft yours, use our AI robots.txt generator.
Then test them. Our robots.txt test suite holds 24 synthetic RFC 9309 cases (wildcards, $ anchors, percent-encoding, merged groups, longest match) with expected output. We re-ran it on 3 October 2026 and it matched byte for byte. It tests parser logic, not bot obedience.
How do you verify an AI crawler in your server logs?
A user-agent string is a claim, so verify the source IP. Every operator below publishes an IP list or a reverse-DNS pattern, and a spoofed "GPTBot" fails both. Common Crawl notes that some crawlers falsely identify as CCBot.
| Operator | How to verify |
|---|---|
| OpenAI | Published lists: searchbot.json, gptbot.json, chatgpt-user.json |
| Anthropic | One published list |
| Perplexity | perplexitybot.json and perplexity-user.json |
| Reverse then forward DNS; the name must be under googlebot.com, google.com or googleusercontent.com | |
| Apple | Reverse DNS in *.applebot.apple.com, or its CIDR JSON file |
| Common Crawl | Reverse DNS in crawl.commoncrawl.org, or ccbot.json |
For a sample of requests, record the exact User-Agent and IP, match the IP to the operator's list or DNS pattern, then confirm the bot fetched /robots.txt first. OpenAI may add a robots.txt marker to its user agent on those fetches, handy when logs lack paths.
Our log analyzer automates the first step. This dependency-free Python script keeps three fields apart: the claimed token, a category from a dated rules file, and an optional IP check against a CIDR file you supply. Without one it reports not_checked, never "verified". Its ten-row log is synthetic; we re-ran it on 3 October 2026 and the output matched.
A crawl is not a citation. Logs prove reachability; whether an engine names you needs answer-level checks like our free audit.
How was the study done, and what are its limits?
We fetched each site once on 3 October 2026 and parsed robots.txt with the same matcher AEOeye's SEO audit uses. Our crawler identified itself as AEOeyeResearch/1.0 and waited one second between requests to a domain.
The sample, picked by us: 194 SaaS brands, 95 e-commerce and consumer brands, 43 AI-visibility vendors (aeoeye.com included), 39 news publishers and 20 UGC or review sites.
Limits:
- Not a random sample. It skews toward established brands; the publisher and UGC groups are small.
- One fetch, one day, main host only. Policies and subdomains differ.
- A block is a request. CDN-level blocking is invisible to robots.txt; we saw it only when our crawler was challenged.
- Counts, not outcomes. Nothing here shows a setting changed whether an engine recommends a site.
FAQ
What is the difference between GPTBot and OAI-SearchBot?+
GPTBot crawls content that may be used to train OpenAI's foundation models, while OAI-SearchBot surfaces sites in ChatGPT search. OpenAI says each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. A third token, ChatGPT-User, fetches pages when a user asks.
If I block GPTBot, will I disappear from ChatGPT?+
No. OpenAI says disallowing GPTBot signals that content should not be used for training; sites opted out of OAI-SearchBot are the ones not shown in ChatGPT search answers. Blocking GPTBot alone leaves search eligibility unchanged. In our study, 19 of 384 sites used exactly that split.
Do AI crawlers obey robots.txt?+
Anthropic, Google, Apple and Amazon state that their automatic crawlers honor it, and OpenAI and Perplexity document robots.txt tokens for managing theirs. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Robots.txt is a request, not enforcement.
How can I tell if a crawler in my logs is real?+
Treat the user-agent string as a claim. Check the source IP against the operator's published ranges (OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl all publish IP lists or DNS patterns) or run reverse and forward DNS lookups where documented. A spoofed 'GPTBot' fails both checks.
Should a brand block AI crawlers?+
Almost never from search bots. If buyers ask AI engines about your category, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are how you become eligible to be named. Blocking training bots is a separate business decision, mostly relevant to publishers with licensable archives.
Sources
Is AI recommending you?
Run a free AI visibility audit and find out in under a minute.