AI Crawlers: Live List + Check Your Site in 1 Click
AI crawlers fall into three groups: training crawlers (GPTBot, ClaudeBot…), search crawlers that let assistants cite you (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers. Enter your site to test all of them at once against your robots.txt and your server.
We read your live robots.txt and request the page as each AI crawler, using the user-agent strings their operators publish. Free, no signup. Updated Oct 24, 2026 by the AEOeye team.
Which AI crawlers exist, and what does each one do?
Every major AI company now runs more than one bot, and they do different jobs. Mixing them up is how sites accidentally vanish from AI answers.
| Crawler | Operator | Job |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results |
| GPTBot | OpenAI | Model training |
| Claude-SearchBot | Anthropic | Search answers in Claude |
| ClaudeBot | Anthropic | Model training |
| PerplexityBot | Perplexity | Perplexity search results |
| Google-Extended | Gemini training and grounding control (token only) | |
| Applebot-Extended | Apple | Apple AI training control (token only) |
| Amzn-SearchBot | Amazon | Alexa and Amazon search answers |
| Amazonbot | Amazon | Amazon services; may train Amazon models |
| Bytespider | ByteDance | Reportedly training data for ByteDance models |
| Meta-WebIndexer | Meta | Meta AI search results and citations |
| meta-externalagent | Meta | AI training and product indexing |
| CCBot | Common Crawl | Open web archive |
Sources: OpenAI, Anthropic, Perplexity, Google, Amazon, Meta, Common Crawl.
Which AI crawlers should I allow?
Allow the search crawlers, and treat training crawlers as a separate business decision. Blocking a training bot doesn't remove you from AI search; blocking a search bot does.
Google is explicit about its own token: Google-Extended does not affect inclusion or ranking in Google Search. The other operators draw the same line between training and search bots in their documentation.
What do real sites do?
Most block nothing. When we checked 391 well-known sites in October 2026 (dataset), 16.7% blocked at least one training crawler and 9.6% blocked at least one AI search crawler.
News publishers are the exception: 84.2% block a training bot and 71.1% block a search bot. For a SaaS or e-commerce brand, an AI search block is usually an accident — a copied robots.txt template or a CDN "block AI bots" toggle that also catches search bots.
How do I read the results table?
Each row shows two separate verdicts. The robots.txt column shows what your rules allow, plus the exact deciding line; the live column shows what your server returned to a request with that bot's user agent.
- Allowed + 200 — the bot can read the page.
- Allowed + 403 or challenge — usually a firewall preset, not your robots.txt.
- Token-only rows (Google-Extended, Applebot-Extended) have no live result, because they never visit.
How do I fix a block?
Start with the search bots, because those cost visibility today. Edit robots.txt with the AI robots.txt generator, relax the firewall rule if the live column failed, then dig into a single bot on its own page, such as GPTBot or ClaudeBot.
Reachable is step one. Does AI actually recommend you?
Run a free AI visibility audit: we ask ChatGPT the questions your buyers ask and show whether your brand comes up.
FAQ
Do AI crawlers follow robots.txt?
+
The major operators — OpenAI, Anthropic, Google, Apple, Perplexity, Amazon, Meta and Common Crawl — document robots.txt controls for their automatic crawlers. User-triggered fetchers such as ChatGPT-User, Amzn-User and Meta-ExternalFetcher may not follow every rule.
Will blocking AI crawlers hurt my Google rankings?
+
Not if you only block AI-specific bots. Google ranks pages with Googlebot, and Google states that the Google-Extended token does not affect inclusion or ranking in Google Search.
Why do some bots show a robots.txt result but no live result?
+
Google-Extended and Applebot-Extended are control tokens, not crawlers. They never request pages from your server, so only your robots.txt rules decide how they are treated.
How often should I re-check?
+
After any change to robots.txt, your CDN or your firewall. OpenAI and Amazon both say robots.txt changes can take about 24 hours to reach their systems, so re-test the next day.
Sources
- 1.OpenAI — Overview of OpenAI crawlers
- 2.Anthropic — Does Anthropic crawl data from the web?
- 3.Google — Google common crawlers (Google-Extended)
- 4.Perplexity — Perplexity crawlers
- 5.Amazon — Amazonbot, Amzn-SearchBot and Amzn-User
- 6.Meta — Meta web crawlers
- 7.Common Crawl — CCBot
- 8.AEOeye AI Crawler Access Study (Oct 2026, 391 sites)
Related free tools
GPTBot Checker
Is GPTBot or OAI-SearchBot blocked? We read your robots.txt and request your homepage as each OpenAI bot, then show which to allow for ChatGPT search.
ClaudeBot Checker
Is ClaudeBot blocked on your site? Check robots.txt and a live request as ClaudeBot, Claude-SearchBot and Claude-User — and what each Anthropic bot does.
AI Robots.txt Generator
Free robots.txt generator for AI crawlers. Choose exactly which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can access your site.
Robots.txt Tester
Free robots.txt tester: test any URL for Googlebot, Bingbot, GPTBot, ClaudeBot and 13 more bots, and see exactly which rule decides (RFC 9309).
AEO Grader
Free AEO grader: we check AI crawler access, llms.txt, schema, answer-first content and entity signals, then score your AI search readiness 0–100.