ClaudeBot: Check If Anthropic's Crawler Can Reach You
ClaudeBot is Anthropic's crawler for collecting training data; Claude-SearchBot indexes pages for Claude's search answers and Claude-User fetches pages on request. Enter your site to see whether robots.txt and your server let each one in — and which line decides it.
We read your live robots.txt and request the page as ClaudeBot, Claude-SearchBot and Claude-User, with Googlebot as a control. Free, no signup. Updated Oct 24, 2026 by the AEOeye team.
ClaudeBot AI Crawler: What It Collects and Whether to Block It
ClaudeBot collects public web content that may be used to train Anthropic's models. Blocking it asks Anthropic to leave your future content out of training — on its own, it does not remove you from Claude's search answers.
Anthropic's help page lists three separate agents, each with its own robots.txt name:
| Agent | Job | If you block it |
|---|---|---|
| ClaudeBot | Training data collection | Future pages are excluded from training |
| Claude-SearchBot | Indexing for search results in Claude | Your visibility in Claude's answers can drop |
| Claude-User | Fetching a page a user asked about | Claude can't read your page for that user |
Anthropic says its bots honor robots.txt directives and support the non-standard Crawl-delay line.
Should I block ClaudeBot?
For most companies, no. If buyers ask Claude about your category, you want Claude to know who you are and what you do. Publishers that license their archives are the exception.
The split between those two groups is stark. When we checked 391 sites in October 2026 (dataset), 11.5% blocked ClaudeBot overall — 78.9% of news publishers versus 1.6% of SaaS companies. Claude-SearchBot was blocked by 6.3% overall.
If you want out of training but still want to be found, block ClaudeBot and leave Claude-SearchBot and Claude-User open.
How do I read the result?
Look at both columns. robots.txt shows what your rules say; the live request shows what your server really did.
- robots.txt allowed, live 200 — nothing stands in the way.
- robots.txt allowed, live 403 or challenge — a firewall rule is blocking Anthropic's user agents. Many bot-protection presets block "AI scrapers" by default.
- robots.txt blocked — we show the exact group and rule, including whether a wildcard
User-agent: *rule caught it.
Robots rules are matched by the most specific user-agent group, then the longest path (RFC 9309). A ClaudeBot group therefore overrides your * group completely.
What rule should I copy?
Use two groups: one that opts out of training and one that keeps search open.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
Run the AI crawler checker to see every AI bot in one table, or build a full policy with the AI robots.txt generator.
Reachable is step one. Does AI actually recommend you?
Run a free AI visibility audit: we ask ChatGPT the questions your buyers ask and show whether your brand comes up.
FAQ
Is ClaudeBot the same as Claude?
+
No. ClaudeBot is a crawler that collects training data. Claude, the assistant, reads the web through Claude-SearchBot for search indexing and Claude-User for pages a user specifically asks about.
Does ClaudeBot respect robots.txt?
+
Anthropic says its bots honor standard robots.txt directives and the Crawl-delay extension, and it documents how to disallow each agent by name. Firewall blocks are a separate layer you control.
Will blocking ClaudeBot hurt my Google rankings?
+
No. Google crawls and ranks with Googlebot, which follows its own robots.txt group. A ClaudeBot rule only changes what Anthropic may collect from your site.
Sources
Related free tools
AI Crawler Checker
Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider and more — what each does, plus a 1-click check of your site.
GPTBot Checker
Is GPTBot or OAI-SearchBot blocked? We read your robots.txt and request your homepage as each OpenAI bot, then show which to allow for ChatGPT search.
Robots.txt Tester
Free robots.txt tester: test any URL for Googlebot, Bingbot, GPTBot, ClaudeBot and 13 more bots, and see exactly which rule decides (RFC 9309).
AI Robots.txt Generator
Free robots.txt generator for AI crawlers. Choose exactly which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can access your site.
AI Visibility Checker
Free AI visibility checker: we ask ChatGPT your buyers' questions and show whether you're named, where you rank and who wins instead. No signup.