ClaudeBot: What It Is and Whether You Should Block It

ClaudeBot is Anthropic's crawler for collecting public web content that could contribute to training its Claude models. It is not the bot behind Claude's search results or user requests: Claude-SearchBot and Claude-User handle those, each with its own robots.txt token. In AEOeye's crawler access study (October 2026, 384 sites), 44 sites (11.5%) block ClaudeBot, including 30 of 38 news publishers but only 3 of 193 SaaS sites.

This guide rests on Anthropic's public documentation as of October 2026; we have not tested the bots hands-on. Our position: if you want Claude to know and recommend you, leave ClaudeBot open, and never block the other two.

What is ClaudeBot?

ClaudeBot is the crawler Anthropic uses to collect web content that could contribute to training its generative AI models. Restricting it signals that your site's future materials should be excluded from those training datasets. A ClaudeBot hit means Anthropic is gathering text, not that Claude is answering a question about you.

A web crawler is software that fetches pages automatically. Anthropic states four principles for its crawling:

• Transparent: it uses three named bots to access web content.
• Not intrusive: it aims for minimal disruption and says it respects Crawl-delay where appropriate.
• Honors do-not-crawl signals: its bots follow industry-standard robots.txt directives.
• Respects anti-circumvention technology: it says it will not try to bypass CAPTCHAs.

How do ClaudeBot, Claude-SearchBot and Claude-User differ?

Anthropic runs three bots with three jobs: ClaudeBot gathers possible training data, Claude-SearchBot improves search result quality, and Claude-User fetches a page when someone asks Claude a question. Each has its own token, so a rule for one does nothing to the others.

Bot	Job, per Anthropic	If you block it, per Anthropic	Sites blocking it (n=384)
ClaudeBot	Collects web content that could contribute to model training	Future materials are excluded from training datasets	44 (11.5%)
Claude-SearchBot	Improves the quality and relevance of search results	Content is not indexed for search; may reduce visibility and accuracy	24 (6.2%)
Claude-User	Retrieves pages when a user asks Claude a question	Content is not retrieved for user queries; may reduce visibility	26 (6.8%)

OpenAI draws the same three-way line: GPTBot for training, OAI-SearchBot for search, ChatGPT-User for user actions.

We also checked how the blocks overlap, using our public CSV. No site blocked Claude-SearchBot without also blocking ClaudeBot. Of the 44 that block ClaudeBot, 20 left Claude-SearchBot open and 18 left both Claude-SearchBot and Claude-User open, which is what a deliberate training-only opt-out looks like. The other 24 probably pasted a blanket "block AI bots" list, though our data cannot say.

How do you block or allow ClaudeBot in robots.txt?

Add a group naming ClaudeBot with Disallow: / to the robots.txt at your domain root, and repeat it on every subdomain. That is Anthropic's own example, and it covers ClaudeBot only:

User-agent: ClaudeBot
Disallow: /

If your worry is server load, slow it down instead. Anthropic supports Crawl-delay, a non-standard extension, with this example:

User-agent: ClaudeBot
Crawl-delay: 1

Watch one trap when you "allow" it. Under RFC 9309, a crawler obeys only the group that matches its token and falls back to * when none does:

User-agent: *
Disallow: /account/

User-agent: ClaudeBot
Allow: /

Here ClaudeBot may crawl /account/, because its own group replaced the * rules. Skip a group for any bot you are happy to treat like everyone else, or copy the Disallow lines into it.

Anthropic's page gives no propagation time, unlike the roughly 24 hours OpenAI states for its search systems, so watch your logs for the change rather than assuming it is instant.

Do not enforce the opt-out by IP address. Anthropic warns that IP blocking may not work or persist, because it stops the bot reading your robots.txt. For rules across all AI bots, see our AI robots.txt generator and the full blocking playbook.

Should you block ClaudeBot?

For a business that wants to be found, no. Blocking ClaudeBot only keeps your future pages out of training data. Block it if your content is paywalled or licensed and you will not accept free reuse. Blocking Claude-SearchBot or Claude-User is what Anthropic says may reduce your visibility in Claude's answers.

For a brand, a block gives up any chance of being part of what the model learns and gains little in return, though we have no data on how large that effect is.

Our own earlier version of this page claimed that a ClaudeBot block leaves Claude unable to recommend you. That overstates it: per Anthropic's documentation ClaudeBot feeds training, and the bots that retrieve content for answers are separate. How Claude finds and cites pages is a different question, covered in Claude SEO.

Who blocks it in our study:

• News publishers: 30 of 38 (78.9%).
• UGC and review sites: 6 of 20 (30%).
• E-commerce and consumer brands: 5 of 90 (5.6%).
• SaaS: 3 of 193 (1.6%).
• AI-visibility vendors: 0 of 43. aeoeye.com allows ClaudeBot too.

Treat those as counts of policy, not proof of outcome. We have no data showing that blocking ClaudeBot changes whether Claude recommends a brand, and we do not claim it does.

Block ClaudeBot if:

• your archive is a product you sell or license;
• you are negotiating with AI companies and want leverage;
• your terms forbid reuse and you need a documented opt-out.

Leave it open if you sell something and want Claude to know you exist. The effect of a block is invisible, so test it: our free audit covers ChatGPT, and the $29 full report adds Claude. Run the free audit to start.

What is the ClaudeBot user agent, and how do you verify a request?

Match the ClaudeBot token in the User-Agent header, then check the request's IP against Anthropic's published list. The name alone proves nothing, since anyone can send it. Anthropic says a crawler whose source IP is on its list is coming from Anthropic.

Anthropic names robots.txt tokens but does not print a full User-Agent string on its page, and we found no reverse-DNS method there. RFC 9309 says the token should appear as a substring of the header, so search logs case-insensitively. claudebot will not match Claude-SearchBot or Claude-User, so search claude too.

Impersonation is real. Common Crawl says it knows of crawlers falsely identifying as CCBot, and the same trick works for any name.

1. Pull the IP from log lines whose User-Agent contains the token.
2. Fetch Anthropic's list. When we fetched it, it held 28 IPv4 prefixes and was dated 2 October 2026. Refetch it rather than pasting it into a rule.
3. Test the IP against the prefixes. Save the file as bots.json, then run this:

import ipaddress, json, sys

with open("bots.json") as f:
    nets = [ipaddress.ip_network(next(iter(p.values()))) for p in json.load(f)["prefixes"]]

for ip in sys.argv[1:]:
    addr = ipaddress.ip_address(ip)
    ok = any(addr.version == n.version and addr in n for n in nets)
    print(ip, "on Anthropic's list" if ok else "NOT on the list")

4. Act on the result. Apply your policy to real hits, and rate-limit or block hits from addresses outside the list.

A verified hit shows that Anthropic reached your site. It says nothing about whether Claude recommends you. For the other AI crawlers and their verification methods, use our AI crawler directory.

What if ClaudeBot is crawling too hard?

Confirm the traffic is genuine, then slow it with Crawl-delay before you block anything. Anthropic says it aims for minimal disruption, and it invites reports from site owners who think a bot is malfunctioning.

Work through it in this order:

1. Verify the source with the IP check above. Heavy traffic from an address outside Anthropic's published ranges is not coming from Anthropic, and a Crawl-delay line will not slow it.
2. Add Crawl-delay to the ClaudeBot group and watch your logs for a few days.
3. Report it. Anthropic asks that you write from an email on the domain concerned, because it finds reports hard to verify otherwise.
4. Block only as a last resort, and only ClaudeBot. Blocking all three Anthropic bots to fix a load problem would also stop the two that serve Claude's answers.