How to Block AI Crawlers (and Which Ones to Leave Open)

Put the training bots you want out in one robots.txt group and end it with Disallow: /. A compliant crawler matches the group by its token and stays out of the whole site. The rules live in a plain-text file defined by RFC 9309, which says plainly that they "are not a form of access authorization".

This recipe opts out of training and leaves search and user-triggered access alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow: /api/

To protect one section instead of the whole site, give a bot its own path, for example User-agent: GPTBot then Disallow: /research/. Each token below comes from the operator's own page: OpenAI, Anthropic, Apple, Common Crawl and Meta.

Five details decide whether the file works:

• A named group replaces *. Under RFC 9309 a crawler obeys its own group and falls back to * only when none matches. Repeat any * rules you still want.
• Rules apply per host. Anthropic asks you to add them to every subdomain you want to opt out.
• Changes lag. OpenAI says its search systems need about 24 hours to adjust; Perplexity says up to 24 hours.
• Two tokens are controls, not crawlers. Google-Extended has no user-agent string, and Apple says Applebot-Extended does not crawl pages.
• Blocking looks forward. Anthropic says restricting ClaudeBot signals that a site's future materials should be excluded from training; it does not recall data already collected.

Sort bots by job, not by company. Training bots feed model datasets, search bots build the index an engine cites from, and user-triggered fetchers load a page because someone asked. For a brand, block only the first group, if you block anything. Our directory of 22 AI user agents has every token; this table is the decision.

One trap catches people who want out of Google's AI answers. Google says a page needs only to be indexed and snippet-eligible to appear in AI Overviews or AI Mode, and that Googlebot rules are the control. Blocking Google-Extended does not remove you. Snippet controls such as nosnippet and max-snippet limit what Search shows from your pages, but they shrink your normal listing too.

If you want AI engines to name your brand, block none of their search bots. Training bots are a business decision that mostly matters to publishers with licensable archives. The real divide is who you are: 84.2% of publishers in our study block a training bot, against 6.2% of SaaS sites.

None of the 43 AI-visibility vendors blocked either kind, including aeoeye.com, whose robots.txt explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. The study counts training bots as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent, and retrieval bots as the search and user-triggered tokens in the table above. It counts policy, not outcomes: nothing here shows a block changed whether an engine recommends a site.

Our positions:

• SaaS, stores, local and service businesses: allow everything. Your risk is being absent, not being crawled.
• Paywalled or licensing-first publishers: block the training tokens and keep search tokens open if referrals matter. Blocking retrieval bots as well is coherent only when licensing income beats citations.
• Worried about server load: rate-limit instead. Anthropic supports Crawl-delay for its bots; Apple says Applebot does not.

For the GPTBot-specific argument, read why blocking GPTBot is usually a mistake.

To enforce a block rather than request it, filter at your CDN. On Cloudflare, AI Crawl Control lets you allow or block each AI crawler, and blocking creates a WAF rule on your zone. Match on user agent plus the operator's published IPs, because a user-agent string alone is only a claim.

Cloudflare gives you three layers:

1. Managed robots.txt. Switch on the setting that blocks training in robots.txt, found under bot traffic in Security Settings. Cloudflare's docs show it prepending Disallow rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent to your own file. That is training bots only, and Cloudflare itself says compliance is voluntary.
2. AI Crawl Control. Available on all plans: set Allow or Block per crawler in the Actions column. The free plan identifies crawlers by user-agent string; paid plans add deeper detection and a configurable block response (403, or 402 Payment Required).
3. Custom WAF rules. Use them for path exceptions or extra user agents.

Two cautions. First, Anthropic warns that blocking by IP address may stop its bots from reading your robots.txt, so opt-outs become unreliable; we would exempt /robots.txt from any firewall block. Second, a firewall is the only thing that stops user-triggered fetchers, but it also turns away the person who asked about you.

The same layer can block bots you want. Perplexity's guide tells sites with a WAF to allow PerplexityBot and Perplexity-User by user agent and IP range. Of 375 homepages in our study, 60 (16%) blocked or challenged our declared research crawler, including 24 of 88 e-commerce sites.

Fetch your live robots.txt, read the group each token matches, then look in your firewall logs for refusals aimed at AI user agents. The file visitors receive is not always the copy in your repository, and a firewall can override a permissive file.

1. Open yourdomain.com/robots.txt and confirm it returns 200. Under RFC 9309, a 4xx response lets crawlers access anything, while a 5xx means they must assume complete disallow. In our study, 30 of 384 sites (7.8%) had no robots.txt at all.
2. Check what a CDN adds. Cloudflare's managed setting prepends its own rules, so read the final served file, not your source.
3. Find the group each token hits. Search for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot and PerplexityBot, then confirm none sits inside a Disallow: / group by accident. Templates, plugins and CDN settings can add them without you noticing.
4. Search your CDN logs for 403, 429 and challenge responses on those user agents. Cloudflare's AI Crawl Control also lists crawlers that violate your robots.txt.
5. Wait a day, then test the outcome. A clean file proves reachability, not recommendation. Our free audit shows whether ChatGPT names you for three unbranded buyer questions, and our AI robots.txt generator drafts a file that separates training from search.