AI Robots.txt Generator
Decide exactly which AI bots can crawl your site. Toggle each crawler, add your sitemap, and copy a ready-to-use robots.txt.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Disallow: / User-agent: * Allow: /
Place at https://yourdomain.com/robots.txt. Blocking trainers won't remove you from past training, but it controls future crawling.
Allow the bots, get the visibility
There’s a reflex to block AI crawlers to “protect” content. For most brands that’s exactly backwards. If your buyers are asking ChatGPT and Perplexity for recommendations, blocking those crawlers guarantees you won’t be one — you’re opting out of the channel. Unless you have proprietary or paywalled content to protect, the default should be to allow the major AI bots and make yourself easy to find.
Know your crawlers
- GPTBot / OAI-SearchBot / ChatGPT-User — OpenAI’s training, search and live-browsing agents.
- ClaudeBot / Claude-Web — Anthropic’s crawlers for Claude.
- PerplexityBot — powers Perplexity’s indexing and citations.
- Google-Extended — controls use of your content for Gemini and AI training (separate from Googlebot).
Once you’re crawlable, the question becomes whether AI actually recommends you. That’s what an AI visibility audit answers.
Crawlable, but recommended?
Allowing bots is step one. See if AI names you with a free audit.
FAQ
Should I allow or block AI crawlers?+
For most brands that want to be discovered and recommended by AI, allow the major ones (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Blocking them keeps you out of the very systems your buyers now use. Block only if you have a specific reason — for example, protecting paywalled or proprietary content.
What's the difference between training bots and search bots?+
Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data that may inform future models. Search/answer bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) fetch pages live to answer a user right now. Blocking training bots won't remove you from models already trained; allowing search bots is usually essential for live citations.
Does robots.txt actually stop AI crawlers?+
Reputable crawlers honor robots.txt. It's a directive, not a hard wall — it won't stop bad actors — but the major AI companies do respect it. It controls future crawling, not data already collected.
Where does robots.txt go?+
At your domain root: https://yourdomain.com/robots.txt, served as plain text. If you already have a robots.txt, merge these user-agent blocks into it.