GPTBot: Is OpenAI's Crawler Allowed on Your Site?
GPTBot is OpenAI's training crawler; OAI-SearchBot is the separate bot that puts pages into ChatGPT search. Enter your site to see whether robots.txt and your server let each OpenAI bot in. Blocking GPTBot does not remove you from ChatGPT search — blocking OAI-SearchBot does.
We read your live robots.txt and request the page as GPTBot, OAI-SearchBot and ChatGPT-User, with Googlebot as a control. Free, no signup. Updated Oct 24, 2026 by the AEOeye team.
OAI-SearchBot: The OpenAI Bot Behind ChatGPT Search
OAI-SearchBot decides whether your pages can be shown and linked in ChatGPT's search answers. OpenAI documents it separately from GPTBot, so you can allow one and block the other.
| Bot | What OpenAI uses it for | If you block it |
|---|---|---|
| GPTBot | Training OpenAI's foundation models | Future content is left out of training |
| OAI-SearchBot | Surfacing sites in ChatGPT search | You can drop out of ChatGPT search results |
| ChatGPT-User | Fetching a page when a user asks ChatGPT to | User-initiated, so robots.txt rules may not apply |
OpenAI's crawler documentation adds one practical detail: robots.txt changes take about 24 hours to reach its search systems. Re-test a day after you edit the file.
Should I block GPTBot?
Only if you have a reason to keep your content out of model training — usually because you license it. For most companies the answer is no, and the safer default is to leave OAI-SearchBot open no matter what you decide about GPTBot.
Our view is simple: GPTBot is a business decision about training, OAI-SearchBot is a visibility decision. Blocking OAI-SearchBot by accident is pure loss.
How do I read the result?
Each bot gets two verdicts: what your robots.txt says, and what your server actually did when we asked for the page with that bot's user agent.
- Allowed + 200: the bot can reach the page.
- Allowed + 403 or a challenge page: a firewall or CDN rule is blocking it even though robots.txt says yes. This is the most common surprise.
- Blocked by robots.txt: we show the exact rule that decided it, so you know which line to change.
Robots groups are matched by the most specific user-agent name (RFC 9309), so a GPTBot group replaces your User-agent: * rules for that bot entirely.
What do 391 real sites do?
Most sites leave OpenAI's bots alone. When we checked the robots.txt files of 391 well-known sites in October 2026 (dataset), 9.1% blocked GPTBot but only 4.2% blocked OAI-SearchBot, and 4.9% made the deliberate split: training off, search on.
News publishers are the outlier, with 52.6% blocking GPTBot. Among SaaS companies it was 2.1%.
How do I allow ChatGPT search but block training?
Give each bot its own group in robots.txt at your domain root:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Then check your firewall. If a bot-protection rule challenges unknown user agents, allow OpenAI's published IP ranges. Test every AI bot at once with the AI crawler checker, or confirm which line matches a path in the robots.txt tester.
Reachable is step one. Does AI actually recommend you?
Run a free AI visibility audit: we ask ChatGPT the questions your buyers ask and show whether your brand comes up.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
+
No. GPTBot only controls whether OpenAI may use your content for training. Whether you appear in ChatGPT search depends on OAI-SearchBot, which has its own robots.txt group you can leave open.
Is GPTBot the same as ChatGPT-User?
+
No. ChatGPT-User only fetches a page when a person asks ChatGPT to look at it. OpenAI says robots.txt rules may not apply to these user-initiated visits, unlike its automatic crawlers.
How long until ChatGPT respects a robots.txt change?
+
OpenAI says it can take about 24 hours for its search systems to pick up a robots.txt update. Make the change, wait a day, then run this check again to confirm the new rule is live.
Why does the live request fail when robots.txt allows the bot?
+
Usually a firewall or CDN bot rule. Some verify crawlers by IP address, so our test can be blocked while the real OpenAI bot is not. Your firewall logs will show which rule fired.
Sources
Related free tools
AI Crawler Checker
Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider and more — what each does, plus a 1-click check of your site.
ClaudeBot Checker
Is ClaudeBot blocked on your site? Check robots.txt and a live request as ClaudeBot, Claude-SearchBot and Claude-User — and what each Anthropic bot does.
Robots.txt Tester
Free robots.txt tester: test any URL for Googlebot, Bingbot, GPTBot, ClaudeBot and 13 more bots, and see exactly which rule decides (RFC 9309).
AI Robots.txt Generator
Free robots.txt generator for AI crawlers. Choose exactly which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can access your site.
AI Visibility Checker
Free AI visibility checker: we ask ChatGPT your buyers' questions and show whether you're named, where you rank and who wins instead. No signup.