Robots.txt Tester: Test Any URL Against the Rules

Each crawler obeys exactly one group of rules: the group whose user-agent line matches its name most specifically, falling back to User-agent: * only if no group names it. That's why a site can block GPTBot while Googlebot sails through.

Within that group, the longest matching path wins. When an Allow and a Disallow match with the same length, Google uses the least restrictive rule — the same tie-break the RFC 9309 standard describes. The table shows, for each bot, the result and the exact deciding rule.

Google is forgiving: lines it can't parse are skipped silently rather than breaking the file. That's exactly the danger. We flag:

• Unknown directives such as Noindex: (Google stopped supporting it in 2019) — they do nothing.
• Rules before any User-agent line, which belong to no group.
• Crawl-delay, which Google ignores (Bing and others use it).
• Relative Sitemap URLs — the Sitemap line needs a full URL.
• Files over 500 KiB — Google reads only the first 500 KiB.

It depends on the status code, and the difference is dramatic. Google treats a 4xx (other than 429) as "no restrictions". A 5xx is a server error: for the first 12 hours Google stops crawling the site while it retries, then uses the last good copy for up to 30 days.

Missing files are more common than broken ones. Among 391 well-known sites we checked in October 2026, 33 had no robots.txt at all and 7 couldn't serve it reliably (dataset).

No — it stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it, just without a description. To keep a page out of Google, allow crawling and use noindex; check it with the meta robots checker.

Managing AI bots? Build a policy with the AI robots.txt generator and test it here.