Meta Robots Checker: Find noindex, nofollow & More
A meta robots checker reads the robots meta tag and the X-Robots-Tag header together and tells you what Google will do: index or not, follow links or not, and which snippet limits apply. It also checks robots.txt, because a blocked page can't show Google its noindex.
We fetch the page as Googlebot (or as a browser if your firewall rejects look-alike Googlebot requests), read every robots meta tag and X-Robots-Tag header, and match the path against your live robots.txt. Free, no signup. Updated Oct 24, 2026 by the AEOeye team.
X-Robots-Tag Checker: Header Directives You Can't See
The X-Robots-Tag is an HTTP header that carries the same directives as the robots meta tag. Google treats the two as equivalent, and the header is the only option for PDFs, images and other files without HTML.
Because it never appears in the page source, it's the classic hidden culprit: a noindex header copied from a staging config can drop pages from search while the HTML looks clean. This checker reads both places at once.
X Robots Tag vs Meta Robots: Which One Wins?
Neither — Google combines them. Every directive from the meta tag and the header is applied, and when they conflict, the more restrictive one wins. A page with index in the tag and noindex in the header is noindexed.
| Directive | Effect |
|---|---|
| noindex | Keep the page out of search results |
| nofollow | Don't follow links on the page |
| none | noindex + nofollow |
| nosnippet | No text snippet (also keeps text out of AI Overviews) |
| max-snippet:[n] | Limit the snippet to n characters |
| max-image-preview:[size] | none, standard or large image previews |
| unavailable_after:[date] | Drop the page after a date |
Why can't Google see my noindex?
Because robots.txt blocks the page. Google has to crawl a URL to read its noindex; if robots.txt disallows it, the directive is never seen and the URL can still be indexed from links. Google's own docs say pages with indexing rules must not be disallowed from crawling, and its robots.txt specification confirms that a disallow only controls crawling.
How often do big sites get this wrong?
Rarely on homepages — but it happens. In our October 2026 analysis of 315 big-brand homepages (dataset), 0.6% carried a noindex robots meta tag on the homepage itself, and 2.9% sent an X-Robots-Tag header.
Check your key templates, not just the homepage. To find which robots.txt rule blocks a path, use the robots.txt tester; to see every header a page returns, use the HTTP header checker.
One page checked. What about the rest of your site?
The free SEO audit crawls your site and scores ~90 checks across 7 areas — with the evidence and the fix for each problem.
FAQ
What does meta robots noindex do?
+
It tells search engines not to show the page in results. Google needs to crawl the page to see the tag, and the page drops out of the index after its next crawl.
Should I use noindex or robots.txt to hide a page?
+
Use noindex to keep a page out of search results, and robots.txt to stop crawling. Never combine them on the same URL, because a robots.txt block hides the noindex from Google.
Does nofollow on a page stop Google from indexing linked pages?
+
No. Page-level nofollow tells Google not to follow the links on that page, but linked pages can still be found and indexed through other links or your sitemap.
Sources
Related free tools
Robots.txt Tester
Free robots.txt tester: test any URL for Googlebot, Bingbot, GPTBot, ClaudeBot and 13 more bots, and see exactly which rule decides (RFC 9309).
HTTP Header Checker
Free HTTP header checker: see every response header for a URL, with SEO-critical ones flagged — X-Robots-Tag, canonical Link, cache-control, HSTS.
Googlebot Simulator
Free Googlebot tester: fetch a page as Googlebot Smartphone and compare it with a browser fetch — status, robots, noindex, canonical and readable text.
Meta Tag Checker
Free meta tag checker: see every tag that matters on a URL — title, description, robots, canonical, viewport, Open Graph, X cards — graded with fixes.