Googlebot Tester: See the HTML Google Actually Gets

Google crawls with two crawlers, Googlebot Smartphone and Googlebot Desktop, and for most sites the smartphone one does nearly all the work. So this test uses the smartphone user agent.

Google's documentation adds two numbers worth knowing. Googlebot fetches up to 15 MB of an HTML file — content after that is ignored — and it reads robots.txt as a whole, so you can't write separate rules for the smartphone and desktop crawlers.

Six things side by side, chosen because each one can quietly change what Google indexes:

• Status code — a page that returns 200 to you and 403 to Googlebot is invisible to search.
• Title and canonical — some setups serve different tags to bots.
• Robots meta — a stray noindex shows up here.
• Readable text — the words in the raw HTML, before any JavaScript runs.
• Links and JSON-LD — how much Google can follow and parse without rendering.

Usually a firewall, CDN or "bot protection" product. They often treat unknown crawlers as threats; some block anything that claims to be Googlebot but doesn't come from Google's IP ranges.

That's common among big sites. When we fetched the homepages of 391 well-known sites in October 2026 with an openly labelled research crawler (dataset), 55 of them (14.1%) answered with a 403 rather than the page.

Yes, Googlebot renders pages with an evergreen Chromium, but rendering can happen later than crawling, and Google still recommends putting important content in the server-rendered HTML. Many AI crawlers don't run JavaScript at all.

If the "readable text" column is nearly empty while your page looks full in a browser, your content depends on JavaScript. Check what the source contains with view page source.

Not quite. Our request uses Googlebot's user agent but comes from our server, not Google's. Firewalls that verify Googlebot by reverse DNS will correctly reject us while letting the real Googlebot through.

Treat a mismatch as a lead, then confirm with the URL Inspection tool in Search Console, which fetches from Google's own infrastructure. For robots rules, use the robots.txt tester.