AEOeyeBot

What it requests

• /robots.txt
• Up to 6 sitemap files (those in robots.txt, /sitemap.xml and /sitemap_index.xml) plus up to 5 child sitemaps of one index
• Up to 20 HTML pages for a free report, 40 for a full one — found through your links and a sample of your sitemap
• Variants of the audited URL — http / https, www / apex, trailing slash, upper case, /index.html and a utm-tagged copy — to check redirects
• One random path such as /aeoeye-404-probe-1a2b3c4d, to check that missing pages return 404
• The start of up to 20 sitemap URLs it didn’t crawl and up to 11 canonical targets — enough to read status, noindex and canonical
• HEAD requests to up to 30 internal link targets it didn’t crawl, to find broken links
• HEAD requests to up to 20 images and the og:image, to read their file size
• HEAD requests to up to 30 links pointing to other sites (5-second timeout), to find dead outbound links

Pages are read as raw HTML, up to 2 MB each. It never runs JavaScript, submits forms or logs in, and doesn’t download scripts, stylesheets or images. When a HEAD request fails it may retry with a GET that stops after the first 4 KB.

In all, roughly 130 requests at most to your site for a free audit and 150 for a full one, spread over a minute or two.

How politely

• Crawls pages 4 at a time and checks links 4 at a time; page fetches and link checks on the same host start at least 250 ms apart
• Follows the robots.txt rules for AEOeyeBot (or *) for every page it crawls and every sitemap URL or internal link it samples
• Honours Crawl-delay while crawling pages, up to 2 s; a longer delay is capped at 2 s and the number of pages is halved
• Stops on a 429, a challenge page or 403 responses and skips the remaining checks on your host. If the audited URL itself is refused, the audit stops there and the report says the site blocked us. It never retries with a different user agent.
• Free audits: at most 6 fresh runs of the same site per hour across all users, and the same URL audited again within 6 hours reuses the earlier result unless a re-run is forced. Paid reports can also be re-run for 30 days after purchase.

How to block it

Stop the crawl with robots.txt:

User-agent: AEOeyeBot
Disallow: /

AEOeyeBot then fetches none of your pages beyond the URL the audit was requested for. Each audit still makes the site-level requests above: robots.txt, sitemaps, that URL with its variants and canonical target, the 404 probe, and size checks on that page’s images.

Refuse every request by blocking user agents containing AEOeyeBot at your firewall or CDN.

The audit then reports the site as blocked instead of giving it a score. Neither option affects Google.

Contact

Questions, or traffic you didn’t expect? Email contact@aeoeye.com with your domain and the time of the requests, and we’ll look into it.

Requests come from Vercel’s cloud and the IP range isn’t fixed, so please match on the user agent. The PageSpeed Insights part of an audit is run by Google: those visits come from Google’s servers with a Chrome-Lighthouse user agent, not from AEOeyeBot.