Crawler information
AEOeyeBot
AEOeyeBot fetches a sample of a website when someone requests a technical SEO audit of it on AEOeye, and for a full report it reads the few pages that rank above that site on Google. It never crawls on its own schedule.
User agents
Both contain AEOeyeBot, so one rule matches both. It never sends a plain browser or Googlebot user agent.
Main user agent
Mozilla/5.0 (compatible; AEOeyeBot/1.0; +https://aeoeye.com/bot)Mobile — one extra request for the audited page, to compare mobile and desktop HTML
Mozilla/5.0 (Linux; Android 14; Mobile) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Mobile Safari/537.36 (compatible; AEOeyeBot/1.0; +https://aeoeye.com/bot)Research crawls (AEOeyeResearch)
Separately from audits, we occasionally run a one-off study of how sites treat AI crawlers. The first covered 391 well-known sites on 3 October 2026 (three passes that day while we fixed our own script); the method and the per-site data are published in our AI crawler access study. Per site and pass it fetches at most four URLs (following redirects): /robots.txt (retried once on the www host if the first attempt fails), then /llms.txt and the homepage only if robots.txt allows them for this user agent — one at a time, a second apart, nothing else.
Mozilla/5.0 (compatible; AEOeyeResearch/1.0; +https://aeoeye.com/bot)To opt out of future studies: User-agent: AEOeyeResearch / Disallow: /.
What it requests
/robots.txt- Up to 6 sitemap files (those in robots.txt,
/sitemap.xmland/sitemap_index.xml) plus up to 5 child sitemaps of one index - Up to 20 HTML pages for a free report, 40 for a full one — found through your links and a sample of your sitemap
- Full reports only: up to 8 of your pages that rank on Google but weren’t reached by the crawl, one request each, plus robots.txt once more to check them
- Variants of the audited URL — http / https, www / apex, trailing slash, upper case,
/index.htmland a utm-tagged copy — to check redirects - One random path such as
/aeoeye-404-probe-1a2b3c4d, to check that missing pages return 404 - The start of up to 20 sitemap URLs it didn’t crawl and up to 11 canonical targets — enough to read status, noindex and canonical
- HEAD requests to up to 30 internal link targets it didn’t crawl, to find broken links
- HEAD requests to up to 20 images and the og:image, to read their file size
- HEAD requests to up to 30 links pointing to other sites (5-second timeout), to find dead outbound links
Pages are read as raw HTML, up to 2 MB each. It never runs JavaScript, submits forms or logs in, and doesn’t download scripts, stylesheets or images. When a HEAD request fails it may retry with a GET that stops after the first 4 KB.
In all, roughly 130 requests at most to your site for a free audit and 160 for a full one, spread over a minute or two.
How politely
- Crawls pages 4 at a time and checks links 4 at a time; page fetches and link checks on the same host start at least 250 ms apart
- Follows the
robots.txtrules forAEOeyeBot(or*) for every page it crawls, every sitemap URL or internal link it samples, and every top-ranking page it reads for a full report - Honours
Crawl-delaywhile crawling pages, up to 2 s; a longer delay is capped at 2 s and the number of pages is halved - Stops on a 429, a challenge page or 403 responses and skips the remaining checks on your host. If the audited URL itself is refused, the audit stops there and the report says the site blocked us. It never retries with a different user agent.
- Free audits: at most 6 fresh runs of the same site per hour across all users, and the same URL audited again within 6 hours reuses the earlier result unless a re-run is forced. Paid reports can also be re-run for 30 days after purchase.
Top-ranking pages (full reports)
A full report compares the audited site with the pages that outrank it. For up to 6 of that site’s search queries we take the top 5 organic Google results (via DataForSEO), and AEOeyeBot fetches each of those pages once per report run:
- It requests that host’s
/robots.txtfirst and skips the page ifAEOeyeBot(or*) is disallowed from it — or if robots.txt can’t be read - Then one GET for the page itself — raw HTML up to 2 MB, 8 s timeout. It doesn’t follow links on the page or fetch its images, scripts or stylesheets
- At most 30 pages per report run, across all sites, 4 at a time; requests to the same host start at least 250 ms apart, longer if robots.txt sets a
Crawl-delay
So if one of your pages ranks for a query an audited site cares about, you may see one request for it plus your robots.txt — nothing else on your site. A re-run of that paid report fetches the same pages again without a new search, unless the site’s owner has added a new target keyword. Blocking AEOeyeBot in robots.txt stops it.
Two parts of a full report send no request to any website. The brand-reputation check is two Google searches run through DataForSEO — the audited site’s brand name, and the name plus “reviews”. Search Console data, read only when the audited site’s owner connects it, comes from Google’s API. Neither shows up as AEOeyeBot in anyone’s logs.
How to block it
Stop the crawl with robots.txt:
User-agent: AEOeyeBot Disallow: /
AEOeyeBot then fetches none of your pages beyond the URL the audit was requested for. Each audit still makes the site-level requests above: robots.txt, sitemaps, that URL with its variants and canonical target, the 404 probe, and size checks on that page’s images.
Refuse every request by blocking user agents containing AEOeyeBot at your firewall or CDN.
The audit then reports the site as blocked instead of giving it a score. Neither option affects Google.
Contact
Questions, or traffic you didn’t expect? Email contact@aeoeye.com with your domain and the time of the requests, and we’ll look into it.
Requests come from Vercel’s cloud and the IP range isn’t fixed, so please match on the user agent. The PageSpeed Insights part of an audit is run by Google: those visits come from Google’s servers with a Chrome-Lighthouse user agent, not from AEOeyeBot.