Crawl Budget

Crawl budget is the practical limit on how many of your pages a bot fetches and how often. Large or slow sites can exhaust it on low-value URLs (filters, duplicates, thin pages), leaving important pages crawled late or not at all — which matters twice over now that AI crawlers like GPTBot and PerplexityBot fetch pages alongside Googlebot.

For AI visibility, the logic is the same as classic SEO: if a crawler never reaches your best answer page, no engine can cite it. Clean sitemaps, fast responses, and pruning low-value URLs keep the budget flowing to pages that deserve it.