Bytespider: Check If ByteDance's Crawler Can Reach You
Bytespider is ByteDance's web crawler, reportedly used to gather training data for its AI models such as Doubao. It is the most-blocked AI crawler we measured. Enter your site to see whether robots.txt and your server let Bytespider in, and copy the rule to block it.
We read your live robots.txt and request the page with the Bytespider user agent, with Googlebot as a control. Free, no signup. Updated Oct 24, 2026 by the AEOeye team.
What is Bytespider?
Bytespider is the crawler run by ByteDance, the company behind TikTok. Cloudflare's 2024 analysis listed it first among the four biggest AI crawlers by request volume on its network, and said it is reportedly used to collect training data for ByteDance's large language models, including Doubao (Cloudflare).
Unlike OpenAI, Anthropic or Apple, ByteDance doesn't publish a detailed page explaining what Bytespider feeds or how to verify its traffic. That gap is a big part of why site owners distrust it.
Should I block Bytespider?
For most brands selling in English-speaking markets, blocking it costs almost nothing. Bytespider isn't the crawler behind ChatGPT, Claude, Perplexity or Google's AI answers, so you give up no visibility in the assistants your buyers use.
Keep it open only if ByteDance products such as TikTok search or Doubao matter to your audience.
How many sites block it?
More than any other AI bot we measured. When we checked 391 sites in October 2026 (dataset), 14.6% blocked Bytespider — ahead of ClaudeBot and CCBot at 11.5%. News publishers blocked it 78.9% of the time; SaaS sites, 4.7%.
How do I read the result?
The robots.txt verdict and the live request answer different questions, so read both.
- Blocked by robots.txt — we show the deciding rule. A
User-agent: *disallow counts too. - Allowed, live request 403 — your CDN or firewall already turns it away.
- Allowed, live request 200 — Bytespider can fetch the page today.
Robots.txt is a request, not a lock. If your server logs still show heavy Bytespider traffic after you block it, add a firewall rule that matches its user agent.
What rule should I copy?
One group is enough, because robots rules apply to the most specific matching user agent (RFC 9309):
User-agent: Bytespider
Disallow: /
Check every AI bot together with the AI crawler checker, or test any path against your rules in the robots.txt tester.
Reachable is step one. Does AI actually recommend you?
Run a free AI visibility audit: we ask ChatGPT the questions your buyers ask and show whether your brand comes up.
FAQ
Does Bytespider respect robots.txt?
+
It identifies itself with a Bytespider user agent, so a robots.txt group can target it by name. Because ByteDance publishes little documentation, back the rule with a firewall block if your logs show it ignoring the rule.
Will blocking Bytespider affect my TikTok account?
+
No. Blocking the crawler doesn't touch your TikTok account, videos or ads. It only stops ByteDance's crawler from fetching pages on your own website.
Why is Bytespider blocked so often?
+
High crawl volume, little public documentation and a training-only purpose. In our October 2026 sample of 391 sites it was the most blocked of the 15 AI and search bots we checked.
Sources
Related free tools
AI Crawler Checker
Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider and more — what each does, plus a 1-click check of your site.
Amazonbot Checker
What is Amazonbot and should you allow it? Check your robots.txt and a live request as Amazonbot, plus what Amazon says it uses the crawler for.
Robots.txt Tester
Free robots.txt tester: test any URL for Googlebot, Bingbot, GPTBot, ClaudeBot and 13 more bots, and see exactly which rule decides (RFC 9309).
AI Robots.txt Generator
Free robots.txt generator for AI crawlers. Choose exactly which AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can access your site.