Bytespider

What it is and why it exists

Bytespider requests pages using a user-agent string that identifies itself as Bytespider, distinct from the crawlers OpenAI, Anthropic and Google operate under their own names. ByteDance has not published the kind of detailed, consistently maintained documentation other major AI labs provide, such as a clear list of IP ranges or a public statement of exactly what collected content feeds into. Trade coverage of the crawler reports a narrower exception: ByteDance does instruct site owners who want to block it to add a two-line robots.txt rule naming the user agent and disallowing the root path.

How Bytespider behaves on a site

Independent reporting on actual behavior tells a different story, and it goes back further than one report. Fortune's Kali Hays reported on 3 October 2024, citing research from bot-management firm Kasada, that "the Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt." TollBit, a content-licensing firm that tracks AI crawler behavior, later published first-half-2026 data showing that Bytespider was one of three agents that reached disallowed pages on close to half of the European sites that had explicitly named them in a robots.txt disallow rule. It was not alone at that rate: ChatGPT-User and Youbot showed the same roughly 50% figure in the same dataset, with ChatGPT-User affecting the largest number of sites overall. Bytespider is one documented case among several, not an outlier.

That still leaves room for doubt about any single number. The same TollBit data found Anthropic's Claude-User reaching disallowed pages on just 9% of the European sites that named it, against 26% in North America, so the gap is real but uneven across companies. Two things complicate it further. Some administrators report the opposite pattern, that Bytespider does stop on disallowed paths once it re-fetches an updated robots.txt file, with the delay between publishing a new rule and the crawler picking it up accounting for part of the apparent violations. And because ByteDance publishes no cryptographic way to verify a request actually came from Bytespider, any client can set that user-agent string, which means some of the traffic blamed on it in server logs may not be ByteDance's crawler at all. Naming Bytespider in robots.txt is still the strongest signal available short of blocking it outright, and two independent reports, nearly two years apart, found that signal not honored as reliably as a publisher might expect.

Why it matters for AI visibility

Bytespider's output feeds ByteDance's own AI and product ecosystem, which is a real, if less visible, channel for how a brand's content ends up represented in AI-generated answers, separate from the more commonly tracked crawlers from OpenAI, Anthropic and Google. The compliance gap found by TollBit means a robots.txt rule aimed at Bytespider is a weaker guarantee than the equivalent rule aimed at a crawler with a stronger measured compliance record, which matters for any brand relying on robots.txt alone to control whether its content reaches a given AI system.

Good practices

• Add an explicit User-agent: Bytespider disallow rule if blocking is the goal, using the documented two-line format.
• Do not rely on robots.txt alone for a reliable block, given the measured compliance gap; enforce it at the CDN or web application firewall level if certainty matters.
• Check server logs directly for Bytespider requests rather than assuming the robots.txt rule is being honored.
• Review the decision alongside other AI crawlers, since a site's overall AI-crawler policy should be a single deliberate decision, not one rule set per crawler in isolation.

Common mistakes

• Assuming a robots.txt disallow rule aimed at Bytespider is honored at the same rate as one aimed at a more consistently compliant crawler.
• Treating the absence of published IP ranges as proof Bytespider cannot be identified or managed; log-based user-agent matching still works, it is just less precise than IP verification.
• Confusing Bytespider, ByteDance's general web crawler, with TikTok-specific bots that serve a different purpose.
• Never re-checking server logs after setting a block, so a rule that stops being effective goes unnoticed.