ByteDance
ByteSpider
An aggressive web crawler that collects data for ByteDance's AI models, including TikTok's recommendation engine and the Doubao chatbot.
Purpose: Generative AI training and search indexing
📊 Popularity & Traffic
#3Ranking among AI crawlers
Accounted for ~90% of AI crawler requests on some platforms in 2024Traffic share
One of the most active AI crawlers globally; often reported as more 'aggressive' than others.
🤖 User Agent Strings
Use these patterns to identify ByteSpider in your server logs or configure your robots.txt file.
Bytespider
Respects robots.txtPrimary ByteDance crawler for AI training
BytespiderTikTokSpider
Respects robots.txtTikTok link-fetcher for previews and queries
TikTokSpider🌐 IP Ranges
Source: ByteDance/TikTok infrastructure (not publicly indexed)
No specific IP ranges published. Identify this bot using the User Agent strings above.
📝 Robots.txt Configuration
Add the following to your robots.txt file to block ByteSpider:
User-agent: Bytespider
Disallow: /
User-agent: TikTokSpider
Disallow: /💡 Important Notes
- ByteSpider has been reported to sometimes disregard robots.txt in certain configurations
- TikTokSpider is user-initiated when links are shared in the TikTok app
- ByteDance uses this data for Llama-like model training in China (Doubao)
Beyond blocking crawlers
See what AI is saying about your brand
Understanding crawlers is step one. With Aiso, you can see the actual conversations happening about your brand inside ChatGPT, Claude, and Perplexity.