Anthropic
ClaudeBot
Anthropic's web crawler that gathers data from the public web to help train and improve Claude's underlying models.
Purpose: Model training data collection
📊 Popularity & Traffic
#2Ranking among AI crawlers
~5.4% of AI crawler traffic (down from 27% in mid-2024)Traffic share
Was #2 AI crawler by volume in mid-2024. Share decreased as OpenAI and Meta increased their crawling.
🤖 User Agent Strings
Use these patterns to identify ClaudeBot in your server logs or configure your robots.txt file.
ClaudeBot
Respects robots.txtPrimary Anthropic crawler for model training
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +https://www.anthropic.comanthropic-ai
Respects robots.txtLegacy/alternative identifier (same as ClaudeBot)
anthropic-ai🌐 IP Ranges
Source: Not published
Identified IP Ranges1 Range
Uses cloud infrastructure: AWS, GCP - IPs may varySingle IP Address
How to read CIDR notation:
The/28 suffix indicates a block of 16 IP addresses. For example,.112/28 covers all addresses from .112 up to .127. Adding these to your firewall will block the entire range used by ClaudeBot.📝 Robots.txt Configuration
Add the following to your robots.txt file to block ClaudeBot:
User-agent: ClaudeBot
Disallow: /
# Also block legacy identifier:
User-agent: anthropic-ai
Disallow: /💡 Important Notes
- Supports Crawl-delay directive to limit crawl rate
- Anthropic does NOT publish IP ranges - use robots.txt for blocking
- Will not circumvent blocks (no CAPTCHA solving, respects disallows)
- Media sites have 'almost universally' blocked AI bots including Claude's
Beyond blocking crawlers
See what AI is saying about your brand
Understanding crawlers is step one. With Aiso, you can see the actual conversations happening about your brand inside ChatGPT, Claude, and Perplexity.