Anthropic
Claude-SearchBot
Crawls the web to index content and enhance Claude's search-based answers.
Purpose: Search indexing for Claude answers
📊 Popularity & Traffic
Similar to PerplexityBot or OAI-SearchBot in function.
🤖 User Agent Strings
Use these patterns to identify Claude-SearchBot in your server logs or configure your robots.txt file.
Claude-SearchBot
Respects robots.txtIndexes content for Claude search features
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0🌐 IP Ranges
Source: Not published
No specific IP ranges published. Identify this bot using the User Agent strings above.
📝 Robots.txt Configuration
Add the following to your robots.txt file to block Claude-SearchBot:
User-agent: Claude-SearchBot
Disallow: /💡 Important Notes
- Blocking reduces presence in Claude's answer citations
- Anthropic doesn't have a public search engine - indexing is for the chatbot only
Beyond blocking crawlers
See what AI is saying about your brand
Understanding crawlers is step one. With Aiso, you can see the actual conversations happening about your brand inside ChatGPT, Claude, and Perplexity.