Known AI chatbot crawlers — user agents & IP addresses
The definitive reference for identifying and managing AI web crawlers. Use this data to configure your robots.txt, firewall rules, or verify bot traffic in your server logs.
AI crawlers documented
major entities covered
data last updated
What you can do with this data
Whether you want to block AI training, appear in AI search results, or just understand your traffic — this reference helps.
Configure robots.txt
Use our examples to allow or block specific AI bots from crawling your site for training or search indexing.
Firewall rules
Reference official IP ranges to whitelist or block AI crawlers at the network level.
Analytics filtering
Identify AI bot traffic in your analytics to get accurate human visitor counts.
Content licensing
Understand which bots respect opt-out directives for AI training vs. search indexing.
Get the data
Download or copy our curated list of AI bot user agents and IP addresses. Perfect for configuring your robots.txt, firewall rules, or analytics filters.
All known AI chatbot crawlers
Click on any bot to see detailed information including user agent strings, IP ranges, and robots.txt configuration examples.
GPTBot
Model training data collection
GPTBotOAI-SearchBot
ChatGPT search indexing and citations
OAI-SearchBotChatGPT-User
Real-time webpage fetching for user queries
ChatGPT-UserGooglebot
Search indexing and AI model training
GooglebotPerplexityBot
Search indexing for AI answers
PerplexityBotPerplexity-User
Real-time content fetching for user queries
Perplexity-UserClaudeBot
Model training data collection
ClaudeBotClaude-User
Real-time content fetching for user queries
Claude-UserClaude-SearchBot
Search indexing for Claude answers
Claude-SearchBotBingbot
Search indexing and AI answer generation
BingbotByteSpider
Generative AI training and search indexing
BytespiderAmazonbot
Alexa accuracy and LLM training (Bedrock)
AmazonbotAI2Bot
Scientific and open-source AI research
AI2BotCCBot
Open web indexing for public datasets
CCBotLAION Crawlers
Multimodal and image AI training
laion-huggingface-processorPetalBot
Search indexing and Pangu LLM training
PetalBotMeta AI Crawlers
Llama model training and AI features
Meta-ExternalAgentCohere-AI
Enterprise-grade LLM training
cohere-aiYouBot
Search indexing and YouChat answers
YouBotDuckAssistBot
Real-time AI summarization (DuckAssist)
DuckAssistBotPhindBot
Search indexing and technical AI answers
PhindBotDevin
Autonomous task execution and coding
DevinTavilyBot
Real-time AI data retrieval for agents
TavilyBotShapBot
Search indexing for AI APIs
ShapBotSee what AI is saying about your brand
Managing bot crawlers is just one piece. With Aiso, you can see the actual conversations happening about your brand inside ChatGPT, Claude, and other AI platforms.