Bot Crawlers Reference
Fresh
This content reflects Metehan Alp's bot crawl research as of 2025-2026.
Reference table for all known AI bot crawlers, their operators, crawl behavior, and verification methods.
Bot Identification
| Bot | Operator | User-Agent String | Purpose |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot/1.0 | Training data collection |
| OAI-SearchBot | OpenAI | OAI-SearchBot/1.0 | Real-time ChatGPT search |
| GoogleOther | GoogleOther | Secondary crawling (AI training, Gemini) | |
| Googlebot | Googlebot/2.1 | Primary search indexing | |
| CCBot | Common Crawl | CCBot/2.0 | Open web crawl for training datasets |
| PerplexityBot | Perplexity | PerplexityBot/1.0 | Perplexity answer sourcing |
| ClaudeBot | Anthropic | ClaudeBot/1.0 | Claude training and retrieval |
| Bytespider | ByteDance | Bytespider | TikTok/ByteDance AI training |
Crawl Behavior from 60K Page Experiment
Based on data from the 60,000-Page AI Website experiment:
| Bot | Relative Volume | Crawl Pattern | Discovery Speed |
|---|---|---|---|
| GPTBot | 470x Googlebot | Burst crawling (thousands of requests in short windows) | Hours |
| OAI-SearchBot | 85x Googlebot | Steady crawling, follows search triggers | Hours |
| CCBot | 12x Googlebot | Systematic, follows sitemap order | Days |
| GoogleOther | 3x Googlebot | Moderate, selective page crawling | Days |
| Googlebot | 1x (baseline) | Conservative, budget-managed | Days to weeks |
| PerplexityBot | 0.4x Googlebot | Light, query-triggered | Days |
Observations
- GPTBot is by far the most aggressive crawler
- OpenAI operates two separate bots with different purposes and crawl patterns
- CCBot is the most-blocked bot by the top 1,000 websites (blocking it removes your content from Common Crawl training data)
- Googlebot crawls conservatively because Google manages crawl budgets based on perceived site quality
IP Verification Approach
User-agent strings can be spoofed. Verify bot identity using this process:
Step 1: Reverse DNS Lookup
dig -x [requesting IP]The result should resolve to a hostname belonging to the claimed bot operator.
Expected domains:
| Bot | Expected Reverse DNS Domain |
|---|---|
| GPTBot | *.openai.com |
| OAI-SearchBot | *.openai.com |
| Googlebot | *.googlebot.com or *.google.com |
| GoogleOther | *.googlebot.com or *.google.com |
| PerplexityBot | *.perplexity.ai |
Step 2: Forward DNS Check
dig [hostname from step 1]The resolved IP should match the original requesting IP. If it does not match, the bot identity is spoofed.
Step 3: IP Range Validation
Some operators publish their bot IP ranges:
- Google: Published at
https://developers.google.com/search/apis/robots/googlebot.json - OpenAI: Published in their documentation
- Common Crawl: Published in their documentation
Cross-reference the requesting IP against the published range for additional verification.
Robots.txt Directives
Control which bots can crawl your content:
Allow All AI Bots
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /Block Specific Bots
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /Allow Bots but Block Specific Paths
User-agent: GPTBot
Allow: /blog/
Allow: /research/
Disallow: /admin/
Disallow: /private/Blocking Trade-offs
Blocking AI bots prevents your content from entering training data and real-time search results. Before blocking, consider whether AI citation is a business goal. If you want AI platforms to cite your content, you need to let their bots crawl it.
Monitoring Recommendations
- Log all bot traffic with user-agent, IP, timestamp, and URL requested
- Verify identity for any bot sending more than 100 requests per day
- Track crawl frequency weekly to detect changes in bot behavior
- Monitor wp-json access specifically (1.6x GPTBot activity correlation)
- Set up alerts for new or unknown bot user-agents