metehan.ai Research & SOPs

Appearance

60,000-Page AI Website Experiment ​

Fresh

This content reflects Metehan Alp's published experiment results as of 2025-2026.

What happens when you generate 60,000 pages of AI content, deploy them on edge infrastructure, and watch what crawls them? This experiment measured bot behavior, traffic distribution, and the economic viability of large-scale AI content generation.

Methodology ​

Content Generation ​

  • Model: gpt-4.1-nano (optimized for speed and cost at scale)
  • Parallelism: 50 concurrent requests
  • Output: 60,000 unique pages covering a broad topic space
  • Content type: Informational articles across multiple categories

Infrastructure ​

  • Hosting: Cloudflare Workers (edge compute)
  • Database: Cloudflare D1 (serverless SQLite at the edge)
  • CDN: Cloudflare's global network (automatic)
  • Domain: StateGlobe.com

Monitoring ​

  • Server-side request logging for all bot traffic
  • User-agent parsing and verification
  • IP-based bot identification
  • Timestamp tracking for crawl patterns

Bot Behavior Discovery ​

The headline finding: GPTBot sent 470x more requests than Googlebot.

Traffic Distribution ​

BotRequestsMultiple vs. Googlebot
GPTBot (OpenAI)~470x baseline470x
OAI-SearchBot (OpenAI)~85x baseline85x
CCBot (Common Crawl)~12x baseline12x
GoogleOther~3x baseline3x
Googlebotbaseline (1x)1x
PerplexityBot~0.4x baseline0.4x

Key Observations ​

GPTBot dominance: OpenAI's GPTBot was the most aggressive crawler by a massive margin. It crawled pages systematically, often hitting the site in bursts of thousands of requests within short time windows.

OAI-SearchBot vs GPTBot: OpenAI operates two distinct bots. GPTBot collects training data. OAI-SearchBot powers ChatGPT's real-time search feature. Both were active, but GPTBot was 5.5x more active than OAI-SearchBot.

Googlebot restraint: Google crawled conservatively compared to AI bots. This aligns with Google's established crawl budget management -- they allocate crawl resources based on perceived site quality and authority.

Speed of discovery: AI bots discovered and began crawling new pages within hours of deployment. The sitemap was the primary discovery mechanism.

Server-Side Verification Implementation ​

To ensure accurate bot identification, the experiment implemented server-side verification:

  1. User-agent string matching -- Initial identification based on the User-Agent header
  2. Reverse DNS lookup -- Verify that the requesting IP resolves to the claimed bot's domain
  3. Forward DNS verification -- Confirm the resolved hostname points back to the requesting IP
  4. IP range validation -- Cross-reference against published bot IP ranges (where available)

This verification is important because user-agent strings can be spoofed. Without server-side verification, bot traffic numbers are unreliable.

Key Findings ​

1. AI Bots Are Hungry for Content ​

The sheer volume of GPTBot requests (470x Googlebot) shows that AI companies are aggressively building training datasets. Any publicly accessible content will be crawled.

2. Speed of Discovery Is Fast ​

New pages were discovered and crawled within hours, not days or weeks. For AI content strategies, this means new content enters the AI training pipeline quickly.

3. Sustainability Warning ​

While 60,000 pages can be generated cheaply and deployed on edge infrastructure, the long-term value is questionable:

  • Google did not reward the content with significant organic rankings
  • The content lacked E-E-A-T signals (no real author, no real expertise)
  • AI-generated content at this scale creates a volume-without-value problem

4. Economic Reality ​

  • Generation cost: Low (gpt-4.1-nano is cheap at scale)
  • Hosting cost: Low (Cloudflare Workers pricing is minimal for static-ish content)
  • SEO value: Near zero (no authority, no links, no E-E-A-T)
  • AI training contribution: High (bots consumed the content aggressively)

The experiment proves that generating massive AI content is technically easy and economically cheap. It also proves that volume alone does not create search value. The content served as bot food, not as a sustainable traffic source.

Not a Strategy Recommendation

This experiment documents what happens, not what you should do. Building 60,000-page AI content sites is not a recommended strategy. The value is in the bot behavior data, not in the content itself.

StateGlobe.com ​

The experiment domain StateGlobe.com remains live as a reference. The StateGlobe tool itself has evolved into a multi-region search results comparison tool available on Metehan's tools page.

Further Reading ​