[FetchDoge]

User Job

As a website owner / local service business buying AI visibility, I want ChatGPT Search, Claude Search, and Perplexity to read, index, and cite my business so new customers can find and call me, without having my proprietary content scraped for bulk AI model training by Common Crawl and model-building bots.

Why this change

PR #12 repositioned AgentReady around AI visibility reports and done-for-you fixes for local service businesses to show up in ChatGPT, Perplexity, and Google AI Overviews.
However, fix.mjs previously:

1. Completely omitted Claude-SearchBot (Anthropic's search-specific crawler), leaving Claude Search citations unconfigured.
2. Hardcoded Allow: / for CCBot (Common Crawl) and GPTBot (bulk model training).
3. Conflated search/citation crawlers with bulk model training scrapers, leaving business owners unable to opt out of training while opting into search visibility.

What changed

• Added --robots-mode search|all (default: search) to fix.mjs.
• In search mode (default):
  • AI Search & Citation crawlers are allowed (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot).
  • Bulk training scrapers are disallowed (GPTBot, ClaudeBot, CCBot, Google-Extended).
• In all mode:
  • Both search and training crawlers are allowed, including Claude-SearchBot.
• Updated FIX.md instructions explaining the search-first default and how to override policy.
• Added comprehensive unit tests in test/scan.test.mjs verifying search vs all modes, presence of Claude-SearchBot, and rejection of invalid modes.

Scores

• Fit: 5/5
• Usefulness: 5/5
• Feasibility: 5/5
• Evidence: 5/5 (Verified)

Sources

• OpenAI Bot Documentation (platform.openai.com/docs/bots): Separates OAI-SearchBot (search) from GPTBot (training). [Verified]
• Anthropic Web Scraping Documentation (docs.anthropic.com/en/docs/agents-and-tools/web-scraping): Separates Claude-SearchBot (search) from ClaudeBot (training). [Verified]
• Google Documentation: Google-Extended controls Gemini training/grounding, while standard search uses Googlebot. [Verified]

Step Size Rationale

AgentReady is a low-ranking repository with unrealized commercial potential (early prototype). Delivering on the core promise of the repositioned homepage (PR #12) requires a structural capability step in the fix generator to provide production-grade, privacy-respecting search citation policies rather than a blunt toy robots block.

Falsifier

If prospective SMB clients indicate they prefer bulk training scrapers to index their site alongside search bots, or if AI search engines begin requiring training crawler permissions (GPTBot / ClaudeBot) to provide citations, this iteration's search-only default is a mistake.

Verification

• Test-first: added failing test assertion in test/scan.test.mjs (failed on missing Claude-SearchBot and missing --robots-mode).
• Passing: node --test test/scan.test.mjs passes 21/21 tests in 3.98s. Zero regressions.
• Zero dependencies maintained.

Non-Goals

• Does not change scan.mjs scoring weights or binary scoring threshold (no scoring inflation).
• Does not modify any user-agent string outside fix.mjs robots generation.
• Does not invent facts not found on the prospect's page.