AI crawlers blocked
robots.txt blocks AI bots
Code: ai_crawlers_blocked
Why it matters
robots.txt blocks AI bots. AI systems are increasingly how people discover information. Sites that make it easy for AI crawlers to access and attribute content gain visibility in AI-generated answers.
How to fix
Allow AI crawlers in robots.txt
http# Allow AI crawlers to index your content
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: anthropic-ai
Allow: /
# Block from sensitive paths only
User-agent: GPTBot
Disallow: /admin/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlRelated checks in AI Readiness
Missing llms.txt
No /llms.txt file for AI crawlers
AI crawler access matrix
Per-bot allow/block status across the 14 major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Bytespider, etc.)
llms.txt missing H1
/llms.txt exists but lacks the `# Project Name` top-level heading required by llmstxt.org
llms.txt has no sections or links
/llms.txt is missing `## Section` headings and curated markdown links — AI crawlers have nothing to ingest
llms.txt links are broken
URLs listed in /llms.txt return 4xx/5xx — wastes AI crawler budget and risks bad citations
Run a free scan to check your site
Get a complete audit in under 2 minutes. No account required.
Start Free Scan