P2 · MediumAI Readiness
AI crawlers blocked
robots.txt blocks AI bots
Code: ai_crawlers_blocked
Why it matters
robots.txt blocks AI bots. AI systems are increasingly how people discover information. Sites that make it easy for AI crawlers to access and attribute content gain visibility in AI-generated answers.
How to fix
Allow AI crawlers in robots.txt
http# Allow AI crawlers to index your content
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: anthropic-ai
Allow: /
# Block from sensitive paths only
User-agent: GPTBot
Disallow: /admin/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlRelated checks in AI Readiness
Run a free scan to check your site
Get a complete audit in under 2 minutes. No account required.
Start Free Scan