What Is llms.txt and Why Your Site Needs One for AI Search
By Elizabeth Stein
Search is fragmenting. Users no longer rely solely on Google - they ask ChatGPT, Perplexity, Claude, and Gemini for answers. These AI systems don't rank ten blue links. They synthesise information from across the web into a single response, and they cite their sources. If your site isn't structured for AI discovery, you're invisible in a growing share of search traffic.
The question isn't whether AI search matters. It's whether your site is ready for it.
How AI Search Engines Find Content
Traditional search engines crawl your pages, index them, and rank them by relevance. AI search engines do something different. They use specialised crawlers - GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (Gemini) - to ingest content, then use it to generate answers.
When a user asks "What's the best way to fix CLS issues?", an AI search engine might pull from your blog post, a competitor's documentation, and a Stack Overflow answer. The sites that get cited are the ones that are crawlable, well-structured, and explicitly AI-friendly.
What Is llms.txt?
llms.txt is a proposed standard - a plain text file you place at the root of your site (like robots.txt) that tells AI systems about your site's content in a structured, LLM-friendly format. It gives AI crawlers a summary of what your site does, what content is available, and how to navigate it.
Think of it as a README for AI. Where robots.txt says "here's what you can and can't crawl" and sitemap.xml says "here's a list of all my URLs", llms.txt says "here's what my site is about and what's most important."
How It Differs From robots.txt and sitemap.xml
| File | Purpose | Audience |
|---|---|---|
robots.txt | Controls crawler access (allow/disallow) | All crawlers |
sitemap.xml | Lists all URLs with priority and frequency | Search engine indexers |
llms.txt | Describes site content and structure for AI | LLM crawlers and agents |
These files complement each other. You need all three.
llms.txt Template
Place this file at https://yoursite.com/llms.txt:
# YourSiteName
> Brief one-line description of what your site does.
## About
A 2-3 sentence overview of your site, your audience, and what problems you solve.
## Key Pages
- [Homepage](https://yoursite.com/): Main landing page with product overview
- [Pricing](https://yoursite.com/pricing): Plans and pricing details
- [Documentation](https://yoursite.com/docs): Technical documentation and guides
- [Blog](https://yoursite.com/blog): Articles about [your topic area]
## Topics Covered
- Topic 1
- Topic 2
- Topic 3
## Contact
- Email: hello@yoursite.com
- Twitter: @yourhandle
The format uses Markdown headings and links. Keep it concise - AI systems don't need your full sitemap here, just the high-level structure and most important pages.
You can also create an llms-full.txt that includes more detailed content for AI systems that want deeper context. The base llms.txt should stay under a few hundred lines.
Configuring robots.txt for AI Crawlers
Beyond llms.txt, you should explicitly allow (or block) the major AI crawlers in your robots.txt:
# AI Crawlers - Allow
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Block specific paths from AI crawlers if needed
User-agent: GPTBot
Disallow: /admin/
Disallow: /api/
If you want AI systems to reference your content (and most sites should), explicitly allowing these crawlers is important. Many hosting platforms and CDNs block unknown user agents by default.
Blocking AI Crawlers
If you don't want AI systems training on or citing your content:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This is a legitimate choice for some sites, but understand the trade-off: you won't appear in AI-generated answers, which is an increasingly significant source of referral traffic.
Why This Matters Now
AI search is growing fast. Perplexity alone handles millions of queries daily. ChatGPT's browsing feature is used by hundreds of millions of users. When these systems generate answers, they cite sources - and those citations drive real traffic.
Sites that are structured for AI discovery get cited. Sites that aren't get ignored. The gap will widen as AI search usage grows.
The steps are straightforward: create an llms.txt file, configure your robots.txt for AI crawlers, and make sure your content is well-structured with proper headings, meta descriptions, and schema markup. These are the same fundamentals that help with traditional SEO - AI readiness is an extension of existing best practices, not a separate discipline.
Rocket Vitals includes AI readiness checks in every scan - verifying that your llms.txt exists, your robots.txt allows AI crawlers, and your schema markup is complete enough for AI systems to understand your content.
Related Checks
Rocket Vitals includes AI readiness checks in every scan. See the full details for each check:
- Missing llms.txt - No /llms.txt file found for AI crawlers
- AI crawlers blocked - robots.txt blocks AI bots like GPTBot and ClaudeBot
- No structured data - No JSON-LD or microdata detected for AI/SEO
- Missing Organization schema - Homepage lacks Organization JSON-LD
- Missing brand entity - No Organization schema for brand recognition
Run a free scan to see how AI-ready your site is and what's missing. Check your AI readiness →