llms.txt is a proposed file that offers AI systems a clean map of your site. No major assistant is confirmed to read it yet. robots.txt is what actually controls AI crawler access, and for most businesses the correct setting there is to allow all of them.
This subject attracts more anxiety than it deserves. The decision is usually simple once you know which crawler does what.
The crawlers that matter
| Crawler | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects data used to train future models. Blocking it keeps you out of what the model knows. |
| OAI-SearchBot | OpenAI | Handles live browsing when ChatGPT searches. Blocking it keeps you out of answers today. |
| ChatGPT-User | OpenAI | Fetches a page when a user explicitly asks about it. |
| Google-Extended | Controls use of your content for Gemini training. Does not affect Search ranking. | |
| PerplexityBot | Perplexity | Indexes for Perplexity results. Blocking it removes you from Perplexity. |
| ClaudeBot | Anthropic | Collects data for Claude. |
| Bytespider | ByteDance | Aggressive crawler with little citation benefit. The one most sites choose to block. |
| CCBot | Common Crawl | Feeds an open dataset used by many model builders. |
The distinction that matters
Training crawlers and retrieval crawlers are not the same thing, and conflating them is where most mistakes happen.
Training crawlers
GPTBot, ClaudeBot, Google-Extended and CCBot gather material that shapes what a model knows. Blocking them is a long-horizon decision with no visible short-term effect.
Retrieval crawlers
OAI-SearchBot, PerplexityBot and ChatGPT-User fetch pages to answer a question being asked right now. Blocking these removes you from live answers immediately. This is the more consequential setting and the one more often blocked by accident.
Security plugins and CDN bot rules block AI user agents by default more often than people realise. Check what your site actually serves before assuming your robots.txt is being honoured.
A robots.txt that works for most businesses
User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Bytespider Disallow: / Sitemap: https://example.com/sitemap.xml
The only entry blocked here is Bytespider, which crawls heavily and cites rarely. Everything else is allowed, because visibility in AI answers is worth more to a service business than the theoretical value of withholding public marketing copy.
When blocking is the right call
- You sell the content itself: a publisher, a research firm, a course library, a paid database.
- You are negotiating a licensing deal and want leverage.
- Crawler traffic is a genuine hosting cost, which happens on very large sites.
- You have a legal or contractual obligation over the material.
If none of those apply, blocking mostly costs you visibility and protects nothing. Your public pages are public.
What llms.txt is for
The proposal is a markdown file at your root that gives an AI system a curated map of your site: what the business is, which pages matter, and where the canonical explanation of each thing lives. Think of it as an index written for a reader with no patience for navigation.
# Example Company > Digital marketing agency in Hafizabad, Pakistan. SEO, Google Ads > and eCommerce work for clients in Pakistan and six export markets. ## Services - [SEO](https://example.com/services/seo): organic search, technical and content - [Google Ads](https://example.com/services/google-ads): search and shopping campaigns ## Reference - [Pricing](https://example.com/pricing): all retainers and what they include - [About](https://example.com/about): founded 2019, team of nine, Hafizabad
It costs an hour to write and cannot hurt. Treat it as cheap insurance against the standard being adopted, not as a lever that does something today.
Questions we get asked
Does llms.txt actually work?
No major AI company has confirmed it reads llms.txt. It is a community proposal, useful as a signal of intent and harmless to publish, but robots.txt is what enforces access today.
Should I block AI crawlers?
For most businesses, no. Blocking them removes you from AI answers without protecting anything, since the content is already public. The case for blocking is strongest when your content itself is the product.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot gathers data used in training. OAI-SearchBot handles live browsing when ChatGPT looks something up. Blocking GPTBot keeps you out of future models. Blocking OAI-SearchBot keeps you out of answers today.
Does Google-Extended affect my rankings?
No. Google-Extended controls whether your content trains Gemini and feeds certain AI features. It does not affect Google Search ranking or indexing.
Where does llms.txt go?
At the root, alongside robots.txt: yoursite.com/llms.txt. It is plain markdown.
Can I block AI crawlers but keep AI Overviews?
Not cleanly. AI Overviews uses Googlebot, so blocking it would remove you from Google Search entirely.
