robots.txt and llms.txt: What Actually Controls AI Crawlers

Two common goals pull in opposite directions. You may want to stop AI companies from training on your content. You may also want AI assistants to find and cite it. Both goals start with robots.txt, and both have limits the file cannot fix.

What robots.txt is

It's a plain text file at the root of your site (/robots.txt) that tells crawlers which paths they should or shouldn't fetch. The format is standardized as RFC 9309. Well-behaved crawlers read it before they visit anything else.

The key word is 'should'. Following robots.txt is voluntary. Reputable crawlers from big companies honor it. Scrapers with bad intentions ignore it. It's a polite sign on the door, not a lock. Never use it to hide anything private. If a page must be secret, put it behind a login.

The AI bots to know

Each company publishes the user-agent names it uses. Names and purposes change, so check the official docs before you rely on this list. As of writing, the common ones are:

Two details trip people up. First, Google-Extended isn't a separate crawler. It's a token you can disallow in robots.txt to opt out of certain Gemini uses, and Google says it doesn't affect your Search ranking or inclusion. Regular Search still uses Googlebot. Second, bots that act on behalf of a live user (like ChatGPT-User) may be treated differently from bulk crawlers. Read each provider's own documentation to see how it handles robots.txt.

Blocking vs allowing: pick a goal first

The choice is a trade-off.

• You want to opt out of AI training but keep search visibility. Disallow the training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) and allow the search ones.
• You want to be cited by AI assistants. Allow the search and user-triggered bots, and make sure the content is easy to read as plain HTML.
• You don't care either way. Leave the defaults alone.

For reference, this site's own robots.txt allows the AI crawlers, because we want our tools and guides to be findable in AI answers. That's a choice. Yours might differ.

What the file looks like

A simple setup that opts out of training bots and allows everything else:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Each User-agent line starts a group of rules that applies to that bot. A bot follows the most specific group that matches its name. Leaving out User-agent: * doesn't block anyone, it just means no default rules exist. The robots.txt Generator builds this for you and includes toggles for the AI bots, so you don't have to remember the exact names.

What blocking can't do

• It doesn't remove content that a crawler already collected earlier.
• It doesn't stop a scraper that ignores the rules.
• It doesn't remove a page from Google's index by itself. A blocked page can still appear in results as a bare URL if other sites link to it. To keep a page out, use a noindex directive and let it be crawled.

llms.txt

llms.txt is a proposal from 2024 for a Markdown file at /llms.txt that gives language models a clean summary of your site: what it is, and links to the most useful pages. It costs nothing to add.

It's a proposal, not a standard that the big platforms have agreed to follow. Google representatives have said Google Search doesn't use it, and the major AI companies haven't committed to reading it. Some tools and agents do fetch it, and it can help when a developer points an assistant at your docs. Don't expect it to boost rankings or guarantee citations.

Add one if you run documentation or a tool-heavy site. It's cheap and might help some agents. Cover the basics first: fast pages, clear headings, and useful content. The llms.txt Generator will draft one from your page list.

A short checklist

1. Decide whether you want AI training opt-out, AI citation, or both.
2. Write robots.txt rules to match, using each provider's current user-agent names.
3. Test the file by opening yoursite.com/robots.txt in a browser.
4. Use noindex, not robots.txt, to keep pages out of search.
5. Re-check every few months. Bot names and policies change.