⚡ Instant Tools

Free llms.txt Generator for AI Crawlers

Build an llms.txt file so ChatGPT, Perplexity, and other AI agents can discover your best pages. This is for AI crawler discovery (GEO), not search-bot crawl rules. Import from sitemap or curate links. Free, no signup.

See our live file: instanttools.io/llms.txt (built with this tool)

Privacy: Site details and sitemaps stay in your browser.

Sections

Preview

# Website

> Description pending.

## Pages


---
Generated by Instant Tools · https://instanttools.io/tools/llms-txt-generator

How to Use

  1. 1Write a clear site name and one-paragraph summary for agents.
  2. 2Add curated sections (docs, tools, products). Quality beats dumping every URL.
  3. 3Optionally import a sitemap, then prune noise pages.
  4. 4Host the file at https://yourdomain.com/llms.txt and link it from your site footer or docs.

llms.txt vs robots.txt: different jobs for AI and search

llms.txt is a proposed site file aimed at AI crawlers and agents (generative engine optimization). It lists the pages, tools, and policies you want models to notice, with short descriptions so agents can choose what to cite. Host it at /llms.txt.

robots.txt is the long- standing crawl-rules file for search bots and many AI scrapers. It uses Allow and Disallow paths, sitemaps, and optional per-bot blocks (for example GPTBot or ClaudeBot). It does not describe what your product is; it only gates which URLs may be fetched.

Use both: set crawl policy in robots.txt Generator, then publish an llms.txt so preferred content stays discoverable for AI answers. Mixing the two formats in one file confuses crawlers; keep them separate.

Good llms.txt habits

Lead with what you are

> One crisp paragraph: product, audience, and privacy stance.

Agents skim the blockquote first.

Curate links

Prefer money pages, docs hubs, and canonical tools, not every blog tag.

Import sitemap, then edit down.

Tips

  • •Keep descriptions action-oriented so agents know when to cite a page.
  • •Include privacy/terms if users will paste sensitive content into your tools.
  • •Update llms.txt when you ship major new tools or docs hubs.
  • •Pair with OpenAPI/MCP tools if your product exposes APIs to agents.

Frequently Asked Questions

What is llms.txt?

llms.txt is a proposed site file (like a human-readable map) that helps AI crawlers and agents discover important pages, tools, and policies. Host it at https://yoursite.com/llms.txt. See our live example at https://instanttools.io/llms.txt.

How do I get Google to index my llms.txt?

Host the file at your domain root (/llms.txt), link it from your homepage footer or docs, include it in your XML sitemap, and request indexing in Google Search Console. This free generator helps you build the file; discovery still depends on crawl signals.

Does this generator call my website?

No. You paste a sitemap XML or enter links manually. Everything runs in your browser so there are no CORS or upload issues.

Is llms.txt the same as robots.txt?

No. robots.txt tells search and AI crawlers which paths they may fetch (Allow/Disallow rules). llms.txt is a human-readable map for AI agents and generative engines: curated links and summaries so tools can cite the right pages. Use robots.txt for crawl policy and llms.txt for GEO discovery. Build robots.txt with our robots.txt generator.

Generate crawl rules with the robots.txt Generator.

Can I import from a sitemap?

Yes. Paste sitemap.xml contents (or a sitemap index’s child sitemap body). URLs are extracted from <loc> tags and turned into a Pages section you can edit.

Is the generator free?

Yes. No signup. Download or copy the file and deploy it on your origin.

Related Tools

robots.txt Generator

Control GPTBot, ClaudeBot, and other AI crawlers

JSON to Schema

Infer Zod and JSON Schema from examples

Cursor Rules Generator

AGENTS.md and .cursor/rules for repos

OpenAPI to MCP

Expose APIs as MCP servers

AI Alt Text Generator

SEO-ready image descriptions

JSON Formatter

Validate sitemap-adjacent JSON configs