robots.txt Generator

Control GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more. Generate or validate — free and private.

Privacy: Everything runs in your browser. Pair with an llms.txt for AI-friendly discovery.

Preset

Sitemap URLDisallow paths (one per line) for User-agent: *Allow paths (optional)Host (optional, non-standard)Add llms.txt tip commentllms.txt path

AI crawlers

Inherit = no explicit block (falls under User-agent: *).

GPTBot (OpenAI)

OpenAI training / retrieval crawler

ChatGPT-User (OpenAI)

User-initiated ChatGPT browsing

OAI-SearchBot (OpenAI)

OpenAI search results crawler

ClaudeBot (Anthropic)

Anthropic Claude web crawler

Google-Extended (Google)

Gemini / Vertex training (not Google Search)

Applebot-Extended (Apple)

Apple Intelligence training (not Applebot search)

PerplexityBot (Perplexity)

Perplexity answer-engine crawler

Bytespider (ByteDance)

ByteDance / TikTok crawler

CCBot (Common Crawl)

Common Crawl open dataset

Amazonbot (Amazon)

Amazon crawler / Alexa-related

meta-externalagent (Meta)

Meta AI training / fetch agents

cohere-ai (Cohere)

Cohere AI crawler

robots.txt preview

# robots.txt — generated by Instant Tools
# https://instanttools.io/tools/robots-txt-generator
#
# Tip: publish an llms.txt for AI assistants (/llms.txt).
# Generator: https://instanttools.io/tools/llms-txt-generator
#
User-agent: *
Disallow: /admin/
Disallow: /api/private/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

User-agent: cohere-ai
Disallow: /

Sitemap: https://example.com/sitemap.xml

Live check of generated file

Score 100/100 — Parsed OK — 4 AI bot(s) allowed/partial, 8 blocked.

AI bot status

Bot	Status	Detail
GPTBot OpenAI	blocked	Blocked via User-agent: GPTBot
ChatGPT-User OpenAI	allowed	Allowed via User-agent: ChatGPT-User
OAI-SearchBot OpenAI	allowed	Allowed via User-agent: OAI-SearchBot
ClaudeBot Anthropic	blocked	Blocked via User-agent: ClaudeBot
Google-Extended Google	blocked	Blocked via User-agent: Google-Extended
Applebot-Extended Apple	blocked	Blocked via User-agent: Applebot-Extended
PerplexityBot Perplexity	allowed	Allowed via User-agent: PerplexityBot
Bytespider ByteDance	blocked	Blocked via User-agent: Bytespider
CCBot Common Crawl	blocked	Blocked via User-agent: CCBot
Amazonbot Amazon	partial	Allowed via User-agent: * (some paths disallowed)
meta-externalagent Meta	blocked	Blocked via User-agent: meta-externalagent
cohere-ai Cohere	blocked	Blocked via User-agent: cohere-ai

How to Use

1Choose a preset that matches your goals (citations vs training scrape).
2Set Sitemap and Disallow paths for private areas.
3Override individual AI bots (Allow / Block / Inherit).
4Download robots.txt to your site root and re-check in Validate mode anytime.

Common policies

Publisher site

Block GPTBot + Google-Extended · allow search bots

Limits training scrapes while keeping Google Search intact.

Docs / open source

Allow most AI bots · Disallow /admin

Maximize discovery; still hide private paths.

Tips

•Googlebot ≠ Google-Extended — you can keep Search while opting out of Gemini training.
•After changing robots.txt, use Bing/Google URL inspection and IndexNow for faster recrawl.
•Publish /llms.txt so preferred pages stay discoverable even when some bots are blocked.
•Never put secrets behind Disallow alone — use authentication.

Frequently Asked Questions

What does this robots.txt generator do?

It builds a robots.txt with sitemap and path rules, plus explicit Allow/Disallow blocks for major AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more). You can also paste an existing file to validate syntax and see which AI bots are allowed or blocked.

Does blocking GPTBot hurt my Google rankings?

No. GPTBot and Googlebot are different. Blocking GPTBot does not block Google Search. Google-Extended controls Gemini/Vertex training use of your content — it is also separate from Googlebot.

Is robots.txt a security boundary?

No. Compliant crawlers honor it; malicious scrapers can ignore it. Keep private data behind auth — don’t rely on Disallow alone.

How does this relate to llms.txt?

robots.txt controls crawling. llms.txt helps AI systems find the pages you want them to cite. Use both: block unwanted training bots in robots.txt, then publish an llms.txt for the content you do want surfaced.

Does my robots.txt leave the browser?

No. Generation and validation run 100% client-side.

Is it free?

Yes. No signup and no watermark on downloads.

Related Tools

llms.txt Generator

Help AI crawlers find the pages you want cited

Token Counter

Count tokens before shipping long agent prompts

Prompt & Rules Linter

Score system prompts and agent rules

MCP Tool Checker

Score agent tool schemas for readiness

JSON Formatter

Validate JSON configs and manifests

Prompt Diff

Compare system prompts and agent rules side-by-side