robot_2

AI Crawler robots.txt Generator

SEO

Create a robots.txt that blocks or allows GPTBot, ClaudeBot, PerplexityBot and 20+ AI crawlers - with one-click presets.

shield_lock Private & on-device
Live workspace
Preparing…
lock Your files and text are processed in this browser and never uploaded.
In-depth guide

About AI Crawler robots.txt Generator

Why You Need an AI Crawler Policy

AI companies crawl the web to build training sets and to ground answer engines. Whether you welcome that traffic or want to reserve your content for human readers is a legitimate business decision - but only if you express it. robots.txt is the standard mechanism crawlers check before fetching pages, and most major AI labs document that their bots honor it.

How This Generator Works

  1. Pick a preset: Block all AI crawlers, Allow all, or Custom to fine-tune each bot individually.
  2. Toggle individual bots: every major AI crawler is listed with the company behind it - OpenAI, Anthropic, Google, Perplexity, Meta, ByteDance, Amazon, Apple, Common Crawl and more.
  3. Add your sitemap: optionally append a Sitemap: directive so compliant crawlers can still find your pages.
  4. Copy or download: paste the output into your existing robots.txt, or download it and upload it to your site root.

Bots This Tool Covers

GPTBot, OAI-SearchBot, ChatGPT-User, anthropic-ai, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, GoogleOther, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, Diffbot, cohere-ai and cohere-training-data-crawler - with links to each operator's documentation where available.

Important Caveats

  • robots.txt is voluntary for well-behaved crawlers; it does not technically prevent scraping. Combine it with rate limiting and terms of service for real enforcement.
  • Blocking GPTBot does not remove existing training data and does not affect Google Search impressions (Google-Extended is a separate control).
  • If you rely on AI referrals, consider allowing search-oriented bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) while blocking pure training crawlers.
Answers

Frequently asked questions

Does robots.txt actually stop AI scraping?

It stops compliant AI crawlers - OpenAI, Anthropic, Perplexity, Apple, Amazon and Common Crawl all publish and honor their bot tokens. It cannot stop scrapers that ignore the standard, and it is not access control. For enforcement, combine robots.txt with WAF rules and terms of service.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects data for potential model training. OAI-SearchBot indexes content to power ChatGPT's search answers. Blocking GPTBot opts out of training; allowing OAI-SearchBot keeps you eligible for ChatGPT search citations.

Should I block Google-Extended?

Google-Extended controls whether your content is used for Gemini model training and grounding - it does not affect your Google Search rankings or indexing. Block it if you want to withhold content from Gemini training; allow it if you want to be part of Gemini answers.

Where do I put the generated robots.txt file?

At the root of your domain - https://yourdomain.com/robots.txt - served as text/plain with no redirects. Merge new AI rules into an existing robots.txt rather than replacing directives your SEO already relies on.

Will blocking AI crawlers hurt my traffic?

It blocks AI answer engines and training - you may lose AI referrals and citations. Many publishers block training bots (CCBot, GPTBot) while allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) to keep AI visibility without feeding training corpora.

Quick tips

  • Aim for a unique title and description on every page.
  • Preview how search and social platforms will display your content.
  • Use the generated markup as a starting point, then refine the copy.