AI Crawler robots.txt Generator
SEOCreate a robots.txt that blocks or allows GPTBot, ClaudeBot, PerplexityBot and 20+ AI crawlers - with one-click presets.
About AI Crawler robots.txt Generator
Frequently asked questions
Does robots.txt actually stop AI scraping?
It stops compliant AI crawlers - OpenAI, Anthropic, Perplexity, Apple, Amazon and Common Crawl all publish and honor their bot tokens. It cannot stop scrapers that ignore the standard, and it is not access control. For enforcement, combine robots.txt with WAF rules and terms of service.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data for potential model training. OAI-SearchBot indexes content to power ChatGPT's search answers. Blocking GPTBot opts out of training; allowing OAI-SearchBot keeps you eligible for ChatGPT search citations.
Should I block Google-Extended?
Google-Extended controls whether your content is used for Gemini model training and grounding - it does not affect your Google Search rankings or indexing. Block it if you want to withhold content from Gemini training; allow it if you want to be part of Gemini answers.
Where do I put the generated robots.txt file?
At the root of your domain - https://yourdomain.com/robots.txt - served as text/plain with no redirects. Merge new AI rules into an existing robots.txt rather than replacing directives your SEO already relies on.
Will blocking AI crawlers hurt my traffic?
It blocks AI answer engines and training - you may lose AI referrals and citations. Many publishers block training bots (CCBot, GPTBot) while allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) to keep AI visibility without feeding training corpora.
Quick tips
- Aim for a unique title and description on every page.
- Preview how search and social platforms will display your content.
- Use the generated markup as a starting point, then refine the copy.