Aller au contenu

robots.txt generator for AI crawlers

Choose which AI crawlers to allow or block, in 1 click or bot by bot. Output as robots.txt, Nginx, .htaccess. Comparator on an existing site.

This tool creates and merges a robots.txt targeting AI crawlers, and analyzes that of an existing site.

Decide which AI crawlers to let through based on their usage: LLM training, AI indexing, user-action. Set it up in 1 click (preset), by category or bot by bot, with export to robots.txt, Nginx and .htaccess. You can import your existing robots.txt: only the AI bot blocks are regenerated, everything else is kept (global rules, per-directory Allow/Disallow, Sitemaps, other robots). 188 AI bots tracked (see the full watch), updated automatically.

Creating and merging robots.txt

1. Choose a preset

2. By bot type

Applies the action to every bot in the category and pre-selects them in part 3. Bots set manually are preserved.

LLM training 0
Indexing 0
User-action / live 0
Other 0

3. By bot (optional)

Optional: by default each bot follows its type. Only open this list to set a specific bot.

Fine-tune bot by bot (200 bots)
Other AddSearchBot
Other AgentDataBot
Other AgentTimes
Other AGI Agent
LLM training AI2Bot
User-action / live AI2Bot-DeepResearchEval
LLM training Ai2Bot-Dolma
Other aiHitBot
Other AIWebIndex
Other amazon-kendra
Other amazon-QBusiness
LLM training Amazonbot
User-action / live AmazonBuyForMe
Indexing Amzn-SearchBot
User-action / live Amzn-User
Indexing Andibot
Indexing Anomura
LLM training anthropic-ai
Other ApifyBot
LLM training ApifyWebsiteContentCrawler
Indexing Applebot
LLM training Applebot-Extended
Indexing Aranet-SearchBot
Other atlassian-bot
Other Awario
Indexing AzureAI-SearchBot
Other bedrockbot
Other bigsur.ai
Other BixelBot
Indexing Bravebot
Other Brightbot
LLM training Brightbot 1.0
User-action / live BuddyBot
LLM training Bytespider
Other CartAI Agentic Commerce
LLM training CCBot
Indexing Channel3Bot
LLM training ChatGLM-Spider
User-action / live ChatGPT Agent
User-action / live ChatGPT-User
Other Claude-Code
Indexing Claude-SearchBot
User-action / live Claude-User
LLM training Claude-Web
LLM training ClaudeBot
Other CledaraBot
Other Cloudflare-AutoRAG
Other CloudflareBrowserRenderingCrawler
Other CloudVertexBot
Other Code
Other Codex
User-action / live cohere-ai
LLM training cohere-training-data-crawler
LLM training Cotoyogi
Other CragCrawler
LLM training Crawl4AI
LLM training Crawlspace
Other Cursor
Other Datenbank Crawler
LLM training DeepSeekBot
User-action / live Devin
LLM training Diffbot
User-action / live Diffbot-User
Other Dot
Other DoubaoBot
User-action / live DuckAssistBot
Other Echobot Bot
Other EchoboxBot
Other ElevenlabsBot
Other ERNIEBot
Indexing ExaBot
Indexing ExaSearchBot
Other ExaUser
LLM training FacebookBot
Other facebookexternalhit
Other Factset_spyderbot
LLM training FirecrawlAgent
LLM training FriendlyCrawler
Other GeistHaus-PageFetcher
User-action / live Gemini-Deep-Research
User-action / live Google-Agent
Other Google-CloudVertexBot
LLM training Google-Extended
Other Google-Firebase
Other Google-Gemini-CLI
User-action / live Google-NotebookLM
User-action / live GoogleAgent-Mariner
Other GoogleAgent-URLContext
Other GoogleOther
Other GoogleOther-Image
Other GoogleOther-Video
LLM training GPTBot
Other Grok Bot
Other GrokAppAndroid
Other GrokBot
Other HarkBot
Other HenkBot
Other Hunyuan
Indexing iAskBot
Indexing iaskspider
Indexing iaskspider/2.0
LLM training ICC-Crawler
LLM training ImagesiftBot
LLM training imageSpider
LLM training img2dataset
Other Instinct
Other ISSCyberRiskCrawler
User-action / live kagi-fetcher
LLM training Kangaroo Bot
Other KeenableBot
Other Kimi-Agent
Indexing Kimi-SearchBot
User-action / live Kimi-User
Other KimiBot
Other KlaviyoAIBot
Indexing KunatoCrawler
LLM training laion-huggingface-processor
LLM training LAIONDownloader
LLM training LCC
Other Lightpanda
User-action / live LinerBot
LLM training Linguee Bot
Indexing LinkupBot
User-action / live Manus-User
LLM training meta-externalagent
LLM training Meta-ExternalAgent
User-action / live meta-externalfetcher
User-action / live Meta-ExternalFetcher
Indexing meta-webindexer
Other MistralAI-Index
Other MistralAI-Training
User-action / live MistralAI-User
User-action / live MistralAI-User/1.0
Other MoonshotBot
Other Mozilla-Tabstack
Other Muse
LLM training MyCentralAIScraperBot
Other NagetBot
Other nbot
Other netEstate Imprint Crawler
Other newsai
User-action / live NotebookLM
User-action / live NovaAct
Other OAI-AdsBot
Indexing OAI-SearchBot
LLM training omgili
LLM training omgilibot
Other OpenAI
Other opencode
User-action / live Operator
LLM training PanguBot
Other Panscient
Other panscient.com
Other Payhawk Invoice Fetching Agent
User-action / live Perplexity-User
Indexing PerplexityBot
Other PetalBot
Indexing PhindBot
User-action / live Poggio-Citations
LLM training Poseidon Research Crawler
Other qodercli
Other QualifiedBot
Indexing Querit-SearchBot
Other QueritBot
Other QuillBot
LLM training quillbot.com
Other QwenBot
Other Reflectionbot
LLM training SBIntuitionsBot
Other Scrapy
Other SemrushBot-OCOB
Other SemrushBot-SWA
User-action / live Shap-User
Indexing ShapBot
Other Sidetrade indexer bot
Other Spider
User-action / live TavilyBot
Other Terra Cotta
Indexing TerraCotta
LLM training Thinkbot
LLM training TikTokSpider
Indexing Timpibot
Other TongyiBot
Other Trae
User-action / live TwinAgent
Other UseAI
Other VelenPublicWebCrawler
Other WARDBot
LLM training Webzio-Extended
LLM training webzio-extended
Other wpbot
Indexing WRTNBot
Other xAI-Grok
Other YaK
Indexing YandexAdditional
Indexing YandexAdditionalBot
Other YiBot
Other YiyanBot
Indexing YouBot
Other ZanistaBot

Analyze a site

Check the robots.txt of an existing site: the tool lists the AI crawlers already blocked and those still getting through, and you can add the uncovered bots to your configuration in one click. Handy for benchmarking a competitor.

Analyze an existing site

Enter a domain (with or without https://): we fetch its /robots.txt and list the AI bots covered / not covered.

Check your robots.txt before publishing it

Once your robots.txt is generated, test it with the Robots.txt Checker: it checks whether your URLs are allowed or blocked, for Googlebot and AI bots, and spots anomalies.

Open the Robots.txt Checker

Learn more

Robots.txt: laissons à Google ce qui lui appartientLes en-têtes HTTP ou comment maîtriser son indexation sur Google