AI crawlers: user agent list and SEO classification

The SEO angle follows from that role. Blocking a training crawler (GPTBot, Google-Extended, ClaudeBot) removes you neither from the search index nor from AI answers that cite their sources: Google states that Google-Extended is not a ranking signal, and citations come from the search crawlers, not from training data. Being cited, and even being recommended by an AI, depends mostly on being picked up by third-party sources (reviews, comparisons, discussions) and on being reachable by the search crawlers, far more than on opening your own site to training.

Then comes actually blocking them, and this is where many get it wrong. The robots.txt file is only a declarative directive: it is honored by operators that choose to follow it (Google, OpenAI, Anthropic), but has no effect on a scraper that spoofs a user agent or ignores it. To genuinely prevent access, an application firewall (WAF) or server rules based on the verified IP (reverse DNS, official ranges) are far more reliable than a Disallow line. In short: robots.txt manages the consent of cooperative bots, not defense against abuse.