Googlebot

robots.txt

Respects robots.txt for all automatic crawling. A Disallow prevents crawling, not necessarily indexing of a URL linked elsewhere: use a noindex tag to deindex.

Verify authenticity

Reverse DNS + forward DNS

Reverse DNS: the IP must resolve to a crawl-*.googlebot.com (or geo-crawl-*.geo.googlebot.com) host, confirmed by a forward DNS lookup. Google also publishes its IP ranges as JSON (common-crawlers.json). The user-agent alone, trivially spoofed, proves nothing.

SEO recommendation

Allow

Allow. Blocking Googlebot means dropping out of Google's index and losing organic traffic. Reserve Disallow rules for areas with no SEO value: cart, unopened faceted filters, back office.

Good to know

Google-Extended is a separate token that controls whether content is used to train Gemini and power AI answers, without affecting Search indexing. Googlebot renders JavaScript through the Web Rendering Service.