Robots.txt Checker
Test whether your URLs are allowed or blocked by a robots.txt, for Googlebot and AI bots alike. Anomaly detection and optimized robots.txt.
Paste a robots.txt or fetch a domain's, list the URLs to test and get their status (allowed / blocked) for each bot. Need to create a robots.txt rather than test one? Use the AI robots.txt generator.
How it works
- Live Mode: enter a domain, the robots.txt is fetched automatically.
- Static Mode: paste robots.txt content or drag and drop a file.
- Add the URLs to test (one per line), choose a User-Agent and run the analysis.
- Compare multiple User-Agents via Compare another UA.
Features
- Allow / Disallow verification compliant with Google specification.
- Anomaly detection (full blocking, contradictions, duplicates, missing sitemap).
- Structural optimization suggestions with copyable optimized robots.txt.
- CSV export and TSV copy of results.
Test configuration
Collez directement le contenu d'un fichier robots.txt ci-dessous, ou glissez-deposez un fichier.