Robots.txt Checker
Check your robots.txt in the browser. Get a 0 to 100 audit, test any URL against Googlebot, Bingbot and more, and see line by line fixes. No upload, no login.
Enter a site URL or paste your robots.txt to begin.
What this tool does
Paste your robots.txt or load it from a site URL and get an instant, scored audit right in your browser, with no login and nothing to upload. This robots.txt checker works as a validator and a tester at once: it parses every directive, flags syntax problems line by line, and lets you test any path against Googlebot, Bingbot and other crawlers to see if it is allowed or blocked. You also get sitemap directive checks and plain language fixes. Search Console is still the place to see the exact file Google cached or to request a recrawl. Here you get the fast, scored second opinion, plus a clear reminder that robots.txt controls crawling, not indexing, so it cannot noindex a page or hide anything.
What you can use it for
- Check whether your robots.txt is accidentally blocking Google from important pages before it costs you traffic.
- Validate the syntax of a new robots.txt and catch crawler rules that sit in the wrong group or before any User-agent.
- Test a single path, like /blog or /wp-admin, against Googlebot, Bingbot, GPTBot or a custom crawler to see if it is allowed or blocked.
- Confirm you declared at least one absolute Sitemap directive so search engines can discover your URLs.
- Hand a client a clear, scored report that explains each robots.txt issue in plain language.
How to use it
- Paste your robots.txt content, or type a site URL and we will try to read its robots.txt from your browser.
- If the browser blocks the URL read (CORS), open the file at https://yoursite.com/robots.txt, copy it, and paste it here. Every check still runs locally.
- Read the 0 to 100 score and the grouped checks for structure, crawlability, sitemaps and hygiene, each with a line reference and a fix.
- Use the URL tester to check any path against Googlebot, Bingbot or a custom crawler and see exactly which rule matched.
- Copy or download the report to share the findings or keep them for later.
Everything runs inside your browser. No file is uploaded to any server. See more tools in this field.
Related tools
SEOFrequently asked questions
Does robots.txt stop a page from showing in Google?
No. robots.txt controls crawling, not indexing. A page you disallow can still be indexed if other pages link to it. To keep a page out of search, use a noindex meta tag or header and leave the page crawlable.
Why did the URL fetch fail when I entered my site address?
Browsers block cross site reads for security (CORS), so reading another site's robots.txt straight from your browser often fails even when the file is perfectly fine. Paste the file contents instead and every check runs locally.
Is it safe to list private or admin paths in robots.txt?
No. robots.txt is public and only advisory. Listing a secret path tells everyone it exists and does not protect it. Use real authentication or a noindex, and keep sensitive paths out of the file.
What does Disallow: / do, and why is the checker warning me?
Disallow: / for User-agent: * blocks crawlers from your entire site. That is fine for a staging or private site, but on a live site it can remove you from search, so the checker flags it as a critical issue.
How do I add a sitemap to robots.txt?
Add a line like Sitemap: https://example.com/sitemap.xml using an absolute URL. You can list more than one. The sitemap directive helps crawlers discover your pages, and the checker warns when no sitemap is declared.
How do I check if robots.txt is blocking Googlebot?
Paste your file, then use the URL tester: enter a path and pick Googlebot. The checker shows whether it is allowed or blocked and which rule matched. Avoid broadly disallowing CSS, JS or image paths, since that can stop Googlebot rendering your pages.