Robots.txt Checker

Check whether a site's robots.txt file is reachable, which sitemap files it declares, and whether crawl rules may be blocking important search engine access.

robots.txt checkercrawl rulessitemap directives

Robots.txt is the first crawl access check.

A page audit cannot explain much if crawlers are blocked before they reach the page. This checker reads the access file first, then shows the rules and sitemap signals that shape search discovery.

See whether robots.txt is reachable.

Confirm the file responds and whether the server returns crawler-blocking behavior.

Read allow and disallow paths.

Review crawl rules before diagnosing missing pages as content or indexing problems.

Find sitemap directives.

Use robots.txt as the handoff into sitemap coverage and URL discovery.

Turn access checks into saved reports.

Use the full app for project history, page audits, and repeat checks after fixes ship.

Open the app

What to do with your robots.txt result

A reachable file tells you what it contains. It does not prove that Google indexed your pages or that every crawler can reach the rest of your site.

User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

This illustrative file asks matching crawlers to avoid /private/ and declares a sitemap. It does not keep private content secure. Use authentication for that.

Finding	Next check
Disallow: /	Check the user-agent group. If it applies to Googlebot, it blocks crawling across that hostname.
No sitemap directive	Your sitemap may still exist. Open the sitemap checker and submit the correct URL in Search Console.
404 response	A missing robots.txt does not itself block crawling. Check page-level noindex and access separately.
Server error or access challenge	Review hosting or firewall rules and retest. A failed fetch cannot confirm which crawl rules apply.

Rules belong to specific user-agent groups. A flat list of allow and disallow paths is not a complete decision for an individual Googlebot URL. Use Google's URL Inspection for that page.

Check the declared sitemap, or follow the missing search visibility diagnosis if the page still does not appear. The technical SEO checklist covers the remaining checks.

See Google's robots.txt guidance for crawl controls and indexing limitations.

Robots.txt checker questions.

Short answers for people checking crawl access before a full SEO audit.

What does this robots.txt checker test?

It fetches robots.txt, reports HTTP status, lists sitemap directives, shows user agents, and surfaces allow, disallow, warning, and blocked-crawler signals.

Can robots.txt block Google?

Yes. A broad disallow rule can keep crawlers out of important paths, so robots.txt should be checked before assuming pages have content or metadata problems.

Do I need a Rank Maxer account?

No. This checker is public. The full Rank Maxer app is for saved reports, project history, and deeper crawl audits.