Home About
Services
⚙️ Technical SEO 📝 Content Strategy 🔍 SEO Audits 🔗 Link Building 📊 Analytics & Reporting 🌍 Local & International SEO 🛒 E-Commerce SEO 🚀 SEO Migration & Relaunch
Tools Blog Contact Get in Touch
SEO Tool

Robots.txt Tester

Paste your robots.txt content and a URL path to check which crawlers are allowed or blocked. Runs entirely in your browser — nothing is sent to a server.

?
Paste your robots.txt and enter a path to test

How robots.txt works

  • User-agent blocks target specific crawlers. The * wildcard applies to all crawlers not matched by a specific block.
  • Allow/Disallow rules are matched by prefix. Disallow: /admin/ blocks /admin/settings, /admin/users, etc.
  • Specificity wins: longer matching paths take priority. Allow: /admin/public overrides Disallow: /admin/ for paths under /admin/public.
  • Robots.txt is advisory — well-behaved crawlers follow it, but it's not a security mechanism.
Related resources

Get crawling right

Robots.txt is just one piece of the crawlability puzzle. For a full technical review — including indexation, rendering, and Core Web Vitals — see the 2026 technical SEO checklist or get a comprehensive SEO audit.

Free 60-Min Consultation → More SEO tools →

Crawl rules are sorted. What about the rest?

Robots.txt is one of fifty things that decide whether your site ranks. If you want senior eyes on the whole picture, let's talk.

Book a Free Strategy Call →