Crawler control
robots.txt Generator & Validator
Build a clean robots.txt file from presets, then test whether a URL path is crawlable for a specific user agent before publishing.
Robots rules
Copy or downloadPreview
Quick test
FAQ
Generate a clean robots.txt and validate if a path is allowed for a given user agent. Use presets to get started, then customize rules. No data is stored.
What should be in robots.txt?
Include User-agent blocks with Allow/Disallow rules for crawl control, optional Crawl-delay (not widely supported), and Sitemap URLs.
How do rules resolve conflicts?
Typically, the longest matching rule wins. If both Allow and Disallow match, the rule with the most specific path takes precedence.
Can I block indexing with robots.txt?
No. robots.txt controls crawling, not indexing. Use noindex meta tags or HTTP headers for indexing control.