robots.txt Generator & Validator
Presets
Validator
No storage
Preview
Quick test
FAQ
Generate a clean robots.txt and validate if a path is allowed for a given user agent. Use presets to get started, then customize rules. No data is stored.
What should be in robots.txt?
Include User-agent blocks with Allow/Disallow rules for crawl control, optional Crawl-delay (not widely supported), and Sitemap URLs.
How do rules resolve conflicts?
Typically, the longest matching rule wins. If both Allow and Disallow match, the rule with the most specific path takes precedence.
Can I block indexing with robots.txt?
No. robots.txt controls crawling, not indexing. Use noindex meta tags or HTTP headers for indexing control.