Crawler control
robots.txt generator and validator
Build a clean robots.txt file from presets, then test whether a URL path is crawlable for a specific user agent before publishing.
Robots rules
Copy or downloadPreview
Quick test
Frequently asked questions
What should be in robots.txt?
User-agent blocks with Allow and Disallow rules, and Sitemap lines. Crawl-delay is optional and Google ignores it.
How are conflicting rules resolved?
The rule with the longest matching path wins. When an Allow and a Disallow rule match with equal length, Google applies the Allow rule.
Can robots.txt keep a page out of search results?
No. robots.txt controls crawling, not indexing. Use a noindex robots meta tag or X-Robots-Tag header, and leave the page crawlable so the tag can be read.