SEO
Robots.txt Guide: What to Allow, Block, and Test
Write robots.txt rules without accidentally hiding assets or confusing crawling controls with indexing controls.
Robots.txt controls crawling, not guaranteed indexing
A Disallow rule asks compliant crawlers not to fetch a path. It does not reliably remove a known URL from search results because the crawler may still know the URL from external links.
Use a page-level noindex directive when an accessible page should not be indexed. Do not block that page in robots.txt before the crawler can read its noindex directive.
Start with the least restrictive file
Most public sites only need a user-agent group, a few deliberate exclusions and a sitemap line. Avoid copying a long template containing paths your application does not use.
Do not block CSS, JavaScript or image assets needed to render public pages. Search systems need enough resources to understand the same page a user sees.
Match paths carefully
Rules operate on URL paths and small punctuation differences can change their scope. Test whether a slash, wildcard or dollar anchor affects more URLs than intended.
Remember that subdomains and protocols can have separate robots files. Publish the file on the exact host that serves the URLs being crawled.
Test after every material change
Fetch the live robots.txt file, confirm it returns plain text successfully and test representative allowed and disallowed URLs in Search Console.
When a migration changes paths, review old rules. A stale exclusion can silently prevent discovery of a new public section.
Ready to apply this guide?
Open the matching tool, test the recommended settings with one real example, then download or copy the final result after previewing it in context.
Open Robots.txt GeneratorFAQ