Robots.txt: What It Can and Cannot Do
Understand crawler rules, common mistakes and the limits of robots.txt.
What the file does
A robots.txt file is a plain-text set of instructions for crawlers that choose to follow the Robots Exclusion Protocol. It is normally published at the root of a host, such as https://example.com/robots.txt. Crawlers look there for rules about paths they may request.
What it does not do
Robots.txt is not a password, firewall or reliable way to hide a URL. A blocked URL may still be known from links elsewhere, and a crawler may still show a URL without a description. Protect private material with authentication and access controls.
Understand the main directives
User-agent identifies the crawler group a rule applies to. Disallow asks a compliant crawler not to request a matching path. Allow can make an exception to a broader disallow rule for crawlers that support it. Sitemap can point crawlers to an XML sitemap.
Avoid blocking important resources
A broad rule can accidentally prevent crawlers from fetching assets or pages that should be available. Review every path before publishing, especially rules that target folders such as /assets/, /wp-content/ or /blog/.
Test after publishing
Open the file directly in a browser and check that it returns the expected text. Use the relevant search engine tools where available to test important URLs. Revisit the file after site migrations, CMS changes or new sections.
A sensible workflow
Start with the smallest set of rules needed. Keep a copy of the previous file, change one thing at a time and monitor crawling after publication. If the goal is to keep a page out of search results, learn about noindex and removal controls instead of assuming a Disallow rule will do that job.
Use the related tools on FrontSEOTools to prepare and review your own website work. Generated results should always be checked against the needs of your site.