Robots.txt Explained: Complete SEO Guide
Learn how robots.txt works, what it can and cannot do, common mistakes, and how to create a crawler-friendly robots.txt file.
What Is robots.txt?
Robots.txt is a text file placed at the root of a website that provides crawler instructions about which paths automated agents may request. It is commonly used to manage crawling rather than as a general page-removal mechanism.
Basic robots.txt Structure
User-agent: * Disallow: /private/ Allow: /private/public-file.html Sitemap: https://example.com/sitemap.xmlCommon Mistakes
Blocking Important Pages
An overly broad Disallow rule can prevent crawlers from requesting important resources or pages.
Using robots.txt to Hide Sensitive Data
Do not use robots.txt as an access-control system. If information must be private, protect it with authentication and authorization.
Forgetting the Sitemap
Including your XML sitemap URL can make discovery easier for crawlers.
Robots.txt vs Noindex
These mechanisms serve different purposes. Robots.txt controls crawling requests, while a noindex directive is used to tell compliant search engines not to index a page. A page blocked from crawling cannot reliably be evaluated for a noindex directive.
For a broader audit, see our technical SEO audit checklist.
Conclusion
Keep robots.txt simple and intentional. Review it whenever you launch new sections, change URL structures, or migrate a website.