What Is a Robots.txt File?
A robots.txt file is a text file placed at the root of a website that can provide crawling instructions to automated crawlers. It can contain rules for specific user-agents and website paths.
How Does Robots.txt Work?
Crawlers can request the robots.txt file before crawling a website and may use its rules to determine which paths they should or should not request.
How to Use This Robots.txt Generator
Select a user-agent, enter any paths you want to disallow or allow, optionally add a crawl-delay value and provide your XML sitemap URL. Then click Generate Robots.txt.
Where Should robots.txt Be Placed?
For a typical website, robots.txt is placed at the root of the host, such as:
https://example.com/robots.txt
Important Note About Robots.txt
Robots.txt is not a security mechanism. A Disallow rule does not make private content secure or inaccessible. Sensitive information should be protected using proper authentication and access controls.
What About Sitemaps?
A sitemap URL can be included in robots.txt to make the location of an XML sitemap discoverable to crawlers that support it.