Robots.txt Crawler

Robots-txt crawls the Web to help publishers protect their site and decrease bot traffic. We save robots.txt files from across the Web and make them available so bots don't need to constantly crawl your site to see if the rules have changed. We never share any of your content, only robots.txt rules. Below is the crawler user-agent string you will see in your logs:





See our tools