The User Agent can be any unique substring, and robots are not supposed to care about capitalization.
Disallow: / User-agent: *
BadBot should uts go away. All other robots can see everything except the "private" folder.
Disallow: /links/listing.html User-agent: *
All other robots can see everything except the tmp and private directories.
If you think this is inefficient, you're right!
Our Searchengine Robot File Code Generator Tool is 100% free to use. Please support this tool by linking to it.
Your link will look like this: Robot File Code Generator Tool
If you are a regular user of this tool for your website please support us by linking to it, We know people do 100s of query per day and don't care to support us. If you cant support us we can't continue to keep this tool Ad free. We are not like others. There are other tool providers we know of who have the same tools we have but like to charge more than 40$ a month just to use them.
We know others who just restrict the usage of tools like 10 queries per hour or 30 queries per day. We never do that or will ever want to do that. But to make this tool Ad free you much support this tool by linking to it. Links bring more attention to our tools and more backlinks. We just want that. We are working hard everyday to bring new tools to our list. We currently have more than 50 tools and the list will keep expanding and one point we are planning to give anywhere between 75 to 100 tools for free.
Remember that all the queries you make uses our server resources and we don't want to restrict anyone from doing it but make sure you link to us which makes us Happy in providing these tools always free and makes us eager to make more tools. Check other sites they just have 10 tools and sell text links, run adsense ads, banner ads all over their site. Check our site we have been providing free tools for more than 3 years and our regular users know we never run any Ads in our tools nor we sell or want to sell text links.