The Robots.txt Generator tool

Cathreena · Joined: 19 Dec 2007 Posts: 49

This tool is used to generate a simple robots.txt file for your website. This file permits you to hide files or directories that you don’t wan the search engine spiders to come across. This generator tool is intended to create the text for the file for you; you can then make changes later should you suppose it necessary.

Generating a robot file with this tool is best if you wan to block some directories or files from search engines. To make use of this generator tool, enter the required information and then click the button. You will then be come across the text for the file. Copy this to a file called robots.txt and put on the root of your website.

John · Joined: 11 Jan 2008 Posts: 31

The Robots.txt Generator tool is otherwise called spider s/w
This usually used in Link exchange right ..... Very Happy

mindi9 · Joined: 02 Jun 2011 Posts: 9

I was trying with google webmaster tool to generate robot.txt file . Here 1st option is "Action"- here I have to select block or allow?
And the second option is "User agent " - here I have to select googlebot? Please guide me. Thanks in advance.

Maddy · Joined: 07 Sep 2011 Posts: 115

thanks for the info. hadnt known it was there till i read this post.

php developer · Guest

If you want to know regarding robots.txt than read follow the link
http://www.robotstxt.org/