|
SEO
Home » SEO
Tools » Robot
File Code Generator Tool
Simply select a robot (or all) and enter the path or paths you
wish to exclude the bot from.
To completely ban a spider from your web site, select it and enter
/ in one of the path/directory boxes.
To exclude different spiders from different areas, use this tool
several times, once for each spider. Simply copy/paste all results,
one after another, into one file.
Search
Engine Optimization SEO Tools |
Robot
File Code Generator |
|
This robots file code generator tool helps to develop robots.txt
file to exclude the pages that you don’t want to index. This
tool has fields to enter the robot to exclude and the directories
to be excluded from indexing. On clicking create the robots.txt
file will be created which you can use directly.
Entry |
Meaning |
User-agent: *
Disallow:
|
The asterisk (*) in the User-agent field is shorthand for
"all robots". Because nothing is disallowed, everything
is allowed. |
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/
|
In this example, all robots can visit every directory
except the three mentioned. |
User-agent: BadBot
Disallow: /
|
In this case, the BadBot robot is not allowed
to see anything. The slash is shorthand for "all directories"
The User Agent can be any unique substring, and robots are not
supposed to care about capitalization.
|
User-agent: BadBot
Disallow: /
User-agent: *
Disallow: /private/
|
The blank line indicates a new "record"
- a new user agent command.
BadBot should uts go away. All other robots can see everything
except the "private" folder.
|
User-agent: WeirdBot
Disallow: /tmp/
Disallow: /private/
Disallow: /links/listing.html
User-agent: *
Disallow: /tmp/
Disallow: /private/
|
This keeps the WeirdBot from visiting the listing
page in the links directory, the tmp directory and the private
directory.
All other robots can see everything except the tmp and private
directories.
If you think this is inefficient, you're right!
|
Search
engine robots that visit your web site
|
