Robots file
User-agent: *
Disallow:
The User-agent field is for specifying robot name for which the access policy follows in the Disallow field. Disallow field specifies URLs which the specified robots have no access to. An example:
User-agent: *
Disallow: /
Here “*” means all robots and “/” means all URLs. This is read as, “No access for any search engine to any URL” Since all URLs are preceded by “/ ” so it bans access to all URLs when nothing follows after “/ “. If partial access has to be given, only the banned URL is specified in the Disallow field.
For detailed information http://www.redalkemi.com/articles/robots-tutorial.php
No comments yet.