Do you need a Robots.txt file for search engines?? Search Engine Optimization Tip 11 February 11th Friday 2005

Robots.txt is a text file placed on the root of the server to tell the search engine crawlers that there are restrictions for the crawlers in crawling the targeted site,

We can perform various tasks with robots.txt, some of them are prevent search engine crawlers from crawling the site, stop crawlers from specific pages/directories etc,

Robots.txt is not necessary for better search engine crawling, it will be good if an empty robots.txt text file is uploaded to the root, that way you can prevent 404 errors when crawlers request fetching the robots file,

Robots.txt file is not for users, it is only for search engines so nothing will be affected if there is no file, But adding one wont make any harm, An empty robots file will be nothing wrong in that,

keep visiting for new tips,

No comments yet.

Leave a comment