Some website owners are worried about using robots exclusion on their sites because they think blocking a crawler from certain pages or certain sections are bad, From search engine genie point of view we recommend to use robots exclusion freely, Make sure you use proper syntax a wrong syntax might confuse the search engine crawler and might get the site dropped from the index,

Excluding robots from certain sections of the site is very good in various ways,

1. You can prevent search engine crawlers from eating up unnecessary bandwidth of your site, For some hosting bandwidth is costly and it is important bandwidth is saved by excluding crawlers from certain sections,

2. Excluding robots from comments section if you run a blog is very good, It prevents robots from seeing or giving credit to any unnecessary links, There might be some bad links left by spammers and those links can be prevented from crawling by excluding those pages from the crawlers, Linking to bad neighbourhood is not good,

3. Excluding robots also helps in preventing sensitive information of your site not getting exposed in search results, There are sites who dont prevent the crawlers properly and those pages contain sensitive data link credit card details, important database etc, So for precautionary measures it is best to exclude the robot from certain areas,

4. Excluding robots also helps in preventing duplicate contents of your site from being crawled, If you have a dynamic site and there are 2 versions of the dynamic pages one search engine friendly and other dynamic with query string it is better to block one version and block the other one that way search engines don’t have to worry about duplicate content on your site,

There are still more benefits from excluding pages / sections from crawlers so do it safely with proper syntax, Use if well if you have big ecommerce dynamic site, Dynamic sites are known to render lots of pages to the crawlers by error, Hope this tip helps,

SEO Blog Team,

                                Earthlink Netscape Netvouz RawSugar Shadows Sphinn StumbleUpon Yahoo MyWeb