MSN webmaster live search blog recently reported how search engines have joined hands on using robots exclusion protocol to not only block them but also allow how users will guide the crawlers. I am sure this is not new but there are some tags described here which are definitely found by Google but not commonly used by other search engine crawlers.

One tag that immediately comes to my mind if the NOODP tag. If this tag is used in meta tag it will tell the Search engine not to show title and description of a Dmoz listing. We at search engine genie use this tag and this was first introduced before around 30 months by Google. Good to know that MSN and Yahoo too understands this tag.

NOODP META Tag

Then we have the crawl Delay settings in robots.txt first introduced by Yahoo. When we set a crawl delay we tell the crawler to fetch the pages from the server at the specified interval of time.
According to MSN Live search blog Microsoft now uses crawl delay and you can set the crawl delay and expect Yahoo and MSN to follow it. Google still doesn't want to use crawl delay. Its understandable on Google's point of view since a crawl delay is something very disturbing for a site.

For detailed reading follow the link above to MSN Live search official blog

Labels: MSN Live Search

                                Earthlink Netscape Netvouz RawSugar Shadows Sphinn StumbleUpon Yahoo MyWeb