Yahoo crawler team have come out with a new crawler that will crawl the web in a more sophisticated way. Yahoo Slup 3.0 is the new version of yahoo’s search engine indexing crawler that has been introduced recently for crawling the web. Yahoo Search engine is the 2nd most important search engine only next to Google and its very important we give attention to yahoo search rankings.<\/p>\n
Yahoo has done a major crawler infrastructure update and this is causing massive indexing in major sites. Things should be settling down very soon but yahoo has reported new crawler activity from a different set of IPs for many sites.<\/p>\n
According to yahoo the same derivatives in robots.txt for New Yahoo slup 3.0 will work,<\/p>\n
“The crawlers will also publish a new user-agent, ‘Yahoo! Slurp\/3.0.’ Existing
robots.txt directives for ‘Slurp’ or ‘Yahoo! Slurp’ will continue to work, but
if you have directives specific to ‘Slurp\/2.0,’ they won’t be recognized by the
new crawler (though usage of the ‘Slurp\/2.0’ user-agent is very rare on the web,
so you won’t likely be affected). We recommend specifying the shorter version
of: User-agent: Slurp. “<\/p><\/blockquote>\nOfficial statement from Yahoo search news blog<\/p>\n