Complaints<\/strong>
Webmasters complain that yahoo slurp is not tripping the filter given in robots.txt for the following<\/p>\n

User-agent: msnbot
Disallow: \/bloop\/
Disallow: \/blop\/<\/p>\n

User-agent: googlebot
Disallow: \/bloop\/
Disallow: \/blop\/<\/p>\n

User-agent: Slurp
Disallow: \/bloop\/
Disallow: \/blop\/<\/p>\n

User-agent: *
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/<\/p>\n

Yahoo slurp obeys the agent specific rule and hence does not crawl the directories \/bloop\/ and \/blop\/directories where as it crawled the directories in the generic rule.<\/p>\n

Discussions and Suggestions
<\/strong>
Webmasters figured out that not only yahoo but even other search engines behaved in the same manner.<\/p>\n

All the SE BOTS and slurps do not obey the Generic rule.<\/p>\n

Suggestion 1:<\/span> Put the wildcard ones first that has generic rule to less specific bots and then place the agent specific code.<\/p>\n

Suggestion 2:<\/span> main theme behind this is that the major bots get to their corresponding user agents and the rest carry on with wild card user agents.<\/p>\n

Conclusion<\/strong><\/p>\n

All bots do trip and filter the specifications. They are tripped by the server configuration.
The fact is that major bots and slurps get to their respective user agents and the rest continue with the generic rule.<\/p>\n

The exact rule for the above case would be<\/p>\n

User-agent: Slurp
Disallow: \/bloop\/
Disallow: \/blop\/
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n

User-agent: msnbot
Disallow: \/bloop\/
Disallow: \/blop\/
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n

User-agent: googlebot
Disallow: \/bloop\/
Disallow: \/blop\/
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n

User-agent: *
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n

This can be shortened to
<\/strong>
User-agent: Slurp
User-agent: msnbot
User-agent: googlebot
Disallow: \/bloop\/
Disallow: \/blop\/
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n

User-agent: *
Disallow: \/shop\/
Disallow: \/forum\/
Disallow: \/cgi-bin\/
Disallow: \/badbottrap\/<\/p>\n","protected":false},"excerpt":{"rendered":"

ComplaintsWebmasters complain that yahoo slurp is not tripping the filter given in robots.txt for the following User-agent: msnbotDisallow: \/bloop\/Disallow: \/blop\/ User-agent: googlebotDisallow: \/bloop\/Disallow: \/blop\/ User-agent: SlurpDisallow: \/bloop\/Disallow: \/blop\/ User-agent: *Disallow: \/shop\/Disallow: \/forum\/Disallow: \/cgi-bin\/ Yahoo slurp obeys the agent specific rule and hence does not crawl the directories \/bloop\/ and \/blop\/directories where as it crawled the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-310","post","type-post","status-publish","format-standard","hentry","category-yahoo"],"_links":{"self":[{"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/posts\/310","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/comments?post=310"}],"version-history":[{"count":1,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/posts\/310\/revisions"}],"predecessor-version":[{"id":1421,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/posts\/310\/revisions\/1421"}],"wp:attachment":[{"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/media?parent=310"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/categories?post=310"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.searchenginegenie.com\/blog-seo\/wp-json\/wp\/v2\/tags?post=310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}