SEO Home » Yahoo Articles » Yahoo Slurp

Yahoo Slurp

Yahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using the Yahoo! search engine. These documents are discovered and crawled because other web pages contain links directing to these documents.
As part of the crawling effort, the Yahoo! Slurp crawler will take robots.txt standards into account to ensure we do not crawl and index those pages that you would not like to have returned via Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, it is neither considered for inclusion nor placed in the search engine's database.
Yahoo! Slurp follows HREF links. It does not follow SRC links. This means that Yahoo! Slurp does not retrieve or index individual frames referred to by SRC links.
Yahoo! Slurp has support for frames and makes an effort to crawl complex URLs such as those generated by forms, content generation systems, and dynamic page generation software.

How can I prevent Yahoo Slurp from following links from a particular page or archiving a copy of a page?

Yahoo! Slurp obeys the noindex meta-tag. If you place:
<META NAME="robots" CONTENT="noindex">
-in the head of your web document, Yahoo! Slurp will retrieve the document, but it will not index the document or place it in the search engine's database.

Yahoo Slurp indexes not only the title and meta tags, but also the full text of webpages. So including quality content in the webpage is as important as including keywords in the title and meta tags.
Yahoo searches for pages and when it finds a page with the required keyword, it lists the page in its SERPs. The position of the page depends on the content. But there are chances for a page with the required keyword being left out due to poor content or because Yahoo could not find the page.

How to attract Yahoo slurp to crawl the site ?

There are 3 ways you attract the Yahoo Crawler in crawling the site:

1. Get links from sites that are regularly crawled by the Yahoo Robot, If that is done Yahoo regular visits the site and crawls it, Regular Yahoo visit is a good sign and helps a lot of getting good Ranking,

2. As yahoo says you can trigger the Yahoo Robot by browsing a site using the Yahoo companion toolbar, Yahoo says this will trigger the Yahoo slurp Bot,

3. Through the Infamous PFI/PPC program sitematch, This type of inclusion guarantee's an inclusion into the Yahoo index, so no problem using it,

For more information on Search Engine Genie’s services, contact us at webpromotions@gmail.com