Friday, February 03, 2006
Recently we had a crawl from Yahoo-Blogs/v3.9 bot, they were searching for our xml feed, think this is new yahoo seem to be planning to start something similiar to blogsearch.google.com ( google's blog search )
This is what yahoo says about their bot
"Yahoo-Blogs/v3.9 is Yahoo!'s blog indexing robot. As part of the crawling effort, Yahoo!'s blog crawler will take robots.txt standards into account to ensure we do not crawl and index content from those pages whose content you do not want included in Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, Yahoo! will not read or use the contents of that page. The URL of a protected page may be included in Yahoo! Search Technology as a "thin" document with no text content. Links and reference text from other public web pages provide identifiable information about a URL and may be indexed as part of web search coverage.
help.yahoo.com/help/us/ysearch/crawling/crawling-02.html
This is what yahoo says about their bot
"Yahoo-Blogs/v3.9 is Yahoo!'s blog indexing robot. As part of the crawling effort, Yahoo!'s blog crawler will take robots.txt standards into account to ensure we do not crawl and index content from those pages whose content you do not want included in Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, Yahoo! will not read or use the contents of that page. The URL of a protected page may be included in Yahoo! Search Technology as a "thin" document with no text content. Links and reference text from other public web pages provide identifiable information about a URL and may be indexed as part of web search coverage.
help.yahoo.com/help/us/ysearch/crawling/crawling-02.html
Earthlink
Netscape
Netvouz
RawSugar
Shadows
Sphinn
StumbleUpon
Yahoo MyWeb
posted by power
@ 7:05 PM
permanent link | Post a Comment |
0 Comments:
Post a Comment
<< Home