Does Google’s crawler active on one day compared to another day?

There are few people who report Google is indexing pages more on weekdays than weekends, also it seems Google’s traffic in much more in first 3 weekdays than towards weekends or Fridays.
I do agree with the traffic point its obvious that weekdays are much more popular than weekends. People tend to use computers more on weekdays especially from work places. We monitor a lot of websites and the pattern remains the same.

But for pages indexed I don’t buy the argument. If you see more pages indexed on some weekdays it could just be a coincidence. What I have seen when Google indexes pages it keeps them in its index for a long time. So when a page is indexed on say Monday it will still remain on Saturday. So the numbers should virtually remain same as of Monday. But from what I have seen, sometimes lot of crawling happens on weekends and sometimes it happens on weekdays. I don’t see much difference; I think it has to be mostly with the person who operates the crawlers.

Google’s influence on Yahoo

Those of us who were in search engine optimization for many years know once yahoo results were completely powered by Google. Google used to have almost 90% market share excluding only MSN and its powered search engines. Where are we now today do we still see any relationships with Google. We recently had a major controversy where Yahoo had a deal with Google to display adwords ads in its search results. But Microsoft was not happy with it.
First we had yahoo make a deal with Google
“Yahoo said it had agreed to let Google put search ads on its site in what it called an $800 million annual revenue opportunity that would boost cash flow by $250 million to $450 million in the first 12 months.
Yahoo’s ads and Google’s would be pitted against each other in an auction style process that could make a deal easier to pass regulatory approval.
“Yahoo is being a reseller of Google whenever it makes sense and that is likely to be a lot of the time given how much more effective Google Web search ads have proven to be,” Global Crown Capital analyst Martin Pyykkonen said.”

http://www.reuters.com/article/topNews/idUSN1247863820080612?feedType=RSS&feedName=topNews

Then Google decided to dump yahoo and the rift began:
“The U.S. Justice Department said on Wednesday it had told Google it planned to file a lawsuit to block the deal, under which Google would have placed its more lucrative ads on Yahoo searches.
“Had the companies implemented their arrangement, Yahoo’s competition likely would have been blunted immediately with respect to the search pages that Yahoo chose to fill with ads sold by Google rather than its own ads,” the government said.
Yahoo regretted Google’s decision, saying it was “disappointed that Google has elected to withdraw from the agreement rather than defend it in court.”

Google suggest one of the find of Google i would say is ever evolving

When first Google introduced it as a beta version everyone liked it and later Google moved suggest option to Google.com regular search. One thing was lagging but. Google was not showing suggestions after we do a search on Google.com homepage. Once you navigate from the page and into results page the suggestion stops. I personally wanted suggest to work both in homepage as well as results page. Now Google has made the changes and it works in both versions now.

Similarly now Google has introduced personalized search which gets saved into web history as preferred suggestions. All the searches you previously made when logged into Google will show up first before the regular suggestions. They also now provide the ability to remove the personalized search keywords which is cool I would say.

Also direct links now appear if you are looking for a specific site. Google’s intelligent algorithm understands your motive to find a website based on your partial keyword input and will show you the correct URL of the site you might want to reach.

Google never missed out commercializing Google suggest. They also have suggestions for sponsored links after all they need to impress their share holders right?

Google webmaster tools new features:

Highlights

• One-stop Dashboard: We redesigned our dashboard to bring together data you view regularly: Links to your site, Top search queries, Sitemaps, and Crawl errors.
• More top search queries: You now have up to 100 queries to track for impressions and click through! In addition, we’ve substantially improved data quality in this area.
• Sitemap tracking for multiple users: In the past, you were unable to monitor Sitemaps submitted by other users or via mechanisms like robots.txt. Now you can track the status of Sitemaps submitted by other users in addition to yourself.
• Message subscription: To make sure you never miss an important notification, you can subscribe to Message Center notifications via e-mail. Stay up-to-date without having to log in as frequently.
• Improved menu and navigation: We reorganized our features into a more logical grouping, making them easier to find and access. More details on changes.
• Smarter help: Every page displays links to relevant Help Center articles and by the way, we’ve streamlined our Help Center and made it easier to use.
• Sites must be verified to access detailed functionality: Since we’re providing so much more data, going forward your site must be verified before you can access any features in Webmaster Tools, including features such as Sitemaps, Test Robots.txt and Generate Robots.txt which were previously available for unverified sites. If you submit Sitemaps for unverified sites, you can continue to do so using Sitemap pings or by including the Sitemap location in your robots.txt file.
• Removal of the enhanced Image Search option: We’re always iterating and improving on our services, both by adding new product attributes and removing old ones. With this release, the enhanced Image Search option is no longer a component of Webmaster Tools. The Google Image Labeler will continue to select images from sites regardless of this setting.

Webmaster tools has now many new features, when you sign into webmaster tools you will see a new home for your site with a message center, and all the sites that you have. To reach a verified site there a one stop dashboard this gives you all the highlights from the data. You can now get your favorite features easily, more navigation and trouble shoot problems can be seen, additionally you can now see more search queries for your site that appears in better than never before. You have robots.txt and URLs in access for some time. But now all tools are together at last under one tab. We already sent messages to your site to webmaster tools Inbox now you can forward those messages to people you know. We all really enjoyed redesigning webmaster tools. This is just a beginning stay tuned for more updates.

Google introduces new features in searcology coference:

As people get more sophisticated at search they are coming to us to solve more complex problems. To stay on top of this, we have spent a lot of time looking at how we can better understand the wide range of information that’s on the web and quickly connect people to just the nuggets they need at that moment. We want to help our users find more useful information, and do more useful things with it.

Our first announcement today is a new set of features that we call Search Options, which are a collection of tools that let you slice and dice your results and generate different views to find what you need faster and easier. Search Options helps solve a problem that can be vexing: what query should I ask?

Let’s say you are looking for forum discussions about a specific product, but are most interested in ones that have taken place more recently. That’s not an easy query to formulate, but with Search Options you can search for the product’s name, apply the option to filter out anything but forum sites, and then apply an option to only see results from the past week. Just last week, at our Shareholders’ Meeting, I had a woman ask me why she couldn’t organize her results by time, with the most recent information appearing first. “Come back Tuesday,” I wanted to say!

The Search Options panel also gives you the ability to view your results in new ways. One view gives you more information about each result, including images as well as text, while others let you explore and iterate your search in different ways. We think of the Search Options panel as a tool belt that gives you new ways to interact with Google Search, and we plan to fill it with more innovative and useful features in the future.

Another challenging problem we have worked on is better understanding the information you get back from a search. When you see your results from a Google search, how do you decide which one has the best information for you? Or, how can we help you make the best decision about where to click?

We call the set of information we return with each result a “snippet,” and today we are announcing that some of our snippets are going to get richer. These “rich snippets” extract and show more useful information from web pages than the preview text that you are used to seeing.

Google competitor Wolfram alpha launching this month May 2009:

The long-expected Wolfram Alpha search engine is due to be launched this month. We are waiting for it anxiously as unfortunately we didn’t get the opportunity to test it, however, others did and it looks amazing. I will start with the fact that many said that this is the Google Killer, but in fact Wolfram Alpha is not a conventional search engine, it is more a computational knowledge engine which is based on ideas from Stephen Wolfram. Recently, Google launched its public data search, and not even that can be compared to Wolfram Alpha.

Don think of Wolfram Alpha as a Google Killer, though, because frankly Google doesn’t really have anything like it—except for maybe Google’s new public data search, which, while impressive, doesn’t look nearly as robust as Wolfram Alpha. (Then again, we’ll have to wait and see how well Wolfram Alpha works when it gets in the hands of the public.) Either way, Google will still corner the market on most normal search. (We’re not always looking for the kind of answers Wolfram Alpha provides when we hit up Google.) As for how this editor uses Google and Wikipedia, I’d actually imagine that Wolfram Alpha could be more of a Wikipedia competitor than a Google competitor.
The system, Wolfram Alpha, was developed by Stephen Wolfram (49), a British physicist, and showcased at Harvard University in the U.S. last week. “Revolutionary new web software could put giants such as Google in the shade,” the daily claimed. Although the system is still new, it has already attracted massive hype among technology pundits, it added.

“Wolfram Alpha will not only give a straight answer to questions such as ‘how high is Mount Everest?’ but it will also produce a neat page of related information – all properly sourced – such as geographical location and nearby towns, and other mountains, complete with graphs and charts,” it said. “Or ask what the weather was like in London on the day John F. Kennedy was assassinated, it will cross-check and provide the answer.”

http://www.wolframalpha.com/index.html

Google Patent to rank personalized pages based on bookmarking:

I am thinking about crawling social media profiles and links shared on those profiles, to which you link on your Google profiles, maybe even Gmail and gchat; and of course sites you join on Google friend connect; and what you share through Google and Google reader. And once they identify your twitter hyperlinked idiosyncrasies, they could then discover those of your followers and rank documents based on what everyone loves… or loves to spam 😉

And ultimately distinguishing what one and one’s followers and friends truly love and love to spam, is the feature measuring you’re ‘linger time:’

1. A computer-implemented method, the method comprising: receiving a search query from a user; receiving a request from the user to personalize a search result; responsive to the search query and the request to personalize the search result, generating a personalized search result by searching a personalized search object; responsive to the search query, generating a general search result by searching a general search object; providing the personalized search result and the general search result for display; selecting an advertisement based at least in part upon the personalized search object; and providing the advertisement for display.

2. The method of claim 1, wherein the personalized search object comprises an article associated with a bookmark.

3. The method of claim 2, wherein an index associated with the bookmark is stored on a server remote from a client with which the bookmark is associated.

4. The method of claim 2, wherein an index associated with the bookmark is stored on a client with which the bookmark is associated wherein searching of the personalized search object is performed by a client-side agent.

5. The method of claim 1, wherein the general search object comprises an index of articles.

6. The method of claim 5, wherein the index comprises an index of articles associated with a global computer network.

7. The method of claim 1, wherein the general search object comprises a plurality of global indices.

8. The method of claim 1, wherein the personalized search object comprises a plurality of bookmarks.

9. The method of claim 1, wherein the personalized search object comprises an annotation.

10. The method of claim 1, wherein the personalized search object comprises a rating.

11. The method of claim 1, further comprising identifying a user cluster based at least in part on the personalized search object and providing to the user a suggestion of another user with which to network based on the user cluster.

12. The method of claim 1, further comprising identifying the personalized search object based at least in part on an implicit measure of the user’s interest.

13. The method of claim 12, wherein the implicit measure of the user’s interest comprises a history of user accesses.

14. The method of claim 12, wherein the history of user accesses comprises at least one of: a period of linger time, a quantity of repeat visits, and a quantity of click-through.

15. A computer storage medium encoded with a computer program, the computer program comprising instructions that when executed cause a computer to perform operations comprising: receiving a search query from a user; receiving a request from the user to personalize the search result; responsive to the search query and the request to personalize the search result, generating a personalized result by searching a personalized search object; responsive to the search query, generating a general result by searching a general search object; providing the personalized search result and the general search result for display; and providing an advertisement for display on a browser based at least in part on one of the personalized search result and the general search result.

16. The computer storage medium of claim 15, wherein the instructions when executed cause the computer to perform operations further comprising identifying a cluster of users based at least in part on the personalized search object.

17. The computer storage medium of claim 15, wherein the instructions when executed cause the computer to perform operations further comprising identifying the personalized search object based at least in part on an implicit measure of the user’s interest.

Google moving to Ajax based result pages:

It seems Google is now moving into Ajax based result pages. As per the Google analytics blog “Starting this week, you may start seeing a new referring URL format for visitors coming from Google search result pages. Up to now, the usual referrer for clicks on search results for the term “flowers”,”

The key difference between these two urls is that instead of “/search?” the URL contains a “/url?”. If you run your own analyses, be sure that you do not depend on the “/search?” portion of the URL to determine if a visit started with an organic search click.

New parameters as per Google blog:
——- old
http://www.google.com/search
hl=en
q=flowers
btnG=Google+Search

——- new
http://www.google.com/url
sa=t
source=web
ct=res
cd=7
url=http%3A%2F%2Fwww.example.com%2Fmypage.htm
ei=0SjdSa-1N5O8M_qW8dQN
rct=j
q=flowers
usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w
sig2=X8uCFh6IoPtnwmvGMULQfw

According to Mattcutts a senior Google employee this change is to make sure search results are retrieved faster than it usually does. Matt says “The team there only thinks about speed. They want to get the results back to users as quick as humanly possible. JavaScript makes the search results a lot faster. Suppose you do a search for flowers, as you’re typing flowers, they can do a query from the back end and fold search results right into the page. You’re still in Google.com and they can pull in the results automatically.”

Stealth links and Googlebot :

Webmaster world owner and senior webmaster Brett Tabke posted an interesting thread what he calls stealth links in Google. Links that are not the same HREF links but still seem to count in Google. He calls them stealth links according to him following are some prominent stealth links:

• another site links to your graphics (img src)
• a site links to your javascript files
• a site links to your css files?
• rss feeds and other xml feeds that people can link to without notice or referrals necc being generated.
• links in email that some se’s can read (yahoo mail, hotmail, Gmail)
• links marked with noindex
• links marked with nofollow
• urls within javascript or js comments
• raw urls within css or css comments
• urls within meta data of graphics and video files
• urls within html comments
• urls within the head section or meta data of a html page
• links or pages that maybe surfed while visitor has page rank engaged on the toolbar
• the target of a constructed, obfuscated, or encrypted js url (hidden until executed)
• links behind pay walls that Google can spider via webmaster tools
• Domains that have been 301’d with links.
• Links in Flash movies (games, quizzes, etc).
• non href’ed url’s. (raw url on page http://www.webmasterworld.com)
• Links in any documents other than web pages e.g. .doc, .pdf, .txt, etc.
• blocking a page in robots.txt should make it blocked from bots, but they still spider it.

According to me most of links from the above sources are not counted by Google atleast for ranking purposes. Brett came up with the above list because of the discussion that was started in another thread about pages getting pagerank without any external links. I feel most of the people who are complaining about pages getting pagerank without external links either don’t know to check backlinks or just rely on yahoo and Google backlink data which is totally unreliable.

Does Protected Whois affect Google rankings?

Many of us prefer to protect our whois data to avoid spammers and scammers getting our email addresses for potential abuse. We protect our privacy on some of our sites though we don’t do it on our company website. Many of us know Google uses whois data in for search engine rankings. They primarily use this to avoid spammers capturing expired domains and using the backlink power of those domains and use it for their own website.

This Google patent already describes Google’s usage of whois for ranking purposes.

(http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=

HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=

1&f=G&l=50&co1=AND&d=PG01&s1=20050071741&OS=20050071741&RS=20050071741 )

Some extra from that Google patent:
1. A geographic information system (GIS) comprising information about a plurality of geospatial entities and configured to prioritize the geospatial entities according to a ranking mechanism.

2. The system of claim 1, wherein the ranking mechanism uses data about a meta attribute of a geospatial entity to determine the geospatial entity’s priority.

3. The system of claim 2, wherein the meta attribute comprises one of: quality of information available about the geospatial entity, an attribute of a description of the geospatial entity, and an attribute of a definition of the geospatial entity.

4. The system of claim 2, wherein the meta attribute comprises an indicator of the geospatial entity’s popularity.

5. The system of claim 2, wherein the meta attribute comprises one of: an age attribute, a stature attribute, and an importance attribute.

6. The system of claim 2, wherein the meta attribute comprises a relationship of a geospatial entity to its place in a hierarchy of geospatial entities.

7. The system of claim 1, wherein an entity of the plurality of entities comprises a collection of geospatial objects and wherein the priority of the entity is determined responsive to a characteristic of the collection of geospatial objects.

8. The system of claim 1, wherein an entity of the plurality of entities comprises a geospatial entity defined in an on-line forum and wherein the ranking mechanism uses data generated in the on-line forum to determine the rank of the geospatial entity.

9. The system of claim 1, wherein the ranking mechanism uses data harvested from a website on the internet about a geospatial entity to determine the geospatial entity’s priority.

10. The system of claim 1, wherein the ranking mechanism determines a geospatial entity’s priority from a combination of weighted data from a plurality of meta attributes of the geospatial entity.

11. A computer-implemented method for ranking geospatial entities, the method comprising: receiving geospatial entity data; evaluating attributes of geospatial entities included in the received geospatial entity data; ranking the geospatial entities based on the evaluation; and storing the ranked geospatial entity data.

12. The method of claim 11, wherein the geospatial entity data comprises data generated in a community forum.

13. The method of claim 11, wherein the geospatial entity data comprises data associated with a specific user and further comprising using the ranked geospatial entity data to generate a map for the specific user.

14. The method of claim 11, further comprising selecting geospatial entities for a geographical display based on the rankings of the geospatial entities.

15. The method of claim 11, further comprising providing the ranked geospatial entity data to a map system configured to generate a map that includes ranked geospatial entities and unranked geospatial entities.

16. The method of claim 11, further comprising selecting geospatial entities to include in navigation instructions based on rankings of the geospatial entities.

17. The method of claim 11, further comprising selecting a geospatial entity to associate with an advertising term based on the geospatial entity’s ranking.

18. The method of claim 11, further comprising providing the ranked geospatial entity data to an application for generating a search result based on the ranked geospatial entity data.

19. The method of claim 11, wherein evaluating is performed responsive to user instructions for providing personalized geospatial entity rankings.

20. The method of claim 19, wherein the user instructions comprise a weighting to be applied to an attribute of a geospatial entity identified in the geospatial entity data.

21. A system for ranking geospatial entities, the system comprising: an interface for receiving ranking data about a plurality of geospatial entities; an entity ranking module for generating place ranks for geospatial entities according to a ranking mechanism based on the ranking data; and a database for storing ranked entity data generated by the entity ranking module.

22. The system of claim 21, wherein the interface is configured to provide the ranked entity data to a requesting application.

23. The system of claim 21, wherein the entity ranking module is configured to evaluate a plurality of diverse attributes to determine a total score for a geospatial entity.

24. The system of claim 21, wherein the entity ranking module is configured to organize ranked entity data into placemark layers.

25. The system of claim 24, wherein each placemark layer corresponds to at least one of: a level of detail, a density, an altitude, and an entity category.

26. The system of claim 21, wherein the requesting application is a map server system configured to use the ranked entity data to generate a map including entities selected on the basis of place ranks.

27. The system of claim 26, wherein the entity ranking module is hosted on the map server system.

28. An entity ranking module hosted on a client device, the module for generating rankings for a plurality of geospatial entities and the module comprising: an interface for receiving entity data that defines a plurality of geospatial entities and ranking data that describes the plurality of geospatial entities; and a ranking engine for generating rankings for the geospatial entities, wherein the rankings are used to select which of the geospatial items to include in a map to be displayed on the client device.

29. The module of claim 28, further comprising a memory for storing data about a user of the client device and wherein the ranking engine is configured to apply a ranking mechanism responsive to the user data.

30. The module of claim 29, wherein the user data comprises user preferences about the relative weightings of attributes evaluated by the ranking engine.

31. The module of claim 29, wherein the user data comprises a user defined geospatial entity.

32. The module of claim 29, wherein the user data comprises an indication of a user’s interest in a geospatial entity and wherein the ranking mechanism assigns a rankings premium to the geospatial entity based on the user’s interest.
What I am trying to say is you need to be careful when changing whois info or trying to suddenly protect your privacy by hiding your whois data. If there is sudden change you can expect some sort of problem but if done correctly I am sure you wont have any problem with whois privacy protection.