Is Robot exclusion bad for a site – Does excluding robots from certain sections of your site bad – seo tip march 17th 2005

Some website owners are worried about using robots exclusion on their sites because they think blocking a crawler from certain pages or certain sections are bad, From search engine genie point of view we recommend to use robots exclusion freely, Make sure you use proper syntax a wrong syntax might confuse the search engine crawler and might get the site dropped from the index,

Excluding robots from certain sections of the site is very good in various ways,

1. You can prevent search engine crawlers from eating up unnecessary bandwidth of your site, For some hosting bandwidth is costly and it is important bandwidth is saved by excluding crawlers from certain sections,

2. Excluding robots from comments section if you run a blog is very good, It prevents robots from seeing or giving credit to any unnecessary links, There might be some bad links left by spammers and those links can be prevented from crawling by excluding those pages from the crawlers, Linking to bad neighbourhood is not good,

3. Excluding robots also helps in preventing sensitive information of your site not getting exposed in search results, There are sites who dont prevent the crawlers properly and those pages contain sensitive data link credit card details, important database etc, So for precautionary measures it is best to exclude the robot from certain areas,

4. Excluding robots also helps in preventing duplicate contents of your site from being crawled, If you have a dynamic site and there are 2 versions of the dynamic pages one search engine friendly and other dynamic with query string it is better to block one version and block the other one that way search engines don’t have to worry about duplicate content on your site,

There are still more benefits from excluding pages / sections from crawlers so do it safely with proper syntax, Use if well if you have big ecommerce dynamic site, Dynamic sites are known to render lots of pages to the crawlers by error, Hope this tip helps,

SEO Blog Team,

Is it bad to duplicate keywords and content in meta description tags? – search engine optimization tip march 16 2005

Meta description tags play a very small role in search engine ranking these days, It doesn’t matter whether the meta description tags are duplicated across your pages, Just make sure your important keywords/ phrases are present in those meta description tags,

It is common for sites to have similar meta description tag, Especially in ecommerce sites it is common to have duplicated meta tags, There is nothing wrong in it, It doesn’t hurt to have duplicated meta description, Also you should remember it doesn’t help much, So it is better to just optimize them and leave it as it is,

SEO Blog Team,

Potential reasons for Sites Crawled But Not Indexed – search engine optimization tip – march 15th 2005

For a more detailed list of search engine crawlers contact us and we will send it to you for free,

Many of us have noticed sometimes search engines keep visiting the site regularly but dont index the site nor show it in the site: command,

There are various reasons for this to happen,

1. The domain is an expired domain, if the domain expires and not registered for a certain period of time google imposes a expired domain penalty on that domain, That domain is left to suffer for certain number of months, in that period googlebot keeps visiting the site but they don’t index it and they don’t show the site in site: command too,

2. An other reason is the domain is a new domain and if the domain is a new domain sometimes the crawler regularly visits the site and it doesn’t show up in the index for a long time, There is nothing wrong with this, probably google index is taking longer time to expand, you just have to wait till google updates its index,

3. An other possible reason is the site is banned from the search engines for any particular onpage factor, In that case search engines periodically checks to see whether the onpage spam tactics is removed and as soon as they see the spam being removed they might reinclude the site into the index, So for people who are complaining that their site was previously indexed and listed in google but suddenly it disappeared from the index and googlebot keeps visiting the site it is good to look at your onpage work and see if there is any spam tactics like hidden text, cloaking, keyword stuffing etc,

4. An other important reason could be that the site was permanently banned from the search engines, Even here search engine crawlers visit the site following existing links to the site but they don’t index the site because the site is banned, This is common with Yahoo slurp yahoo’s robot, yahoo slurp is known to visit the site and don’t index the site if the site is banned,

List of top search engine user-agents – search engine optimization tip march 14th 2005

List of top search engine user-agents

We get lot of mails from people who want to know the names of the leading user-agents, We will be pleased to give the information, Identifying the useragents is a very important criteria in search engine optimization, A regular visit by search engine robots like Googlebot, yahoo slurp etc is a good sign,

Here is the list of top search engines crawlers,

Googlebot/2.*ooglebot@googlebot.com)
Yahoo – Yahoo Slurp
Msn – Msnbot
Lycos – Lycos_Spider_(T-Rex)/3.
Teoma- Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

For a more detailed list of search engine crawlers contact us and we will send it to you for free,

Google’s new feature gives answer to your questions, –

Google blog says,

Just the facts, fast
Have you ever needed a piece of info right now? Today we’re excited to introduce Google Q&A. We’ve pulled together facts from all over the Web to help give you the fastest possible access to the quick bits of information you need every day; just type a query into the search box, and you’ll get back the answer at the top of your search results. Q&A knows about a lot of areas: celebrities, countries of the world, the planets, the elements, electronics, movies, and anything else we’ve thought of so far (including enabling you to get answers on your mobile device). Try it out, and keep checking back. This is only the beginning.

Google Q and A gives great accurate answers, try some queries like Capital India

who discovered car

Google Adds Satellite Imagery to maps.google.com

Google now adds a cool feature the satellite image with their google maps,

Please check it out here, http://maps.google.com

Google acquires urchin web analytics software – Google Agrees To Acquire Urchin

press release on Urchin site,

Google Agrees To Acquire Urchin
MOUNTAIN VIEW, Calif. – March 28, 2005 – Google Inc. today announced it has agreed to acquire Urchin Software Corporation, a San Diego, California based web analytics company.
Financial terms of the deal were not disclosed.
Urchin is a web site analytics solution used by web site owners and marketers to better understand their users’ experiences, optimize content and track marketing performance. Urchin tools are available as a hosted service, a software product and through large web hosting providers. These products are used by thousands of popular sites on the Internet.
Google plans to make these tools available to web site owners and marketers to better enable them to increase their advertising return on investment and make their web sites more effective.
“We want to provide web site owners and marketers with the information they need to optimize their users’ experience and generate a higher return-on-investment from their advertising spending,” said Jonathan Rosenberg, vice president of product management, Google. “This technology will be a valuable addition to Google’s suite of advertising and publishing products.”
The acquisition is subject to customary closing conditions. Google anticipates that the acquisition will close before the end of April.

http://www.urchin.com/company/news/03282005.html

Does search engines crawl Javascript links ? – search engine optimization tip – march 13th 2005

Scripts are complex codings which difficult for some browsers and crawlers to read, Some tough javascripts are read only by advanced browsers, search engine crawlers are not advanced browsers some of the browsers that search engines use find difficult to index high graphics, So it is best to avoid too much javascript,

Prevent creating menus in javascript or in any scripting language, Create menus in simple html or other crawler understandable language, Javascript menus wont be crawled by search engines, Best is to avoid using clickable menus and other important inner page links in javascript,

if you cannot avoid using javascript menus just add the links to a sitemap and attach the sitemap to the homepage or any other important page,

SEO BLog Team,

Does outgoing links affect your site – seo tip march 12th 2005

Outbound links are links going out from a site to an other site, Whole internet/WWW was built upon links, Inbound links and Outbound links make up the web, Outbound links are good to maintain the quality of a site, search engines like links, they like outbound links too, If you link out to a collection of quality sites then definitely there is a small boost to that page,

Jon M. Kleinberg proposed that hubs are a collection of quality links, More information on hubs and authorities in this paper, http://www.cs.cornell.edu/home/kleinber/auth.pdf

Outbound links are good for usability too, For certain references it is important to give the source so that people are guided in the correct way, Especially non commercial sites need to link out freely for people to find relevant information if they are found elsewhere on the web,

SEO Blog Team,

Does H1 header tags help in search engines? – search engine optimization tip – march 11th 2005

Header tags used to carry weight in search engine algorithms, Today header tags carry very low weightage in search algorithms, Infact you can build sites without it, It is never needed infact, Quality contents and high quality backlinks are more important than header tags,

It never hurts to use header tags, It doesnt help too much too,

SEO BLog Team,