For a more detailed list of search engine crawlers contact us and we will send it to you for free,

Many of us have noticed sometimes search engines keep visiting the site regularly but dont index the site nor show it in the site: command,

There are various reasons for this to happen,

1. The domain is an expired domain, if the domain expires and not registered for a certain period of time google imposes a expired domain penalty on that domain, That domain is left to suffer for certain number of months, in that period googlebot keeps visiting the site but they don’t index it and they don’t show the site in site: command too,

2. An other reason is the domain is a new domain and if the domain is a new domain sometimes the crawler regularly visits the site and it doesn’t show up in the index for a long time, There is nothing wrong with this, probably google index is taking longer time to expand, you just have to wait till google updates its index,

3. An other possible reason is the site is banned from the search engines for any particular onpage factor, In that case search engines periodically checks to see whether the onpage spam tactics is removed and as soon as they see the spam being removed they might reinclude the site into the index, So for people who are complaining that their site was previously indexed and listed in google but suddenly it disappeared from the index and googlebot keeps visiting the site it is good to look at your onpage work and see if there is any spam tactics like hidden text, cloaking, keyword stuffing etc,

4. An other important reason could be that the site was permanently banned from the search engines, Even here search engine crawlers visit the site following existing links to the site but they don’t index the site because the site is banned, This is common with Yahoo slurp yahoo’s robot, yahoo slurp is known to visit the site and don’t index the site if the site is banned,

                                Earthlink Netscape Netvouz RawSugar Shadows Sphinn StumbleUpon Yahoo MyWeb