Losing pages in search engine index a concern

According to a member " I have been doing a lot of digging lately because of a site I have that has been losing pages in the index… at least I thought it was losing pages. In the coarse of investigating, I have been finding a lot of discrepancies and have come to the conclusion that though tools and search operators may be helpful, they seem to be far from accurate and do not fully portray what is in the data and returns. What I found fascinating is that while I perceived that I was losing pages in the index, I actually have been increasing position for some relatively hard to get keywords and phrases. In fact, the site in question just went to #6 for widgets. It seems the more I search and investigate, the more glaring the discrepancies.



I was having a lot of problems with the site and duplicate content. It seems there were several ways of getting to the same page (different URLs) and as we know, this can be a bad thing. The site has a forum that has generated 16,000 topics (some of them on multiple pages) so in essence, I am going to estimate that I have around 19,000 pages total on the site. Now at the height of the duplicate problem, when I did a site:mysite.com, I was getting over 80,000 pages returned. WOW! I fixed all the dupe content issues and now each page has one URL and each has a uniquely generated title, description, keywords and of course, the content is different since it is user generated. I used robots.txt to get rid of the duplicated pages and started to watch what would happen. This seemed to have corrected the problem.



Pages started going supplemental and dropped, as far as I can tell. But the pendulum seemed to have swung too far! Within the past month, the number of pages returned using site: have been slowly dropping. Now when I do a site:mysite.com, it only shows 4000 pages. Huh? What's the deal with that? Not only that, when I do a site:mysite.com/*, I only get about 800 pages. So I am confused, of course. But are the missing pages really not there? I conducted about 200 searches for the pages that I thought were missing and found every single one of them, though the searches were fairly specific. So what does this tell me? The site: operator does not work. All of my pages are there, it's just Google doesn't want to count them all with this operator. What does this mean? Not sure, but it is what it is. For every page I find missing, I can find in a search. The tool seems to be broken - like a lot of the tools on G. "

What causes drop of indexed pages?

Labels: Google