As much as it pains me to have to give ground to the industry giant, Google appears to still have the edge in index size. Despite Yahoo!’s claims to over 19 billion web documents indexed, in a very good report from earlier this week, Matthew Cheney and Mike Perry from the Univ. of Illinois note that:
Based on the data created from our sample searches, this study concludes that for a random set of words a user can expect, on average, to receive 65% more results using the Google search engine than the Yahoo! search engine. In fact, in the 10,034 test cases we ran, only in 16% of the cases (1606) did Yahoo! return more results. In 83.7% of the cases (8399) Google returned more results. In less than 1% of the cases both search engines returned the same number of results.
The methodology of the study is reasonably bulletproof, and it would be my guess that rather than a “lie”, what has happened here is that Google has purged “duplicate” documents while Yahoo! has counted them towards index size. While Yahoo! is clearly making progress, reports like this without the ability to back it up via experimentation is probably bad for publicity…
