Google’s Farmer/Panda Update: Analysis of Winners vs. Losers

By now, everyone in the SEO world is aware of the algorithmic update Google launched last Wednesday, February 23rd. Several posts on the topic are worth reading, including Danny Sullivan’s take, Aaron Wall’s assesment, SearchMetrics’ analysis and Sistrix’s data-driven post.

Here at SEOmoz, we’ve been analyzing the shift with help from our friends at Distilled, staff research scientist Dr. Matt Peters (whom you may remember from our Google Places analysis and who’s now joined our staff full time – welcome!), and several other contributors. While there’s no way to be precisely sure what Google changed to impact “11.8%” of queries, we’ve got some ideas that fit a number of the data points and we hope to contribute to the discussion on the topic and help search marketers gauge the update’s impact on their own sites.

How to See the Farmer Update’s Effect on Your Site(s) with Google Analytics

Step 1: Use GA’s date comparison feature to see the same days before and after the update on Wednesday

Step 2: Exclude “branded” keyword traffic – you don’t want those terms potentially gunking up the data, since they were very unlikely to be affected:

Step 3: Check out the traffic line comparison:

Step 4: Remove the comparison to see keyword counts, too (this data will also be in your web app traffic tab at the end of the week):

Step 5: Check out the week before, too:

Note some of the key filters applied – the exclusion of branded terms, the use of “non-paid” keywords only, and selection of “Google” as the engine (Bing didn’t have an update). Using this process, you can check to see how your sites were (or weren’t) affected. It appears that SEOmoz is virtually unaffected, as the past few weeks, our search traffic has been rising similarly to this pattern.

Who Lost in the Farmer Update?

Thanks to some datasets, we’re able to get good data on the winners and losers.

First, there’s Sistrix, which monitors 250,000 keywords in Google up to the first 100 ranking positions for each:

_

Next is SearchMetrics, which monitors ~25 million keywords in Google: 

As you can see, there’s quite a bit of crossover between the two data sources on which sites have lost substantial traffic, and I suspect that both sources are at least correct that the sites listed have lost out in the update. Note that they both have unique ways of calculating a “visibility” score based on the rankings where a page/site appears, and these, along with the differences in which keywords they’re monitoring, likely lead to imperfect overlap.

Who are the Winners in the Farmer Update?

Again, we’ve got some data from Sistrix:

And from SearchMetrics: 

This data’s a bit more curated from SearchMetrics (they appear to be excluding many winners – possibly clients?), and Wikihow is curious because it appears to have gained so strongly in their measurement, but doesn’t appear on Sistrix’s list.

Nonetheless, reviewing these winners and comparing them against the losers does suggest some potential causes, which we’ll discuss more below.

Do Link Metrics Correlate with the Losers or Winners?

Before tackling other types of differences, we wanted to answer a question that Linkscape’s web index is uniquely qualified to help handle – are links or link analysis responsible for Google’s algorithmic shift?

To answer that, we turned to the metrics available in the Linkscape API (most of which you’ll see in the mozBar, Open Site Explorer and the Web App). Matt pulled data for each of the winners and losers from Sistrix’s data (as we had that several days ago, but only today saw SearchMetrics’ post). We then looked at Spearman’s correlation coefficient for the winning vs. losing sites against various metrics.

The chart below compares the correlations with Linkscape metrics to Sistrix’s visibility scores from before and after the update:

There’s a few interesting takeways from this data:

• Generally speaking, link metrics on the domain level (Domain Authority, Domain mozRank, etc.) are very well correlated with Sistrix’s visibility scores, which suggests that, broadly, links are (probably) quite important for large domains seeking to rank large quantities of content. That’s nothing we didn’t know, but it’s a nice validation of both datasets and conforms to expectations.
• Prior to the Farmer Update, Linkscape’s metrics were considerably better correlated with visibility scores, suggesting that link metrics are no longer as predictive of rankings/visibility as prior to the update (at least, for these sites).
• We generally interpret that to suggest that link analysis likely wasn’t the culprit here – Google’s Farmer Update is looking at something else on these sites – perhaps user/usage data, content analysis, page formatting, uniqueness/quality of user experience, etc.
• The bottom section of the chart is based on an intuition of Tom Critchlow’s – that perhaps distribution of links to home vs. internal pages was a metric used in the Farmer Update. The data we collected, however, suggests (though doesn’t prove) that’s not the case.

Remember that correlation is not causation – this data leads us to believe link analysis isn’t the culprit and certainly suggests that having good links correlates well with higher rankings/visibility, but there’s always the possibility that other factors are at play. Those caveats aside, Tom, Matt and I all feel it’s most likely that link analysis wasn’t at work here and Google’s using something else.

What Factors Could Have Caused Lost Rankings?

In reviewing the sites that got hit, we were struck by a few interesting, potential culprits.

_
An eHow page on the left-hand side and an EzineArticles page on the right
 

1. It seemed that sites whose pages had fewer and/or less intrusive blocks of advertisements on them tended to be in the winner bucket, while those with more and more intrusive advertising tended to be in the loser group.
2. Likewise, sites whose UI/design would likely be described as more modern, high quality, thoughtful and “attractive” were winners vs. the “ugly” sites that tended to be in the loser bucket.
3. When it came to user-generated-content (UGC) sites, those that tended to attract “thin” contributions (think EzineArticles, Hubpages or Buzzle) lost, while those with richer, often more authentic, non-paid, and not-intended to build SEO value or links (think Etsy, DailyMotion, LinkedIn, Facebook) won.
4. In the “rich content” sector, pages with less usable/readable/easily-consumable content (think AllBusiness, FindArticles) tended to lose out to similarly content-rich sites that had made their work more usable (think LOC.gov, HuffingtonPost)

Based on these, we have some guesses about what signals Google may have used in this update:

• User/usage data – signals like click-through-rate, time-on-site, “success” of the search visit (based on other usage data)
• Quality raters – a machine-learning type algorithm could be applied to sites quality raters liked vs. didn’t to build features/factors that would boost the “liked” sites and lower the “disliked” sites. This can be a dangerous way to build algorithms, though, because no human can really say why a site is ranking higher vs. lower or what the factors are – they might be derivatives of very weird datapoints rather than explainable mechanisms.
• Content analysis – topic modeling algorithms, those that calculate/score readability, uniqueness/robustness analysis and perhaps even visual “attractiveness” of content presentation could be used (or other signals that conform well to these).

More detailed analysis, particularly of individual pages that won vs. lost, may help to get more insight into these.

If you’re an SEOmoz PRO member, there’s a great discussion going on in the Q+A section and several sites have shared their week-over-week traffic graphs. While some patterns are emerging, there’s conflicting signals on virtually everything, so we’re not yet confident about solutions. That said, we’ll be looking more deeply into this over the weeks to come, and hope to have more to report soon.