Do Improved Social Signals Cause Improved Rankings?

Everyone in search is by now aware that certain social signals are well-correlated with rankings.

In each major study published on the subject, the authors point to how correlation does not imply causation (see, for example SEOmoz and Searchmetrics). Dr. Pete even wrote a whole post on the subject.

I wanted to see if it was actually plausible for these correlations to arise without social signals being a direct ranking factor. I built some Excel models to test this out and see if I could build a model that achieved the observed correlations without assuming social signals as a ranking factor.

---

The punchline: it’s possible there is no causation

I have a suspicion that this could be the most misinterpreted post I have ever written, so I thought I’d start with a prominent “Cliff notes” to be explicitly clear about what I am saying and more importantly what I am not saying.

I am saying

You can tweet any of the following without misrepresenting me:

• Social signals *may* be correlated with better rankings but not cause them [tweet this]
• Facebook Likes and rankings could achieve high correlation without Likes being a ranking factor [tweet this]

I am not saying

If you tweet any of the following attributed to me, I will write “does not follow instructions” on your forehead in magic marker:

What is this based on?

I have built a simplified Excel model of how pages accrue Likes over time. With no assumption of them being a ranking factor, I nevertheless demonstrate that we could see a strong correlation between Likes and ranking position.

Why focus on Likes?

The modelling works equally well with any of the social signals. I simply chose Likes to make the example more concrete – you could build the exact some correlation model with Tweets, Facebook Shares, Google +1s, or any other signal where accruing more social shares makes it even more likely that you will accrue more in the future.

---

Starting at the beginning

Every time we see a correlation study, I see evidence that some people haven’t completely taken on board the correlation/causation subtleties. This is unsurprising – the mathematics behind the calculations in these study is typically undergraduate level (with some of the advanced analysis verging on graduate level) – most people’s intuition lets them down horribly when confronted with probability and statistics. (Don’t believe me? Check out the Monty Hall Problem).

So let’s start from the beginning:

What are these studies looking for?

When we say correlation in this context, you can imagine that what we are looking for is similarity. We are looking for evidence that two things happen together (and don’t happen together).

In the context of these studies, we are typically looking to see if “ranking well” happens together with “strong social signals.”

Now – the mathematical part comes in when we try to define “happens together with” properly. The human brain is a remarkably powerful pattern matching device. For example – how many sportsmen and women have a pre-game routine involving a specific pair of lucky socks because of a sequence of events something like:

• Wore a new pair of socks today. Kicked ass.
• Wore the same pair of socks as last week. Kicked ass.
• New socks in the wash. Grabbed a different pair. Got whupped.
• Socks successfully cleaned and dried. Kicked ass again.

Pretty compelling evidence for those socks, huh?

From that point onwards, the athlete refuses to surrender the lucky socks. Any future losses are attributed to other factors (“I did everything I could – I even wore my lucky socks”).

Michael Jordan apparently started wearing longer shorts to cover his UNC “lucky shorts”

But let’s look at this a little more closely and skeptically. Are there any other explanations for this sequence of events? Imagine that the athlete in question is good – winning roughly 75% of his or her games on average. Imagine also that the socks are, in fact, not magic and that they have no impact on the result (shocking, I know). The odds that the single loss of a set of 4 games will coincide with a single wear of a different pair of socks is then: 0.75 x 0.75 x 0.25 x 0.75 = 0.11

In other words, roughly one in ten pairs of socks would randomly look this lucky.

Given all this evidence, most of us would probably chalk it up to chance (but keep wearing our lucky socks just in case).

Add to this the fact that we can’t help but be always on the lookout for these patterns (it’s just how our brains are wired) and it’s unsurprising that there is always some pattern to be seen somewhere.

Given all of this, we apply pretty high standards of proof before stating that there is correlation [i.e. that two things tend to happen (or not) together]. This is measured with a “confidence” which is similar to the layman’s definition but is measured in probabilities. We express our confidence in terms of “the probability that we would see a correlation at least this strong even if there were no underlying correlation.” Statisticians typically talk in 95% or 99% confidence ranges (though note that a 95% confidence interval is still wrong one time in 20).

The ranking factor studies undertaken by SEOmoz and others have shown a non-zero correlation with high confidence. In other words, there is a correlation between certain social signals and higher rankings. I don’t think anyone is seriously disputing that at this point.

Correlation is not causation

This tricky phrase gets wheeled out with every study. What does it mean?

It means that the mathematical techniques we have applied to be confident that there is a relationship between these two variables says nothing about whether one causes the other.

It’s easy to think of correlations that are not causative. More ice creams are sold in months when more sun lotion is sold. Sun lotion sales don’t cause ice cream sales and ice cream sales don’t cause sun lotion sales. Both are caused by sunny days.

While measuring correlation is straightforward based solely on raw data, this is generally not sufficient to judge causation. This is especially true where neither variable is in your control (such as the sunshine example above). Measuring or understanding causation is a topic for another day.

The important thing to note is that the size of the correlation or the degree of confidence in the correlation have no bearing at all on the likelihood of a causal relationship.

This is one of the common misconceptions with the interaction between social signals and rankings – when people say things like:

“But the correlation is too high – social signals must be a ranking factor”

I’m afraid my response is

“I’m sorry to inform you that you have been taken in by unsupportable mathematics designed to prey on the gullible and the lonely.”

Sorry for the Sheldon moment there

Seeking alternative explanations

I believe in a healthy skepticism when presented with bold evidence. I can see lots of arguments why search engines could view social signals as ranking factors (though at least in the case of tweets, I’ve long supported an algorithmic discounting of nofollow). For all the reasons outlined above, however, I’m not convinced we have seen real evidence that this is in fact what is happening.

Assuming we take the correlation studies at face value, there are three possible explanations:

1. Social signals are a ranking factor (and apparently a strong one at that)

This appears to be the hypothesis of Searchmetrics:

These findings come from a study by search and social analytics company Searchmetrics aimed at identifying the key factors that help web pages rank well in Google searches

From Searchmetrics (emphasis mine).

2. The causation goes the other way – ranking well results in better social signals

Although it’s hard to know how strong this effect could be, it’s easy to believe there is some kind of effect here. Just think about your searching/liking behaviour:

Carry out a search:

Click on a link:

Recommend the page:

I only used this example because I know how disappointed Mike was when we had to move this page – while the redirect carried across much of the link equity, it reset the social signals – this content has been tweeted and Liked thousands of times. Sorry, Mike.

3. There is a hidden causal variable (some kind of “page quality” signal?) that causes better rankings and increased social signals

The research Dan Zarrella published here last week on the relationship between social signals and links indicates that this is a plausible explanation – since we see that there is a fairly strong relationship between the two. The challenge with this approach is that if we believe social signals’ correlation with rankings comes entirely from their correlation with a real ranking factor, it’s surprising that we often see a stronger correlation between rankings and social signals than with any other single factor:

Can the alternatives account for the observations?

Whenever this has come up in conversation, I’ve had people express doubts that #2 or #3 could be strong enough effects to give the results we see.

My intuition said that #2 could be. Mainly based off the fact that any effect that is there will compound over time under an assumption that “Likes beget Likes” which seems reasonable given the way that Facebook edgerank and visibility work. If we have compounding growth to magnify small effects, then over time we could see remarkable correlation appear from relatively small effects.

So I decided to see if I could build a plausible model of #2.

Imperfect models

What do I mean by a plausible model?

I mean that I’m going to simplify a whole raft of stuff from the real world (I’m going to think about a single SERP, for example, and I’m going to think only in time units of months). I’m going to attempt not to have these oversimplifications bias the answer in my favour. My default position (known as the “null hypothesis” in statistics and probability) is that these effects are not strong enough. I’m going to construct a model that biases towards that being true and see if I can still produce a strong enough effect.

Hacking the Excel

Editor’s note: The Excel model discussed below is no longer available.

I built this model in Excel [warning: macros]. It’s very hacky – just designed to find an answer rather than to be a robust model. It takes a set of simple assumptions (none of which include a causal link from Likes to rankings) – you can see these on the “Input” sheet – and you can substitute your own values if you would like to see the impact these have:

• Top ranking pages get 400 visits / month from search (the model over-simplifies to think about a single keyword/SERP getting 1,000 searches a month – this is a proxy for all organic traffic to the page)
• Organic traffic drops off through the ranking positions according to an averaged traffic distribution
• Each website is labelled as doing “Facebook marketing” (whatever that entails exactly) with a 30% probability. Facebook marketing doubles the rate of “random” Like acquisition. Sites not doing FB marketing in the model accrue “random” Likes according to a Poisson distribution with a mean of 10
• Likes –> more Likes at a rate of 3% (i.e. for every 100 Likes a page has, it’ll get 3 more in the next month)
• Traffic –> Likes at a rate of 1%

This is what the Poisson distributions look like for the geeks in the audience:

It creates a really simple time series of Likes for each page in each month. The model runs for 36 time periods. Each refresh of Excel runs a new scenario and results in a single Spearman rank correlation at the end of the time period. Spearman rank correlation is the same measurement tool used in the SEOmoz and Searchmetrics studies.

This is what the growth of Likes looks like for a single example run (note that the lines are not ordered by ranking position despite the fact that the ordering is correlated with ranking across many runs):

I then ran the same model a hundred independent times to get a fair assessment of the correlation we could expect as a result of the simple assumptions above. (There is an embedded macro that does this for you if you would like to reproduce it – I’m not a macro expert – there’s no doubt loads wrong with this):

I got a correlation of 0.44

This is actually higher than the correlation found in most studies I’m aware of and was based off my first pass of “finger in the air” assumptions. It’s easy to tweak the assumptions to get way higher. I’d be interested in a discussion about realistic assumptions and/or flaws in the methodology.

Since there is definitively no causation in my model, unless someone can find a flaw in my method (a very real possibility – I’m a little rusty at this), I’m going to declare that it absolutely is possible for the factors we described above to be strong enough to result in the measured correlation without Likes causing better rankings directly. (Remember – you could build this exact same model applied to any of the social signals so this applies equally well to Tweets, Facebook Shares, Google +1s, etc.)

I’d love to hear if you think I’ve missed something or got something wrong in building my model.