This post is of the ‘short and sweet’ variety but the lesson learned may well help others so I thought it was worth writing.
Earlier this year a number of our sites tanked and when I say tanked I mean THEY TANKED. Not only did visits dry up, as a result of their rankings evaporating, but all toolbar PageRank disappeared reverting to n/a. I’m talking a site with a homepage toolbar PR 7 showing PR n/a! It was a very strange experience, made all the more strange by the fact that sites that ranked on pure age (been there since god was a boy with decent content) tanked alongside sites we’d worked on.
Skipping to the end of the story (When Harry Met Sally style) everything was back to normal within a week but to demonstrate what happened here’s a traffic graph.
All the normal thoughts went through my head. Negative SEO, something we’d done was now against Google’e T&C’s (unlikely), the server was flaky, it was a real mystery. Then I remembered about the Fetch As Google tool and the mystery started to unfold.
The Fetch As Google tool returned a 406 error. Eh? The website’s showing just fine in my browser, what’s going on? Some further digging through the logs found that Google’s spider was using an IP address that was listed in the Realtime Blackhole List (RBL) our server was using. Argghh! How Google’s IP address had finished up in the list I’m not sure but there it was, by using a service designed to keep bad people away from our server we were banning Google from our server!
Let me recap. Our server was doing its job and blocking requests from IP addresses in an RBL. The Google spider’s IP address was in the RBL so was being blocked, Google then decided the sites weren’t available or the server was blocking it, either way it looked suspicious, so the sites were dropped from the index. Scary or what?
We quickly unblocked Google and watched things go back to normal, in the case of the high PR sites this happened very quickly. A week later and things were pretty much sorted with rankings (and traffic) and PR all as they were before the incident occurred.
It was quite an experience and something that’s worth considering if like us you lock your servers down. It certainly proved the worth of the Fetch As Google tool!
A Mozzer suggested we test this again by re-blocking Google, just to prove the theory, but my heart couldn’t stand the stress to be honest. Has anyone else witnessed this behaviour?
