MSN’s blog on web search yesterday featured the following fascinating paragraph on the subject of a shift in their technology (probably responsible for their vastly improved results):
Our team focuses obsessively on the relevance of the web results. We want to get the most relevant result in the top position all of the time. The ranker we released in February served us well, but had some flaws that we weren’t happy about. In collaboration with Chris Burges and other friends from Microsoft Research, we now have a brand new ranker. The new ranker has improved our relevance and perhaps most importantly gives us a platform we think we can move forward on quicker than before. This new ranker also is based on technology with an awesome name – it’s a neural net, which we internally call “RankNet”.
I’m guessing they’re sad they didn’t have this system operational during the RustySearch test period. I’m certain they would have very much liked to have scored better than last place. MSN’s current results are very, very impressive, showcasing an uncanny ability for finding important related pages for search queries that would seem quite difficult for an automated system to find relevant. Try searches for randfish, SEOmoz, and black hat seo – fairly remarkable, no?
Supposedly, this new technology is the result of neural network that MSN has instituted as part of its ranking algorithm. Since I was unfamiliar with this term and thought many of my readers might be as well, I’ve found an excellent definition at (where else?) Wikipedia:
A neural network is an interconnected group of artificial or biological neurons. It is possible to differentiate between two major groups of neural networks:
- Biological neural networks, for example the human brain or parts thereof.
- Artificial neural networks originally referred to electrical, mechanical or computational simulations or models of biological neural networks. The field has expanded so that some applications do not clearly resemble any existing biological counterpart.
In modern usage the term most often refers to artificial neural networks, especially in computer science and related fields. There exist hybrids, incorporating biological neurons as part of electronic circuits, so there is not always a clear delineation.
What we can garner from this is that neural nets are more similiar in how they function to an organic brain – with thousands of parallel connections, than to an artificial computer processor, which uses a central core through which all operations are routed. A neural net allows for an important leap in computing technology – learning. Learning computers are based around a neural net system and we can assume that Microsoft has done much the same thing with its ranking algorithm – causing this incredible leap forward in quality of ranking results. The question that remains is how the neural net is trained – obviously the corpus of data is the www’s many sites and pages, but is there massive human input that helps to train the algorithm, or is it operating independently. My money’s on the former, but there could well be implementations of both.
In any case, we are looking at a clear step forward in search technology – while Google, Yahoo! and AskJeeves may indeed use neural nets to search for and eliminate unwanted results, my guess is that MSN is the first to use this system so fully as a ranking methodology. Now it’s up to the marketing team to steal back user from the other SEs and see if Microsoft can leverage its technology to gain share.
ADDED: I’ve started a discussion about this, including some good searches to compare MSN & Google at SEOChat.
