The Real Impact of the Google SmartPhone Crawler (Part 3): Avoiding Mobile Mis-Indexing

This is the third and final installment in this mobile SEO blog post series, covering the impact of the new Google smartphone bot and how you can use it to make the most of your mobile content. The first article in the series discussed how the new smartphone bot works and which sites will be most affected. The post from last week discussed how to author redirects correctly to ensure that your mobile content will be properly indexed by the smartphone bot. This final post will review common search engine indexing problems that mobile sites and mobile platforms have, and how you can prevent them.

Some SEOs insist that we must believe what Google tells us about how and why they index things the way they do; that indexing is consistent, predictable and flawless. Unfortunately, that is not the case, especially in mobile where there are more pages and more potential for things to go wrong. Believing that indexing will always happen correctly, and that you need not mitigate risk factors for mis-indexing will not create the ideal SEO scenario. It will leave your sites (mobile/desktop/tablet) exposed when there are changes to the algorithm, or when new crawlers are evaluating your site for the first time.

If you are trying to ‘dot all your ‘i’s’ and cross all your ‘t’s’ in the world of mobile search engine indexing, here is what you need to know to prevent mis-indexing:

Avoid Duplicate Content

Google has never and will never like duplicate content. Google’s new smartphone bot, and their decision to index and cache mobile redirects may be a way for Google to avoid or minimize the need to index entire mobile pages (possibly), but it is still hard to tell how it all works. Adding mobile pages into a mix will always presents the RISK that something will be misunderstood as ‘duplicate’ and cause problems.

To keep Google happy, in the mobile world it is especially important to avoid the sneakier kinds of duplicate content that some webmasters forget about, otherwise known as DUST. The acronym stands for Duplicate Url Same Text [Acronym shared with my by the awesome Lindsay Perkin-Wassle, of Keyphrasiology]. DUST happens any time more than one version of a URL will resolve in the address bar but the browser shows the same page. The easiest example to understand is a page rendering with or without inclusion of the ‘www’ in the URL (the canonical v. non-canonical discussion usually stops here), but DUST can also be seen when there are multiple versions of a home page or category level page, as in the examples below:

In the mobile SEO world, it is quite common for mobilization platforms to control the servers and databases that generate the mobile content, and they are infamous (maybe only in my mind) for generating lots of DUST. Even the best mobilization platforms typically have minimal understanding of SEO; they try to set their servers to be very flexible with what page requests they can correctly render, and render as many different variations of a URL as possible. Instead of doing this, the platforms should be setting up the servers to 301 redirect any version of the URL that is not the canonical ‘chosen’ to redirect to the ‘chosen’ version of the URL. This is also how you can set up your own servers to prevent DUST.

Avoid 404 Errors and Misdirects

There is a risk that Google’s new smartphone crawler may be overly literal at first, and rely exclusively on the redirects that are in place, but not evaluate other signals or algorithmic elements. This means that it will probably also have a heightened the sensitivity to errors that are present on a site or in a redirect.

In general having lots of errors on your site can hinder crawling and indexing and cast your mobile site (and possibly desktop site too) in a bad light. Be sure that you check the content frequently for indexed 404 errors in Webmaster Tools, especially if you are generating dynamic mobile pages or using a hosted mobile solution to generate your mobile pages. To make finding and fixing 404s easy, you should set your mobile content up in a separate Webmaster Tools account. This way, you can see just the errors and information related to the mobile content, and not have to subtract out desktop figures to generate meaningful information.

Many 404’s in mobilization platforms are caused by improperly expired mobile content, but you should also watch for 404 errors caused by a lack of capitalization normalization and trailing slash rules set up on the server. See the example below, where one version of a URL is working fine, but the same URL with a capital letter is understood as missing, and being redirected to the mobile home page. (This is also DUST – your server can automatically normalize URLs to remove capitals.)

Capital letters in the URL cause a 404 or redirect to the mobile home page:

The presence or absence of trailing slashes can also cause problems, as shown below:

Whether the page is 404 or just redirecting to the home page, this is a problem. Stuff like this REALLY happens all the time, especially when the mobilization platforms are in charge of the server, so if you are working with an external mobilization vendor, go check this stuff out when you are done reading the article. Error-based redirect to the home page could be somehow mis-indexed as the mobile redirect.

Mobilization platforms will usually not archive mobilized pages for long periods of time, especially for sites that generate new content on a daily basis, but they also generally don’t have a proper mechanism to expire the content in a way that is good for SEO so a very similar scenario could happen with a 404 error on a page that has expired.. Mobilization platforms will generally just remove the content and leave a 404 error, which makes the mobile site look bad, because as you are constantly generating new content, you are also constantly generating new 404 errors at the same rate.

What if Google took the 404 errors on the mobile pages seriously? What if Google somehow associated the errors on the mobile pages with the corresponding desktop pages even though they were still live and fine? Hopefully Google would not let the 404 on the mobile page drag down the credibility and rankings of the desktop page that was redirecting to it, but it is not worth the risk! If you are worried about it, there is a Mobile SEO Tool to help you check indexing of one domain across the desktop and WAP index. 

When you are optimizing your mobile content, the best bet is to always play it safe, and keep your content and your server settings as neat and tidy as possible. Avoid the risk of mis-indexing by checking your URLs and watching for errors. When you add more pages and more redirects, and potentially even more servers and different companies to the mix to achieve a good mobile user experience, you increase the risk of mis-indexing.

Thanks for tuning in to this mobile SEO series about optimizing for Google’s new smartphone bot! If you missed the previous articles, they cover important information like how the new bot works, which sites will be affected and how to generate the right kinds of redirects to ensure that your content is correctly indexed by the new bot. Good luck with all of your SEO efforts and stay mobile!