Update Jan. 5, 2009 – I’ve actually significantly changed my mind about this advice. Yes, sitemaps can still obsfucate architectural issues, but given the experience I’ve had over the last 1.5 years, I now recommend to all of our clients (and nearly everyone else who asks) that sitemaps be submitted. The positives in terms of crawling, indexation and traffic simply outweigh the downsides.
It sounds bizarre, almost counterintuitive, but many of best minds in the world of SEO appear to be rallying around the idea that submitting a feed to Google Sitemaps and Yahoo! Site Explorer is actually a terrible idea. The logic behind the practice is simple, if you follow the steps:
- Without sitemaps, a search engine visits your site’s pages through links on and off the site, indexing and ranking those pages it deems worthy of being indexed and ranked.
- When a search engine crawls your site and fails to index particluar pages, you have a signal from the engines that those pages lack the necessary components for inclusion, be they architectural, link strength, content-related, etc.
- Sitemaps enables search engine to crawl and index pages that they might not ordinarily include in a normal crawl process.
- If a page lacks the link juice, internally or externally, or has content that engines wouldn’t normally deem worthy of indexing, Sitemaps may overlook these weaknesses and include those pages in their indices.
Why are so many SEOs recommending against submitting a feed to Sitemaps? Because the data you get from the natural crawl IS valuable, and submitting an XML feed (or any other format) can cause that natural process of inclusion to be lost. If a page isn’t accessible, doesn’t carry enough link juice, or lacks unique, valuable content, I want to know about it, and the Sitemaps process can be a hinderance.
Enormously big sites, who will see more value from having thousands of extra pages included in the index, even if it means a few stragglers are left behind are exempt from this rule. So, too, are sites managed by a team who is unwilling or unable to take the time to detect and fix omissions.
Don’t get me wrong – Sitemap submission is an amazing and valuable tool in a webmaster’s arsenal, but it’s also one that should be wielded with careful knowledge of the side effects. I’d love to hear your opinions on the subject.
BTW – Full credit to DaveN for first introducing me to this idea back in Chicago.
p.s. Jan. 5, 2009 – I’ve actually significantly changed my mind about this advice. Yes, sitemaps can still obsfucate architectural issues, but given the experience I’ve had over the last 1.5 years, I now recommend to all of our clients (and nearly everyone else who asks) that sitemaps be submitted. The positives in terms of crawling, indexation and traffic simply outweigh the downsides.
