Yesterday night I stayed up way too late authoring a post on Google’s Indexation Cap. Today, despite getting up way too early, I wanted to follow up and answer some of the questions from the comments, Twitter and my email. I think SEOs who read the post rightly asked for more direction in solving this problem – a fair request. Below, I’ve done my best to tackle these problems visually, as I believe we all think about site architecture and crawling issues in a visual structure.
First off, here’s a sample site hierarchy to set down the concept and give the colors I’m using in the following diagrams more context:
Next, I’ve illustrated in a more representative fashion, how those hierarchies might look on a website, and noted the external link potential of each:
In this next piece, I’m trying to explain a very important concept and something that’s frequently misunderstood by SEOs. Once upon a time, search spiders would crawl the web largely recursively – hitting a homepage that had been submitted to its index (remember way back when search engines had submission?!), then crawling in an outward fashion based on the links they discovered there. That hasn’t been the case for a long time, and as we all see with crawl paths (if you’re looking at the requests Google/Yahoo!/Bing make to your domain), multiple entry points are nearly universal and crawling pushes “outward” from those priority URLs. It looks a bit like Minesweeper, right? 🙂
Finally, I’ve got a graphic to help understand how to positively approach these problems and solve them.
There are certainly more recommendations that can be provided around these issues, and I look forward to a discussion of them in the comments.
p.s. I covered site architecture and navigation in a good bit of detail at the PRO Training this summer, but I like this image format so much, I think I might re-craft something new for next year. It feels like structuring sites properly is still a big pain point for SEOs (but possibly that’s less to do with lack of knowledge and more to do with lack of influence during the design phase?)
