The SEO Cyborg: How to Resonate with Users & Make Sense to Search Bots

SEO is about understanding how search bots and users react to an online experience. As search professionals, we’re required to bridge gaps between online experiences, search engine bots, and users. We need to know where to insert ourselves (or our teams) to ensure the best experience for both users and bots. In other words, we strive for experiences that resonate with humans and make sense to search engine bots.

This article seeks to answer the following questions:

• How do we drive sustainable growth for our clients?
• What are the building blocks of an organic search strategy?

What is the SEO cyborg?

A cyborg (or cybernetic organism) is defined as “a being with both organic and
biomechatronic body parts, whose physical abilities are extended beyond normal human limitations by mechanical elements.”

With the ability to relate between humans, search bots, and our site experiences, the SEO cyborg is an SEO (or team) that is able to work seamlessly between both technical and content initiatives (whose skills are extended beyond normal human limitations) to support driving of organic search performance. An SEO cyborg is able to strategically pinpoint where to place organic search efforts to maximize performance.

So, how do we do this?

The SEO model

Like so many classic triads (think: primary colors, the Three Musketeers, Destiny’s Child [the canonical version, of course]) the traditional SEO model, known as the crawl-index-rank method, packages SEO into three distinct steps. At the same time, however, this model fails to capture the breadth of work that we SEOs are expected to do on a daily basis, and not having a functioning model can be limiting. We need to expand this model without reinventing the wheel.

The enhanced model involves adding in a rendering, signaling, and connection phase.

You might be wondering, why do we need these?:

• Rendering: There is increased prevalence of JavaScript, CSS, imagery, and personalization.
• Signaling: HTML tags, status codes, and even GSC signals are powerful indicators that tell search engines how to process and understand the page, determine its intent, and ultimately rank it. In the previous model, it didn’t feel as if these powerful elements really had a place.
• Connecting: People are a critical component of search. The ultimate goal of search engines is to identify and rank content that resonates with people. In the previous model, “rank” felt cold, hierarchical, and indifferent towards the end user.

All of this brings us to the question: how do we find success in each stage of this model?

Note: When using this piece, I recommend skimming ahead and leveraging those sections of the enhanced model that are most applicable to your business’ current search program.

The enhanced SEO model

Crawling

Technical SEO starts with the search engine’s ability to find a site’s webpages (hopefully efficiently).

Finding pages

Initially finding pages can happen a few ways, via:

• Links (internal or external)
• Redirected pages
• Sitemaps (XML, RSS 2.0, Atom 1.0, or .txt)

Side note: This information (although at first pretty straightforward) can be really useful. For example, if you’re seeing weird pages popping up in site crawls or performing in search, try checking:

• Backlink reports
• Internal links to URL
• Redirected into URL

Obtaining resources

The second component of crawling relates to the ability to obtain resources (which later becomes critical for rendering a page’s experience).

This typically relates to two elements:

1. Appropriate robots.txt declarations
2. Proper HTTP status code (namely 200 HTTP status codes)

Crawl efficiency

Finally, there’s the idea of how efficiently a search engine bot can traverse your site’s most critical experiences.

Action items:

• Is site’s main navigation simple, clear, and useful?
• Are there relevant on-page links?
• Is internal linking clear and crawlable (i.e., )?
• Is an HTML sitemap available?

• Side note: Make sure to check the HTML sitemap’s next page flow (or behavior flow reports) to find where those users are going. This may help to inform the main navigation.

• Do footer links contain tertiary content?
• Are important pages close to root?
• Are there no crawl traps?
• Are there no orphan pages?
• Are pages consolidated?
• Do all pages have purpose?
• Has duplicate content been resolved?
• Have redirects been consolidated?
• Are canonical tags on point?
• Are parameters well defined?

Information architecture

The organization of information extends past the bots, requiring an in-depth understanding of how users engage with a site.

Some seed questions to begin research include:

• What trends appear in search volume (by location, device)? What are common questions users have?
• Which pages get the most traffic?
• What are common user journeys?
• What are users’ traffic behaviors and flow?
• How do users leverage site features (e.g., internal site search)?

Rendering

Rendering a page relates to search engines’ ability to capture the page’s desired essence.

JavaScript

The big kahuna in the rendering section is JavaScript. For Google, rendering of JavaScript occurs during a second wave of indexing and the content is queued and rendered as resources become available.

Image based off of Google I/O ’18 presentation by Tom Greenway and John Mueller, Deliver search-friendly JavaScript-powered websites

As an SEO, it’s critical that we be able to answer the question — are search engines rendering my content?

Action items:

• Are direct “quotes” from content indexed?
• Is the site using links (not onclick();)?
• Is the same content being served to search engine bots (user-agent)?
• Is the content present within the DOM?
• What does Google’s Mobile-Friendly Testing Tool’s JavaScript console (click “view details”) say?

Infinite scroll and lazy loading

Another hot topic relating to JavaScript is infinite scroll (and lazy load for imagery). Since search engine bots are lazy users, they won’t scroll to attain content.

Action items:

Ask ourselves – should all of the content really be indexed? Is it content that provides value to users?

• Infinite scroll: a user experience (and occasionally a performance optimizing) tactic to load content when the user hits a certain point in the UI; typically the content is exhaustive.

Solution one (updating AJAX):

1. Break out content into separate sections

• Note: The breakout of pages can be /page-1, /page-2, etc.; however, it would be best to delineate meaningful divides (e.g., /voltron, /optimus-prime, etc.)

2. Implement History API (pushState(), replaceState()) to update URLs as a user scrolls (i.e., push/update the URL into the URL bar)

3. Add the tag’s rel=”next” and rel=”prev” on relevant page

Solution two (create a view-all page)
Note: This is not recommended for large amounts of content.

1. If it’s possible (i.e., there’s not a ton of content within the infinite scroll), create one page encompassing all content

2. Site latency/page load should be considered

• Lazy load imagery is a web performance optimization tactic, in which images loads upon a user scrolling (the idea is to save time, downloading images only when they’re needed)
• Add tags in
• Use JSON-LD structured data
  • Schema.org “image” attributes nested in appropriate item types
  • Schema.org ImageObject item type

CSS

I only have a few elements relating to the rendering of CSS.

Action items:

• CSS background images not picked up in image search, so don’t count on for important imagery
• CSS animations not interpreted, so make sure to add surrounding textual content
• Layouts for page are important (use responsive mobile layouts; avoid excessive ads)

Personalization

Although a trend in the broader digital exists to create 1:1, people-based marketing, Google doesn’t save cookies across sessions and thus will not interpret personalization based on cookies, meaning there must be an average, base-user, default experience. The data from other digital channels can be exceptionally useful when building out audience segments and gaining a deeper understanding of the base-user.

Action item:

• Ensure there is a base-user, unauthenticated, default experience

Technology

Google’s rendering engine is leveraging Chrome 41. Canary (Chrome’s testing browser) is currently operating on Chrome 69. Using CanIUse.com, we can infer that this affects Google’s abilities relating to HTTP/2, service workers (think: PWAs), certain JavaScript, specific advanced image formats, resource hints, and new encoding methods. That said, this does not mean we shouldn’t progress our sites and experiences for users — we just must ensure that we use progressive development (i.e., there’s a fallback for less advanced browsers [and Google too ☺]).

Action items:

• Ensure there’s a fallback for less advanced browsers

Indexing

Getting pages into Google’s databases is what indexing is all about. From what I’ve experienced, this process is straightforward for most sites.

Action items:

• Ensure URLs are able to be crawled and rendered
• Ensure nothing is preventing indexing (e.g., robots meta tag)
• Submit sitemap in Google Search Console
• Fetch as Google in Google Search Console

Signaling

A site should strive to send clear signals to search engines. Unnecessarily confusing search engines can significantly impact a site’s performance. Signaling relates to suggesting best representation and status of a page. All this means is that we’re ensuring the following elements are sending appropriate signals.

Action items:

• tag: This represents the relationship between documents in HTML.
  • Rel=”canonical”: This represents appreciably similar content.
    • Are canonicals a secondary solution to 301-redirecting experiences?
    • Are canonicals pointing to end-state URLs?
    • Is the content appreciably similar?
      • Since Google maintains prerogative over determining end-state URL, it’s important that the canonical tags represent duplicates (and/or duplicate content).
    • Are all canonicals in HTML?
    • Is there safeguarding against incorrect canonical tags?
  • Rel=”next” and rel=”prev”: These represent a collective series and are not considered duplicate content, which means that all URLs can be indexed. That said, typically the first page in the chain is the most authoritative, so usually it will be the one to rank.
  • Rel=”alternate”
    • media: typically used for separate mobile experiences.
    • hreflang: indicate appropriate language/country
      • The hreflang is quite unforgiving and it’s very easy to make errors.
      • Ensure the documentation is followed closely.
      • Check GSC International Target reports to ensure tags are populating.
• HTTP status codes can also be signals, particularly the 304, 404, 410, and 503 status codes.
  • 304 – a valid page that simply hasn’t been modified
  • 404 – file not found
  • 410 – file not found (and it is gone, forever and always)
  • 503 – server maintenance

• Google Search Console settings: Make sure the following reports are all sending clear signals. Occasionally Google decides to honor these signals.
  • International Targeting
  • URL Parameters
  • Data Highlighter
  • Remove URLs
  • Sitemaps

Rank

Rank relates to how search engines arrange web experiences, stacking them against each other to see who ends up on top for each individual query (taking into account numerous data points surrounding the query).

Two critical questions recur often when understanding ranking pages:

• Does or could your page have the best response?
• Are you or could you become semantically known (on the Internet and in the minds of users) for the topics? (i.e., are you worthy of receiving links and people traversing the web to land on your experience?)

On-page optimizations

These are the elements webmasters control. Off-page is a critical component to achieving success in search; however, in an idyllic world, we shouldn’t have to worry about links and/or mentions – they should come naturally.

Action items:

• Textual content:
  • Make content both people and bots can understand
  • Answer questions directly
  • Write short, logical, simple sentences
  • Ensure subjects are clear (not to be inferred)
  • Create scannable content (i.e., make sure tags are an outline, use bullets/lists, use tables, charts, and visuals to delineate content, etc.)
  • Define any uncommon vocabulary or link to a glossary
• Multimedia (images, videos, engaging elements):
  • Use imagery, videos, engaging content where applicable
  • Ensure that image optimization best practices are followed
• Meta elements (