Custom Extraction Using an SEO Crawler for CRO and UX Insights

From e-commerce to listings sites to real estate and myriad verticals beyond, the data you can harness using custom extraction via crawler tools is worth its weight in revenue. With a greater granularity of data at your fingertips, you can uncover CRO and user experience insights that can inform your optimizations and transform your customer experience.

In this episode of Whiteboard Friday, we’re delighted to welcome Luke Carthy to share actionable wisdom from his recent MozCon 2019 presentation, Killer CRO and UX Wins Using an SEO Crawler.

Video Transcription

Hey, Moz. What’s up? Wow, can I just say it’s incredible I’m here in Seattle right now doing a Whiteboard Friday? I can’t wait to share this cool stuff with you. So thanks for joining me.

My name is Luke Carthy. As you can probably tell, I’m from the UK, and I want to talk to you about custom extraction, specifically in the world of e-commerce. However, what I will say is this works beautifully well in many of the verticals as well, so real estate, in job listings. In fact, any website that can pretty much spit out HTML in a web crawler, you can use custom extraction.

What is custom extraction?

Let’s get started. What is custom extraction? Well, as I kind of just alluded to, it allows you, when you’re crawling using like Screaming Frog, for example, or DeepCrawl or whatever it is you want to use, it allows you to grab and extract specific parts of the HTML and export it to a file, a CSV, in Excel, or whatever you prefer.



As a principle, okay, great, but I’m going to give you some really good examples of how you can really leverage that. So e-commerce, right here we’ve got a product page that I’ve beautifully drawn, and everything in red is something that you can potentially extract. Although, as I said, anything on the page you can. These are just some good examples.

Product information + page performance

Think about this for a moment. You’re an e-commerce website, you’re a listing site, and of course you have listing pages, you have product pages. Wouldn’t it be great if you could very quickly, at scale, understand all of your products’ pricing, whether you’ve got stock, whether it’s got an image, whether it’s got a description, how many reviews it has, and of the reviews, what’s the aggregate score, whether it’s four stars, five stars, whatever it is?

That’s really powerful because you can then start to understand how good pages perform based upon the information that they have, based upon traffic, conversion, customer feedback, and all sorts of great stuff, all using custom extraction and spitting it out on say a CSV or an Excel spreadsheet file.

Competitive insights

But where it gets super powerful and you get a lot of insight from is when you start to turn the lens to your competitors and you think about ways in which you can get those really good insights. You may have three competitors. You may have some aspirational competitors. You may have a site that you don’t necessarily compete with, but you use them on a day-to-day basis or you admire how easy their site was to use, and you can go away and do that.

You can fire up a crawl, and there’s no reason why you couldn’t extract that same information from other competitors and see what’s going on, to see what pricing your competitors are selling an item at, do they have that in stock or not, what are the reviews like, what FAQs do people have, can you then leverage that in your own content. 

Examples of how to glean insights from custom extraction in e-commerce

Example 1: Price increases for products competitors don’t stock

Let me give you a perfect example of how I’ve managed to use this.

I’ve managed to identify that a competitor doesn’t have a specific product in stock, and, as a result of that, I’ve been able to increase our prices because they didn’t sell it. We did at that specific time, and we could identify the price point, the fact that they didn’t have any stock, and it was awesome. Think about that. Really powerful insights at massive amounts of scale. 

Example 2: Improving facets and filters on category pages

Another example I wanted to talk to you about. Category pages, again incredibly gorgeous illustrations. So category pages, we have filters, we have a category page, and just to switch things up a little bit I’ve also got like a listings page as well, so whether it’s, as I said, real estate, jobs, or anything in that environment.

If you think about the competition again for a second, there is no reason why you wouldn’t be able to extract via custom extraction the best filters that people use, the top filters, the top facets that people like to select and understand. So you can then see whether you’re using the same kind of combinations of features and facets on your site and maybe improve that.

Equally, you can then start to understand what specific features correlate to sales and performance and impacts and really start to improve the performance of how your website performs and behaves for your customers. The same thing applies to both environments here. 

If you are a listing site and you list jobs or you list products or classified ads, is it location filters that they have at the top? Is it availability? Is it reviews? Is it scores? You can crawl a number of your competitors across a number of areas and identify whether there’s a pattern, see a theme, and then see whether you can leverage and better that and take advantage of that. That’s a great way in which you can use it. 

Example 3: Recommendations, suggestions, and optimization

But on top of that and the one that I am most fascinated with is by far recommendations.

In the MozCon talk I did earlier I had a statistic, and I think I can recall it. It was 35% of what people buy on Amazon comes from recommendations, and 75% of what people watch on Netflix comes from suggestions, from recommendations.

Think about how powerful that is. You can crawl your own site, understand your own recommendations at scale, identify the stock of those recommendations, the price, whether they have images, in what order they are, and you can start to build a really vivid picture as to what products people associate with your items. You can do that on a global scale. You can crawl the entire of your product portfolio or your listing portfolio and get that. 



But again, back to powerful intelligence, your competitors, especially when you have competitors that might have multivariable facets or multivariable recommendations. What I mean by that is we’ve all seen sites where you’ve got multiple carousels. So you’ve got Recommended for You.

You might have People Also Bought, alternative suggestions. The more different types of recommendations you have, the more data you have, the more intelligence you have, the more insight you have. Going back to say a real estate example, you might be looking at a property here. It’s at this price. What is your main aspirational real estate competitor recommending to you that you may not be aware of?

Then you can think about whether the focus is on location, whether it’s on price, whether it’s on number of bedrooms, etc., and you can start to understand and behave how that can work and get some really powerful insights from that. 

Custom extraction is all about granular data at scale

To summarize and bring it all to a close, custom extraction is all about great granular data at scale. The really powerful thing about it is you can do all of this yourself, so there’s no need to have to have meetings, send elaborate emails, get permission from somebody.

Fire up Screaming Frog, fire up DeepCrawl, fire up whatever kind of crawler you want to use, have a look at custom extraction, and see how you can make your business more efficient, find out how you can get some really cool competitive insights, and yeah, hopefully, fingers crossed that works for you guys. Thank you very much.

Bonus resources:

Video transcription by Speechpad.com

This is a meaty topic, we know — if you enjoyed this Whiteboard Friday and find yourself eager to know more, you’re in luck! Luke’s full presentation at MozCon 2019 goes even more in-depth into what custom extraction can do for you. Catch his talk along with 26 other forward-thinking topics from our amazing speakers in the MozCon video bundle:

Access the sessions now!

We recommend sharing them with your team and spreading the learning love. Happy watching!