Help Google help you.

Learn how to identify common crawling, rendering and indexing problems and how to fix them.

Why technical SEO and content marketing should come before link-building.

For most websites, link-building should be the last thing you should do. I always tell clients that links are like a steroid injection. When done poorly, link-building can hurt your website.

Your top priority should be building out your website’s topical relevance. This means mapping out your services and/or products, identifying all possible questions that a person may have before they become a customer, and authoring content that solves their questions.

But all your valuable content marketing won’t benefit you if search engines cannot find and understand the information on your pages. This is where technical SEO comes in.

Generally speaking, Google is pretty good with crawling, rendering and indexing most CMS platforms (Wordpress, Squarespace, Shopify, Wix, Magento). But most websites have thin content, duplicate content, and inefficient javascript that make it difficult for Google to crawl and render. And sometimes, someone has forgotten to remove a noindex tag.

In conclusion, technical SEO and content marketing are the most important tasks to prioritize.

Things you should know.

Crawling refers to the discovery process that a search engine must take in order to find all the URLs on a website. Search engines such as Google must be able to crawl a website before it can consider adding URLs to its index.

Therefore, in the context of technical SEO, you want to reduce all possible barriers that are making it difficult for search engines to discover pages, images, and other files that you want shown on search engine result pages.

Rendering is the process where a search engine visits a URL and sees what information is found on the page. To do this, it will run the code on the page and spend a few seconds to see what is displayed.

However, many things can prevent search engines from efficiently seeing the content on your webpages and the goal of technical SEO is to reduce/remove these as much as possible.

Indexing is the process how and where a search engine places a URL on its result pages. This is often referred to as ‘ranking’.

Most search engines do not publicly disclose how their indexing system works and Google is frequently updating its indexing algorithm to provide better search results for their users.

Unlike crawling and rendering, indexing is not something you can directly influence with ease.

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site (source).

To learn more about how to implement robots.txt based on best practices, read the following blog posts:

  • How to understand your robots.txt file
  • Common robots.txt file mistakes you should avoid
  • Robots.txt for Shopify – best practices
  • Robots.txt for Wordpress – best practices
  • Robots.txt for ecommerce sites – best practices

A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them (source).

You can upload one or more sitemap files in Google Search Console.

I suggest that you read these blog posts:

  • Should your sitemap be compressed?
  • Are sitemaps necessary?
  • Can you have more than one sitemap?

When you start removing discontinued products or irrelevant pages on your website, you will come across a 404, 301 and 410 decision tree.

Each decision and the correct course of action will depend on the context of what you’re trying to achieve.

A 301 redirect should be used to point users to a better experience on another URL. For example, a CMS-generated category or tag archive page tends to offer very little value to a web visitor. In these instances, you can create a custom page for each category or tag taxonomy and 301 these to the custom URLs. I do this a lot for WooCommerce websites so that category pages are keyword targeted and offer useful information to the user when they land on it from a Google search.

A common mistake people make is 301 redirecting a deleted page to their homepage of category page. This should only be done if the page has inbound backlinks from noteworthy websites.

In most cases, an outdated page with no backlinks should serve a 404 ‘page not found’ status.

If you’re updating a website’s site structure (e.g., introducing a physical subfolder for a category), you should use a 301 to point users and search engines to the new location of the page within your website.

For discontinued products, I recommend creating a custom page that communicates to the user that the item is no longer available and provide them with suitable alternatives.

Most CMS platforms will serve a 404 by default when you delete a page on your website. After a certain period of time, search engines will stop showing the URL in its SERPs as it understands that the page no longer exists.

When deleting a page, ensure that all internal links to the deleted page are removed. There is nothing more frustrating for a user than clicking on a link and landing on a ‘page not found’ greeting.

There aren’t many reasons to use a 410 these days. You’re better served by using a 404 and using the URL removal tool in Google Search Console so that the URL is no longer served on Google SERPs.

To understand this better, read this blog post.

A canonical URL is the URL of the page that a search engine thinks is most representative from a set of duplicate pages on your site.

Click here to read the Google Search Central documentation on canonical URLs.

Pagination is the process of dividing a document into discrete pages, either electronic pages or printed pages – (source).

It is common for ecommerce sites to have hundreds if not thousands of product URLs. Similarly, publishers, comparison websites and sites that are content marketing focused will have hundreds of URLs within a category/archive page.

To address this, pagination was introduced to show an archive of all these URLs where recent products/articles are shown earlier in the pagination series.

For most ecommerce websites, pagination is used for splitting up product category pages into easy-to-consume pages where a maximum of 10-25 products are shown on each URL. Subsequent products are then shown in a paginated series of pages.

For publishers, pagination is used to categorise related URLs and show users and search engines all the pages that are relevant to a given topic.

Let’s get started.

Crawling

  • Is Google crawling your website?
  • Does crawl budget affect your website?
  • How to understand your robots.txt file
  • Common robots.txt file mistakes you should avoid
  • Robots.txt for Shopify – best practices
  • Robots.txt for Wordpress – best practices
  • Robots.txt for ecommerce sites – best practices
  • Should your sitemap be compressed?
  • Are sitemaps necessary?
  • Can you have more than one sitemap?
  • How to use Screaming Frog to identify common sitemap issues
  • How to use Sitebulb to identify common sitemap issues
  • For publishers: pagination in the context of crawling
  • For ecommerce sites: pagination in the context of crawling

Rendering

  • 3 tools to help you diagnose whether Google can render your page’s content
  • Common rendering problems and how you can fix them

Indexing

  • How to know if Google has indexed the pages on your website?
  • What are the common problems that prevent Google from indexing the pages on your website?
  • Are noindex tags bad?
  • When should you use a noindex tag?
  • Should you noindex a page or apply a 301 redirect or canonicalize it to another URL?
  • Doing ecommerce pagination right: a framework you can follow
  • What is duplicate content?
  • How duplicate content on your website can hurt your SEO
  • What should you do with duplicate content?
  • Wordpress: how to fix a trailing slash or non-trailing slash issue
  • What is thin content, why thin content can hurt your SEO, and what steps should you take?
  • How to identify thin content URLs using Screaming Frog
  • How to identify thin content URLs using Sitebulb
  • What are redirect chains, why are they bad, and how you can find redirect chains and fix them using Screaming Frog and Sitebulb
  • Should you worry about URLs marked as “indexed, not submitted in sitemap?”
  • How to make sense of “excluded” URLs in Google Search Console coverage
  • How to make sense of error messages in Google Search Console coverage

Web speed optimization and core web vitals:

  • Should you move to a faster web host?
  • Why I recommend Cloudways
  • Image optimization for ecommerce websites
  • Image optimization for creative services websites (e.g., wedding photographers)
  • How to embed YouTube videos without slowing down your webpage
  • How to fix “eliminate render blocking resources” on a Wordpress site
  • How to fix “avoid chaining critical requests” on a Wordpress site
  • How to fix “avoid large layout shifts” on a Wordpress site
  • Should you prioritize core web vitals?

Site migration:

  • What is a site migration and why should you consider doing one
  • Common site migration mistakes
  • Pre-launch checklist
  • Staging checklist
  • Launch-day checklist
  • Post-launch checklist

Structured data:

Don’t have time to learn all this?

Follow my exact processes to achieve customer acquisition potential for yourself.

Content marketing

Learn how to capture and engage your audience.

Technical SEO

Learn how to identify and remove issues that make it difficult for Google to rank your pages.

E-A-T

Learn how to show up in YMYL SERPs.

Local SEO

Learn how to dominate local search intent without resorting to spammy tactics.

Link building

Learn how to earn links from noteworthy sources with minimal risk.