Monitoring, Optimizing, and Staying Ahead

What Factors Impact Your Crawl Budget?

According to Google, the biggest issue that affects crawl budget is low-value URLs.

Having too many URLs that present little or no value, yet are still on the crawler’s path, will use up the available budget and prevent Googlebot from accessing more important assets. 

The problem is that you might not even realize that you have many low-value URLs as many are created without your direct impact. Let’s see how that typically happens.
 

How Websites Generate Low-Value URLs

#1. Faceted Navigation

Faceted navigation refers to different ways users can filter or sort results on a web page based on different criteria. For example, any time you use Ross-Simons’ filters to fine-tune the product listing, you use faceted navigation.

Although helpful for users, faceted navigation can create issues for search engines. 

Filters often create dynamic URLs which, to Googlebot, might look like individual URLs to crawl and index. This can use up your crawl budget and create a lot of duplicate content issues on the site. 

Faceted navigation can also dilute the link equity on the page, passing it to dynamic URLs that you don’t want indexed. 

There are ways to overcome this:

Option 1 - Nofollow Tag

Add a “nofollow” tag to any faceted navigation link.

This will minimize the crawler’s discovery of unnecessary URLs and therefore reduce the potentially explosive crawl space that can occur with faceted navigation. 

Option 2 - Noindex tag

Add a “noindex” tag to inform bots of which pages are not to be included in the index.

This will remove pages from the index, but there will still be crawl budget wasted and link equity that is diluted.

Option 3 - robots.txt disallow

Use a robots.txt disallow. For URLs with unnecessary parameters, include a directory that will be robots.txt disallowed. This lets all search engines freely crawl URLs that you want a bot to crawl.

For example: we could disallow prices under $100 in the robots file.

Disallow: *?prefn1=priceRank&prefv1=%240%20-%20%24100

Option 4 - cannonical tags

Canonical tags allow you to instruct Google that a group of pages have a preferred version. Link equity can be consolidated into the chosen preferred page utilizing this method. However, the crawl budget will still be wasted.

#2. Session Identifiers/On-site Duplicate Content

Similarly, URL parameters - like session IDs or tracking IDs - or forms sending information with the GET method will create many unique instances of the same URL. 

Those dynamic URLs, in turn, can cause duplicate content issues on the site and use up much of the crawl budget, even though none of those assets are, in fact, unique.

#3. Soft 404

A "soft 404" occurs when a web server responds with a 200 OK HTTP status code, rather than the 404 Not Found, even though the page doesn’t exist.  

In this case, the Googlebot will attempt to crawl the page, using up the allocated budget, instead of moving on to actual, existing URLs.
 

#4. Hacked Pages

Unfortunately, pages that have been hacked can increase the list of URLs a crawler might attempt to access. If your site got hacked, remove those pages from the site and serve Googlebot with a 404 Not Found response code. 

Seeing hacked pages isn’t anything new to Google and the search engine will drop them from the index promptly, but only if you serve it a 404. 

#5. Infinite Spaces and Proxies

Infinite spaces are near-endless lists of URLs that Googlebot will attempt to crawl.

Infinite spaces can occur in many ways - but the most common include auto-generated URLs by the site search.

Some sites list on-site searches on pages which leads to the creation of an almost infinite number of low-value URLs that Google will consider crawling. 

Another common scenario is displaying a calendar on a page with a “next month” link. Each URL will have this link, meaning that the calendar can generate thousands of unnecessary infinite spaces.

Google has suggested methods to deal with infinite spaces, such as eliminating the entire categories of those links in the robots.txt file. Doing so will restrict Googlebot from accessing those URLs from the start and save your crawl budget for other pages.

How to Optimize Your Site's Crawl Budget

Based on the information above, you can see how severe issues with your site's crawl budget can be. The good news is that you can also maximize the time crawlers allocate for your website. 

There are some generic things that help such as improving overall site speed, avoiding duplicate content,  eliminating broken pages, or simplifying the site’s architecture.

However, below are other factors you should optimize to save the crawl budget from going to waste.
 

Reduce the number of crawlable URLs

As previously stated, the key to optimizing for the crawl budget is ensuring that the number of crawlable URLs doesn't exceed the budget. Otherwise, Google will cut off crawling the site once the budget is up. 

But with fewer URLs to crawl than the allocated number of requests, you stand a much greater chance for crawlers to access all your content. 

You can achieve this in many different ways, but here are some of the most common approaches:

#1. Fix 30x Redirects

Any broken link or redirect is a dead-end for Googlebot. When it comes to broken links, the crawler might consider that there is nowhere else to go and move to another website. With the redirects, it can travel through some hops. However, even Google recommends not to exceed five hops, otherwise, the crawler will move on. 

To avoid those issues, ensure that all redirected URLs point to the final destination directly, and fix any broken links.
 

#2. Remove 4xx - URL No Longer Active - Links

Removing any links to 404 pages also optimizes the crawl budget usage. As the website gets older, it's a higher risk to have internal links on your site that point to URLs that are no longer active.

#3. Optimize the Faceted Navigation

We’ve talked about the issue of faceted navigation already. You know that filters on a page can create multiple low-value URLs that use up the crawl budget. 

But that doesn’t mean that you can’t use the faceted navigation. Quite the contrary. However, you must take measures to ensure that crawlers don't try to access dynamic URLs that the navigation will create. 

To solve this faceted navigation conundrum, there are a few solutions that you can implement depending on what parts of your site should be indexed.

No Index

"Noindex" tags can be implemented to inform bots of which pages not to include in the index. This method will remove pages from the index, but there will still be a crawl budget spent on them and link equity that is diluted.

Canonicalization

Canonical tags allow you to instruct Google that a group of similar pages has a preferred version of the page.

Nofollow

The simplest solution is adding the “nofollow” tag to those internal links. It will prevent crawlers from following those links and attempting to crawl the content.
 

#4. Remove Outdated Content

You do not have to physically delete those pages. However, blocking crawlers from accessing them will immediately reduce the number of crawlable URLs to free up crawl budget. 

#5. Block Crawlers from Accessing URLs that Should Not Be Indexed

You can save much of the crawl budget from going to waste by simply blocking crawlers from accessing URLs that you don’t need to be indexed.

These could be pages with legal information, tags, content categories, or other assets that do not provide much value for searchers. 

The simplest way to do so is by either adding the “noindex” tag to those assets or a canonical tag pointing to a page you want to index instead.
 

#6. Cleaning the Sitemap

As we’ve discussed already, Google will prioritize URLs in the sitemap over those it discovers while crawling the site.

Unfortunately, though, without regular updates, the sitemap might end up clogged with inactive URLs or pages you don’t necessarily need to have indexed. Regular updates to the sitemap and cleaning up those unwanted URLs will free up crawl budget as well. 

#7. Using the Robots.txt File

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. Typically, the file is used to prevent crawlers from overloading sites with requests, however, it can help steer Googlebot away from certain sections of the site and also free up the crawl budget. 

One thing to remember, though, is that the robots.txt is only a suggestion to Googlebot - not a directive it has to unconditionally follow every single time. 

#8. Improve the Site Speed

Google has stated openly that improving site speed doesn't just provide a much better user experience, but also increases the crawl rate. Making pages load faster, therefore, is bound to improve the crawl budget usage. 

Optimizing page speed is a vast topic, of course, and involves working on many technical SEO factors.

At seoClarity, we recommend that you at minimum enable compression, remove render-blocking JavaScript, leverage browser caching, and optimize images to ensure that Googlebot has enough time to visit and index all of your pages.
 

#9. Improve the Internal Linking Structure

Search engine bots find content on a site in two ways. First, they consult the sitemap. But they also navigate the site by following internal links.

This means that if a certain page is well interlinked with other content, its chances of having the bot discover it are significantly higher. An asset with little or no internal links can, most likely, remain unnoticed by the bot.

But this also means that you can use internal links to direct crawlers to pages or content clusters that you absolutely need to have indexed. 

For example, you can link those pages from content with many backlinks and high crawl frequency. By doing so, you’ll increase the chances that Googlebot will reach and index those pages quickly. 

Optimizing the entire site’s architecture can also help with freeing up the crawl budget. Having a flat but wide architecture, meaning that the most critical pages are not too far from the homepage, will make it easy for Googlebot to reach those assets within the available crawl budget.

How to Monitor Your Site's Crawl Budget

An important aspect of optimizing your crawl budget is monitoring how Googlebot visits your site and how it accesses the content. 

There are three ways to do it, two of which lie within GSC:

Google Search Console

Google Search Console includes a breadth of information about your site’s stance in the index and the search performance. It also provides certain insight into your crawl budget. 

First of all, in the Legacy tools section, you’ll find the Crawl Stats report, showing Googlebot activity on your site within the last 90 days. 

From the report you can see that, on average, Google crawls 48 pages a day on this site. Assuming that this rate remains consistent, you can calculate the average crawl budget for the site with this formula:

Average pages per day * 30 days = Crawl Budget

In this case, it looks like this:

48 pages per day * 30 days = 1440 pages per month. 

Naturally, this is a crude estimate but it can give some insight into your available crawl budget. 

Note: Optimizing the crawl budget using the tips above should increase the number.

The Coverage Report in GSC will also show how many pages Google has indexed on the site and excluded from indexation. You can compare that number with the actual volume of content assets to identify if there are any pages the Googlebot has missed.

Server Log File Analysis

Without a doubt, the server log file is one of the biggest sources of truth about a site’s crawl budget. 

This is because the server log file will tell you exactly when search engine bots are visiting your site. The file will also reveal what pages they visit the most often, and the size of those crawled files. 

11.222.333.44 - - [11/Mar/2020:11:01:28 –0600] “GET https://www.seoclarity.net/blog/keyword-research HTTP/1.1” 200 182 “-” “Mozilla/5.0 Chrome/60.0.3112.113”

On Mar 11, 2020 someone using Google Chrome tried to load https://www.seoclarity.net/blog/keyword-research. The “200” means the server found the file, weighing in at 182 bytes.

Now, it's true that server log-file analysis is not a simple task. You have to go through thousands of rows of server requests to identify the right bot and analyze its activity. 

At seoClarity, you have access to Bot Clarity, our site audit tool that, among other functionalities, provides a detailed log-file analysis.

With Bot Clarity, you can:

• Understand what the most important pages on your site are to the search engine crawl.
• Optimize crawl budgets to ensure bots crawl and index as many important pages on your site as possible.
• Find broken links and errors that search engine bots have encountered while crawling your site.
• Audit your redirects.
• Tie bot activity to performance, indicating which areas of the site you should focus your efforts.
  
  

Frequently Asked Questions About Crawl Budgets

Do URLs I disallowed through robots.txt affect my crawl budget in any way?

No, disallowed URLs do not affect crawl budget. 

Does the meta noindex tag help to "save" crawling budget?

Unfortunately, no. But it can reduce the number of crawlable URLs and affect crawl budget positively.

Does the nofollow directive affect crawl budget?

It actually depends. Any URL the Googlebot crawls will affect crawl budget.

The nofollow directive on a URL does not restrict the crawler from accessing the page as if another page on your site, or any page on the web, doesn't label the link as nofollow.

Can I control Googlebot with the "crawl-delay" directive?

No, the "crawl-delay" directive in the robots.txt is not processed by Googlebot.

Do alternate URLs and embedded content count in your crawl budget?

Yes, they do. Any URL that the crawler accesses counts against the crawl budget. As a result, any alternate versions of the content, AMP, hreflang tag defined content, or content triggered by JS or CSS will affect the crawl budget. 

Is crawling a ranking factor?

Google's ability to access and crawl the site’s content impacts its ability to rank it for relevant phrases.

So, although crawling may not be a ranking factor per se, it is the first step to getting the search engine to discover the content, understand it, index, and rank. 

Does site speed affect my crawl budget? 

Yes, and this has been confirmed by Google.

Do canonical tags save crawl budget?

Probably not (or not much). You have to pick a canonical and have to crawl the dups to see that they're dups anyway.

How do I increase the crawl rate?

You can adjust your crawl rate in the Google Search Console by using the Crawl Rate Settings option.