One SEO professional was concerned about their crawl rate, and asked John Mueller during a hangout about it.
During the last few weeks, an SEO professional noticed a large drop in their crawl stats from 700 to approximately 50 per day.
Is there a way to understand from the Search Console report what could be the cause of this drop?
Could it be source page load? How can they correctly read the crawl request breakdown?
John explained that on their side, there are a few things that go into the amount of crawling that Google does.
On one hand, they try to figure out how much they need to crawl from a website to keep things fresh and useful in their search results.
And that relies on understanding the quality of their site and how things change on that site. Google calls it the crawl demand.
On the other hand, there are also limitations that they see from the server, from the site, and from the network infrastructure. And this applies to how much Google can crawl on a site.
Therefore, they try to strike a balance between those two.
The restrictions tend to be tied to two main things. On the one hand, it’s response time to requests to the website. And on the other – the number of errors, specifically server errors that Google sees during crawling.
So if they see a lot of server errors, they will slow down the crawl, because they don’t want to cause any more problems.
If they see that the server is getting slower, then they will also slow down crawling. Because again, they don’t want to cause any problems with crawling.
So these are the two main things that come into play.
The difficulty, John believes, with the speed aspect is that they have two different ways of looking at speed.
These can be confusing when you look at crawl rate.
And then there comes things like Core Web Vitals, and how quickly the page loads in the browser.
But, the speed at which the browser loads the URL is not directly related to the speed that it takes for Google to fetch an individual URL on their site.
John also discussed several other interesting points, and explained further that any time you make a change to your website’s hosting, you must anticipate that the crawl rate is going to drop as a result.
This happens at approximately the 35:09 mark in the video.