One SEO professional was curious about why they were having issues with their CDN implementation.
Their site is mainly based out of India. And their users are also from India.
A few days back, they moved all of their dynamic traffic behind the CDN (content delivery network). But, what they have been observing is that, in Search Console under their crawl stats, their response times actually ended up going up from 300 milliseconds to around a second.
They aren’t understanding why their crawl rate has dropped from around 2 million requests per day to around 80,000 requests.
They aren’t understanding how moving to a CDN would impact how Google crawls their website.
Also, does the network latency that was introduced by the CDN service impact the crawl rate, and their ranking?
John explained that from a ranking point of view, the CDN implementation would not change anything.
Perhaps that’s the first question. But it can be, if you change your hosting significantly, what will happen on Google’s side is the crawl rate will move to a more conservative area first, because they saw a bigger change in hosting.
This could include a move to a CDN or from one CDN to a different CDN. Then over time, perhaps over a couple of weeks to perhaps a month or so, they will increase the crawl rate again to see where they think it will settle down.
So, the drop in crawl rate overall, for a move to a CDN or the change in CDN, can be normal.
The crawl rate itself doesn’t necessarily mean there is a problem. Because, if Google was crawling 2 million pages before, it is unlikely that the site will have 2 million pages that change every single day.
So it’s not necessarily the case that they will miss all of the new content on the site. They would just try to re-prioritize, and figure out which of the pages they actually need to crawl on a daily basis.
Just because the crawl rate drops is not necessarily a reason to be concerned.
What worries John more is the overall change in the average response time.
Google’s crawl rate is one that they choose based on the average response time, and it’s also based on server errors.
If the average response time goes up significantly, then Google will stick to a lower crawl rate.
This happens at approximately the 26:26 mark in the video.