During a hangout, one SEO professional asked John Mueller during his question and answer segment about their crawls in Search Console.
In their Search Console Report, 97 percent of the crawler requests are refreshed crawls, and 3 percent are discovery crawls.
Their main question is what can they do to optimize this and have Google discover more pages?
Johne explained that they don’t have any guidelines on how the balance should be and how you should tweak it.
In general, it’s normal for an older, more established website to have a lot of refresh crawls because they willlook at the amount of pages they know about that grows over time.
And the amount of new pages that come in tends to be stable. So it’s pretty common, especially for a site that’s established and just slowly growing to have a balance like this that most of the crawling is on the refresh crawling side and not so much on the discovery crawling side.
John thinks it would be different if you had a short-lived site. Maybe classifieds, or for local news where you have a lot of new articles that come in and the old content becomes irrelevant very quickly.
Then he thinks they would focus more on Discovery.
But especially if you have something like an e-commerce site where you’re just growing the amount of content that you have slowly and most of the old content remains valid.
John sees the refresh crawling there is likely going to be higher.
This happens at approximately the 33:32 mark in the video.
John Mueller Hangout Transcript
John (Question) 33:32
In the Search Console report, 97 percent of the crawler requests are refreshed and only 3 percent is discovery. How to optimize this and let Google discover more pages?
John (Answer) 33:42
We don’t really have any guidelines on how that balance should be and how you should try to tweak it. In general, it’s kind of normal for, especially an older, more established website to have a lot of refresh crawls because we will look at the amount of pages that we know about that grows over time.
And the amount of new pages that comes in tends to be fairly stable. So it’s pretty common, especially for a website that is kind of established and just kind of like slowly growing, to have a balance like this that most of the crawling is on the kind of refresh crawling, and not so much on the discovery crawling.
I think it would be different if you had a very short-lived website. Maybe I don’t know, classifieds or kind of local news where you have a lot of new articles that come in, and the old content becomes irrelevant very quickly, then I think we would tend to focus more on discovery.
But, especially if you have something like an e-commerce site where you’re just growing the amount of content that you have slowly and most of the old content remains valid, I tend to see that the amount of refresh crawling there, that’s probably probably going to be a bit higher.
