One SEO professional was curious about Google’s types of crawls.
They explained that when they posted one article per day on their blog , Google crawled their website every day.
When they went through Google Search Console, they examined the sitemap, and they saw that Google recrawled it every day.
However, after becoming more inconsistent with publishing habits, they found Google crawled the site less.
John explained that this is possible. He then said that Google doesn’t crawl websites, but they crawl pages of a website.
And where crawling is concerned, they have two types of crawling.
One type is a discovery crawl, where they try and find brand-new pages on your site.
The other is a refreshed crawl where they update existing pages that they are aware of.
They can, for example, crawl the home page with a refresh crawl once a day or every few hours or something to that effect.
If they find new links, then they will go off and crawl those with a discovery crawl as well.
Because of this, you will always see a mix of discover and refresh crawls happening.
However, if they find that certain pages are not updated as frequently, or they crawl very rarely, then they realize they don’t have to crawl these pages all the time.
Another example is a news website. If they have an article that’s published hourly, then they learn that they should crawl the site hourly.
If it updates once a month, then they learn that they should crawl that website once a month.
It’s purely a technical thing, and not a sign of crawling or ranking functions.
This happens at approximately the 5:27 mark in the video.
John Mueller Hangout Transcript
Okay, I have another question. When I used to post, as I said, one article every day, on my website, I used to see that Google crawled my website every day.
And when I went to Search Console, and when I saw this sitemap, I saw that Google recrawled it every day.
But when I became inconsistent, then I saw that Google crawls the site once in every two days, or maybe less. Is this a fact?
John 6:04
That can happen. Yes. So it’s, I mean, it’s not so much that we crawl a website, but we crawl individual pages of a website. And when it comes to crawling, we have roughly two types of crawling. One is a discovery crawl, where we try to discover new pages on your website.
And the other is the refresh crawl where we update existing pages that we know about. So for the most part, for example, we would refresh, crawl the homepage, I don’t know, once a day, or every couple of hours or something like that.
And if we find new links on their homepage, then we’ll go off and crawl those with kind of the discovery crawl as well. And because of that, you will always see a mix of discover and refresh happening with regards to crawling, and you’ll see some baseline of crawling happening every day.
But if we recognize that individual pages change very rarely, then we realize we don’t have to crawl them all the time. Or, like, for example, if you have a news website, and you update it hourly, then we should learn that we need to crawl it hourly.
Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour. And that’s not a sign of quality, or a sign of ranking or anything like that. It’s really just purely from a technical point of view. We’ve learned we can crawl this once a day or once a week, and that’s okay.
