One SEO professional asked John Mueller in a hangout about how sitemaps are crawled.
Their main question is: when Googlebot crawls the sitemap in the server address, does it crawl the submitted sitemaps in the GSC backend, or does it go directly to the site owner’s server address to crawl these sitemaps?
John explained that they don’t crawl the sitemaps all at the same time. So it’s not that there is a specific order to them. It’s more that for individual sitemap files, Google tries to figure out how often they change and how often they give Google useful information.
Based on this, for individual files, Google will crawl them at different rates.
It’s possible that what may happen is one sitemap in their Google Search Console account is crawled very frequently, but also one that they may submit directly is crawled frequently, and maybe another sitemap in Search Console is crawled very infrequently.
It doesn’t depend on where you’re submitted. It really depends more on the sitemap file itself and what value it shows Google.
This happens at approximately the 11:42 mark in the video.
John Mueller Hangout Transcript
SEO Professional 3 11:42
So I’ve got two questions. The first one is about the sitemap. When Googlebot crawls the sitemap in the server address, does it firstly crawl the submitted sitemaps in the GSC backend, or it goes directly to our server address to crawl all sitemaps?
John 12:10
We don’t crawl them all at the same time. So it’s not that there’s a specific order to them. It’s more that for individual sitemap files, we try to figure out how often they change and how often they give us useful information. And based on that, for individual files, we will crawl them at different rates. So it can happen that one in your Search Console account is crawled very frequently, but also one that you submit directly is crawled frequently, and maybe another in Search Console is crawled very infrequently. So it doesn’t depend on where you’re submitted. It really depends more on the sitemap file itself and what value it gives to us.
SEO Professional 3 12:52
So even though one like one useless sitemap we delay in the GSC backend, the Google bot still can crawl in our server address? Yeah…
John 13:06
Yeah. Yeah. So especially when you delete a sitemap file from Search Console, we will still have it in our systems, and we will check it probably for, I don’t know, I’m making a guess, like several months, we will probably try to recrawl that sitemap file just to make sure that it’s not temporarily gone.
SEO Professional 3 13:27
Yeah. So is there any priority like maybe Googlebots will firstly crawl the submitted one, and then maybe crawl the–?
John 13:40
No, no, not really. It’s really about the individual sitemap files and kind of the value that that our systems address to that sitemap file. And sometimes it’s, it’s due to how frequently we see the sitemap file changing. Sometimes it’s just due to other ways that our systems try to understand the value of the content that is in the sitemap file.
