One SEO professional asked John Mueller in a recent hangout about the reason why server logs do not match the Google Search Console crawl report.
They were having issues with different numbers showing up between the server logs and the crawl report in Google Search Console.
For example, in their server logs they have three times the number of crawls from Google.
In Google Search Console, they have one of the three bars. Maybe they have something wrong in the server logs or perhaps they haven’t anticipated Googlebot, because of how the numbers are way off.
John explains that he believes the numbers will always be different. This is simply due to the fact that in Search Console, they report on all crawls that go through the infrastructure of Googlebot.
This also includes other requests, however.
For example, John thinks the Google Adsbot also uses the Googlebot infrastructure.
And, they have different user agents. If they examine their server logs, and only look at the Googlebot user agent that Google uses for web search, then these numbers will never match what they show in Google Search Console.
This happens at approximately the 00:54 mark in the video.
John Mueller Hangout Transcript
SEO Professional 1 0:54
Hello, good morning, everybody. And my question is about the crawling of our website. We have different numbers from the crawling of the Search Console in our server logs. For instance, we have three times the number of the crawling from Google, in our server, and in Search Console, we have one of the three bars. Could be possible, maybe we have something wrong, or we don’t anticipate the Googlebot, because the numbers are different.
John 1:33
I think the numbers would always be different. Just because of the way that the–in Search Console, we report on all crawls that go through the infrastructure of Googlebot. But that also includes other types of requests. So for example, I think the Adsbot also uses the Googlebot infrastructure. Those kinds of things. And they have different user agents. So if you look at your server logs, and only look at the Googlebot user agent that we use for web search, those numbers will never match what we show in Google Search Console.
SEO Professional 2:16
But we have more in our server logs than in Google Search Console.
John 2:18
Okay, if you have more in your server logs, then to me that would sound like either you have multiple sites combined in the server logs. Or you’re also counting requests by other people who use the same user agent. And so the user agent isn’t something that is locked and only limited to Googlebot. Anyone can say like I am Googlebot, but even if they’re not Googlebot. So there is one way to double-check that with the IP address. And I think we have that in our Help Center. How to kind of double-check that it’s really a Googlebot.
SEO Professional 1 2:57
Okay. Okay. Thank you.
