One SEO professional asked John Mueller in a recent hangout about the reason why server logs do not match the Google Search Console crawl report.
They were having issues with different numbers showing up between the server logs and the crawl report in Google Search Console.
For example, in their server logs they have three times the number of crawls from Google.
In Google Search Console, they have one of the three bars. Maybe they have something wrong in the server logs or perhaps they haven’t anticipated Googlebot, because of how the numbers are way off.
John explains that he believes the numbers will always be different. This is simply due to the fact that in Search Console, they report on all crawls that go through the infrastructure of Googlebot.
This also includes other requests, however.
For example, John thinks the Google Adsbot also uses the Googlebot infrastructure.
And, they have different user agents. If they examine their server logs, and only look at the Googlebot user agent that Google uses for web search, then these numbers will never match what they show in Google Search Console.
This happens at approximately the 00:54 mark in the video.