In John Mueller’s hangout on 09/10/2021, one webmaster asked about their site’s pages and their indexing.
The webmaster is not experiencing a very high crawl demand and wanted to know what else they could do to improve their overall pages.
John explained the differences between crawl budget and crawl demand. This problem is something that falls under both but mainly under the crawl budget category of issues.
If it’s a smaller website, they—more than likely—can crawl everything.
If there’s a bunch of pages that the webmaster created and they’re not being indexed, John’s hunch is maybe they are creating too many pages.
Perhaps they should be focusing on fewer pages and more on the page quality.
As the pages get indexed better and faster, then they can expand on that significantly.
This occurs at approximately the 35:31 point in the video:
John Mueller 09/10/2021 Hangout Transcript
I’ve been waiting for my pages to get crawled for more than one month. I’ve tried to improve my overall pages outside of those non-crawled pages, what would your advice be to improve crawling on a website?
John (answer) 35:34
It is really hard to say without knowing the website itself. Usually, this is something that falls roughly into the category of crawl budget, not in the sense that it is like a problem for a website. But essentially, there are two sides that are always at play there. On the one hand, the capacity for crawling, like, how much can Google crawl? If this is a very small website then probably we can crawl everything.
And then there’s a crawl demand. How much does Google want to crawl? And the crawl demand is something that you can help us with things like internal linking to let us know about your relative importance of pages, but also is something that we can pick up over time by recognizing, well, there’s a lot of really good and important content here, so we should perhaps be investing more time in more crawling here.
If you’re looking at a bunch of pages that you’ve created, and they’re just not being indexed. My hunch is that maybe you’re creating too many pages. And you should be focusing on fewer pages and making sure that they’re better first. And then at some point, when you realize that the pages that you create, they actually get indexed fairly quickly, then go off and create more pages.
So first, improve the quality. And then, as a second step, go off and kind of grow your websites, step-by-step.
