One SEO professional was worried that they were on the wrong track. They started with creating a website that had around 350,000 URLs, and were worried about how Google may crawl it.
Creating a website this large, they ran into issues. Only around half of their URLs are getting indexed.
While everything was indexed at first, when you go to the excluded list of URLs in Google Search Console, this would show that half of the URLs would be excluded, to the tune of 160,000.
They were not 100 percent sure why this was happening. So they wanted to ask John: could this be because of the language being in English, or could it be because of the different URL structures?
John explained that, with a website of that size, Google is unable to handle that volume of URLs.
With 70 different country and language versions on top of each other, combined with the existing large quantities of URLs, this is shooting yourself in the foot from a technical perspective.
By submitting so much content that’s multiplied by 70, this means that Google has to start somewhere by indexing some things and ignoring other things.
He recommends, in these situations, to begin creating your site with a smaller number of pages, rather than all of the English language versions as well.
Then, once everything is confirmed to be correct, you can move forward with the international implementation in small sections at a time.
If you go too large, then Google may end up not crawling or indexing the site at all because of its tremendous volume.
And this just makes the entire process of crawling, indexing, and ranking all that much harder.
If you’re not already a well-known site with those large page counts, then doing something like this is going to lead to a very different result where you may not get everything crawled at all.
This happens at approximately the 15:49 mark in the video.