An SEO professional presented John with a rather interesting problem. They have an e-commerce site of around 2-3,000 pages.
When a few pages on the site were developed, they were set to noindex when they should not have been.
This issue has been fixed for several months now.
However, in Search Console, approximately 80 percent of these pages were still set to noindex. Googlebot even crawled the page back in June.
They attempted to repair this in development, but this has not changed the situation.
They kept requesting indexing via Search Console and they tried submitting new sitemaps. However, they could never get these pages indexed again.
The SEO pro was wondering if there were any problems with Search Console indexing pages right now, or if any glitches existed in this regard.
John answered that they don’t think there are currently any indexing problems.
He explained that there are two possibilities. The first one is that they are more conservative when it comes to indexing requests.
For example, if a page is left noindexed for a long period of time, then they will automatically slow down the crawling process for that page.
It’s also possible that the Google Search Console reports are out of date, making the report look worse than it actually is.
By using the filtering in Search Console, you can examine the performance report for these pages for URL patterns to see if they match the number of high noindex pages in Search Console.
It could also be that Search Console is reporting on pages that aren’t really all that important.
John then said that improving internal linking to explain to search engines that these pages are really important is something else that you can do as well.
This happens at approximately the 8:14 mark in the video.
John Mueller Hangout Transcript
SEO Professional 2 8:14
I have a small Search Console related question. So for reference, it’s about a commerce website of 2,000 or 3,000 pages in total. And next to the product pages, it also contains a couple of course pages. Now, when these were developed, they were wrongly set to no index. And this was fixed a couple of weeks or even a couple of months ago.
But in Search Console, most of these pages, and we’re really talking about only a handful of pages, like maybe 10 pages. But like 80% of these pages are still set to no index and Search Console. And I can see that the Googlebot last crawled it months ago, back in June. We tried, we fixed it in development.
So it’s been fixed for a couple of months. The noindex meta tag is gone. Everything can be indexed. They are indexed in other search engines as well, so there are no technical issues, really. But if we check Search Console, it still says, well, Googlebot last passed this page in June, and it still contains a noindex tag, which isn’t the case.
So we tried to request the indexing via Search Console. We tried resubmitting sitemaps. But still we don’t get these pages indexed. And I don’t know if you have any ideas as to what might cause Googlebot not to listen to the indexing requests, or if there are any known issues in Search Console with indexing.
John 9:47
I don’t think there are any known issues in that regard. But some, sometimes we’re a little bit conservative with regards to submitting to indexing requests, which is probably partially what you’re seeing there? I think there might be two things. Well, I guess a couple things. On the one hand, if we see that a page is noindexed for a longer period of time, then we usually slow down with crawling of that.
So I think that’s what you’re seeing there. That also means that when the page becomes indexable, again, we’ll, we’ll pick up crawling again. So it’s essentially that one kind of push that you need to do. One other thing is that, since Search Console reports on, essentially, the URLs that we know, for the website, it might be that the picture looks worse than it actually is.
That might be something that you could look at, by for example, looking at the performance report and filtering for that section of the website, or those those URL patterns to see if kind of that number of high no index pages in Search Console is basically reporting on on pages that weren’t really important.
And the important pages from those sections are actually indexed. The other thing that you could do, I think the sitemap is essentially a good start. But another thing that you could do is make it clear with internal linking that these pages are very important for the website so that we crawl them a little bit faster.
And that can be a temporary internal linking, where you say like for, I don’t know, a couple of weeks, we link to individual products from our homepage, and we call it, I don’t know, whatever section you want to call it, there’s like special products or whatever. And essentially, when we find that the internal linking has significantly changed, then usually we go off and double check those pages too.
So that could be kind of like a temporary approach to pushing things into the index again. ith internal linking, it’s not that you’re saying these are important pages across the web, but rather important pages relative to your website.
So if you significantly changed the internal linking, it can happen that other parts of the website that were maybe kind of like, just barely indexed, that they drop out at some point. So that’s why I would do that on a temporary level and kind of say, like, I want to push these back into the system so that they get recrawled at the regular rate.
And then I’ll change the internal linking back so that everything is more normal again.
SEO Professional 2 12:40
Okay. That’s a good idea. So for example, a footer link as well, or adding the pages to the footer would be adding internal links.
John 12:48
I think that would work too. I think it’s usually better or best if we can find it on really important pages on the website, where we know, like, usually like on your homepage or something where we can tell, okay, this is something and you’re saying that this is important for you, therefore, we’ll go double check that.
SEO Professional 2 13:09
Okay, and what do you think about rewriting the URLs for these pages? So having a new URL, a fresh URL and submitting that via sitemap, would that speed up the process? Or?
John 13:22
I don’t know. I would guess it might do that for some, and it might not for others. So I think overall, I would try to keep the old URLs unless there’s really a severe problem with the old URLs. And just kind of try to push them through that way. Changing the URLs is something where you lose any of the information that was associated with the old URL.
And it’s like, the advantage is Google doesn’t know that there wasn’t noindex there. But on the other hand, Google doesn’t know anything about that page. So it’s kind of hit and miss if it actually changes anything for your situation.
