On the John Mueller hangout on 09/03/2021, one webmaster asked about a problem they are having with their e-commerce website using JavaScript. Google cannot access it and cannot render it.
The webmaster is seeing these pages as duplicates or they see less indexing than they would expect.
John said that the webmaster will need to make sure that their website actually can be rendered for search first. One of the best ways to do this is to use a site query for these specific pages.
It’s important to do this because the testing tools have much stricter guidelines regarding how fast a server should respond compared to how Google actually acts.
It’s possible that the testing tool says that it’s not working when, in fact, it actually does work properly on Google itself.
This is important to keep in mind.
This conversation occurs at approximately 20:58 in the video:
John Mueller Hangout Transcript 09/03/2021
Another question is maybe something a bit too complicated for you to give me much help. But I’d like some ideas. So basically, we have a huge problem on one of our e-commerce websites we look at. On listing pages, the JavaScript that’s responsible for loading all the listing products—Googlebot cannot access it, cannot render it.
Apparently, according to mobile-friendly test tool and search console. So we get like zero products. Robots.txt doesn’t block it. We checked server logs, they say that Googlebot isn’t blocked, and it can access the resources. Other parts of the page are absolutely fine. Other JavaScript on the website’s fine. Another absolutely identical website we have, which is in a different country, it’s also fine.\
It doesn’t have this issue. And the issue just resolved itself for like one month when developers didn’t change anything. And the server folks also didn’t change anything. So we have no idea what’s happening. And I don’t really know where else to look. So if you have any ideas?
John 22:07
Is this something that you only see in the testing tools? Or is it something that you also see in the search results?
Webmaster 8 22:14
We see these pages being seen as duplicate, for example, or we see less indexing that we expect. So I think it’s actually not rendered correctly.
John 22:27
Okay. I mean, what what I would double-check is to see if it can actually be rendered for search first. And I think one way you could do that, potentially, is to do a site query for those specific pages. And some of the keywords that are only added with JavaScript. Just to kind of double-check the, the assumption that it’s really like not being taken into account by Google because especially with the, the testing tools that we have we, we set them up in a way that they have much stricter deadlines with regards to how quickly a server should respond because we want to give you a result as quickly as possible.
And when it comes to search, we don’t have those kind of deadlines, because we can say, “Well, we will just kind of like take time to cache these scripts and then execute them separately.” We don’t have to do it all within a couple of seconds. So it can certainly happen that a testing tool will say it didn’t work, but, actually, in search, it did work. So I’d kind of watch out for that. But usually also, when, when this happens, it can be a sign that things are kind of on the edge, in the sense that sometimes it works and sometimes it doesn’t work. And that I think is still something worth trying to figure out. What, what I would do, especially if this is a JavaScript site, I would join one of Martin’s sessions.
He does office hours specifically for JavaScript-based websites, and maybe submit one of those URLs to his question and answers ahead of time so that he can take a look and get his input on that. Because it might be that there’s something small that you can do to improve, kind of, the whole process there. It might be that when he looks at it, he says, Oh, it’s, it’s a non-issue. It’s the testing tool that is confusing you.
But that’s, that’s kind of the direction I would take there. So I would first check if it’s really a critical issue and then also check in with Martin to see if it’s maybe something on the edge where sometimes it works, sometimes it doesn’t work.
Webmaster 8 24:43
Okay, thank you. So just quickly to check because we did see when the issue self-resolved itself, the indexes went up. And then when it went back, it—they dropped. So assuming there the issue exists, what do you think is… What are some of the big reasons when Google cannot access JavaScript, aside from being blocked with Robots.txt or being blocked by servers?
John 25:11
Yeah. So I think probably what is the—what happens the most there is that it’s more of a general rendering speed issue, in the sense that maybe, maybe you have something set up that the JavaScript or the API responses are created in the way that they almost have session IDs associated with them so that we can cache them properly or anything where, essentially, rendering the page takes a significant amount of resources extra.
So what, what I’ve done in the past, when looking at these kinds of issues, is to look at webpagetest.org or in Chrome at the waterfall kind of diagram there to see is this, is this a page that has a significant amount of JavaScript and other content that needs to be loaded? Or is it something where maybe they’re just like 50 requests in total? And it’s maybe a slightly different question then.
But at least in the cases I’ve looked at it, it is more something where they’re almost like hundreds or 200 300 400 500 requests that are required to render a page. And when you have that many in place, even if we cache a lot of those requests, it’s—it can be a bit hit and miss if the timing works out that we can actually complete all of those requests in time for rendering.
