Google: What If Speed Were a Ranking Factor?

John Mueller
Hi, everyone, and welcome to the next episode of the Search Off The Record podcast. Our plan is to talk a bit about what’s happening at Google Search, how things work behind the scenes, and who knows, maybe have some fun along the way. My name is John Mueller. I’m a search advocate on the search relations team here at Google in Switzerland. I’m joined by Martin and Gary, who are also search advocates on this search relations team.

So one topic that has come up over and over again externally is everything around core web vitals and kind of speed and things around that nature. And it feels like we don’t really have all of the details that we can share publicly at the moment. So one of the ideas that I had is, maybe we could take this as a chance to create something like a hypothetical search engine. For example, if we three were to go off and create a search engine, how would we do something like this? What do you think, guys?

Martin
That’s awesome, but the biggest question is, then, what should it be called?

Gary
Steve?

John
Steve, what do you mean, Steve?

Gary
I mean, Steve, Steve, like Steven. Steve, we call our search engine. Steve.

Martin
Okay. What’s the verb for it?

Gary
To Steve, something that exactly when you want to Steve something, then you go to Steve, and you type in the things that you want to Steve and you click search, or you hit enter?

John
I don’t know. I think we need like an A B test. Like, do we have an alternative that we can?

Gary
Yes.

Martin
Yeah. Searchy McSearch face.

John
Okay.

Gary
Yeah, that will go with totally rolls off your tongue. We’ll go with Steve Evans.

John
I think that’s much easier. Yeah.

Martin
Sorry, Steve. We could call it Ask Steve.

Martin
Ask Steve. No, no, we’ll just go with Steve. That’s like, “you can, you can go to Steve.”

Gary
How old are you Martin?

John
Just remembering old times? Oh, my gosh. I mean, we can make an official version like Mr. Stevens.

Gary
That will be the grown up person.

John
Okay.

John
Now that we have the most critical parts covered, which is surprisingly quick this time, what do you think about using speed as a ranking factor in Steve? So we kind of have like this gigantic index, like Gary has been talking about the index for a while now and how we have all of these shards that go around, shard does shards, shards, shards, shards, charts, charts, it’s kind of like kale, right?

Gary
So many floppy disks, so many floppy disks.

John
Anyway, we have all of this index stuff happening and all of this crawling stuff happening. So kind of the next step, I guess, is talking about the ranking side of things.

And if we wanted to use speed as a ranking factor, how would we go about that?

Gary Illyes
So first, I would talk about how we, Google, release our ranking signals and how we decide what should be a ranking signal and what not, because that might help us further down the road decide what can we do for Steve, because there are quite a few quirks there that you have to navigate, before you can release a ranking factor that will be useful in some way. And I think we talked about this on the house search works site, as well, as we talked about this at conferences. And I know Matt, Matt Cutts, organized a video recording of one of our search quality team meetings awhile ago, which was heavily publicized as well, and people found it very interesting. And all these give little sense in what goes into releasing a ranking signal.

One of the things that we should consider for Steve, is the weight of the ranking signal, meaning that will it be heavyweight, meaning that it will shake up the results quite a bit, or it will be minor, and it will be more like a tiebreaker? Because sometimes, that’s what you need, actually. And to make that decision, we at Google use data from our experiments and raters basically, who are all these things reflecting how users interact with search results. And we look at the metrics that they generate, basically how they interact with the search result page, or direct evaluation from raters from humans. And then we’re taking that into account, we decide whether to learn something or not.

Or stepping back, we can even decide whether to go to the search quality meeting for approvals to launch this or not. So for example, if you have a ranking change that shakes up the results a lot, and you don’t think it should, as a ranking engineer, then you can take a step back and ask yourself, “do I want to go with this to the ranking leads or not?” So if you think about it, there are two big categories of ranking changes. One is where you launch something to fix a persistent problem, or a bigger problem.

If I wanted to give an example for this is fake news, for example, where there are sites that are ranking better than authoritative sources, and you want to fix that. The problem with that is that in your experiments, when you push down non authoritative sources, then that may actually look in your metrics as a negative change, meaning that the metrics generated by the experiment will look negative. And that’s because users might be more willing to click on something for example, that’s more clickbait-y.

Martin
Hmm, that makes sense. Yeah.

Gary
And then if you push those down, then that might look negative in your experiments. And then on the other end of the spectrum is lightweight changes, where you perhaps just need to fix something very minor, or you want to influence something HTTPS, for instance, yeah, HTTPS would be a good example for that, where we specifically designed it to be a tiebreaker. Because we didn’t want to compromise the quality and the relevancy of our search results.

But we still wanted to give a little boost for HTTPS results in our search results, enough to break a tie, for example, where that happens, which actually happens more often than not. So those would be the two categories. And if we designed a speed algorithm or speed ranking change for Steve, then we would have to decide where in this spectrum defined by these two endpoints, would we want it to land? My take on it is that it should be closer to the HTTPS ranking change that Google did. Because you don’t want to sacrifice relevancy for speed. But maybe you guys think differently.

John
Yeah, I think one example that is sometimes used is you can make a really fast, empty page. But that doesn’t mean that it’s necessarily a good result for people. So just because something is super fast, doesn’t make it actually useful or helpful for people. So I think when it comes to Steve, we should make sure that it’s not like super high weight. But at the same time, I also don’t see it as something that is purely influential kind of thing, because it is something that people watch out for. And if we can show people ahead of time that what they’re going to click on is going to be slow, or is going to be reasonably fast and useful, then I think that does help people to figure out a little bit better what is useful for them.

So I guess, similar to kind of the tiebreaker scenario that you mentioned, especially in situations where you can find similar answers on multiple sites, then, like clearly having those on top that are actually a very good user experience with regards to speed usability.

Martin
I think that kind of makes sense.

Gary
Yeah, I would agree with the tiebreaker nature of it, because in the end, users have a first and foremost goal, and that is to get an answer to whatever intention they came with. And that answer should come in a delightful fashion, not like, slow and annoying.

John
Yeah. I mean, just personally, when it comes to Steve, I think it’s something that we should probably test before we nail everything down and say he’s like, it’s gonna be this strong kind of thing. Because it might be that there are situations where there are pages that are not purely identical or purely equivalent, whereas speed does play a bigger role, but it’s tricky to kind of figure out.

Martin
I like the experiments part that Gary mentioned, because I think it’s super tricky to get all the weightings right. You probably want to look at more than just speed and relevancy or authority or something. So for Steve, we probably need lots of signals and figure out how to weight them all somehow.

Gary
Now, so how we did with Google with the HTTPS ranking boost was that we came up with a weight, basically something that we thought would be appropriate. And then we ran an experiment, and we saw how people react to it, how the results changed. And then, based on those results, we made a coal wetter, or do we want to accept those changes? Do we want to convince the ranking leads that that’s good for our users, for Google users, or we go back to the drawing board and decrease the weight, for example, or increase the weight and push it more.

And in reality, we went back I think four times or five times and kept decreasing it, based on one of the ranking leads recommendation to the point where it is at the moment where it acts as a tiebreaker. So we could do that on Steve as well. Basically, we come up with a number, how much it should change the results, and then adjust the weight. Based on experiment results, maybe more than five and less than seven. I think it should be at least seven.

John
At least seven. Okay. I have no idea of the order of magnitude but fine. Seven sounds good. Does speed have any kind of technical role as well with these things? So if we say like, “speed is important for ranking”, is there already kind of some aspect within Steve that also takes speed into account? I’m thinking kind of like of the crawling stats where I see it externally, that people are sometimes confused. They look at their crawling stats, and it says like, your site is fast. And then they look at their speed report. And it’s like your site is slow.

Is there some connection there that there’s already kind of some aspect of speed taken into account or less so?

Gary
There is a technical aspect to speed. And it starts with the question, what do you mean when you say speed, and speed is not always the same thing, depending on what you’re asking? If you look at it from a crawling perspective, Steve probably wants to crawl sites as quickly and as often as possible, because we want to have the latest and freshest information. So if something gets added or removed or changed on the page, we want that change to be reflected in our index. But then again, Steve has to make trade offs because we might run Steve off of a bunch of Raspberry Pi’s, for instance. And then Steve can only crawl so much on a date. And we need to decide what to crawl.

Also maybe even Steve takes very long to connect to a site server and we can only have a limited amount of connections at the same time. Then Steve would conclude that waiting for the slow server means it can’t make connections to other servers while it’s waiting.

Martin
Yeah, you mean Steve bot?

Gary
Oh, yes. Steve bot why? Yes, of course, the crawling part of Steve. Steve bot. And then there’s the Steve index tiers, very cheerful, and Steve index shards, and of course, the Steve rendering service, when we want to support dynamic website content any way.

So yeah, Steve bot has a pending connection waiting for a slow server. That’s one challenge. The other one is when servers fast to connect to but it takes a while until it actually sends over the response. So again, that would be holding up Steve bot. And then Steve bot would probably want to crawl other sites more often when they’re faster. But that’s just some of the aspects of how speed can affect crawling of Steve bot but a completely different topic that happens later on as JavaScript when you have the fast side. But do we actually want to support JavaScript?

Martin
No, no, no. JavaScript is evil.

Gary
Oh, okay. No JavaScript, no Steve rendering service. So sad.

John
Okay. Okay, fine. Like, hypothetically, if we were to expand Steve, and make Steve understand JavaScript, okay.

Gary
So that’s Steve 2.0. So Steve 2.0, has the Steve rendering service. And some site might be really fast and crawling, quick connection, quick response, but then uses JavaScript to create the actual content on screen. So we need to actually start a headless browser and run the JavaScript.

But then how long do we wait for the JavaScript to finish? Do we say 60 seconds? Do we wait five minutes, or just 10 seconds? And I guess we can use some heuristics to like figure out a good value. And also some common sense guidance, like no user is going to wait 10 minutes, so we don’t wait for 10 minutes either. But still the question that is like, what do we measure in terms of speeds on the user side of things, right?

John
Well, we can just take the rendering speed, right. Like you said, we’re going to render the pages and then we wait, like, can’t we just take that number?

Gary
Or yeah, we could take that number from rendering. But what do we exactly measure there until the content shows up? I think that’s great for pages like Wikipedia articles we just read.

John
Okay. Yeah.

Gary
And that’s what we call LCP, or the largest contentful paint.

But what about if it’s like a pizza website, so the content shows up quickly, I can see the pizzas. But when I click on one to add it to cart, nothing happens. So I clicked like, 1000 times, and suddenly I have 1001 pizzas in my cart. And you have to awkwardly remove 1000 pizzas.

John
That’s not bad, right?

Gary
No, no, no, you don’t remove 1000 pizzas. You buy the 1000 pizzas.

Martin
You have interesting eating habits, Gary.

Gary
I would give pizza to everyone. And I would put pineapples on it.

Martin
Oh, wow. Oh, my goodness, you’re mean. But you know, I’d still eat that. Pizza with pineapple is still pizza. And there’s no such thing as a bad pizza.

Gary
What? What’s wrong with you? I will add this to all this stuff. Horrible things Martin did oh my that must be long. How many shards does it take to store that?

Martin
At least 7 floppy disks. That’s big data, man.

John
Okay. Okay, back to our pizza delivery service. Like you’re saying 1000 pizzas would be a bad signal.

Gary
No 1000 pizzas itself isn’t the best signal it’s just, if we were just to look at the largest contentful paint, we’d miss out on the interactivity issue on that website. Or another thing, if you’re like trying to tap on something, and then something on the page makes everything move as you tap on something completely different. Like, I don’t know, a pizza with pineapple. And you don’t want that. That’s also super frustrating. So I think we need more metrics than just like a single speed metric.

John
Okay, so you’re saying we can’t just use the Steve rendering service to get a number and then stick that number into our ranking system?

Martin
Yeah, we might have to get multiple numbers, and then figure out how to mix these numbers together into one signal.

John
Okay.

Gary
What if we create a browser called Steve Nicol, and use data from that browser that would feed into our measurements for speed?

Martin
That’s not a terrible idea, actually. But it seems like a kind of a long term plan. It’s like, we’re just three guys. I don’t know if we can John, who got to think big. Got to think big.

John
Okay. Very big, very big. Steve Nicol. Okay.

Gary
It’s a long shot, but it’s worth it.

John
Okay. But I mean, if we didn’t have like Steve Nicol, until we have Steve Nicol everywhere.

Gary
And we have an interesting problem. Where do we measure speeds?

John
I can do it on my computer. I have this speed testing tool.

Gary
Yeah. Okay, great. But what if the user is somewhere with a terrible mobile internet connection? Like, Germany? Your computer probably has a stable fast fiber connection, doesn’t it?

John
It’s possible.

Martin
Can’t we buy that kind of data? Basically, connection times and perceived speed?

John
Okay. Okay, we can probably get that I think like the internet connection speeds in different regions. That seems like something that could be doable. And then you’re saying, like, basically, we just scale the numbers that we get?

Gary
Yes.

John
Internally. It’s like, “oh, slower here, faster here.” And that, how do we apply that then? Is it? Like, do people in different regions have different rankings or—

Gary
Yeah, but you’d probably have that to begin with due to things like different language regions? I mean, getting results in the language I don’t speak just because it’s ranking high elsewhere isn’t going to be great. So there will probably always be ranking differences anyway.

Martin
Yeah, we will definitely have that.

John
Okay.

Gary
We need to have that. Otherwise, I would get German results. In Switzerland, for example,

John
I already get…

Gary
Wait, wait.

Martin
Well, that was an example of how things actually work. Well, good job.

Gary
I would get results from Germany in Switzerland, for example.

John
So we should introduce a Steve Lang meta tag, what do you think?

Gary
I would advocate for not introducing more meta tags. We already have enough meta tags on the internet, please.

John
Okay. Thank you. Okay. I think like one approach we could also do is based on the data that we see from people using Steve, like to try to figure out like, which regions are even relevant for individual sites, and maybe use that to scale the numbers.

Do you think that would make sense or is that tricky with things like global sites that everyone uses?

Gary
Yeah, that’s probably tricky. But the reality is that there are not that many truly global sites compared to the bulk of the internet. We were exploring this quite a bit when I was on the country promotion language demotion team a while ago, when I was working on ranking, and there aren’t all that many truly global sites.

John
But sea shanties are global.

Gary
They are?

John
Yeah.

Gary
Okay, now it’s in my ear. Again. Don’t tempt him. Do not tempt.

John
Soon may the ranking change come now, soon may the Wellerman come to bring us sugar and tea and rum one day when the tonguing is done, we’ll take our leave and go.

Gary
If you keep doing that, it will also be in my ear eventually.

John
That’s the plan.

Martin
That’s fantastic.

Gary
Yeah, fantastic indeed.

Another thing we should look into when we use this kind of data, like regional data, and so on, is we need to prevent people from abusing that like feeding us bad information through data sources and experiments and like influencing how we measure speed.

John
So you mean make a page fake-fast?

Martin
Yeah, fake-fast or fake? Slow?

John
Why would you make a page fake-slow?

Gary
It’s like, well, if it’s your competitor, you kind of go ha-ha, I’ll make a connection from the flicker mobile network in a bunker, you do know that we have 4G receptors in the bunkers, here in Switzerland, right? Not surprisingly, I actually do know that after all, our basement is an airway shelter and I have fantastic mobile connection in there. I even make video calls from down there. So that would not work.

John
I guess we’re not gonna have spammers in Switzerland, in that case, which is kind of good. But I’m sure there are creative people everywhere. I don’t know, like, how would we deal with people trying to game these signals?

Gary
Well, one thing that we have to consider is how many people will actually try to do this, basically, how feasible it is to do it. Because if it’s not feasible to do it, then there can be all that many people will do it, right, which renders the whole problem kind of moot, and basically low priority, something that you don’t necessarily have to fix in the first release, for example.

Then if you see an update, for example, like some genius creates this service, which would enable others to, I don’t know, run a competitor site on iPhone to German bunkers, then that might actually change our mind. But in the first release, when we don’t know or don’t think that it’s feasible to spend, we might not need to fix anything, that can be a problem for future Steve.

John
Yeah, it’s kind of like letting the perfect be the enemy of the good or like, however, that saying goes. And as like, theoretically, like, we could be trying to solve all of these problems at the same time. But if we don’t even know it’s actually going to be a problem. Like, let’s look at it when it does come up.

Gary
Right, I think that the most important thing is that we have to figure out whether it is feasible to spend, because it might very well be that it’s not even feasible to spend. Basically, you have to invest so much in equipment, for example, or infrastructure, spending infrastructure, that it might not be feasible for spammers to actually create that with web spam, that’s completely different. Because you literally can create websites for free, at no cost whatsoever, besides your own time, and spam the living hell out of search engines. But with speed, it might actually not be feasible.

John
It might also be one of those things where the best way to game it is to make your site fast, which is kind of like, Okayyy…

Gary
Good job for doing the right thing for the wrong reason. But hey, you’re doing the right thing. Yeah.

John
So cool. So I guess the next step would be how do we tell people about this?

Gary
So that’s a big one.

John
So do we put like a banner on Steve and say, like, by the way, today, all the results that you see will be fast? Or what would you do there? Martin?

Martin
I will create a podcast called Steve Off The Record.

Gary
We can have a Steve relations team, Steve relations team.

John
Okay. Steve Central. I mean, is it something that we would even tell people about like which metrics that we would use? And what we’re kind of thinking behind this? Or how would you see that.

Gary
I think it’s okay to tell people what we are looking for in broad terms, like your site should have relevant content, it should be fast, and so on.

But I would probably not disperse too much on the exact factors and weights, that stuff is just used to game the system normally. And it’s a pretty gnarly situation where giving out too much information makes the ecosystem worse for everyone in a race to the bottom because everyone tries to like game the system, then yes, it’s tricky.

John
I think it’s tricky, but at the same time, I also like being able to give people some clear guidance and some mechanism for them to double check. Is your site speed-friendly or not? Kind of, I don’t know testing tools or understanding of what exactly we’re looking for so that they can work on things a little bit in the right direction. It also feels like something where probably the exact metrics and the waiting will evolve over time. On the one hand for, I don’t know, potential abuse situations, but maybe also just because the different possibilities at some point with regards to what we think really annoys people or not, because I guess it all ultimately goes back to like, why would we even use speed as a ranking factor on Steve, and it’s not like we have nothing better to do. And speed is a number, therefore, we will put it into our system of kind of calculations.

But it’s more like, well, if we make a search engine like Steve, then we kind of rely on people actually using the web, because otherwise, why would you use a search engine? So we kind of need to make sure that the experience on the web is a good one for people, which I think kind of also matches how Google would probably look at this. In terms of like, well, we need to make sure that when people go online, they actually see going into a browser searching for things, looking at random websites as something that’s fun, that gives them information, and is not super annoying in the sense that you accidentally end up ordering 1000 pizzas when you just wanted one.

Martin
Or worse, ordering a pizza with pineapple on it, according to Gary.

John
Yeah. So you’re saying like that shifting around metric, we should make that really high weight?

Gary
Yes. Yeah. So important. Make it at least seven

John
At least seven? Maybe it depends on the type of site that you’re looking at.

John
Okay. So I think Steve is shaping up fairly well, that gives us some work to do. Now, we just need someone to actually do the coding. What do you think? Should we go to some consultants who can just help us bash out some Python?

Gary
Python.

John
Now we use, do we use C? Or like, what do we use for coding? nowadays?

Martin
You can use JavaScript.

Gary
JavaScript. Well, that would be the shortest career I’ve ever had.

Martin
Oh, you would show up, say no and leave?

Gary
Yes.

John
Okay, maybe not JavaScript. Maybe we will pass the discussion of programming language on to another podcast episode or someone else’s podcast, because that probably could get quite controversial.

John
Wow. Okay. Cool.

Gary
This was a good idea. This was very much fun.

Martin
It was a great idea. Indeed.

Gary
You should have come up with this sooner, John. What’s wrong with you?

John
Okay.

Martin
Oh, Gary. so supportive.

Gary
Cool.

John
All right. One day the Wellerman comes…

Gary
Oh no.

John
It worked. It worked.

Gary
You did it.

John
And I think we should cut here.

John
Yeah, that’s it for this episode. Thank you for joining us here, folks. It’s been fun doing these podcast episodes. And I hope you the listener have also found them entertaining, and maybe even insightful. Regardless, let us know what you think. Drop us a note on Twitter. Drop us a note in the YouTube comments where we’ll also be uploading these podcast episodes. And let us know what else we should be covering and maybe what other ranking factors Steve should be taking into account. And of course, don’t forget to like and subscribe and share this podcast with everyone everywhere so that we get at least seven viewers or listeners. Yeah. All right. Thank you and goodbye everyone!