← Voices of Search // A Search Engine Optimization (SEO) & Content Marketing Podcast
Voices of Search // A Search Engine Optimization (SEO) & Content Marketing Podcast
SearchPilot’s Biggest Launch Yet: The Future of SEO Testing
01:03:52
Loading summary
Transcript92 lines
- [00:00]
A
The Voices of Search Podcast is a proud member of the I Hear Everything Podcast network. Looking to launch or scale your podcast, I Hear Everything delivers podcast production, growth and monetization solutions that transform your words into profit. Ready to give your brand a voice then visit iheareverything.com welcome to the Voices of Search Podcast. A member of the I Hear Everything podcast network, ready to expedite your company's organic growth efforts. Sit back, relax, and get ready for your daily dose of search engine optimization wisdom. Here's today's host of the Voices of Search podcast, Jordan Cooney.
- [00:45]
B
Over half of all US searches, 56% now end without a click. That means users are getting answers directly in AI summaries, not from links, not from your site, but right there in the search experience. And that stat, it's not hypothetical, it is real. And the SEO teams are facing a seismic shift because when traffic doesn't happen, what does performance even mean? How do you test what works when the old rules no longer apply? I'm Jordan Cooney and joining me is Will Critchlow, CEO at SearchPilot. SearchPilot is the leading SEO AB testing platform for enterprise sites and they just launched the industry's first geo testing system, purpose built for AI search. Will, welcome to the Voices of Search podcast.
- [01:34]
C
Thanks for having me. Jordan, great to be here.
- [01:36]
B
Yeah, thrilled to have you on the show. I think that this is a highly under discussed topic which is like what are we, what are we doing? Like what are we measuring now? And I think that, you know, there's been a lot of just hype and discussion in the market around a lot of different metrics and how you think about what it really means to be discovered in organic and how do you attract users, buyers through organic channels. And I really want to understand like with geo, how is testing different today than what we used to think when it was just an age old 10 blue link SEO world.
- [02:21]
C
Yeah. So I mean the first thing I should say is I'm super excited about this. It's great to be working in such a vibrant industry and space and I love that there's competition for the first time in decades. Feels like that's holding Google's feet to the fire, making them innovate, making them do exciting things and even if we just look at them and of course there's all the energy coming out of all the startups doing exciting discovery things as well. So to answer your question, let's start with what's the same. So when we're doing the kind of testing that we're talking about here for Geo, some things are the same. We're still fundamentally talking about website changes, we're still talking about changes typically to scalable site sections. So most of our work is with E commerce, retail and travel businesses. And so we're talking about changes to product pages, listing pages, category pages, those kinds of things. We're talking about hypotheses that are similar, they're related, they may have a different why, but the fundamental change is going to be something like, you know, we're adding this new information to the page and our hypothesis is that it's going to make it more valuable, more attractive, ultimately get us more business. The measurement is actually quite similar because we're still, most of our tests are run against analytics data. In the case of Google, we can also augment that with search console data and we can pull in other data sources, but the data maturity of the new platforms is much lower than, you know, there isn't a search console for ChatGPT yet. Maybe we'll talk about how that might come along one day, but those things are the same. And our approach is similar to how we've worked in the SEO space, that we're taking that hypothesis, applying it to some pages and not to others, and measuring the relative performance of those groups of pages. The key difference when it comes to the AI powered searcher journey is that as you alluded to in the, in the intro, so much more of that journey happens on the search engine or on the, in the chat window and less of it happens on your website. So it's almost like we're testing against the user journey as a whole rather than just that, you know, rather than every click. And it's a little bit analogous to thinking, you know, when you as a user are researching a product that you might like to buy and you open up a whole load of different tabs in your browser all at once and then you go and read all the tabs.
- [04:58]
B
Are you like looking at my screen right now?
- [05:00]
C
What's going on? We all do this, I think, and you know, it's okay guys, you can, you can shut some of those tabs, you're not going to get back to them. But you know that process where you're doing research, you're on a whole load of tabs in different windows and then at the end of it you come back to the one place where you're going to buy from. The LLMs are kind of doing that process for you when what we call the fan out query is them going and Doing that and sometimes even doing multiple queries, Lots of queries behind the scenes, opening lots of tabs in their browser windows, reading all of them and kind of coming back. And so a GEO test is a blend of an SEO and a conversion rate test in a sense. But the conversion rate is the robot conversion. It's the idea of like, okay, the robots read your page, did it find it compelling, did it present that information back to the end user? And you know, you ask one of these AI tools to recommend some running shoes or something, and it's not as simple as a ranking algorithm, which was complicated enough. It's a recommendation process. And so it's not just does your page rank, it's does the page rank in those fan out queries and convince the robot to include it and include that information in its summary and recommendations to the user. So, you know, there's some philosophical differences, but kind of the fundamentals of how we actually run the test are very similar.
- [06:23]
B
So let's talk about one of these concepts because you brought it into this, which is fan out query. And kind of this has been a really very hot discussion topic across the SEO content digital marketing community lately. And I think that this is an important one as a, as a backbone to thinking about testing fundamentally, because as we better understand what LLMs are looking at, it probably better gives us an understanding of what content or what experiences we need to have on our sites. And so I'm curious to get a little bit of a unpacking and as to back to the original question. Like some of these traditional SEO ways of working or doing things, we're probably still doing some of those today because they're valuable for the user and they're valuable for the user because they map to this concept of fan out query. What are your thoughts on that relationship of trying to get to the user and how is it that search pilot or even just any form of testing in general helps helps us get to better understanding of what to prioritize.
- [07:34]
C
So I think the user needs to be kept at the center of all of this. You know, one day maybe we'll have these, you know, magic robot agents off buying things on our behalf. Maybe they'll even earn the money for us to go and buy the things on our behalf. Who knows?
- [07:50]
B
That would be great. Will? Yes, please.
- [07:52]
C
Yeah, but we are still the ones that want the stuff, right? You know, if your robot agent is off buying you shoes that don't fit or whatever it might be, that's not going to waste. That's not going to help anyone. And so the human is still at the center of why we're doing this work. And so I think you're absolutely right. We need to ground our hypotheses in how it benefits the user ultimately. And that to my mind is the kind of big picture societal level good of the work that we do as SEOs or in the generative world is if we do our job well, we're kind of helping the economy run better. We're helping people buy the products that they should. Right. I'm a big fan of trade. You want the money more than I want the shoes. I want the shoes more than you want the money. So we swap the money in the shoes. Right. It makes both of us better off. And the more that we can facilitate that, remove those frictions, get people buying the right things, then the more that incentive there is on both sides for the vendor to describe these things accurately and to build the best products and to source the best things and for the user to go put that work in to choose the best product for them. And, you know, and so again, we're getting a bit philosophical, but I think keeping the user at the center is absolutely critical. And then I think when we think about how it works from a testing standpoint, one of the big things that we're finding as we work with these large organizations, because our customers tend to be very, very large retailers, is one of the things they really want is the insights. It's not just winners. Everybody celebrates the winners, of course, but what we're really looking for is to say, look, this worked. So let's go back to the hypothesis and let's understand why we thought this was going to work. And let's extrapolate that hypothesis and say, well, look, this worked because we gave the user more product information, for example. Well, can we take that a step further? Is there another source of product information that we can add in and see if that's even better? And so following that kind of that breadcrumb trail, if you like, not HTML breadcrumbs, but that trail that leads us from one insight to another to continually improve and ultimately end up on that best user experience, again, keeping that user at the center of things, just like.
- [10:19]
B
Last question on this kind of segment of this GEO world and the SEO world of testing, what is new? What are some of the new elements that you are introducing as we look at a GEO world? In particular, what I'm, I'm kind of curious to, to, to understand is how are you thinking about things now that it is a little more fragmented than before. It's not only just more fragmented within Google with concepts like AI mode, AI overviews and just a broader SERP experience, but it's now even fragmented where you have different discovery engines like ChatGPT. Perplexity. And we can go through a very long list of all these different LLMs. But like, I am curious to get your, your, your perspective on how it is changing today.
- [11:09]
C
Well, I think of this as the ecosystem getting healthier with the diversity of those sources. And it's early days, right? Google is still the behemoth. And you know, as, as I'll talk about, no doubt I'm pretty bullish actually on Google's prospects. I think, I think they're, they're doing great right now for their business. But one of the things that we. Probably the biggest thing that is changing is the move to the multimetrics world. So for the longest time, search part's really been focused on a single metric, which is your performance in Google. And that typically meant clicks through to conversions from Google. And we are more and more finding that people are interested in multiple metrics on the same test. And that might be guardrail metrics. Right. So anything from merchant center impressions to search console impressions to you might have a pure LLM hypothesis, but you want to have a guardrail that your search clicks don't, aren't harmed by whatever you might have done. Interesting. And so, you know, we've always joked, people used to ask me in sales processes, you know, how much do you care about Bing? And it's like, well, how much do you care about Bing? And the, the answer was typically not a great deal. But now the answer to how much do you care about ChatGPT is people care about that a lot, disproportionately, even to its size and its growth rate. I think, because it feels like the future, because it is growing so fast and because it represents more than just a search ranking. It represents a whole, I want to use the word vibe, but you know, a whole feeling about how your brand is portrayed and how your products are recommended. And it's not just that kind of clinical list of links. And so, yeah, I would say the biggest shift for us is that move to multimetrics. It's the fact that people genuinely care about more than one thing now. And ironically, that then goes full circle to what. Where we're ending up is that you roll it back into Net Impact. And so we're starting to say, okay, what you should really care about when you make A change to your website is how does it affect the overall performance? And that combines all of those channels you talked about and it doesn't just look at them in isolation and say is this good for chatgpt or is this good for Google? It's trying to say in aggregate, in total across all those things, what's the net impact?
- [13:39]
B
One of the things that I love that you're talking about is what's happening on websites, right. A lot of the metric based conversations that I see right now online and that I think a lot of the industry is feeling are external. Right. But like what you're talking about is like let's take a look at what's actually happening on your website today. And so let's spend some time on LLM traffic and I'd love to just start very broad on this theme and just get your perception of the shift over the past year towards LLM traffic. Obviously there's a lot of noise about the volume. We just did a study here at Pre visible analyzing 2 million LLM sessions across our portfolio. And I think that it's really eye opening as you look at how this traffic is different, what it means to a brand. And so you've brought that up but, but just give us kind of a broad strokes view on like what does it mean to get an LLM session or visit or click or whatever definition you want to use to your site.
- [14:41]
C
I think the great thing about that is that the answer today might be different to the answer tomorrow. And again back to the, the fun and enjoyment of this stuff. It's, it's just great to be alive in this era. It feels like we've seen the invention of the printing press or something. I love that. So I'm just excited by it. But the answer for me is that it's kind of still too soon to tell in many ways. I need to see that study that you just mentioned. What we're seeing for most of our customers is single digit percentage volume. And you know, that's fine, it's interesting but you double it and you still have single digit percentages probably. And so it is a different kind of thing. My personal view, this ties into how I'm kind of bullish on the Google position in the market is that it's not actually a surprise that we ended up with search engines looking kind of how they do with a list of links. I think actually links are, links are great, hyperlinks are amazing. The invention of the World Wide Web was a big deal and I sound like a technology Fanboy. But I like links, I like clicking through to somewhere else and getting to where I wanted to get to and I actually think that a lot of these user interfaces are going to evolve further in that direction and I think we're seeing this with AI mode. I think AI AI overview through to AI mode at launch through to AI mode today. You can see some of that evolution from. There are certain things that people want to get instantly right and, and we've seen that in even in traditional search results. You know it's been, it's been a long time that you've been able to get you know the time in la it just tells you the answer. It doesn't send you to what's the time in LA.com or you know, whatever. And same with the weather, same with recipes. Yeah, exactly those one box type things. I think for a lot of things you need more than that and you need more diversity, you need more multiple options. My brother folks have read any of his stuff. His name's Tom Critchlow. He did a great proof of concept that I really enjoyed where he was prompting Gemini to do a better AI mode and getting it to. It's like you've heard the classic sales interview technique of like sell me this pen. He was asking Gemini like sell me this website. So convince me to go and visit this website was his, his kind of part of his prompt and it was coming up with great stuff and, and I want that because I want to visit websites I like, I like reading things I like, I like visiting stuff. So that's part of it. I suspect we converge more and more on there actually being more outbound traffic from some of these resources, particularly the ones that are used as discovery engines. And I actually, I'm not sure that that's a stable long term future for chatbots. Exactly that chatbots are great for what I'm finding more and more of my interactions with the LLMs being being more interactive, even more interactive than a chat. Right. We're co creating something or we're reviewing something or you know, the vibe coding something and if I just want to find a pair of sneakers I actually want something more search like and so anyway we'll see where that goes. Final closing thought on that train of thought for me is we also at Search Pilot are really focusing heavily as I said on E commerce and travel and those kind of sectors. And I think in those sectors at some point in the journey the user has to interact with the vendor. Right. Chat GPT is not going to ship you a Pair of sneakers is not going to, you know, take you on an airplane to another country. Those things have to be provided by a different business, and there has to be an interface with that business. And we can discuss what the future of how agentic that interface looks or whatever else it looks like, but that's where we're playing. The future of media is a very different space. And I am, I'm not sure what it ends up looking like and how many clicks there are to news sites or to, you know, entertainment sites and the like. That's less my game these days. So I'm not quite as expert on that. But. But yeah, in our space, I think.
- [19:04]
B
Those clicks are remaining on this point of verticalization. Like, I mean, our, our study did also highlight that it's 1% or less of traffic, but it is very different by verticals. And I think that that's one of the interesting components of understanding LLM traffic. And in the spirit of, like this, this, this position that we're in in the market right now, 1% of the traffic, there's a lot of, like, mystery conversations about what's actually happening on the other side of that 99%.
- [19:32]
C
Right.
- [19:33]
B
What is the visibility, what is the impression rate, what's actually happening with users behind it? Now, now there are some verticals that were outsized, even in our, our category, Healthcare, finance. A few of these traditionally have been called in the SEO space, those YMYL categories, they had outsized percentages of LLM traffic. But, but to your point, I think this is a really astute one, which is LLMs. They are like a window behind the click. We used to have to do a lot of clicking. Now we have to just do less clicking because the LLM gives us that window behind the click. And so I really think that how you understand that, how you test for that, and really getting back to the core of like maybe just understanding how Search Pilot, and maybe even you yourself will think about the purpose of that click, the value of that LLM click and is that, does it have an outsized weight compared to all those old SEO clicks? Just want to get your, your value.
- [20:34]
C
In the long run. I think there have been various studies, I've seen studies in different directions on this. I think it's hard to tell at the moment because there's such a selection bias, right. That the people who are using the LLMs more heavily than Google is not a random subset of the population. So we can't, you can't exactly compare those clicks because they're different people. So that selection bias kind of comes into play. But if you think about it from first principles, my expectation is that yes, the better the search results get, the more AI powered all of these things get, including Google. I should repeat a thing I've said many times, which is your biggest AI powered channel is Google. Google is AI powered now and is only going to get more AI powered. And I don't just mean AI mode, I mean all of it is, is, is and is going to be increasingly AI powered. The better that stuff gets, the fewer clicks there will be in all these verticals. Are people gonna buy a. Buy fewer sneakers, take fewer flights? Like, I don't think so. If anything, I think the opposite. I think people will take more flights, buy more sneakers, because they will be getting better recommendations, finding them at better prices, happier, with the result if everybody does their job correctly, including us. And I think therefore, one of the things that I've been talking to in house teams about is you need to be setting your organization up for looking at real value and not just looking at some of those vanity metrics that we might have been excited about in years gone by of, you know, your year over year traffic might be down while the business is up, while the business is up, driven by organic discovery. And you need to be careful not to kind of shut yourself in a dead end where you're trying to make that argument too late. You want to make that argument now. These are the trends that we're seeing, the trends we're expecting, and yet if we do our job well, more people can buy from us than ever before in 2026, 2027 and beyond.
- [22:32]
B
I got to ask you this question on this theme of like, you know, LLM metrics and not just the traffic, but like, even more broadly speaking, like, there's a lot of discussion in our industry around kind of the footprint that LLMs use to better create context, right? Whether it be the Reddit hype, a bunch of other conversations around how they're kind of utilizing different sources to form their responses. But at the end of the day, like all of this stuff boils down to this concept of authority as a testing platform, as a testing product that helps SEOs, marketers, executives, E commerce managers and leaders make decisions. How do you bring the authority conversation into this testing environment and how do you make it something that's actionable or useful?
- [23:26]
C
So we are very focused on on site, so we're looking at the authority that comes from the things you say about yourself that everybody knows are the things you're Saying so this is, yeah, this is not PR or you know, social media, Reddit, whatever. And so my approach would be to come at that from a, from a hypothesis lens. You know, if we can put expert content in, we can put trustworthy content in, we can demonstrate that trust in various ways, whether it's third party testimonials, whether it's review information, whether it's other trust, data delivery times, statistics, whatever. It might be that we're almost coming back to the eat world.
- [24:09]
B
Yeah, exactly.
- [24:10]
C
Which is, which feels like a different era already.
- [24:13]
B
It is a different era.
- [24:15]
C
As I, as I say that, I realize I haven't said those words for a little while, but it actually ties also into why I think Google is well positioned here. Because you, I'm sure will have seen many of your, your audience will have seen these like crappy listicles that people have been putting out with their own brand number one, drive you nuts. And it's like the reason people are.
- [24:38]
B
Doing that is thanks for being selfish. Thanks for being selfish and putting yourself as number one.
- [24:41]
C
Well done for putting yourself well done for winning your own award.
- [24:44]
B
We can swear here. But those are assholes right there.
- [24:46]
C
I mean, that's just, I hate that content. And me too. While I respect certain bits of the game and there are certain things that I've, you know, it's not wildly different to some of the SEO content like, I don't know, you know, alternatives to Dropbox. Right. So Dropbox probably has pages on their website.
- [25:06]
B
The versus content.
- [25:07]
C
Yeah, Dropbox versus box or, you know, alternatives to Dropbox or whatever it might be. And it's not wildly different to that. And actually I think there are ways to do it in a wholesome way of just kind of, you know, listing, listing alternatives that are different and kind of saying why, you know, Search Pilot, we might say we believe we're the number one option in enterprise E commerce. And if you want to be doing other kinds of testing, if you want to be doing, you know, conversion rate testing or you want to be doing website performance testing or whatever else, then here are some other options and we might, you could, you could imagine some legitimate content around that. But the, the reason I bring it up though is that the reason people are doing that is that the LLMs are pretty naive right now. They are, they are trained on data and they just ingest data at face value in many cases. And I've seen, I think Lily Rae did some experiments where she put some stuff on LinkedIn and two weeks later, sure enough, you could go and ask those obscure questions of the LLMs and they would just verbatim recite this thing that is from one source on one social media post. And uh, I, I, I often say to like friends and family, you know, somebody recite a fact. And I was like, I, I read that on the Internet and I know some of the people who write the Internet and I wouldn't necessarily, wouldn't necessarily believe everything you read on there, but.
- [26:31]
B
Now I know how to correct all those Thanksgiving conversations.
- [26:34]
C
So I know the people who write the Internet. And but this is where I think Google has an advantage because Google has been an adversarial information retrieval environment for 25 years. Yes. And yeah, despite that, I am fascinated by the fact that the other LLM providers don't seem to have realized that they're walking into an adversarial environment. Or I mean, in some cases, I mean grok maybe just like they relish it, right? They, they, they, they like, oh no.
- [27:04]
B
They'Re just eating it up.
- [27:05]
C
Yeah, yeah, yeah. They like the fact that, that people are abusing their, their system. But I'm surprised that the anthropic and OpenAI folks who I would have thought would have seen coming the Google experience, like, you're going to need a spam team, you're going to need an abuse team, you're going to need a, you know, probably a webmaster liaison kind of role. And but the, the adversarial piece is particularly interesting to me that it's like they never met an SEO and maybe they haven't ever met an SEO, but you know, they have it. But you, you put some of the, you put any of these things in front of, you put Google Maps in front of an SEO and they start thinking about how to abuse it. You put YouTube search in front of an SEO and they start thinking about how to abuse it. And I think they, they need to realize that the Internet is full of people writing stuff because of the incentives they've created. And you know, Google created that virtuous loop through AdSense revenue which essentially funded the creation of the content that Google themselves wanted to index to make their searches happy. And they were paying people to write that knowing that they could, it would monetize end to end and everybody would do well out of that. And that was, you know, that decade worked very well for all the kind of content creators on the web. Google still has that mindset of spam detection and authority and relevance. And I think, yeah, as I say, the fact that they have that search engine to ground Things on is one of the things that makes me more bullish in, in their, their engine over, over the, the next year or two.
- [28:40]
B
Let's jump into this. I mean you've said it now a couple times. You're super bullish on, on Google. I, I have a, I have a 13 year old son and just a few nights ago he came up to me and he asked me this question, dad, why does ChatGPT need OpenAI? So I mean, obviously he uses ChatGPT. He has no conceptual understanding that this is the parent company that owns this product or tool.
- [29:05]
C
And if the behind the right, the Microsoft shareholding and the nonprofit and the Elon and anyway, yeah, I was gonna.
- [29:15]
B
Go down that path and explain all that, but I stopped myself.
- [29:20]
C
Maybe save that for another day. Yeah, it's a great question. And actually look at the, that generation. So my kids are the same kind of age and there's a few things that are interesting. I was driving my son and a couple of his friends home from, from something last night and they were having a big argument among themselves because one of them was quoting stats to the others that he'd got off AI and the others were not believing him because, because they were like, it's just hallucinated that stuff like these things make things up. So they, you know, they're being taught in school. Some of the things that we were, we were taught about. Well, we weren't taught about Wikipedia in school, but we weren't, we weren't being.
- [29:57]
B
We weren't talking about this back when we were kids. We weren't, this was not a. We, we, we. We took the Internet at face value and largely it was somewhat accurate. Right. But that was, I think, because of who invented the Internet.
- [30:09]
C
Although it's interesting. And so, I mean, I don't, I know nothing about your, your family. My parents, when I was, when I was the age that, you know, our kids are now. Yeah, yeah. My parents were the ones telling me, don't believe everything you read. Right. And they were telling me that because of tabloid newspapers, it wasn't the Internet. Right. It was tabloid newspapers.
- [30:28]
B
Exactly.
- [30:28]
C
It was. Um, and I find it fascinating that it's that generation that got completely like befuddled by things they read on Facebook or WhatsApp or, you know, whatever. Anyway, that's a side time for a.
- [30:43]
D
One minute break to hear from our sponsor, Pre Visible. So you're looking for SEO help and you got a couple of options. You could start replying to spam from agencies that claim they can get you to rank number one on Google, you could pay an hourly rate for a consultant who will inevitably nickel and dime you with hourly charges. Or you can work with a cookie cutter agency to quickly launch a strategy less project with low success rate. None of those sound very good now do they? Well, that's where Pre Visible's integrated consulting model comes in. Pre Visible draws From a collective 40 years of SEO and digital marketing experience to unlock your organic growth opportunities. They build custom solutions that combine strategy, technical expertise, content and reporting to effectively operationalize SEO for your business. Pre Physical's four stage approach ensures that your SEO programs thrive by starting off with a strategy first approach. Then they support you in your efforts to create quality content, help you identify technical issues, and most importantly, they'll work with your cross functional teams to integrate your SEO strategies to make sure that your SEO budget actually drives results, not just your agency's bottom line. So join brands like Yelp, eBay, Canva, Atlassian Square, all who rely on the SEO consultants at Pre Visible. For more information, go to Pre Visible IO That's Pre Visible. P R E V I S I B L E I O.
- [32:11]
C
I mean.
- [32:11]
B
I made my, when I gave, when I gave my son chatgpt the paid version, I made him read the article by OpenAI on why LLMs hallucinate. So I said you're going to use this thing, but you're going to sit down and read this first. Because this is the doctrine of what you're using. Like you are using something that inherently is going to make shit up. And in that, in that, in that.
- [32:36]
C
You will tell you what it is.
- [32:39]
B
I mean sometimes it bolds it now but like still it's not great. But so like you, you have to be the steward of it. Then you, you run the tool, not the tool. The tool doesn't run you.
- [32:49]
C
Yeah, right, right. It's a, it's a bicycle for the mind. And I think. But my point is that generation, they are in my experience pretty savvy about the, pretty skeptical about some of these things in, in ways that actually I think people who haven't had it around them so much have, haven't quite, quite grasped, but at the same time they're naive about how and why it works. Like you say, you know, your son doesn't have the distinction between ChatGPT and OpenAI, which is obviously fine. But I think the, the, we see it with, with relatively less technical users all the time. What's the difference between the browser you use, the search engine you use your Internet connection, your ISP your phone. It's like, you know, I use Apple, I use Safari, I use Google, I use, you know, like which of these things is, is what is doing which bit in the stack? You know, we kind of take for granted that we understand that stuff. The other thing I've joked about, just funny that we're kind of the same age with similar age, similar age kids, is how did our generation end up doing tech support for the generation above us and the generation below us? That's such a great question. Like, are we the only ones who. I mean literally, I think it's because we were the ones who had to build our own computers, right? Like my parents, my parents didn't do that and my kids don't need to do that. And so somehow we end up telling the kids how, how to do stuff and our parents how to do stuff.
- [34:21]
B
That is so true. That is so true. No wonder I'm, I have no time, right? Doing tech support for both generations.
- [34:29]
C
Although for everyone. For everyone listening, who's got kids about that age, they're great at doing user interface tech support for grandparents. So. Oh yeah, hand over, you know, hey, how do I pull up, you know, how do I get my favorite show on my iPad? Right, right. Kids can do that for the grandparents and so delegate that and they can record little video and the grandparents love it and you know, that kind of stuff. Great. But when you're like, why doesn't the WI fi work? They're like, I don't even understand the difference between WI fi, mobile, Internet and whatever. So point being, I think why is Google winning?
- [35:08]
B
Why is Google going to win this? So thinking about even these generational gaps, even these generational components, right, Bringing that into this picture, why, why will Google win?
- [35:17]
C
So I think, I think they fundamentally win with the best product that the, the, I think they will end up with the best product for a combination of the things I was talking about, about adversarial information retrieval, right? They get spam detection, they get how to evaluate quality, authority, relevance, etc, etc, so they can train their data, they train their stuff better. They're vertically integrated, so they've got the, the tpu, they've got the hardware all the way through to their own model and they have their own search engine and everybody else is grounding in somebody else's search engine. And so I, that's another reason why I'm pretty bullish on them. That's why I think their stuff ends up best.
- [35:56]
B
Yes.
- [35:56]
C
In addition to that they have distribution, right?
- [35:59]
B
They still have will Can I, can I, can I ask you, can you talk more about grounding? Because I think this is such an underrated discussion, not just in our industry, but like, just every single industry that touches AI in one way, shape or form or needs discovery from AI, needs to understand grounding. And I'd love to get your perspective on what that means and then even bring it back to, like, how do you test for that and how do you, like, learn and, like, become better at understanding why grounding is important to your own brand, for sure.
- [36:26]
C
So I actually think grounding is crucial to why we can even test against these things in the first place. If we had to wait for, you know, the next model to see whether our changes had been beneficial, right? So you imagine that we made a change to our website and now we had to wait for GPT6 to see if it was any good, that would not work, right? That's too slow. The feedback loop would be disconnected and that whole, whole thing wouldn't work. So the reason that grounding is important is my understanding, and I'm not an ML engineer, but my understanding is the machine learning folks are. What they really want is they want intelligent models more than they want knowledgeable models, right? So in particular, I read a very compelling article about this being a big part of the gap between GPT4 and GPT5 is that the latest model is smarter, but actually knows fewer facts than GPT4 knew. And the reason for that is that it's actually very hard to tell which facts are mutable, which facts change over time. And you don't want to encode into the neurons who won the super bowl, because who won the super bowl changes every year. And you don't want to encode into the neurons what day is it, or who's the president, or who's the King of England or any of these kind of things, because those facts are things that change. And so you want the reasoning capabilities, you want the knowledge, you want the intelligence rather, and you want the smarts. But you'll find that it looks up the facts and we need to find facts broadly. Like, you know, is this sneaker in stock in my local footlocker? That's a fact that you don't want encoded in the training data, right? You need to look it up that might have changed since half an hour ago. You want to look that up in real time, whether it's via an API, whether it's via the website, however that might operate. And so grounding wants to be as up to date as possible and this is another reason I'm bullish on Google is you look at the evolution of everything Google's done over the last 25 years has all been about freshness, accuracy, relevance. And that freshness piece is way under discussed in my opinion. When we saw all of the Query deserves, query deserves freshness. I forget when that was. Mid 2000s, right, when you started seeing search results change. Regular 10 old 10 blue link search results change because a query was trending. I remember, I remember vividly. So this was Michael Jackson dying was the query that. I remember the Chicago Tribune writing about how they had leapt on this, right? They'd seen the news, it had come out in the LA Times or somewhere. And the Chicago Tribune very, very quickly got the page live, redirected a whole bunch of their topic pages to the same place and was ranking for the non news section, right? The old style link for a whole load of those like Michael Jackson queries. And I can't remember when this would have been. Would that have been 2008? Maybe, I don't know. Anyway, point being, it was a kind of watershed moment where everybody started to realize that the search results themselves not only aren't static, they shouldn't be static. Users are happier with dynamic search results that are up to the moment. And whether that's breaking news, whether it's in stock, whether it's price information, whether it's recalls or whatever it might be, any of this stuff, users want the most up to date, the most fresh information and it really drives a ton of stuff and it dry, it drove infrastructure improvements with Google. It was caffeine, it was, you know, the real time indexing. It was the enhancements to be able to recalculate page rank and those kind of metrics on the fly without having to do the whole big batch recalculation. And we're in the early days of this for the LLMs. They are still training a massive model all in one go, putting that out there as Gemini 3 or GPT 5 or whatever it might be. And one of my strongest predictions is that we're going to see more freshness and that will come from a combination of training these models as we go and better grounding better tools for them to go and say what is this fact? What is this latest information? What is this breaking news? What is this stock level or whatever it might be. And users are not going to tolerate out of date content and out of date information. And so let's. Yeah, I think that drives a lot of it.
- [41:10]
B
Let's talk more about this, I think, and you know, maybe this is a little off script, but I want to change one of the topics that we're going to talk about because I think it's really relevant to your perspective on where Google's going. But even more importantly, I fundamentally think that it's super relevant to the current media cycle that's happening across these LLMs. So this past week Google announced Universal Commerce Protocol. It's very much a piggyback off of ACP as well as the fact that Microsoft Copilot announced their, their Copilot Commerce I believe is what they coined it. So there's this, there's this growing need for protocols or ways of sharing that accuracy, that consistency, that freshness that you're talking about through these, these, these systems. And then, and then at the same time, you know, ChatGPT announces ChatGPT Health and these other verticalized environments where again to your point, accuracy really matters when you're asking a health related prompt. And so I want to get your view, especially working with a lot of commerce, working with a lot of travel, how these partnerships, protocols and verticalization are going to play out in 2026.
- [42:36]
C
Fascinating. Big topic. I think the historical grounding I would add to this is that some of this isn't new and obviously it's new that it's an agent doing it. But I remember, do you remember Google Checkout which was, I think, what do they call that before it's these days called Buy with Google or something? Many iterations of it over the years. It was when PayPal was on the rise and you had the, you know, you could buy with PayPal, you could buy with Apple, Amazon. There was checkout with Amazon button that used to show up various places and you know, it's finally working. I think with Apple Pay. Apple Pay is probably the, the canonical, best example of this. Kind of like bypass the checkout bit and have that seamlessly work and again back to understanding the stack. It obviously works best in Safari on a, on an Apple device, et cetera, et cetera. But Google Checkout was relatively unsuccessful, I think. And the bits that are, it's interesting to figure out what are the bits that are powerful, valuable and compelling. And I think that is not having to enter your checkout information, right? Nobody really wants to fill in shipping address information, your email, your phone, whether you, whether your browser auto completes that for you, or whether Apple Pay just sends it over or whatever it might be. Nobody wants to be typing into those forms. But what people do want to do is see lots of product information before they buy. And I think that was where the old Google checkout fell down was they used to have it in the AdWords, if you remember. So you could, you could do the search and you could check out from the AdWords ad itself. And basically nobody did because it turns out everybody wants to see the page, right? They want to see high quality product images, they want to see, they want to read the reviews, they want to look at the specifications. And it may or may not be that that holds true across the LLM innovation because maybe you trust that ChatGPT has read those things and is making that recommendation for you. But I think some of the style stuff doesn't go away. I mean it'd be interesting to see what like vibe style looks like, you know, hey chatgpt, dress me like a, you know, dress me like a banker or dress me like a skateboarder, I don't know. But I think people have those style opinions. So my suspicion, especially if we're talking 2026, right, so if we're literally saying this next, next 12 months, I predict websites are not as screwed as people think, right? Websites are going to remain important. Humans are still going to visit web pages and what we're going to see, where we're going to see that friction reduction is it's going to look more like Apple Pay when you finally decided that that is the one you want. Super quick, super easy, don't have to re enter all the information. That's where the machine to machine comes into play. I want the machine to reduce all the friction. I don't necessarily want it to pick which color I want when I'm buying.
- [45:52]
B
A pair of sneakers as it pertains to these, these, these connection points, right? Like I think that there is this, this hope or there's this belief that agentic commerce is going to be that frictionless, you know, reality. So in, in your AdWords scenario, you know, it's going to be a moment in time where you just purely do all of your shopping within ChatGPT, you know, and you voice prompt your way through this entire shopping journey of the greatest vacation you've ever had. And, and, and I think that that's, that's, that, that's the, that's the belief, that's the want. In order to do that really well you need these like data endpoints to have accurate pricing on flights and hotel prices and is what's the proximity to this beach and all of these key data points that allow you to really truly experience that and My main question here is does the user still want to see booking.com do they still care enough to want to be on that webpage? Do they still want to feel and touch and see that logo and go through the different, you know, room options in that hotel and read the aggregated reviews that all say great place. Johnny at the front desk was really nice. Does that still really matter?
- [47:14]
C
Somebody needs something is going to want to consume that information, whether it's the, whether it's the human or whether it's their agent. And I think that question is critical to the business model, isn't it? And I think probably the biggest concern, so to say we, most of our work is with the retailers who do maintain that customer relationship because they have to ship the stuff. The OTAs have a big challenge on their hands of how do they stay relevant in an agentic world. They're extremely important to the agent, right, because the agent wants that price comparison information, the agent wants those review information and so on and so forth. But they could easily get cut out of the middleman of like once it's used that information, scraped that information learn which is best. The booking could happen directly, potentially. So I think there's some business model stuff around. You know, do they have, do they have favored relationships? Do they have unique deals? Do they have, is, are they genuinely bringing some value to that beyond just being a ranking sort order or you know, a pure recommendation? And as I said earlier in media where there isn't even the transaction, right? So if you're talking about a, the wire cutter for example, New York Times making, saying, you know, this is the best pen you can buy, it's like, well, sure, thank you. I'm gonna go and buy that pen now. Am I going to click on your affiliate? Is my, is ChatGPT going to click on your affiliate link? Like that doesn't necessarily make sense. And so, but the thing that somebody in that process is going to have to figure out, as I say, this is not my game anymore, is if you lose the incentive, you lose the site. So if nobody clicks on Wirecutter's affiliate links, Wirecutter is going to stop doing that in depth product research. And if they stop doing that in depth product research, where does the data come from to feed GPT 6, 7, 8, 9, 10. And there's going to have to be some sustainability around that whole ecosystem because if take it to the extreme, there's no journalist left, then well there's, there's no content being written to even. You don't even get the news to aggregate up into all these other things.
- [49:31]
B
So if you don't have local and regional journalism and we're already seeing that great national. Yeah, yeah, it's already been gutted entirely.
- [49:40]
C
But, but I think the danger is that there were all these areas that were sustained by Google traffic, whether it's, you know, whether it's recipes, whether it's blogging, whether it's any kind of media that you can think of. And if that incentive goes away and what happens to local news, happens to.
- [50:04]
B
Those kinds of online information and content and insight.
- [50:08]
C
Yeah. Then you lose that training data to be able to make the next generation of these things. And some of that is solved by the same technology. Right. Because some of those things, maybe we're going to be having AI generated recipes, but the AI can't taste stuff yet. And so there's again back to grounding. There's no grounding in actual human reality of like well, but was that a good soup recipe that you just gave me or was that like entirely hallucinated? And yeah, there's a lot of tough engineering business model strategy questions and I think the, those middle. Middlemen sites, right, the, the, the OTAs and the affiliates have some of the hardest questions to answer, like what, what does their business model look like in a year, two years, five years? And yeah, definitely excited to see what innovations they come up with because I don't think some of these are huge businesses. Right. These are, these are multibillion dollar businesses in many cases. Yeah.
- [51:06]
B
Publicly traded companies.
- [51:07]
C
Yeah, they're not giving up on that stuff. Right. They're building a little bit of their own agent technology. They're doing B2B deals with the LLMs, with Google and I think we're just, I think innovation is good. We're going to see some, some great consumers are going to benefit from this ultimately because of the creative innovation that comes from it all.
- [51:32]
B
I want to spend some time in our last topic here talking about that innovation and specifically what you and the team at Search Pilot are doing. How are you evolving testing and what's new for you all? When it comes to looking at the geo world that we're now in, I.
- [51:50]
C
Said the main thing we're focusing on is this multimetrics kind of idea that the data needs to come from more places and all be segmented in new ways. And so we're saying, yeah, let's look at analytics data, but let's also segment the analytics data to look at the performance in the different channels. I actually have a, this is a bit speaking out of line. We haven't actually released this yet, but one of the things I've been thinking about is dark traffic. You know the. When you think about traffic that lands deep in your website. So I'm not talking about homepage here, I'm talking about specific product pages for example, but it lands with no referrer and we think of that as direct, right? That gets called direct in every analytics platform I've ever seen. But it's not direct. It's not. People are not typing in 150 character long URL.
- [52:39]
B
What that is with question mark equals and a sands clicked on a link somewhere.
- [52:44]
C
But where have they clicked on a link? They clicked on a Link in Slack, WhatsApp or ChatGPT desktop app or mobile app or these are still referrals, we just don't know where they're referrals from. And I am increasingly coming to the opinion that we should be grouping that dark, deep, dark traffic as I call it, right? So not dark traffic to the homepage because people do type in, you know, Amazon.com or whatever. I'm not talking about homepage traffic here, I'm talking about deep product page traffic. That deep dark traffic. We should count that as referral and a discovery. And that is stuff that increasingly is not going to be from humans, it's going to increasingly be from tools. Right? So this is going to be an LLM recommended that you click on this link, but they recommended it in an app, not in a browser. And so we need to, we need to see that as a discovery link. And so I think we're going to be doing more testing against dark traffic as well. And anyway, so there's a whole data play. I think that there's, I'm excited that I think the future will hold some good things in terms of getting more data in search console, getting new the equivalent of search console for ChatGPT and Claude and the like, particularly as some of them start to introduce advertising. The advertisers will need data so we'll get better data out probably. And yeah, so there's a whole host of stuff that we're doing around those kinds of things. We're evolving the hypotheses so that what do we test? We're having more and more hypotheses be LLM specific hypotheses. So we think if we add this extra information it will give the context needed to the LLM agent to be able to recommend our products more often, for example. So it's data, it's Hypotheses and then it's technology. And I'm excited about all the ways that AI can help us build faster, build better things, integrate. It's amazing that computers could do things today that there's that famous XKCD comic about identifying birds that was like at the time it was made was, computers can't do that. And of course now they can. And I'm just excited both to build faster and to build things that we previously wouldn't have been able to.
- [55:11]
B
Last question about kind of where Search Pilot's going and just in general, like advice for our listeners, like for many of our listeners who aren't at a big enterprise. Right. And can't necessarily integrate a testing stack. What kind of advice do you have for them and where do you encourage them to get started when it comes to experimenting, testing within their own data set?
- [55:31]
C
It's one of the areas we try and pay it forward a little with publishing our test results. And so what I often encourage small orgs to do is not try and do their own tests, but piggyback off information that other people have been able to test on much larger websites. Now this is not foolproof. You know, what works for Amazon doesn't work for everyone and but it gives you at least, at least gives you ideas and helps you prioritize your backlog. I actually, I'm in this boat for my own business. Right. Search Pilot is not in searchpilot's own target market. Right. We can't. We're too small a startup to run to run the kind of testing that we offer our enterprise customers on our own site. So what I try and do is learn from case studies to stay grounded in the user stories. So ask ourselves, what are we trying to service here? What are we trying to achieve for whom? Stay focused. So a big part of what we've been trying to do over the last year to 18 months in particular is really focused. And this is not something that I'm naturally good at. I'm not a very naturally focused person. But as an organization, we're trying to be super focused on SEO testing, GEO testing for large enterprise retailers in particular. And you know, you've heard me repeat that a lot. This is part of. I'm embedding in my own brain as much as anything else.
- [56:53]
B
You're welcome.
- [56:53]
C
So. So I think, yeah, for folks in small orgs, focus, think about the user and read those case studies and those testimonials and those test results from places that can test. You probably can't run the test yourself and that that's okay. And you just have to accept that you're following your nose and you've got a little bit more but, but you can informat with, with data. And so we try and publish those things because I do want to have an answer for smaller retailers or smaller sites and it can't just be sorry, you know, you can't afford Azor tech doesn't work on your, your scale and so that's why we publish so much data. We've got a couple of the free tools coming soon as well, so people keep an eye out or sign up to our email list. We'll be publicizing a couple of free tools, including a forecaster that is designed to help forecast search traffic and some small sites can get involved with that as well.
- [57:45]
B
Okay, well, now let's move into our lightning round. I'm going to ask you some search geo related questions from the topics we discussed this week. Are you ready?
- [57:55]
C
Let's do it. Awesome.
- [57:57]
B
What's the biggest SEO best practice that you've debunked through testing, but our industry is just still clinging onto it?
- [58:06]
C
I mean, I hate it because I, I like the concept and the technology, but you alt text, like we've never seen a successful alt text test. It doesn't seem to matter for SEO. Use it for accessibility, still do it. You know, people need it, robots probably need it in the future, but it's not helping your rankings. Sorry. I love it. Love it.
- [58:30]
B
All right, next question here. You know, you've been in this industry for two decades now and you've seen a lot of things that have changed. What's the most shocking thing that you've seen as it pertains to LLM and ranking content?
- [58:49]
C
I'm going to actually just say that the most shocking thing is that it works at all. I have, my background, my academic background was in math and statistics and I studied neural networks before they were cool when they didn't work famously. Yeah, they famously were a dead end. And then it was only when you threw like billions and billions and billions of Data points and CPUs and GPUs and all this other stuff that suddenly, suddenly gradient descent worked. So I'm just mind blown that we're, we're getting anything that resembles intelligence out of, out of this nonsense.
- [59:29]
B
And do you think that these maybe more, more eloquently said, these LLM responses, not necessarily rankings, do you think that these LLM responses are going to become more opinionated, more broad, sweeping, or are they going to become more and more narrow, focused and more and more specific in their nature over time.
- [59:52]
C
I'm going to take a specific version of that question which is around product when you're looking for a thing to buy. Because that's the space that we're playing so much, I think in that specific niche. Actually we're going to see them look more like search results. I think we're going to see more links, I think we're going to see more, you know, read more about. And those links may be to sub conversations or, you know, expanded information. But I think that as humans when we're researching particularly big ticket purchases or things that are important to us, we're going to want that information ourselves. And what these things are going to provide us is a fast route to those, to that information and summarizing and so forth rather than, rather than abstracting that all the way and saying, hey, I bought you one, love it.
- [60:43]
B
You know, you've made a big bet on yourself launching Search Pilot. You've been in the consulting space and in other parts of the search, search marketing, now, AI discovery industry. Why are you betting on testing and why is testing so important right now?
- [61:01]
C
So we started building what became SearchPilot because of our consulting work. Actually the origin story is that we by that point had a decade of experience in the industry and had worked with a lot of these large organizations and we just seen that we found the same problems over and over again. We found the challenges getting things done, challenges proving the value of the work that they'd done, challenges even understanding whether there had been value in the work that they'd done. We saw this in our consulting work, we saw it in their own work, their in house work as well. And that was the problem that we set out to solve. And since there we've kind of followed our nose a little bit to kind of say that's, that's what we're trying to do. Do we feel like we're achieving that? And particularly of this last two or three years, it really feels like we're making some breakthroughs. And this is why we've honed in on where we specialize because that's where we're finding that we're adding the most value. And I'm very excited that the growth of all of these new discovery channels that we've been talking about today is bringing this topic to the conversation, into the boardroom, into the C suite, into the executive layer in a way that it hasn't been really for a long time. And that's proving good for us. And so, yeah, I think it's important because the more it changes, the less you can rely on yesterday's best practices, the more it's a black box and the more that these algorithms are unknown and unknowable. The only way to operate, in my opinion, is to test. And you've got to basically stop guessing because, yeah, it's not a case of a set of ranking factors that we could speculate about like we could in 2009.
- [62:42]
B
Yeah, absolutely. You're betting big on Google. What is the biggest threat to Google's ability to win in AI discovery?
- [62:54]
C
I think it's the kids. It's the thing that you were alluding to. If they lose the kids, they lose the taste makers and it becomes the thing that people do to do something else, then it's hard to come back from that. I think there's some more tactical stuff. We've seen them do a deal with Apple this week. I think that's huge for them. Both sides. I love when two monopolies both get to exploit each other at once. It's not actually clear to me who's exploiting whom and in fact, which one should be paying which one. I love that. Had they not done some of those deals, there would be some threats in that kind of direction. But I think the big macro threat is just what if the tide.
- [63:41]
B
Yeah, I love it. I absolutely love it. Last question. In 2026, what is the one testing aspect that you are betting on big?
- [63:51]
C
It's no surprise. It's the thing I've mentioned before, but it's the multi metrics. I think that we are betting that by optimizing our tests to be able to be measured against different things, that will enable people to get more insights faster, with more confidence, build better tests, make smarter decisions, and that then pervades everything. So we're, we're doing, we're designing the tests with those multimetrics in mind. We're ingesting metrics from many, many different sources. And the, the exciting stuff that's just, just starting to come out in our product right now is that it also enables things like segmented analysis, which is where imagine you've got like a major retailer that sells many different product lines and you run a test that is, that is positive across the board, but you want to be able to report on it and say, how good was it in home furnishings, how good was it in fashion, how good was it in whatever else it might be? Kids, kids, clothes. And we can do that as well because of the same technology that powers the multimetrics.
- [64:53]
B
Love it. And that wraps up this episode of the Voice of the Search podcast. Thank you to Will Critchlow, CEO at SearchPilot, for joining us. If you'd like to contact Will, you can find a link to his LinkedIn profile in our show notes or on the voicesofsearch.com you can visit his company website, searchpilot.com if you haven't subscribed yet and want a daily stream of SEO and content marketing knowledge in your podcast feed, hit the subscribe button in your podcast app or on YouTube and we'll be back in your feed every week.
- [65:22]
C
Week.
- [65:23]
B
Okay, that's all for today, but until next time, remember, the answers are always in the data.