Podcast Summary: SearchPilot’s Biggest Launch Yet: The Future of SEO Testing
Podcast: Voices of Search
Host: Jordan Cooney
Guest: Will Critchlow, CEO at SearchPilot
Date: February 2, 2026
Episode Overview
This episode dives into the evolving landscape of Search Engine Optimization (SEO), focusing on how AI-powered search, Large Language Models (LLMs), and the rise of new discovery engines are reshaping testing and performance measurement. Jordan Cooney hosts Will Critchlow, CEO of SearchPilot, to discuss the company’s new geo testing system for AI search, the changing metrics of SEO, and how enterprise sites are adapting in a world where over half of searches end without a website click.
Key Discussion Points & Insights
The Clickless Search Era and AI’s Impact on SEO
- The Click Decline: Over half of US searches (56%) now end without a click, as users get answers directly in AI-powered summaries (00:44).
- Challenge for SEO: Traditional metrics and measurements are becoming less meaningful. Marketers must redefine what performance means (01:00).
- Geo Testing for AI Search: SearchPilot introduces the first purpose-built geo testing system for AI-powered search, enabling enterprise-level testing beyond Google’s traditional blue links (01:34).
What Has Changed in SEO Testing
- Core Similarities:
- Testing still revolves around website changes, especially on scalable sections like product/category pages (02:21).
- The process—hypothesis, application to pages vs. controls, and analytics-based measurement—remains largely the same.
- Key Differences:
- Much more of the “searcher’s journey” happens within the search interface or LLM chat window—less on owned websites (03:15).
- Now, SEOs must consider if AI “robots” find and recommend their content in answers, not just if they rank (05:00).
- Fan Out Query Explained: LLMs emulate user comparison behavior by querying multiple sources before recommending or summarizing, making testing analogous to a blend between SEO and conversion rate optimization but for “robot conversions” (05:00).
“A GEO test is a blend of an SEO and a conversion rate test... but the conversion rate is the robot conversion.”
— Will Critchlow (05:00)
Keeping the User at the Center
- User-Centric Hypotheses: Successful testing still hinges on delivering user value. Will highlights the need to ground hypotheses and improvements in user benefit (07:34).
- Continuous Insight: Focus is on extracting insights, not just winning tests—each result should inform and iterate on product/content improvements (07:52).
“We need to ground our hypotheses in how it benefits the user ultimately... that is the kind of big picture societal level good of the work that we do as SEOs.”
— Will Critchlow (07:52)
The Rise of Multi-Metric Testing
- More Fragmented Discovery: With multiple discovery engines (Google, ChatGPT, Perplexity, etc.), the ecosystem is diversifying (11:09).
- Multi-Metrics Era: Businesses are now interested in multiple metrics—search clicks, merchant center impressions, LLM-based metrics, and more (11:25).
- Net Impact Focus: Aggregate performance matters most; optimizing for one channel (e.g., ChatGPT) in isolation is less relevant (13:09).
“The biggest shift for us is that move to multimetrics. It’s the fact that people genuinely care about more than one thing now.” — Will Critchlow (12:30)
The State and Future of LLM Traffic
- LLM Traffic is Small but Growing: Single-digit percentage of overall sessions, very vertical-dependent (14:40), with outsized impact in healthcare, finance, etc. (19:03).
- Clicks Behind the Curtain: LLMs act as windows “behind the click” by providing direct answers, which changes how value is ascribed to clicks and traffic (19:32).
“LLMs... are like a window behind the click. We used to have to do a lot of clicking. Now we just have to do less clicking because the LLM gives us that window.”
— Jordan Cooney (19:32)
- Selection Bias: Don’t equate LLM clicks directly with SEO clicks; users are different and behaviors diverge (20:34).
- Need for Real Value Metrics: Orgs must prioritize end business value over legacy traffic metrics as AI makes traffic less direct and more interpretive (21:30).
Authority & Trustworthiness in the LLM Era
- Authority Remains Key: Testing for trust signals—expert content, reviews, delivery stats, etc.—remains central (23:25).
- LLMs Still Susceptible: LLMs may naively ingest content, leading to abuse (e.g., brands putting themselves as “#1”). Google’s adversarial experience gives it an edge in spam detection and authority measurement (24:14; 26:31).
“I’m surprised that the anthropic and OpenAI folks... don’t seem to have realized that they’re walking into an adversarial environment.”
— Will Critchlow (27:04)
Why Google Is Likely to Win AI-Powered Search
- Institutional Advantages: Advanced in spam detection, vertical integration, freshness of results, distribution, and user trust (35:08).
- Grounding: Google’s search index grounds LLM responses, keeping them up to date—crucial for accuracy and for rapid testing (36:26).
“You want the reasoning capabilities... but you’ll find that it looks up the facts. And we need to find facts broadly.”
— Will Critchlow (37:07)
- Generational Shifts: Google’s staying power may depend on retaining younger users as the primary discovery tool (62:54).
The Importance of Freshness & Protocols
- Freshness Underrated: Dynamic, up-to-the-moment information is increasingly crucial for search and LLMs; this links to Google’s Caffeine update and real-time indexing (39:00).
- Emerging Commerce Protocols: Google’s Universal Commerce Protocol, Microsoft’s Copilot Commerce, and ChatGPT Health all reflect the need for accurate, standardized, up-to-date data across LLMs (41:10).
Agentic Commerce, Media, and Business Models
- Agentic Shopping Uncertainties: While the vision is frictionless commerce via agents (voice, LLMs, etc.), users still want to see product details and verify information themselves (45:52).
- Role of Aggregators and Affiliates: OTAs and companies like Wirecutter face existential questions if LLMs bypass them or fail to drive traffic/affiliate revenue (47:13).
- Sustainability Concerns: If incentives to create content disappear, so does the training data and ecosystem for LLMs—raising long-term sustainability issues (49:40; 50:08).
Innovations at SearchPilot
- Multi-Metric & Channel Segmentation: SearchPilot is building analytics to segment performance by channel and to handle “dark” traffic (unattributed deep links which may originate in LLMs or chat apps) (51:49; 52:38).
- Testing for LLM-Specific Hypotheses: Actively running more tests designed to optimize for LLM recommendations, not just traditional SEO (54:30).
- Advice for Smaller Sites: Lean on published case studies and test results from enterprises; focus on user intent, read available data, and specialize (55:31).
“What I often encourage small orgs to do is not try and do their own tests, but piggyback off information that other people have been able to test on much larger websites.”
— Will Critchlow (55:31)
Lightning Round – Notable Takeaways
- SEO Best Practice Debunked: Alt text has shown no measurable SEO value in testing; use for accessibility but not for search rankings (58:05).
- Most Shocking LLM Finding: The fact that LLMs work at all—“mind blown” at their emergent intelligence from statistical models (58:49).
- LLM Response Direction: For products, LLM outputs will resemble search results with more links and deeper research, not simply direct recommendations (59:51).
- The Case for Testing: Rapid change means best practices will not be enough—testing is the only way to navigate black-box algorithms (61:00).
- Threat to Google: If Google loses relevance for the next generation (“the kids”), it risks losing the search market, despite technical and distribution strengths (62:54).
- 2026 Bet: Multi-metrics testing—measuring tests against numerous metrics and channels for greater insight and business alignment (63:51).
Memorable Quotes
-
On AI-powered SEO Testing:
“We’re almost testing against the user journey as a whole rather than every click. It’s a little analogous to opening multiple tabs, researching, and circling back to where you’ll buy.”
— Will Critchlow (03:15) -
On Grounding and Freshness:
“Freshness is way under-discussed in my opinion... Users aren’t going to tolerate out-of-date content.”
— Will Critchlow (39:00) -
On Multi-Metric Evolution:
“What you should really care about... is how does it affect the overall performance?... in aggregate, across all those things, what’s the net impact?”
— Will Critchlow (13:09) -
On Testing vs. Guesswork:
“The more it changes, the less you can rely on yesterday’s best practices... the only way to operate, in my opinion, is to test.”
— Will Critchlow (62:10)
Important Timestamps
- 00:44: The rise of clickless searches, intro of Will & SearchPilot’s geo testing
- 02:21: What has and hasn’t changed in SEO testing
- 05:00: Fan-out query concept and LLM journey
- 11:09: The shift to a multi-metrics world
- 14:40 – 20:34: State, significance, and context of LLM traffic
- 23:25 – 28:39: Authority, trust, adversarial SEO, and LLM susceptibility
- 35:08 – 41:10: Why Google is positioned to win, the technical nuance of “grounding,” and LLM freshness
- 41:10 – 47:13: Commerce protocols, agentic shopping, and the future of product/brand presence
- 49:40 – 51:06: The sustainability of the content ecosystem in an LLM-driven world
- 51:49 – 54:30: New directions in test segmentation, dark traffic, and LLM-specific testing
- 55:31 – 56:52: Advice for small teams and leveraging big-customer learnings
- 58:05 – 64:52: Lightning round insights on SEO myths, the surprise of LLMs’ effectiveness, and predictions/bets for 2026
Conclusion
The SEO game is rapidly evolving: with AI-driven, clickless search experiences, measuring and optimizing performance has never been more nuanced. Testing is the anchor in an uncertain landscape, and SearchPilot is expanding its toolkit to enable enterprise and, eventually, smaller teams to keep up by focusing on user value, authority, and the overall performance across a growing array of digital platforms. The future is multi-metric, user-centered, and adaptive.
Contact & Resources: