Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

In this episode, Alessio and Swix delve into the seismic shifts happening at the intersection of AI and internet search. Their guest, Robert from Scrunch.ai, brings a pragmatic and technical perspective to how Large Language Models (LLMs) and AI search agents are revolutionizing discovery, SEO, and the broader digital customer journey. The discussion covers the origin and mission of Scrunch, the transformation in how people interact with the web, evolving strategies for content creators and brands, AI’s encroachment on traditional web browsing, technical insights into AI search ranking, and practical tips for businesses adapting to the new landscape.

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

Background: Robert and his co-founder, both ex-Hearsay execs, initially noticed demand among enterprises for tools to help manage their AI presence, especially as LLMs were threading into mainstream consumer interactions.

"The elephant in the room was AI...if you started a company that wasn’t in some way LLM-native, you’d already be behind." — Robert [01:29]
Initial Observations:
- Most companies wanted on-site chatbots, but Robert found this unappealing, believing most users don’t want to interact with site-specific chatbots.
- The key realization: customers prefer staying within their chosen AI environments (ChatGPT, Claude, etc.), instead of interacting with per-site solutions.
- The insight was less about SEO or performance marketing, and more about a profound change in how people discover, research, and interact online.
  
  "The consumer experience of the internet is changing really, really quickly for the first time in quite a long time." — Robert [05:00]

2. Monitoring the Shifting AI Search Stack

How Scrunch.ai Works:
- Simulates consumer interactions—regularly prompts major AI platforms to monitor how brands and products are surfaced.
- Focuses on platforms like ChatGPT, Gemini, Perplexity, Claude, and Meta AI due to their broad consumer reach.
- Emphasizes tracking what "normies" (mainstream users) are actually seeing and doing.
  
  "Our ambition is to be wherever the consumer is, wherever our customers' audiences are." — Robert [06:52]
Search vs. Pre-training:
- It’s much easier for brands to influence AI search (using live web results) than what’s embedded in an LLM’s pre-training corpus.
- Brands should consider what they expose to web crawlers, especially regarding transient information like sales or promotions.
  
  "Should [a seasonal sale] go into the trained knowledge of a model? I would argue it’s probably counterproductive." — Robert [08:36]

3. The Future of Content Access & Paywalls

Cloudflare & AI Bot Paywalls:
- Cloudflare’s move to paywall AI bots is seen as a partial technical solution, not a panacea.
- The power dynamics remain: websites want protection, but also crave traffic—especially as AI-driven channels become a greater referral source.
  
  "There’s a fine line...unless you’ve really got a strong native organic audience, you need to be careful about how restrictive you are." — Robert [10:40]
AI as Web Browsing Replacement:
- Rather than simply replacing Google as the search gateway, Robert forecasts AI tools replacing much of traditional web browsing itself.
  
  "Thinking about what’s going on in AI that’s really replacing search is the wrong take...AI is replacing web browsing." — Robert [13:36]

4. AI Search Ranking: How It Works & How to Win

Underlying Mechanics:
- Despite the AI wrapper, most ranking is still handled by traditional search algorithms (BM25, TF-IDF, PageRank) before LLMs summarize and answer.
- AI then re-ranks for relevance before content is consumed.
  
  “There is still a search ranking...traditional SEO techniques do still work.” — Robert [16:19]
Optimization Tips:
- Purely human-optimized, clickbaity metadata may harm AI curation—LLMs prefer descriptive, factual metadata and content.
- Clear writing and context about who a product is for and what it does, is crucial.
  
  “…just being a little bit clearer about like here’s why, like here’s who buys these type of leggings and for what can make a big difference in getting surfaced in ChatGPT.” — Robert [18:42]
What Doesn't Work:
- Over-focusing on embeddings for content similarity doesn't help most AI search interactions.
- LLM-specific files (like llms.txt) are currently ineffective for discovery by AI search bots.
  
  "Embeddings aren’t that relevant. You’re better off focusing on…good structured content that makes sense to a human with clean HTML." — Robert [39:29]
Technical Gotcha:
- Most AI crawlers do not execute JavaScript. If your page’s content is client-side rendered, it may be invisible to AI.
"If your content is not rendered on the server side, it’s typically not going to be available." — Robert [40:26]

5. Privacy and Real-Time Personalization

Personalization:
- ChatGPT and similar tools do personalize results based on user settings, memories, and explicit preferences.
- However, referral data (the prompt used to find content) isn’t passed downstream to protect user privacy.
  
  "I don’t think…search is going to pass the prompt the user is using to the website." — Robert [20:55]

6. New Games: Black/Grey-Hat SEO and Generative Optimization

Prompt Injection:
- Prompt injection can manipulate AI outputs today but is not a durable or recommended long-term strategy.
  
  “It works until it stops working...as things get more sophisticated, black hat stuff tends to get penalized.” — Robert [36:19]
Programmatic SEO:
- AI-generated content for SEO is widespread. When done thoughtfully (e.g., turning real customer support tickets into how-tos), it can provide value to users and LLMs alike.
"If it’s bringing some kind of insight that actually is particular to you...that content is like the almost the exact ideal content you could give to an LLM." — Robert [43:53]

7. Case Studies and Tactical Outcomes

Successful Examples:
- Clerk (authentication SaaS) improved both traffic and conversions dramatically by optimizing content for AI agents (6x growth in ChatGPT-driven traffic, 9x in conversions).
  
  "They’ve seen like a 6x growth in ChatGPT traffic...but they’ve seen a 9x lift in conversions." — Robert [48:10]
Feedback and Experimentation:
- Scrunch.ai acts as a feedback loop: helping clients understand what’s being surfaced by AI and what audiences are looking for, enabling rapid iteration—not as a content generator itself.
Uplift Ranges:
- “Double digit improvements in traffic within a month or two...if actively improving content.” — Robert [50:49]

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Market Share:
- ChatGPT is the clear category leader for consumer AI-driven search and interaction; AI Overviews from Google and Meta AI have massive reach due to native distribution.
- Perplexity is a strong #2 among AI-native search tools, especially among technically savvy users.
  
  “Nobody has this sort of durable relationship with tons of people like ChatGPT does in this space so far...Perplexity...is really, really strong.” — Robert [53:15]

Notable Quotes & Memorable Moments

On website chatbots:
"Nobody wants to use a chatbot on your website…I fly into a visceral rage." — Robert [02:26]
On AI as a transformative force:
"AI is replacing web browsing. I think that’s actually a more meaningful, more fundamental shift." — Robert [13:36]
On SEO game evolution:
"We’re actually in one of these rare windows of time where it’s positive sum all the way around." — Robert [36:39]
On practical advice:
"Clear writing matters a ton...Most of the AI search indexers and retrievers don’t execute JavaScript." — Robert [40:26]
On prompt injection:
"If you want to prompt inject ChatGPT through search, it’s definitely achievable...these systems are like 100th of the sophistication they’ll eventually be." — Robert [37:45]
On programmatic SEO:
"If it’s bringing some kind of insight…making it easier to consume for AI…that content is like the almost the exact ideal content you could give to an LLM." — Robert [43:53]

Timestamps for Key Segments

[01:29] — Scrunch.ai’s origin story and recognizing AI’s impact on discovery
[06:52] — Scrunch.ai’s monitoring strategy: what LLMs & platforms really matter
[08:26] — The challenges (and limits) of influencing LLM pretraining via your content
[09:54] — Cloudflare's AI bot paywall and the evolving content-power struggle
[13:36] — “AI is replacing web browsing:” the real shift in user behavior
[16:07] — How AI search ranking works; where traditional SEO still matters
[20:55] — Personalized search, privacy, and why you can’t dynamically rewrite content per AI query
[24:04] — The coming “AI SEO”/GEO meta-game and why it’s still positive sum
[36:10] — Prompt injection & grey-hat tactics: effective (for now), but risky and temporary
[39:29] — What doesn’t work: embeddings, LLMs.txt, and over-engineering for AI
[40:26] — Server-side rendering is essential: “AI doesn’t execute JavaScript”
[43:53] — Programmatic SEO using real user data: when it’s high-value, not spam
[48:10] — Case study: Clerk, meaningful uplift from thoughtful AI optimization
[51:53] — Platform rankings: ChatGPT dominance, Perplexity’s passionate user base
[55:10] — Final thoughts: be AI-compatible, avoid snake oil, and Scrunch is hiring

Final Takeaways

AI-powered search and discovery are fundamentally rewriting how content needs to be structured, optimized, and measured online.
Clear, descriptive, server-side content wins; gaming the system is temporary; meaningful, user-centered programmatic content can be very effective.
ChatGPT leads the way as the default AI interface for mainstream users, with Perplexity as a notable runner-up among power users.
For brands and developers: This is a once-in-a-generation shift—those who invest in understanding and adapting to AI search now will have an edge as both tactics and platforms continue to rapidly evolve.

Resources & Show Notes:
For more, visit https://latent.space

Interested in shaping the future of AI compatibility for the web? Scrunch.ai is hiring!

Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

Background: Robert and his co-founder, both ex-Hearsay execs, initially noticed demand among enterprises for tools to help manage their AI presence, especially as LLMs were threading into mainstream consumer interactions.

"The elephant in the room was AI...if you started a company that wasn’t in some way LLM-native, you’d already be behind." — Robert [01:29]
Initial Observations:
- Most companies wanted on-site chatbots, but Robert found this unappealing, believing most users don’t want to interact with site-specific chatbots.
- The key realization: customers prefer staying within their chosen AI environments (ChatGPT, Claude, etc.), instead of interacting with per-site solutions.
- The insight was less about SEO or performance marketing, and more about a profound change in how people discover, research, and interact online.
  
  "The consumer experience of the internet is changing really, really quickly for the first time in quite a long time." — Robert [05:00]

2. Monitoring the Shifting AI Search Stack

How Scrunch.ai Works:
- Simulates consumer interactions—regularly prompts major AI platforms to monitor how brands and products are surfaced.
- Focuses on platforms like ChatGPT, Gemini, Perplexity, Claude, and Meta AI due to their broad consumer reach.
- Emphasizes tracking what "normies" (mainstream users) are actually seeing and doing.
  
  "Our ambition is to be wherever the consumer is, wherever our customers' audiences are." — Robert [06:52]
Search vs. Pre-training:
- It’s much easier for brands to influence AI search (using live web results) than what’s embedded in an LLM’s pre-training corpus.
- Brands should consider what they expose to web crawlers, especially regarding transient information like sales or promotions.
  
  "Should [a seasonal sale] go into the trained knowledge of a model? I would argue it’s probably counterproductive." — Robert [08:36]

3. The Future of Content Access & Paywalls

Cloudflare & AI Bot Paywalls:
- Cloudflare’s move to paywall AI bots is seen as a partial technical solution, not a panacea.
- The power dynamics remain: websites want protection, but also crave traffic—especially as AI-driven channels become a greater referral source.
  
  "There’s a fine line...unless you’ve really got a strong native organic audience, you need to be careful about how restrictive you are." — Robert [10:40]
AI as Web Browsing Replacement:
- Rather than simply replacing Google as the search gateway, Robert forecasts AI tools replacing much of traditional web browsing itself.
  
  "Thinking about what’s going on in AI that’s really replacing search is the wrong take...AI is replacing web browsing." — Robert [13:36]

4. AI Search Ranking: How It Works & How to Win

Underlying Mechanics:
- Despite the AI wrapper, most ranking is still handled by traditional search algorithms (BM25, TF-IDF, PageRank) before LLMs summarize and answer.
- AI then re-ranks for relevance before content is consumed.
  
  “There is still a search ranking...traditional SEO techniques do still work.” — Robert [16:19]
Optimization Tips:
- Purely human-optimized, clickbaity metadata may harm AI curation—LLMs prefer descriptive, factual metadata and content.
- Clear writing and context about who a product is for and what it does, is crucial.
  
  “…just being a little bit clearer about like here’s why, like here’s who buys these type of leggings and for what can make a big difference in getting surfaced in ChatGPT.” — Robert [18:42]
What Doesn't Work:
- Over-focusing on embeddings for content similarity doesn't help most AI search interactions.
- LLM-specific files (like llms.txt) are currently ineffective for discovery by AI search bots.
  
  "Embeddings aren’t that relevant. You’re better off focusing on…good structured content that makes sense to a human with clean HTML." — Robert [39:29]
Technical Gotcha:
- Most AI crawlers do not execute JavaScript. If your page’s content is client-side rendered, it may be invisible to AI.
"If your content is not rendered on the server side, it’s typically not going to be available." — Robert [40:26]

5. Privacy and Real-Time Personalization

Personalization:
- ChatGPT and similar tools do personalize results based on user settings, memories, and explicit preferences.
- However, referral data (the prompt used to find content) isn’t passed downstream to protect user privacy.
  
  "I don’t think…search is going to pass the prompt the user is using to the website." — Robert [20:55]

6. New Games: Black/Grey-Hat SEO and Generative Optimization

Prompt Injection:
- Prompt injection can manipulate AI outputs today but is not a durable or recommended long-term strategy.
  
  “It works until it stops working...as things get more sophisticated, black hat stuff tends to get penalized.” — Robert [36:19]
Programmatic SEO:
- AI-generated content for SEO is widespread. When done thoughtfully (e.g., turning real customer support tickets into how-tos), it can provide value to users and LLMs alike.
"If it’s bringing some kind of insight that actually is particular to you...that content is like the almost the exact ideal content you could give to an LLM." — Robert [43:53]

7. Case Studies and Tactical Outcomes

Successful Examples:
- Clerk (authentication SaaS) improved both traffic and conversions dramatically by optimizing content for AI agents (6x growth in ChatGPT-driven traffic, 9x in conversions).
  
  "They’ve seen like a 6x growth in ChatGPT traffic...but they’ve seen a 9x lift in conversions." — Robert [48:10]
Feedback and Experimentation:
- Scrunch.ai acts as a feedback loop: helping clients understand what’s being surfaced by AI and what audiences are looking for, enabling rapid iteration—not as a content generator itself.
Uplift Ranges:
- “Double digit improvements in traffic within a month or two...if actively improving content.” — Robert [50:49]

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Market Share:
- ChatGPT is the clear category leader for consumer AI-driven search and interaction; AI Overviews from Google and Meta AI have massive reach due to native distribution.
- Perplexity is a strong #2 among AI-native search tools, especially among technically savvy users.
  
  “Nobody has this sort of durable relationship with tons of people like ChatGPT does in this space so far...Perplexity...is really, really strong.” — Robert [53:15]

Notable Quotes & Memorable Moments

On website chatbots:
"Nobody wants to use a chatbot on your website…I fly into a visceral rage." — Robert [02:26]
On AI as a transformative force:
"AI is replacing web browsing. I think that’s actually a more meaningful, more fundamental shift." — Robert [13:36]
On SEO game evolution:
"We’re actually in one of these rare windows of time where it’s positive sum all the way around." — Robert [36:39]
On practical advice:
"Clear writing matters a ton...Most of the AI search indexers and retrievers don’t execute JavaScript." — Robert [40:26]
On prompt injection:
"If you want to prompt inject ChatGPT through search, it’s definitely achievable...these systems are like 100th of the sophistication they’ll eventually be." — Robert [37:45]
On programmatic SEO:
"If it’s bringing some kind of insight…making it easier to consume for AI…that content is like the almost the exact ideal content you could give to an LLM." — Robert [43:53]

Timestamps for Key Segments

[01:29] — Scrunch.ai’s origin story and recognizing AI’s impact on discovery
[06:52] — Scrunch.ai’s monitoring strategy: what LLMs & platforms really matter
[08:26] — The challenges (and limits) of influencing LLM pretraining via your content
[09:54] — Cloudflare's AI bot paywall and the evolving content-power struggle
[13:36] — “AI is replacing web browsing:” the real shift in user behavior
[16:07] — How AI search ranking works; where traditional SEO still matters
[20:55] — Personalized search, privacy, and why you can’t dynamically rewrite content per AI query
[24:04] — The coming “AI SEO”/GEO meta-game and why it’s still positive sum
[36:10] — Prompt injection & grey-hat tactics: effective (for now), but risky and temporary
[39:29] — What doesn’t work: embeddings, LLMs.txt, and over-engineering for AI
[40:26] — Server-side rendering is essential: “AI doesn’t execute JavaScript”
[43:53] — Programmatic SEO using real user data: when it’s high-value, not spam
[48:10] — Case study: Clerk, meaningful uplift from thoughtful AI optimization
[51:53] — Platform rankings: ChatGPT dominance, Perplexity’s passionate user base
[55:10] — Final thoughts: be AI-compatible, avoid snake oil, and Scrunch is hiring

Final Takeaways

AI-powered search and discovery are fundamentally rewriting how content needs to be structured, optimized, and measured online.
Clear, descriptive, server-side content wins; gaming the system is temporary; meaningful, user-centered programmatic content can be very effective.
ChatGPT leads the way as the default AI interface for mainstream users, with Perplexity as a notable runner-up among power users.
For brands and developers: This is a once-in-a-generation shift—those who invest in understanding and adapting to AI search now will have an edge as both tactics and platforms continue to rapidly evolve.

Resources & Show Notes:
For more, visit https://latent.space

Interested in shaping the future of AI compatibility for the web? Scrunch.ai is hiring!

wavePod

AI is Eating Search

Powered by Wave AI

Summary

Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

2. Monitoring the Shifting AI Search Stack

3. The Future of Content Access & Paywalls

4. AI Search Ranking: How It Works & How to Win

5. Privacy and Real-Time Personalization

6. New Games: Black/Grey-Hat SEO and Generative Optimization

7. Case Studies and Tactical Outcomes

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Notable Quotes & Memorable Moments

Timestamps for Key Segments

Final Takeaways

Summary

Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

2. Monitoring the Shifting AI Search Stack

3. The Future of Content Access & Paywalls

4. AI Search Ranking: How It Works & How to Win

5. Privacy and Real-Time Personalization

6. New Games: Black/Grey-Hat SEO and Generative Optimization

7. Case Studies and Tactical Outcomes

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Notable Quotes & Memorable Moments

Timestamps for Key Segments

Final Takeaways

AI is Eating Search

Powered by Wave AI

Summary

Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search Date: July 23, 2025 Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI) Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

2. Monitoring the Shifting AI Search Stack

3. The Future of Content Access & Paywalls

4. AI Search Ranking: How It Works & How to Win

5. Privacy and Real-Time Personalization

6. New Games: Black/Grey-Hat SEO and Generative Optimization

7. Case Studies and Tactical Outcomes

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Notable Quotes & Memorable Moments

Timestamps for Key Segments

Final Takeaways

Summary

Latent Space: The AI Engineer Podcast

Episode: AI is Eating Search Date: July 23, 2025 Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI) Guest: Robert (Co-Founder, Scrunch.ai)

Episode Overview

Key Discussion Points and Insights

1. The Origin of Scrunch.ai & Identifying the Opportunity

2. Monitoring the Shifting AI Search Stack

3. The Future of Content Access & Paywalls

4. AI Search Ranking: How It Works & How to Win

5. Privacy and Real-Time Personalization

6. New Games: Black/Grey-Hat SEO and Generative Optimization

7. Case Studies and Tactical Outcomes

8. The Competitive Map: ChatGPT, Perplexity, Claude & Others

Notable Quotes & Memorable Moments

Timestamps for Key Segments

Final Takeaways

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)

Episode: AI is Eating Search
Date: July 23, 2025
Host: Alessio (Partner & CTO at Decibel), Swix (Founder of Small AI)
Guest: Robert (Co-Founder, Scrunch.ai)