Podcast Summary: Insights Unlocked

Episode Title: All the questions you had about synthetic users but were afraid to ask
Release Date: October 6, 2025
Host: Leah Hogan (UserTesting)
Producer: Nathan Isaacs
Guest: Dr. John Whalen – Cognitive Scientist, Founder of Brilliant Experience, Author of Design for How People Think

Episode Overview

This episode dives into the rapidly evolving world of customer research, focusing on the intersection of artificial intelligence (AI) and user research—particularly the provocative concept of “synthetic users.” Dr. John Whalen joins host Leah Hogan to share grounded, candid insights on how his team is integrating AI into research, where synthetic users fit (and don’t fit), and how AI-driven tools are reshaping the workflow for researchers, UX, and product leaders.

Key Discussion Points & Insights

1. Dr. John Whalen’s Origin Story and Passion for Human-Centered Research

Timestamps: 01:43–03:28

• Dr. Whalen began his career during the dot-com era, intrigued by the rough user experiences of early browsers (e.g., Netscape Navigator).
• Originally trained as an experimental psychologist, he found the dynamic nature of consulting allowed him to merge his love of technology with a deep curiosity about human thinking.
• He authored Design for How People Think to bridge technology and human cognition.

Notable Quote:

“I found that in consulting, I could have a new thing I do and make a contribution, like every two months or every three months… It’s endlessly curious to see how people are thinking and how can we align what we’re doing with how people think.”
— Dr. John Whalen [02:25]

2. Integrating AI into Research Processes—Benefits and Boundaries

Timestamps: 04:22–09:26

• Dr. Whalen’s team at Brilliant Experience carefully adopted AI, starting with hands-on trials, interviews with tool founders, and rigorous comparison studies:
  • Human researchers vs. AI moderators in conducting and analyzing interviews
  • White-labeling findings to senior researchers for unbiased evaluation
• Surprised by AI moderation’s strong performance: it matched seasoned researchers’ findings about 80–85% of the time.
• AI’s Role: Currently best for accelerating scale, language inclusion, and generating “some insight” where time/budget prohibits traditional research.
• Boundaries: High-stakes research (e.g., $300M decisions) should still heavily involve human expertise and rigorous checks.

Notable Quote:

“We were all set to hate AI moderation. And we were sort of dumbfounded that it hit about 80 or 85% of what we found as seasoned researchers… So we were like, wow, we at least have to take this seriously.”
— Dr. John Whalen [05:55]

3. Synthetic Users: Why Take Them Seriously?

Timestamps: 09:40–16:57

• Initial skepticism is normal—even Dr. Whalen “guffawed” at the idea.
• Good research demands measured, empirical evaluation; his team used synthetic users in parallel with real user interviews.
• Findings: Synthetic users offered similar major answers (6–7 out of 7 key points), without introducing irrelevant ideas.
• Practical Applications:
  • Useful for designers or product owners needing fast “inspirational” data to inform early concepts.
  • Not a replacement for primary data; best as a supplement, especially for rapid prototyping, hypothesis development, or stakeholder engagement.
  • Great for “impossible questions,” like “Which of my questions offended you the most?” or “What would you never tell me in an interview?”

Notable Quote:

“The way I think of synthetic user data is not that it’s actually data or facts. I think of it as inspiration or a way to broaden my thinking and a way to prepare for being with the real users.”
— Dr. John Whalen [12:42]

Notable Moment:
Synthetic users can even boost stakeholder excitement, paradoxically nudging them to invest in richer, human research.

“Ironically, I have to use a synthetic user to do it. But it’s a tool to draw everyone into the humane nature that we really want in the end.”
— Dr. John Whalen [15:59]

4. Where Synthetic Users and AI Agents Fit into the Research Lifecycle

Timestamps: 18:10–22:58

• Early Process: Use synthetic users in stakeholder workshops for instant feedback, clarifying research priorities.
• During Recruiting: Test screeners with synthetic users to avoid unintentional exclusion of the target audience.
• Throughout Analysis: Employ synthetic “experts” to bring new perspectives (e.g., brand, accessibility) even if you lack in-house expertise.
• Preparation for Presentations: Spin up synthetic versions of key stakeholders to anticipate tough questions or uncover overlooked points.

Notable Quote:

“When we’re doing recruiting, we get our synthetic users to answer the questions to the screener to see, would we have accidentally knocked out our target audience because they would think of that question in a different way than I did?”
— Dr. John Whalen [19:53]

5. Building Trust and Setting Guardrails with AI Research Agents

Timestamps: 22:58–27:41

• Cautions against over-trusting AI outputs; emphasizes “trust but verify.”
• Tools like Coloop and Quali provide hyperlinking between summarized insights and source transcript data, facilitating quick verification.
• Challenges:
  • AI lacks a strategic mindset—may surface obvious “red ocean” needs instead of deep insights.
  • Averaging insights can mask crucial segments (e.g., privacy-concerned vs. enthusiastic users).
  • Human researchers still critical for robust study design, quality, and nuanced interpretation.

Notable Quote:

“Every one of those points it makes is a hyperlink... this is a trust but verify moment. Maybe this is a good way to get a quick sense of what happened. But obviously there’s more nuance to it.”
— Dr. John Whalen [24:16]

6. The Continued Importance of Human-to-Human Research

Timestamps: 30:27–32:57

• AI and synthetic users expand the researcher’s toolkit and enable more iterative, global, and inclusive research, but not at the expense of authentic human insight.
• Suggests a revised mixed-methods model:
  • Start with a few “human-to-human” interviews
  • Scale up with AI moderation
  • Use findings to inform where additional human depth is needed

Notable Quote:

“We might do a handful of human-to-human interviews just to be sparkly clear and then be able to use, say, AI moderation to scale up our work… those distinctions aren’t as black and white as they used to be.”
— Dr. John Whalen [31:38]

7. The Future: Always-on Research, Synthetic Users as Routine, and Researcher Identity

Timestamps: 33:16–39:42

• Predicts that by 2028, most product/marketing/design folks will routinely use synthetic users for quick “what if” checks (“the highest number of hits on a research tool in 2028 will be synthetic users!”).
• Data repositories empowered by AI will allow for continuous, layered analysis across customer touchpoints—far beyond interviewing.
• Warns of a transition period: excitement, overwhelm, and even “grief” as researcher identity shifts with AI integration.

Notable Quote:

“[Synthetic users] are super inexpensive... risk-free. The average engineer, product person, marketer, designer will have at their disposal these synthetic users to do a quick one-off test on.”
— Dr. John Whalen [33:44]

Notable Quote:

“My heart beats a little faster thinking about this too. It’s a new world and a little scary… almost everyone is saying ‘I’m like excited and terrified.’”
— Dr. John Whalen [39:10]

Practical Guidance & Memorable Takeaways

• AI Tools To Explore:
  • SyntheticUsers.com; Vervey; Coloop; Quali; Next (for agentic AI cross-channel experience insights)
• Concrete Steps for Researchers:
  • Learn prompt/context engineering
  • Try AI-moderated interview tools
  • Familiarize with AI-powered analysis
  • Experiment with synthetic users—both for early prep and as ongoing inspiration
• Vital Advice:
  • “Be curious, but skeptical.”
  • “Have an educated opinion on these things and understand what works—and what doesn’t—and then think to the future of suppose these got really good. What would I do then?”
    — Dr. John Whalen [40:17]

Further Resources

• Dr. John Whalen:
  • Podcast: AI for UX (YouTube: Brilliant Experience)
  • Course: AI for Customer Research (via Maven)
  • LinkedIn: [John Whalen]
• Show Notes and Curated Clips: usertesting.com/podcast

Final Thoughts from the Episode

Even the most skeptical should challenge their assumptions. Synthetic users and AI-powered analysis are here to stay—but their best value comes not from “replacement” but as augmentation, inspiration, and a locus for deeper, more human-centered exploration.

“I really encourage folks to keep learning… be prepared for the future and evolve the way we need to.”
— Dr. John Whalen [40:04]