The Growth Podcast – Episode Summary

Episode: How to Do AI-Powered Discovery (Step-by-Step with Live Demo)
Host: Aakash Gupta
Guest: Caitlin Sullivan (Expert in AI-driven user research)
Date: February 13, 2026

Episode Overview

This episode is a masterclass with Caitlin Sullivan on conducting rigorous user discovery and customer research with the aid of AI, especially large language models (LLMs) like Claude, Gemini, and ChatGPT. Caitlin dives deep into the practical workflows, showcases live prompts and demos, debunks myths about AI “hallucination,” and explains how step-by-step prompting can replicate (and accelerate) the best practices of human researchers. The episode focuses on both survey and interview analysis—walking listeners through every phase, from preparing context to running audits, both interactively (step-by-step) and in agentic, code-driven workflows.

Key Discussion Points & Insights

1. The Mess and the Promise of AI in User Research

• AI for user research is "messy and unreliable" if handled poorly—but with the right workflow, you can halve your analysis time without hallucinations.
  • Caitlin:

    “The key is actually replicating the way that you would do things in a rigorous way as a human and just doing it like that with AI.” [00:05]

• Critical insight: Don’t skip straight to synthesis. True rigor comes from mirroring the methodical steps humans take.

2. Selecting Your LLM Workspace

• Caitlin prefers Claude for its nuanced, thorough analysis by default—but regularly cross-tests with Gemini and ChatGPT.

  • Quote:

    “By default [Claude] does a more thorough, more nuanced analysis than the other two platforms... Gemini seems to be fine-tuned a bit more for accuracy...but you have to push [it] more than Claude to give you the complete picture.” – Caitlin [01:03, 11:47]

• Markdown transcripts are essential for preserving structure and enhancing accuracy in AI analysis.

  • Tip:

    “Transforming your interview transcripts into markdown files...not just better for file handling, it actually helps these models do a more accurate job.” [12:44]

3. Replicating Rigorous Human Analysis (The Core Workflow)

Breakdown of Tasks and Workflow

• The process echoes classic research methodology:
  • Analysis (dig through, pick apart data)
  • Verification/Stress Testing (ensure insights hold up)
  • Synthesis (only at the end)
  • Caitlin:

    "Most people...jump straight ahead to synthesis. And that’s exactly what we don’t want to do." [02:50]

Phase Details:

I. Load Context:

• Present background/project info alone in a first prompt (do not combine with tasks).
• Sample Instruction:

  “Internalize this only, do not run analysis yet.” [16:19] II. Analysis (Per-Participant/Per-Response Digging):

• Extract “value anchors” and “fragile points” for interviewees.
• Explicitly define quote selection, ratings, and coding rules.
• Quote:

  “I spell out what a quote looks like to me.” [17:40] III. Verification/Audit/Stress Test:

• Instruct the model to find contradictions or inconsistencies in its own findings.
• Quote:

  “The truth is they do find mistakes. They find mistakes all the time.” [27:11] IV. Synthesis (Summarize themes/action items):

• Only after verification.

**Notable Segment: “Two/Three/Four Step Prompting”

• Context load → Per-participant analysis → Verification → (Optional) Synthesis. [16:44-34:34, 54:21-55:17]
• Aakash:

  “Step by step prompting is how you get the AI to follow your human process. And that’s our North Star here.” [46:04-46:12]

4. Survey Analysis in Detail

Dataset Example: Churn survey with open-ended feedback.
Process:

1. Coding: Assign mutually exclusive labels to open-text responses (not software ‘coding’ but categorization). Use inductive coding to let themes arise from the data.
   • Provide coding rules:

     “Each response needs one primary code...mutually exclusive, not overlapping.” [38:49]

   • Watch for math errors, instruct to use code for calculations.

     “If you want to be safe on your math, tell it to code it.” [40:52]

2. Quantification: Count the frequency of each code.
3. Intensity Ratings: For emotional strength—differentiate between ‘soft exits’ (circumstantial) and ‘angry exits’ (fixable product issues).
   • Tip: Give few-shot examples plus rationale:

     “Few shot is reasoning. Not just showing the example, but why as well.” [49:21-49:35]

4. Audit: Have AI review and correct its own coding and ratings.

5. Agentic / Parallelized (“Mega”) AI Workflows

Running Parallel Agents:

• Set up Claude Code (in your terminal) to automate and parallelize both interview and survey analysis via agent markdown files and context documents.
  • Summary: Three types of input: context document; agent markdown prompt (e.g., Interview Analyzer Lite); and actual data files. [60:02]
• File structure and explicit context/rules in markdown = higher accuracy and speed.
• Time savings: Analyze interviews + survey in parallel, slashing overall time.

  • “You just let your agents...do both things at once...cutting out half the time by parallelizing this.” [63:33-64:11]

File Output:

• Produces ready-to-use markdown summaries with executive summaries, per-user breakouts, traceable quotes, and intensity ratings—prime for synthesis or reporting.

6. AI-Moderated Surveys & Limitations

• AI-moderated interviews/surveys are improving but vary in quality. Still, human skill in research is irreplaceable.
  • Caitlin:

    “Some [AI tools] are asking really, really good questions...some are not quite there yet.” [06:36]
    “I also wasn’t particularly impressed [with Anthropic’s results].” [06:50]

  • Aakash:

    “Even Anthropics interviewer couldn’t do a good job...think about this as a skill.” [07:52]

Notable Quotes & Memorable Moments

• Respect the Craft:

  "Respect the craft of user research. Respect the people who are working on it in your company." – Aakash [07:52]

• Two-Step Prompting:

  “First step context, second step analysis. Most people try to shove it into one prompt.” – Aakash [16:50]

• Why Separate Context Load:

  “If you try to paste a two page long prompt in one go...they drop some of the instructions...So I just copy my first prompt, which is only about context.” – Caitlin [09:22]

• ‘CYA’ Audit:

  “I’ll call this the CYA way to use AI – cover your ass – have AI check that out.” – Aakash [53:18]

Timestamps for Key Segments

| Topic | Timestamp | |:---------------------------------------- |:---------------:| | Initial framing on AI analysis | 00:00 – 02:41 | | Importance of “not skipping steps” | 02:50 – 03:43 | | Caitlin on preferred LLMs | 11:47 | | Markdown transcripts approach | 12:44 – 14:54 | | Step-by-step prompting demo (Interviews) | 16:44 – 24:22 | | Tools: Dovetail, Breda, Reveal | 24:22 – 26:14 | | Multi-step prompting: Audit explained | 27:11 – 31:38 | | Survey analysis: Coding/Quantification | 35:02 – 44:14 | | Coding vs. Quantification vs. Verification | 54:21 – 55:17 | | Sentiment/Intensity Rating | 48:13 – 50:36 | | Agentic workflow via Claude Code | 58:14 – 67:00 | | Output formatting/markdown review | 67:13 – 70:39 | | Host summary wrap-up | 70:50 – 71:44 |

Practical Takeaways & Advanced Tips

• Replicate, don’t shortcut, the proven human analysis workflow with AI: Always split out context, explicit per-response analysis, and verification. Synthesis should be last.
• Be explicit in rules and give examples: Key to avoiding bias and AI misinterpretation.
• Audit/falsify the AI’s own reasoning: Push the model to critique itself, spot contradictions, and recode where necessary.
• Consider survey/interview data as separate parallelizable jobs. Agentic workflows in terminal environments can halve your analysis time for large, multi-modal discoveries.

Closing Wisdom

“The best products are built with the best user understanding. And this is your roadmap.” – Aakash [70:50]

Guest resources:
Caitlin Sullivan’s cohort-based course (more depth, advanced prompt engineering) – see show notes for discount.
Host resources: Bundle of AI product tools for listeners at buildle.akashg.com

For anyone serious about using AI to power up their user discovery, this episode is the definitive playbook.