A (1:39)

of Maubin, Arise, Relay App, Dovetail, Linear Magic Patterns, Deep Sky, Reforge, Build, Descript, and Speechify.

C (1:45)

So be sure to check that out@buildle.akashg.com

A (1:48)

and now into today's episode. Quick note for my audio listeners. There are some things that we showed which you can see on Spotify or

C (1:58)

YouTube, but we've edited this audio so.

A (2:01)

So that it's a really good listening experience nonetheless. Caitlin, welcome to the podcast.

B (2:06)

Hey, Aakash, thanks for having me.

A (2:08)

What does good AI customer research and analysis actually look like?

B (2:14)

So I think there's actually a really simple answer, which is it looks like replicating the way that we do rigorous analysis as humans. I'm not reinventing the wheel with pretty much Anything I do, I have just taken what I know to be true and doing really solid research as an individual contributor, as the head of user research and replicating those processes with AI.

A (2:41)

Amazing. So let's break this down for people in terms of what the actual tasks are and what the actual tools are.

B (2:50)

So analysis actually looks fairly similar whether you're analyzing interview transcripts or survey responses or other kinds of short customer feedback. We're going to use a similar set of, of kind of phases of analysis across all of them. So good analysis, rigorous analysis with AI, or as an individual, as a human doing it, is going to involve actual analysis where you dig through data and you pick it apart, you look at what's in there closely, put it back together in different ways and see what pops at you. And then verification or stress testing, basically seeing how the findings you have hold up under pressure, and then finally synthesizing. But what most people do, both as humans and also especially with AI, is jumping straight ahead to synthesis. And that's exactly what we don't want to do.

A (3:43)

And then there's two big types of analysis that people tend to run, surveys and interviews. What are we going to do today on the surveys front?

B (3:52)

I'm going to demo how you can use any of the large language models to run a few kind of key steps that I just talked about to get really solid results out of any analysis that you do with surveys or short customer feedback. And I'm going to do that in Claude in the normal web UI, but you can replicate this process in ChatGPT or Gemini. I do that all the time as well.

C (4:16)

Awesome.

A (4:17)

And then we're going to show you everything here in blue. So we're going to cover the Claude analysis process, but also show you the agentic version with the ChatGPT codecs. You could use other tools like Zapier, N8N. And are people using those frequently in the fields?

B (4:36)

So I'm going to do the agentic part in Claude code in my terminal, actually. But yeah, people are using the other tools, the agentic. The, the more advanced agentic tools, I would say, are not commonly used by most of my clients, but what a lot of people are already doing, because it's kind of more. A more familiar space, is automating a lot of these processes rather than making them fully agentic. So you can use n8n, zapier, make.com, any of these automation tools to link up with your chosen large language model and sort of put some of these prompts in place to happen Automatically for you.

A (5:15)

Awesome. And then the other side, of course, everybody knows, are interviews. And so there are the human moderated side of interviews. What are we going to cover for people? There's.

B (5:25)

I'm going to basically replicate the same sort of process across survey analysis and interview analysis. What's worth knowing for interview analysis is that things can get really complicated really fast. AI is built to understand structured information best. So with really long transcripts and sometimes messy transcripts where speaker labels aren't particularly good, we can run into a lot of problems with AI. So I want to show you how, being systematic and terms of the sequence of prompts we use and the types of prompts we use can actually ensure that you get much better results, even when we're dealing with a tougher space like interview transcripts.

A (6:07)

Awesome. So we're going to use transcripts for ChatGPT and Claude. If you had a video that is actually possible now with Gemini 3, we're not going to demo that, but that is something to keep in mind. And then the agentic version, we'll show you guys how to do that in Claude code and do that in parallel with the survey analysis. Now, a lot of people have been talking about AI moderated surveys, so can you just talk to us about what these are, what the tools are and how you might use these?

B (6:36)

So there are already a lot of AI moderation tools on the market. There's a huge list of them. They are tools that run sessions with customers for you. So the idea is sort of that you can scale your team's ability to talk to customers by having all of these sessions run when you're sleeping or making your coffee or spending time with your family instead of having to line up, you know, 15 people to talk to every day. So, as we've talked about before the recording started, there has been a little bit of discussion this past week where Claude, the team behind Claude Anthropic, launched their own study where they're using an AI moderation tool of their own that has surprised some people in our space in terms of the quality of the questions that were being asked. So this space is a little bit tricky. There's a huge range of quality between the different AI moderation tools. To be honest, some of them are asking really, really good questions and handling bias, not asking leading questions, and also doing decent analysis after collecting data. Some of them are not quite there yet. And I've seen the questions that Claude was asking in the Anthropic team study, and I also wasn't particularly impressed.

A (7:52)

I Can't emphasize this for you guys listening enough. In my 16 years of product management, one of the things I learned is to respect the craft of user research. Respect the people who are working on it in your company. Try to get some time shadowing the best PMs who do user research and the best user researchers in your company. Because even Anthropics interviewer couldn't do a good job. Right? And so you want to think about this as a skill that you're going to continue to improve throughout your career. We could cover it all, but we don't think the AI interviewers are quite the most important thing for you to learn. So we're going to cover what's in blue for you guys, how to analyze surveys, how to analyze interviews. So without further ado, can you get us started, Caitlin?

B (8:33)

Yeah. Okay. So as I said, we have to go through multiple steps in the analysis process to get something really rigorous and reliable out. And I mentioned that we have to start with kind of picking apart the pieces of the data before we synthesis before we do synthesis, which is unfortunately what most people don't do. Most people jump straight to synthesis. Another piece of the puzzle here is of course giving Claude or any of these large language models enough context that it can do a really good job for us. And I know that this metaphor is really overused at this point of saying that it's like your junior assistant or something. I prefer to think of these models at this point not as juniors, but like someone who's new to your team. So if someone new joined your team, they don't necessarily have no skill in this space, but they don't really have the expertise in the domain or about your customers that you have. They might not even know exactly what kind of feature functionality you have built in if they haven't been fully onboarded. So the first step that I use in a lot of especially interview analysis is just loading context and I make this an individual prompt rather than building it into anything else. Because large language models actually can be a little bit overloaded. If you try to paste a two page long prompt in one go, a lot of the models will still have trouble handling that and you'll see that they drop some of the instructions. Even if you said this is really important. So I'm just going to copy my first prompt here, which is only about context.

C (10:07)

If you're enjoying this masterclass in AI analysis, you will love Caitlin's code cohort based course. This runs for the month of March, where Caitlin will walk you through not just these first level workflows that we've shown here, but the next layer deeper of complex workflows. The Agent markdown files everything you need. She will help you get to the next level of craft using AI for discovery and user research. For my money, it's one of the most important skills for PMs. But not just PMs. Designers, engineers, go to market people and obviously user researchers. So if you enjoy what she's saying now, take her course. You will learn even more. She'll walk through all of this in detail and the best thing is you can get a discount on the course using my code link in the show notes. So check her course out and back to today's episode. Today's podcast is brought to you by Pendo, the leading software experience management platform. McKinsey found that 78% of companies are using Gen AI, but just as many have reported no bottom line improvements. So how do you know if your AI agents are actually working? Are they giving users the wrong answers? Creating more work instead of less, improving retention or hurting it? When your software data and AI data are disconnected, you can't answer these questions. But when you bring all your usage data together in one place, you can see what users do before, during and after they use AI, showing you when agents work, how they help you grow, and when to prioritize on your roadmap. Pendo Agent analytics is the only solution built to do this for product teams. Start measuring your AI's performance with agent analytics at Pendo IO Akash. That's P E N D O.

A (11:40)

And obviously you've chosen Claude. If people have the option to choose between all the AI models, why have you chosen Claude?

B (11:47)

I've chosen Claude. It's my favorite. It has been for a long time and I keep checking myself on that, doing tests across the three major ones, meaning Gemini, Claude and ChatGPT. And I keep coming back to Claude because by default it does a more thorough, more nuanced analysis than the other two platforms. I will say though, I do do like Gemini quite a lot because it seems to be fine tuned a bit more for accuracy and focusing on frequency of findings. So you're more likely to get kind of a top five themes or patterns from Gemini really quickly that are solidly evidenced than you will in Claude or ChatGPT just because that's kind of what it lifts up. But you'll have to push Gemini more than you have to push Claude to give you the complete picture and all of the smaller themes or the outliers and things like that.

A (12:41)

Okay, excited to see how Cloud does.

B (12:44)

So I'm with my context prompt here, I'm just going to get all of my files ready here. I'm just dragging them over. And I want to quickly point out as I send this context prompt that I'm using markdown files. And this is important because in Claude, if you don't have a max account like I'm using right now, if you have Pro even, you've probably experienced uploading a bunch of transcripts and hitting this token wall, even though claude's token limits have been expanding and expanding and you know it can handle a lot more at once. Now, you can kind of circumvent that issue by transforming your interview transcripts into markdown files. But this is something that I teach in my course because you have to do it right. You have to do it in a way that really keeps maintains all of the most important information from those transcripts and carries them over into mark files. But what you get out of a markdown file, instead of just using the transcript, is you've turned it into a format that's more structured than the original transcript is, which means that it's not just better for file handling, it actually helps these models do a more accurate job as well.

A (13:55)

Love it. And just the high level, like, what does that transformation into markdown look like? What do people need to focus on?

B (14:02)

You're basically telling it to take certain types of information and it's not going to take everything from the transcript. There's a lot of bloat in your transcripts. Not every conversation necessarily needs to be carried over. Some people will argue with that. But for, especially for a lot of the sort of product research that I've done, you have a specific focus. You're not always doing exploratory research. So you can usually have a large language model with you, comb through a transcript and extract the right kinds of information that are most relevant to the topic areas you're exploring or the hypotheses you have, and turn that into a document that already as tagging or groupings of conversations under certain headers and things like that. Putting in metadata, a little bit of extra coding that makes it easier for the model to make sense of it.

A (14:54)

And then how would you break down this context prompt? What are the key elements here?

B (14:58)

So it looks like a lot at first glance, I realize because some people share a half of this or a third of this within a prompt with actual tasks. But I would call out what's most important here is I do have A bit of project context. I have a business goal, reduce churn. The percentage is kind of arguable here though. I would say in my experience saying 15% versus 50% like 5,0 will make a difference. And then I have product context. So just like I was saying about onboarding this new team member, I want to make sure that it fully understands what kind of functionality is involved in the service that we offer, what the differences are between the product tiers. So that in its reading through of people's experiences or comments about the product, what they're using, it fully understands what kind of functionality would actually be present and what they would be using when they have that experience. Otherwise, if we don't include something like this, I've seen hugely wrong guesses. Gross assumptions by large language models about what someone has as part of their account, kind of making things up or just trying to piece together bits of information it had that were entirely wrong.

A (16:15)

Point it in the right direction of what your product is actually doing in that case.

B (16:19)

Exactly. Yeah. What's important here though is I said that for a context prompt we actually are just loading context. So at the end I said internalize this only do not run analysis yet. It's really important that you don't let it jump ahead and start doing tasks because I haven't given any instructions here for how to do analysis. So it hasn't done analysis yet. I'm just going to jump to the next prompt and send that now.

A (16:44)

And I think this is a really cool lesson for people. I don't think many people have done

C (16:47)

this

A (16:50)

two step prompting, right? Yeah. First step context, second step analysis. Most people try to shove it into one prompt.

B (16:59)

Exactly. But this will get you much better results. I mean it will basically help the large language model focus on one thing at a time, which means that each individual step will be done more thoroughly. So let me start working on it on this step so that I can talk through it at the same time because it will take a minute. What's really important here is I've started analysis, but I'm not saying, hey, go find me some themes and patterns. So what I said earlier about starting rigorous analysis means doing actual analysis and not synthesis, it actually means that we need to tell it to go look at at every single file that I just uploaded. I'm basically asking it to pull out for each participant value anchors. And I'm telling it that means what's keeping them subscribed and then fragile points here the formatting is a little wonky. What frustrations exist despite subscription. And I'm spelling things out in detail. So where I say I want it to give ratings, right? Like, how big a deal is this? How strong a fragile point is this? I'm explaining what a high fragile point means. Like, what is a high rating? So I'm not letting it make any assumptions here. I'm trying to be pretty explicit in every case about how I would rate things. I've also started this prompt with quote selection rules, which seems a little funny because I think most people just assume large language models know what a quote looks like. But if you prompt in Gemini and then you prompt in Claude, you will see that they interpret quotes completely differently. So Gemini will give you much shorter snippets, Gemini will give you much shorter snippets of customer statements, and Claude will give you, you know, a longer sentence or even a short passage. So I like to, you know, avoid making the assumption that Claude, in this case, has the same understanding of quotes that I do. And I spell out what a quote looks like to me.

A (18:59)

Very interesting. Yeah. I've also noticed that sometimes the AI will, like you've had here, take away some of the hedges and qualifiers. It'll try to, like, clean up the quote as if it were cleaning it up for a news article. But in user research case, we don't want to clean up the code, so we're giving that instruction to the AI.

B (19:17)

Yeah. And in that case, you could even line. Add a line here that says something like leave, ums, uh, pauses, you know, fumbled language, things like that. I sometimes include that in here, too, to keep things exactly as they are with the customer's voice.

A (19:33)

And can you define some of these words for me? Because I'm not totally familiar with them or I can imagine a lot of the audience is trying to guess at exactly what they mean. Value anchors, fragile points, stability rating. What are those that seem like cool terms of art in the user research discipline? People could benefit from learning.

B (19:49)

Let me give you a background quickly about the data I'm using here, too. So I have eight interview transcripts from a synthetic user study that I created based on a study that I actually ran. And it's all about improving retention and decreasing Churn. So these are all people who had a premium tier account on a fictional app called Flow that is a meditation and anxiety management app that also has on demand therapy. And so what I'm doing here is I'm trying to figure out. I'm doing a bit of a retention analysis where I'm trying to figure out what's keeping people subscribed and then what's likely to break their experience? So what frustrations exist despite subscription basically means the flip side of the things they're finding really valuable about the service. What could actually push them to cancel anyway? And I want to collect both sides of that because I'm not assuming that when someone is super happy, they're just happy. No one is. So one dimensional as that as a customer or as a human being. So I want to know, okay, if they're happy about using the service and they continue to pay, that's great. But what would cause them to churn anyway if it continued or increased over time?

A (21:05)

Makes sense.

B (21:06)

So we have some results here. And I've told it to put this into a table. So it followed my instructions really well. Basically, this looks like the value anchors are in a table here for Participant 1. This is by participant. So this is the first participant here, value anchors, then fragile points. And then I also asked for a stability rating. So overall, how risky or how much risk is there that this person will churn? And then I have a few key themes that I asked it to call out too. And then we start participant two. And it just looks like that for every participant. So what I said earlier about making sure we actually pick through the data and kind of look at it closely first is what I've forced the large language model to do here. But if you jump to asking for themes, patterns, pain points, whatever it is, you're actually forcing it to jump into like synthesis mode instead of real analysis picking through the data mode. So I usually start with something like this, where I force it to look at every file and kind of look for specific pieces of information so that it actually comes through properly.

A (22:13)

And that's replicating the human process. That was the high level advice you gave us at the beginning, which is we are always going to analyze the specific interviews. So let's not have the AI jump that step, let's have it analyze those first.

B (22:25)

Exactly. So as a very experienced user researcher, you always comb through. You know, we used to just sit with one interview transcript and look through it manually. Every line of copying, pasting into Miro or sticky notes or whatever. And so I'm basically replicating that process here.

A (22:44)

Awesome.

B (22:46)

So the kind of information that we get here is going to be much more granular, much more specific. It hasn't really, other than this bottom section here, it's not mapping things into themes, it's really just picking up the foundational elements that a really good analysis will be built on. So we have things like this full passage about someone who is doing breathing exercises with this app every day, usually between meetings, and so on, all of the things that she's using. And so the strength of this in terms of stickiness with the product is strong because there is daily integration into the work routine, multiple features used regularly, and so on. Let me see, what else is there? It's maybe more than what she expected because she thought it would just be meditation, but it's become this whole system for managing my new role. So dealing with stress at work is kind of now dependent on this app. That's a good sign. And then the therapy sessions can be booked spontaneously. I don't love that this means I won't always talk to the same person. So there's something interesting there. It's labeled this as moderate. I would almost say, okay, well, this is where my perspective comes in, because I don't just have AI do analysis without thinking about it myself. But this seems very positive. And the rest is starting to get a little bit more into the fragile points. So this is the sort of thing that I want to be pulled out because if I were doing this as a researcher, I would be pulling out all of these quotes myself and then trying to categorize them, label them, give them ratings and so on. So this is really what it looks like to do good analysis, whether you're doing it with AI or not. But it's doing it much faster for me.

A (24:22)

And there are a lot of tools that I think help with this. When I was at a firm, we used a tool called Dovetail. What is your take on some of those tools and which ones do you like like?

B (24:32)

So I use a few that are less talked about. The two that I've continued to return to because I think they're doing a particularly good job. Are Brayda AI. I can send you that for the show notes and reveal. They have completely different approaches. And I think there's so many different tools out there that really do things just very differently than each other. So it's hard to compare them all kind of apples to apples. But Breda is getting to a pretty good nuanced level of analysis fast because they have really good prompting in the back end and reveal maps the customer comments in a bit of a grid and does a bit of hypothesis testing for you. So it lays things out in a way that you, you know, kind of like this. I'm almost mimicking what they do as well. Being able to see things yourself and make your own comparisons in Reveal versus just having a finished analysis for you dovetail. I honestly hear mixed reviews about, from researchers especially, and I have had mixed experiences with it. So I haven't tested it in about five months. So I wouldn't dare to kind of give it a final rating right now. But a tool like that is really meant to get you to insights as fast as possible. But you don't have control in any of these platforms over exactly what you're looking for and exactly the type of analysis that you want it to do. So I tend to stick to the large language models for the most part, unless the time to insight is so extreme that I, you know, feel better uploading things into a platform and having it do that for me. But it's rare.

A (26:14)

So the platform is going to take some of this control that we're getting from these prompts, from this step by step prompting process out of your hands. And that's the power of this workflow. We're kind of showing you guys the advanced, this is what an advanced user researcher would do.

B (26:27)

Yeah, and obviously the, the teams behind all of those tools have worked tirelessly on those prompts. They've also worked with the team at Maze to kind of develop the prompts behind their AI analysis and AI moderation. So I know what it looks like behind the scenes and they're all working very hard to kind of figure out the right prompting sequences to do this for you. But I prefer, as someone who's used to doing research especially, I prefer to kind of have more control over the granularity in the analysis or focusing it on a retention analysis in a certain way and so on over using a tool to do it for me makes sense.

A (27:04)

So is there a step three now?

B (27:06)

There is a step three.

A (27:07)

So I had said two step before, but it's actually a three step prompting process.

B (27:11)

It's three step here and I would say it's four if synthesis is added. So the context is almost like a step zero. I would say you kind of just load the background information. So I would call this three steps in this workflow flow, which is condensed compared to what I usually do. So for the most rigorous possible version of this, you have more steps because you need more feedback loops with AI. But you'll see one version of this. Here, let me just space this out so people can see this more easily. Just going to let it do its thing while I talk. So this is what I referred to earlier as a stress testing or verification step. We need this because even if you think everything looks really good which, you know, most people go, oh, AI did this analysis for me and it looks really solid. It's usually not as solid as you think it is. And what happens down the line is then a product manager or designer or researcher, you know, presents some findings in a meeting and then can't trace them back or realizes there are actually lots of holes in this insight or set of insights. But if you push your model through an additional step for verification, or I usually call this some kind of audit step, then it very often catches its own mistakes. And I get questions all the time from people who are really rigorous or kind of know that these models have a lot of bias built in. They ask me, yeah, but how is it going to critique its own work? You know, it's bias toward its own work. The truth is they do find mistakes. They find mistakes all the time. And I've tried this kind of splitting the process between tools too. But this actually using it in the same model that you're doing analysis in does find issues. So let's see if it found some things. So, yeah, so in this particular prompt, actually, let me bring this back up again. This version of a verification check, I'm asking for contradictions in user statements. So it's very common that one person in a transcript will be telling a story one way. Like they say, you know, well, first I got my shopping cart when I was going grocery shopping and then I looked for the apples or whatever in a certain order. The second time around, in a different part of the conversation, they tell the story of the user journey in a completely different order or they contradict whatever they said earlier in some way. Well, you know, I use this every day, the equivalent of like I go to the gym every day or five times a week. And then, well, when was the last time you went to the gym? Oh, well, it was probably Saturday and it's Wednesday today. Right. So very often if you're doing a good interview, you find some contradictions in what someone does and says or what someone said the first time around and then what they said the second time around. But AI is very likely to blur that kind of cherry pick the stories that it likes or that match what you asked for most closely and forget about or ignore the second thing they said. That is actually the opposite. So I'm putting it through this pass where I'm checking for contradictions and that will make the full analysis in the end much more bulletproof, kind of preventing the cherry picking of the story that best aligns and getting the full picture. So to spare you the details. I think people can look at this on the screen, stop the video if they want to, but but I'm basically asking for contradictions and I'm defining what contradictions look like to me. I've even spelled out what is not a contradiction and a few examples. So I'm consistently giving examples, definitions and then what that looks like to me to be really, really clear and just so you see a few contradictions here. So something was revised. The rating was revised from original no, this is confirmed. Sorry, this one is moderate risk in the first place, like at risk of churning and then it was revised to high risk. So upgraded because it double checked what the person was saying, the stories they were telling and actually realized that there's more risk involved. So this is the sort of correction that I'm looking for. I want to make sure that it has combed through the data again, made sure that the claims, the statements that it made were actually verified.

C (31:38)

You know what kills momentum? Spending 30 minutes on a brainstorm with Claude and then copy pasting it into

A (31:43)

a slide tool to create the slides.

C (31:45)

That loop is dead. Gamma now works inside Claude through Claude connectors so you can go from thinking to a finished presentation without ever leaving the conversation. Here's what that actually looks like. You're brainstorming a product strategy in Claude mid conversation you say turn this into a deck. Gamma generates a right there. You keep refining your thinking and the

A (32:03)

presentation updates with you.

C (32:04)

No copying, no pasting, no starting over in another tool and it gets better. Connect Gmail, Notion or GitHub alongside Gamma and Claude will use those connections to pull live data straight into your slides. Weekly business reviews feature PRDs all generated from the data you're already using. Your deck finally keeps up with your ideas.

A (32:23)

Try it now.

C (32:23)

Connect Gamma to Claude app. Go from thinking to present, presenting in one flow.

A (32:27)

Claude helps you think.

C (32:28)

Gamma helps you show it. Today's episode is brought to you by the experimentation platform Chameleon. Nine out of 10 companies that see themselves as industry leaders and expect to grow this year say experimentation is critical to their business. But most companies still fail at it.

A (32:43)

Why?

C (32:44)

Because most experiments require too much developer involvement. Chameleon handles experimentation differently. It enables product involvement growth teams to create and test prototypes in minutes. With prompt based experimentation. You describe what you want. Chameleon builds a variation of your web page, lets you target a cohort of users, choose KPIs and runs the experiment for you. Prompt based experimentation makes what used to take days of developer time turn into minutes. Try prompt based experimentation on your own Web apps. Visit chameleon.com prompt to join the waitlist.

A (33:16)

That's K A M E E L

C (33:18)

e o o n.com/Today's episode is brought to you by amplitude. Replays of mobile user engagement are critical to building better products and experiences. But many session replay tools don't capture the full picture. Some tools take screenshots every second, leading to choppy replays and high storage costs from enormous capture sizes. Others use wireframes. But key moments go missing, creating gaps in your understanding. Neither approach gives you a truly mobile experience. Amplitude does things differently. Their mobile replays capture the full experience every tap, every scroll, and every gesture with no lag and no performance hit. It's the most accurate way to understand mobile behavior. See the full story with amplitude.

A (34:02)

So this is how we increase the certainty that it hasn't hallucinated. And it's giving us the right analysis based on the quotes.

B (34:08)

Exactly. So it's a bit of an analysis check in the first place. It's also an exaggeration check. In some cases, you can run an audit to make sure that you're staying as close to the data as possible, that no statements were made by AI in the process that are a bit over or under exaggerated. So you can use it for lots of different things. This is just one version of it.

A (34:31)

Excited to see where we go from here.

B (34:34)

So I would end this process for this kind of very condensed workflow version with synthesis. This, I think if it's okay with you, I would just skip a synthesis step because I think most people know how to do synthesis. Most people would just say summarize this or give me the key themes, the top three action items, that sort of thing. And I would hop into survey analysis. Does that work?

A (34:57)

Let's do it.

B (34:57)

Okay, so let me open a new window.

A (35:02)

The key takeaway here, what we did is we didn't jump into synthesis. We did this, whatever you want to call it, two, three, four step process to make sure that we increase the certainty in the results, in the quality of the analysis and replicate the human process exactly.

B (35:16)

So this is what I'm dealing with here. This is a very basic churn survey, but pretty similar to what I've ended up working with in the past. We've got some automatically logged information. We've got a column like Reason Text where the customer chose a category from a dropdown. And these are a lot of other categories. So this isn't particularly well organized in the first place. Then we have Reason Detailed Text and this was an open text field where people could just say something about why they were canceling. Pretty basic. And then we have what tier of service they were on at the end. So that's the file that I will be uploading now, like what all of you guys have. So let me get my file in here, just dragging it over. And what most people would do here is similar to what I said before, kind of toss in a CSV file, say, show me the biggest pain points from the column with the open responses, or count all response types in this other column where, you know, we had the type of cancellation logged. But that's actually not what I want to do here because a lot can go wrong if you start there. Similarly to the interviews, just jumping into pattern finding, a lot can go wrong, a lot can be missed, missed, and a lot can be miscategorized. So when we're working with survey responses or short customer feedback, we want to code first.

A (36:41)

And for people who don't know, what does coding mean? It's not software engineering.

B (36:45)

No, definitely not. I'm not a particularly technical person, so definitely not. That type of coding. We're talking about applying labels to things, putting things in buckets that help us make sense of them. So normally if you're doing data analysis, you're looking at all of these responses and there is an open text response field, like, you know, customer feedback form or feature request form, that sort of thing. You have to kind of make sense of what people are talking about in those message fields, in those message responses and label them with something before you can start grouping and saying, well, most people complain about this or most people are requesting this. That coding needs to have an approach, right? You need a bit of a methodology for the coding in the first place. You can do inductive coding where you look at the data available and you try to make sense of it from what's there. Or you can do deductive coding when you already have a taxonomy, you already have parts of the product, and you want to, you know, label all of your data with those. Those features or customer types or things like that. So you might already have something that you like to use as labeling. But in this case, I'm using inductive open coding, where I'm saying let themes emerge from the data rather than forcing preset categories. And what's important about this, I very often use inductive coding, at least at first, because I let AI try to see what I may not have noticed about the data and apply codes from a more open perspective first if you just apply your own tags and say, here's the list of tags I want you to code this with. What very often happens is it will start force fitting. It will take the quotes that it sees and try to make them make sense in the set of kind of required buckets that you gave it. And that doesn't lead to the most accurate coding and analysis.

A (38:42)

Makes sense. So this goes back to that craftsmanship of research that you want to be applying. And so we have this first step here where we code.

B (38:49)

And I'm going to let it start working because it will take a couple of minutes on the. But I also have rules here. So I'm not just saying go ahead and code this however you want. You know, be open, find some interesting things for me. I'm also saying each response needs one primary code. They need to be mutually exclusive, not overlapping. I want really tidy little boxes. I want all of the quotes to be in one box categorized right. In a way that I can make sense of later not having too many kind of messy overlapping things. Things. This is really up to the person doing the analysis. With AI, you could, you know, anyone watching could decide you would take a completely different approach. And that's fine. But what's most important here is just that there are rules, that there are criteria, that there are kind of lines that we need AI to paint within. We don't just kind of let it run this, however it feels like running it. And one last thing, while it's still working on this, most likely, yep, I finish with a line that is maybe becoming obsolete, but not quite yet. I've always included this as a bit of a fail safe. In the past, all of the models have made mistakes with math calculations. And a lot of the time the calculations on a dataset like this come from sometimes running tasks with math in text instead of in code, which is a really goofy thing because we know that all of these tools have been coding for many, many months at this point. But if you know, they are originally text based machines. And so I've seen so many cases in my students projects that they've submitted. And in my own work of saying, you know, calculate how many people said this and then seeing that the answer is different if you prompt it three times because it doesn't do it in an actual math equation.

A (40:31)

Yeah, they're notoriously bad at math unless you tell them code use coding fault.

B (40:36)

Yeah. So I'm hoping I can remove that. In my latest tests with all the models, I, you know, got really consistent results without having that in as well, but I still am a little anxious about taking it out, so I, I just leave it in all of my prompts. It's better. Better safe than sorry.

A (40:52)

Good takeaway for anybody watching. If you want to be safe on your math, tell it to code it.

B (40:56)

Yeah.

A (40:57)

What I love about quad these days is how much of the thinking is exposed and how much planning it's doing and how much step by step and how you can even click into some of those and just watch it work. I find myself just like getting fascinated with this and getting better at coding and stuff like that, but doing that.

B (41:12)

Yeah. And you don't really need to be a Python expert here to just see, you know, how it's thinking through things like team specific issues. It's already kind of providing an overarching code group here. Invoice reimbursement, primary, primarily a team issue and then, you know, logging. If you find any of these terms, then code it like that. Right. So this is how it's thinking through. I found these topics and this is how I'm going to start coding them.

A (41:38)

Seem to have brought a lot of the Claude code functionality breathe into the CLAUDE user interface. So if people have just been stuck in Claude code, the user interface version has gotten quite a bit better in the last couple months. People are curious. It's almost always going to import Pandas, which is like a really common data science library. So as you watch this think and learn, you get to learn a little bit more deeply too.

B (41:58)

Yeah, exactly.

A (41:59)

And so if you were doing this as a human, what would have been the process?

B (42:02)

More or less the same. I mean, again, what I'm showing you is a bit of a condensed process compared to what I would usually do. I have more steps involved. So one way I might have done this as a human or with AI as well is actually to break this prompt into multiple steps. So human wise. Right. I would be reading all of the responses. If we didn't have a code book already, if we didn't have a taxonomy or a set of tags that we wanted to apply, and we really were doing this from inductive open coding, I would read through all the data, or, you know, a sample of it at least, and start coming up with the labels that I think most apply from what I see. So that's this creating a code book, like what are the labels that I'm am identifying the codes? Why do I think that they should exist? What's the definition of them? How would we know if something else should fit into this Group or not giving some examples and then applying that code to other responses. You could even say in between these two, there's a step of testing it, like taking another sample from the data set and checking. Do these codes that I just came up with actually apply to this new sample? Or am I missing something? Right, so you're doing this kind of chunk, look at it, create code codes, apply it to the next chunk. Did that fit? Yes or no? Okay, iterate, keep going. This is really how I would have done things manually as well. So I've just laid out the steps that I would follow and I've given them to Claude. But here then is where things do get a little bit better. If you have a slightly more complex, stretched out process where you separate these steps into individual prompts. Like first going through the data and thinking about what the code book might be, and then I could give it another set of data and say, do they still fit for this, for example? And so on.

A (43:48)

Nice. Okay. And looks like it's got a response for us.

B (43:50)

It's got a response. I'm going to close this version because it has things down here hopefully. Yeah, it's still generating but so dominant turn driver is clear, 23.5%. What I actually am more interested in is this. So there's some quantification. I want to see the code. Let me see. Usually it gets me a code book.

A (44:10)

This is already so much more than I would have done as a pm. This is cool.

B (44:14)

So you've got a bit of the code book in here. And what's funny is it didn't give me a code book. Usually it gives me a codebook, but it has basically coded everything again. So I have a downloadable version of my CSV file with all of the codes applied to it. So every single line, every row is now coded. And then I have my code book here. I think this is the code book. Usually from this sort of prompt it gives me a codebook, it gives me the recoded CSV file and then it also gives me me it's kind of summary of the analysis. So that's what this is. What I actually don't need yet from my perspective is this full analysis. But I want the quantification, I want to see how it like first how it coded things, the codes that it came up with, to see do I agree or do I feel like it coded things in a funny way that I wanted to iterate on. And then I want to see the how, you know, how many responses it coded as each. Each of those codes. So missing favorites or save functionality because this fictional app didn't have a way to kind of save your favorite meditation sessions so you can return to them. This was a big chunk, so 23.5%. And yeah, that's, that's like a clear that rises to the top. And then we have some other things like switch to competitor content gaps didn't resonate. But to be clear, what's really important about the way that I structured this prompt is actually the order of things. I'm forcing it to go through the coding and the analysis in this order. I'm not saying, hey, provide frequency counts. I'm saying first look at the data, then create the codes, then apply them, and then decide what the frequency counts are. Because if you jump to give me some patterns, it hasn't coded anything or made sense of it in a way that, you know, we would agree with or make sense to us before it starts counting things. And that doesn't make sense. Makes sense.

A (46:04)

I think this is a more generalized advice for anybody watching this. Step by step prompting is how you get the AI to follow your human process. And that's our North Star here.

B (46:12)

Yeah. So let me jump into the next step. You can keep asking me questions about that if you want to, but just to keep things moving, copy pasting prompts. Does it?

A (46:22)

So yes, if the last step was coding.

B (46:26)

So this, I'm actually adding a little bit of a, of a step here for, for the survey analysis, because I want to compare this a little bit with, for example, sentiment analysis. A lot of people. For a lot of people, sentiment analysis is a natural thing to do on survey responses like NPS or churn. The problem with, with sentiment analysis is and, and even a prompt like this is that we are assuming that a large language model will understand sentiment and rate things in the way that we would. So first of all, with any, any rating task, like does this fit into, you know, negative, positive, or neutral sentiment, or you know, which buckets do customer statements fit into? We need to explain what we would rate things at or how we would label them. So that's what I'm doing here. But also with the CHURN survey, for example, there is a pretty strong likelihood that a large language model will decide that everything is negative. And that doesn't really help us. So instead of saying, hey, categorize or group things in by sentiment, I'm having it add intensity ratings so that we can make more sense of which of the problems that are causing turn are actually most worth us working on. As a product team. So if people are having a soft exit where they're circumstantial, like things in my life just change, like there's no hard feelings, I. I'll come back if things change back again. Well, this isn't necessarily something that we're probably going to want to fix. Whereas if something is really frustrated, if it's, you know, a group of people who very much fit in our ICP group, or they are high value customers and they're frustrated or angry, then these are probably the things we want to

A (48:09)

focus on and what's happening in this calibration example.

B (48:13)

So all of this is about how to rate things. What do we call a soft exit versus an angry exit versus, you know, a vocal detractor. What's the difference between frustrated and angry? This, it would be making a mistake if we assume that Claude or any of the large language models know the difference or interpret the difference the same way that we would. And this is where it gets a little bit subjective. But you know, as someone doing a lot of research, I have beliefs about some of these things or I have an idea about what an angry customer looks like from, from my domain. Right. Like from this Flow app or whatever I've been working on. I know the difference between angry and frustrated. Or you could kind of come up with your own definition. So I want it to follow my lead or the way that I've grouped people in the past so that I have comparable results, something like that. So it's really important to me that it doesn't just kind of come up with its own rating scales or ways of labeling things, but that it kind of follows, follows my lead and way of thinking.

A (49:12)

So when you're defining something, put some few shot examples so that it's very explicit and clear what that means for the LLM. That's what I'm taking away here. Is that right?

B (49:21)

Yeah, yeah. And what's actually really important about few shot is reasoning. Not just showing what the example looks like, but why as well.

A (49:31)

Ooh, okay. So giving it that reasoning so it can replicate you.

B (49:35)

So here today it's putting things into. Txt files, which is really funny because usually it outputs everything in the normal view. But here we have analysis here, but we see most people are clustered in these like level 2, 3 and 4 here there are far fewer angry exits than neutral and frustrated, which at least is a good thing. But we want to see how they line up, how they stack up with these different codes as well. Right. So this is kind of a Stacking of prompts, getting more specific or more detailed in our analysis or adding a layer to the analysis analysis. So we did coding and now we're determining the emotional strength of these different codes. So this is the point here. What I wanted to see was, okay, how frustrated are people who are missing favorites or save functionality, for example? Pretty frustrated. I mean, it's in the middle level three, but there are more level threes than anything else. So that means they're pretty frustrated. And that would say to me, hey, this is something we need to work on.

A (50:36)

Makes sense. Sense.

B (50:37)

But we have all of the codes ranked by intensity. We have a little bit of a visualization because Claude decided to go a little bit beyond the brief here. I didn't ask for any visualization, but hey, I'll take it.

A (50:49)

So overall, you're happy with these results?

B (50:51)

I am happy with these results because I can also check them. So because I had that exportable CSV that was recoded for me, I always ask for these kind of outputs at the end where I can can double check first of all, that from the interview analysis that the quotes are real. If it gives me, hey, this quote came from this participant on this page, or it was quote, number, whatever in my markdown file, then I can go back and I can make sure that that quote exists, that it came from the participant it said it did. And in this case when it quantifies things, but it's already given me the full set of coded recoded responses. I can go do my own math there too if I want.

C (51:31)

Awesome.

A (51:31)

Where do we go from here?

B (51:34)

So we still need an audit. That verification step, just like we did with the interview analysis, we need to check or we need AI to check its own work here.

A (51:48)

So it's going to select random responses, check for code consistency.

B (51:52)

Yeah.

A (51:53)

Okay.

B (51:53)

What's really important here is simply, first of all, simply that you do a step like this. Like I said before, we just need to force it to kind of do another pass through its own work. And it will usually find something that's a little off. Sometimes what it finds is some big calculation errors. It's rare if you have really good prompts and you push it through this step by step process. It's rare to find big calculation errors. Sometimes they're small though, sometimes it goes, hey, I would change the coding a little bit here because I think there's no another group that deserves another code that I kind of mixed up in some other groups. And then, you know, the quantification changes a little Bit So that's the sort of thing I want to see here because I want to stay as close to the data as possible. I don't want to exaggerate things. I don't want to blow up responses to super angry exits, you know, from the previous step if they weren't super angry exits. I want to be as realistic as possible. So it's, for example here it's rechecking level four because I asked it to specifically check level here for the intensity rating audit. Recheck all responses rated 4 or 5. We don't want to go kind of running around saying, oh, people are super, super angry and we have to fix this thing right away because they're all level four or five. We want to know is that true? Like should this actually be priority number one? Are they actually really angry or did we over exaggerate a little bit?

A (53:18)

I think this is a really interesting way to use AI. I'll call this the cya. UA to use AI Cover your ass asks what is anything that could come back to bite you when you're using AI in the project? Have AI check that out. I love this lesson.

B (53:31)

Oh, see why. Yeah, that's a good, that's a good name for it. I've heard so many people, it's less common with researchers because I think they're, they're feeling more cautious with AI and a bit more protective of this work naturally. But I've heard a lot of product managers I've worked with and friends and designers as well kind of say like, yeah, I came up with all of these, these really nice sounding insights. And then when I went digging into the data myself or I presented it and someone asked questions I couldn't answer and you know, they didn't do a step like this. And if they had, I bet you that a few of the issues, a few of the holes that could be poked in the insights would have been, you know, highlighted here.

A (54:13)

So if we review the three prompts so far, the first prompt was coding, the second prompt was quantification and the third prompt is verification. Is that the right summary?

B (54:21)

So the first prompt was coding and quantification in one. The second one was rating the emotional intensity, you could say. And the third is auditing the results of everything so far.

A (54:33)

Got it.

B (54:34)

So here you can see it's corrected things. It says survey final corrected at the top. So it's recoded potentially the primary codes and also the intensity levels here. Let me see. So corrections made the coding, eight corrections. So some things were missing. Oh, Missing Favorites was the old code and the new code is actually lack of control over selection. So it's gotten more specific about the code that it feels should be applied there. And because it's lifting this up and because it had recoded the whole thing originally, I can go through this myself. And I would, in a normal workflow scenario, I would go through and I would double check. Do I agree with these? Like, based on my understanding of the data, do I agree with these? And would I kind of label them the same way?

A (55:17)

And I think this is a really important step because it gave us a preview of the synthesis, even though we didn't ask. And its synthesis was like missing Favorites is like driving all the churn. And I was like, wow, like if that's true, then you can really quickly solve a lot of your churn problems. But going back and coding it, it realized, oh, there's some nuance here beyond

B (55:34)

just favorite and here, exactly as I expected, the intensity rating for at least two things was redone. So it kind of over exaggerated the frustration. Right. It dropped things down from angry to frustrated, which could be meaningful, especially if it found more issues like that. So now we have move got it

C (55:53)

to check its work.

A (55:54)

What is the next step?

B (55:55)

So from here, same as with interviews. I would just do a synthesis. Like I said. I would say wrap up the top, the top issues that are worth fixing. The. I would maybe ask for things per tier, like per segment based on the service tiers that they, they were using, like the people on individual premium accounts versus team accounts and things like that. Because those, those might be a little bit different. We see things like billing account admin issues or technical issues. Right. That might be across both of them. But billing an account would only be with team account users, for example, or proper invoice issues issues, things like that. So I might do a synthesis like that. But I think again, most people are decent enough at asking for a wrap up of the insights and not as good at pushing through this. So I would jump to the mega version of this, I would say, which is if it's okay with you that I skip synthesis. Again, I would judge.

A (56:50)

Yeah, no, I think people can imagine it on their own. Right. And I'd say like how a PM and a user researcher do it might be different too. And so that's, that's build it to your own use case. Like as a pm, I'd probably be saying like, you know, okay, draft up my first version of my product strategy document based on this. Like, what are the key Problems we need to solve across those. Please put one or two quotes from the specific survey and make it link back a bowl so that I can go trace it if needed. But a user researcher might have some different goals and so build the synthesis prompt for your goals.

B (57:24)

Exactly. Yeah. And all of these prompts can be modified for your goals. Right. I've done this in a particular way with, you know, retention and churn as the focus. But this can be used for any kind of analysis you're doing, any kind of data sets. It's really. It's supposed to be a flexible model. That's kind of why I developed it this way. But I developed it, like I said, based on the way that people like me have been doing analysis for decades anyway, so it's not. Not reinventing the wheel here, here.

A (57:51)

Awesome. So what does the agentic parallelized mega version of this look? It's somewhat scary, but awesome. Cloud code is here. You just install it with like a simple command. It's like NPM something. Something you might need to install npm. It's great. That'll have you use Homebrew to install that. So it can feel a little bit intimidating to install this. But once you have it installed, you just type in the word Claude and you're here.

B (58:14)

Yeah. So I'm not using anything like a special code viewer, like VS Code or anything. I. I usually use the terminal exactly as it is, especially for demo videos, because most people who aren't used to using the terminal will download it and look at it like this. So I want to kind of keep it consistent. What's worth knowing in the first place is you have slash commands and agents is what I'm going to be working with right now. So if I just look at my agents, I have a bunch set up already. I kind of add new ones, delete them, and reiterate them all the time. But what I'm going to be using right now are my Survey Analysts Analyzer Lite and my interview Analyzer Lite. And this is exactly what you just saw. I've pretty much set up the same workflows or mostly, but just to work a little bit more on autopilot. So I set up those agents beforehand. The only difference between that or the biggest difference between what you've seen me do step by step in the web UI versus this is I have programmed those agents with system prompts that refer to and retrieve information from more files. So instead of being in the web UI and just uploading things every time and having an individual prompt for everything, I actually have a context document that it's going to retrieve from instead of me prompting, hey, here's my context. And I can change that context over time. So I can kind of use this agent for different things and update the context. When our team, team information, project information, customer information, whatever it is, feature set changes, I can keep that updated, but it's kind of doing things a bit more automatically than my step by step process in the web ui.

A (60:02)

So just to summarize for folks, I heard three inputs. Your context document, which is similar to the context prompt. We saw your agent markdown file. In this case, she's got two, she's got a survey and an interviewer log light, agent markdown file and then your actual data responses.

B (60:19)

Yeah, and for some reason my pasted prompt wasn't showing for you. I really want you to see it. Give me a second. There we go. So what I'm saying here, this prompt, what you see is really simple compared to, to the other things. Right. So what I just said was I'm. I'm basically loading all of the steps automatically in the system prompt. I built in a little bit of extra kind of feedback loop type stuff in there as well. And then it's pulling from documents. So I don't need to break this up into individual prompts. I just send it one prompt to kick off using my pre programmed agents. I'm calling the Interview Analyzer Lite here. It's not a slash command. I just write it in and the Survey Analyzer Lite and I'm just basically telling it to refer to the markdown files here and I can hit enter and it will just in a moment tell me. Okay, it's starting that. So it's reading the markdown files that I just referred to for each agent. Oh, it's searching for them. This happens all the time. If my, if my paths. If I didn't give a file path here and I just name the markdown file, then it has to go hunting for them. Sometimes I. They write the full path wrong too. And then it has to go hunting for them. But the great thing about CLAUDE is it will troubleshoot that on its own. So you don't normally have to go, oh wait, go look over here, it's there. Instead it will just go find it in your files for you. So it found them.

A (61:54)

Sometimes it might ask you for like a permission or something. You usually just hit like two I think, and then it'll just like keep going forward.

B (62:01)

Yeah, I have something I don't necessarily Recommend to everyone. But you can see at the bottom this red tag text bypass permissions on. So you can actually type in a command to remove that request. Which means, unfortunately, if I were to go beyond this prompt, it can kind of do anything it wants on my computer and destroy all of my files. So I don't recommend that you do that every time, but for a demo like this, it's really helpful because it won't ask me every five seconds whether I really want it to do the next action or not.

A (62:28)

And I think it's called YOLO mode, right? You can you just type in like slash yellow and it'll do that.

B (62:33)

So you can see it's hunting through my files in my Dropbox projects. AI tests, Live demos. Akash. Right. I have these files set up here so you can see the paths and it's reading everything it needs to based on what I told it to do in those system prompts that you didn't see. And it says now I have the paths here for interviews and it's reading all of the files with the names here so I can can make sure that it is read the right interview files. What's really cool about this, I mean, the reason I'm showing this in the first place is you just saw me do these analysis processes where, you know, if I weren't talking so much in between the survey analysis and the interview analysis processes individually for that amount of data. And that number of prompts took about 20 minutes each, but that means you would spend 40 minutes on that, which is still remarkably fast if you felt that the that was complete with a short synthesis prompt afterward. And usually it takes me a little longer than that because I have an even more rigorous process with more steps and feedback loops built in, but that's still just really fast. This allows us to do both of those things in parallel. So because the interview analysis and the survey analysis here don't build on each other, they don't need to be sequential. So instead of saying I have one hour to analyze all of of this stuff, will I have enough time? Well, yes, you have enough time if you do it like this, because you just let your agents with a pretty solid process do both things at once and then you can triangulate on your own at the end or you can put that into AI and triangulate for you, but you're cutting out half the time by parallelizing this.

A (64:11)

And how did we parallelize it?

B (64:13)

So simply because I have agents set up that run the processes I went through manually in Claude in The Web ui, I can call them both here. So in this prompt where I say, number one, use Interview Analyzer Lite. This has all my prompts that I just showed you in Claude, plus a few extra feedback loops. And so it's basically saying, do that, what I did in Claude's Web ui, and then also do the survey version, but do them at the same time.

A (64:43)

Nice. Yeah, sometimes I did this, like I would open this in cursor and I'd have multiple terminal windows. But I think this is actually an even cleaner way.

B (64:50)

It is really clean. You can just follow the whole thing here too. There will be sections of the process where it says expand with command O or something like that. So you can really look at the whole process it's going through, just like you would in the Web ui and make sure that it's kind of following the instructions, doing things properly. And I've told it to output everything, like output the final analysis to markdown files as well. So I'll be able to see them in my files and kind of review them there. I also kind of like seeing how many tokens.

A (65:22)

Yeah, you always want to know how many tokens you're burning. While this is running, could we look into one of those agent markdown files maybe and start to teach people about what those look like?

B (65:31)

Yeah, gotta stop sharing that. So this is. I mean, this should look familiar if anyone really stops and looks at the prompts that I've used. So this is the interview methodology markdown, and I've got the product, the project context, and the product context and everything that was already in those prompts before. Right. So I'm just feeding it all of that background information in this file and then also the definitions of contradiction types, things like that. So all of those pieces of information where I'm trying to teach it to analyze or identify information or label it the way that I would. That's basically built in here as a sort of context load that it can retrieve from.

A (66:10)

And does an agent markdown file need any special formatting or instructions?

B (66:14)

I mean, it depends on what you're trying to do. There are ways of getting a little bit trickier about it, but generally it's still all grounded in being specific, being clear and specific, just the way that you would be in prompting. And for markdown files specifically, it's about structure. Right. So the reason we use markdown files is really to help the large language model retrieve information in structured, organized ways that will make it easier for it to do the processes faster and more accurately as well, okay.

A (66:50)

And Claude can help you create these. Any files. If you guys are nervous about them, you can put in some of these prompts and say, hey, create me a nice interview analyzer light and it'll work with you on it.

B (67:00)

Exactly. So let's see how Claude is doing over here. Still working. You can see it's running interview analyzer light here. So it says both agents are still analyzing the data. Let me wait for them to complete.

A (67:13)

Generally, do you find yourself using Claude code more or the UI more for this?

B (67:18)

These days it really depends on what I'm trying to do. But if I have everything based in my files, then I find Claude code in the terminal to just be a sort of more efficient way of doing things. I'm. Like I said earlier, I'm. I don't consider myself a super technical person person. I have a marketing and then design and then research background. So I'm not coming from coding. And I, once I got used to this, like the whole interface not being particularly, you know, designy, I, I started really appreciating just being able to use slash commands to kind of call things, start tasks with a couple of keystrokes. I think it's really interesting to automate these processes that I would normally do step by step manually, but just learn how to make it call the right information from different documents, or the fact that it can find the right information within my documents without me having to go, oh, where did I put that? Or even saying, do we have, you know, for repository cases where someone's saying, what do we already know about X? Well, if you have Claude code and it's actually synced to all of your company files, your project files, then it can just go figure that out for you. You need to know how to prompt really well. You need to be clear and explicit and you need to know how to structure documents in a way that makes it easier for your large language model to call all of them. But you know, if you figure that out, then a process like this goes really, really fast and much easier. Easier. So interview analysis is complete here. Survey analysis is complete and it has a few quick findings. But what's more important is that it has output things to a file. So I'm going to stop the screen share here and I'm going to go over to where I hope it saved it. So here's our Meditation app Premium user retention analysis. There's an executive summary in here because of my instructions. So we have how many stable subscribers, moderate risk subscribers there are, and so on from the Interview analysis. Right, so this is my interview analysis results, Markdown file phase one data confirmation phase two individual retention analysis. So there's a little bit more in here, but it's basically doing the same thing. We have the value anchors analysis and the results here. This particular therapy chat feature directly addresses a gap that traditional therapy couldn't fill. And so on one we've got quotes and timestamps from this participant. So it's done exactly what it did when I was doing this step by step in the browser. UI per participant, pulling out everything. Then we've got P2, Participant 02 and all of the value anchors from this person. It's all of this, the output you've already seen but just now in a markdown file. So what I usually do from here is I kind of format this differently, put it in pages or Google Docs or something and make it look a little bit nicer and then I'll pluck out things that kind of stood out to me. It has, you know, that executive summary at the top. So most of the time that's pretty ready. I can kind of copy paste that and share it with someone, but it has all of the details that I would need if I want to review this on my own as well. Like you saw in, in the browser and even some tables.

A (70:39)

Epic.

C (70:39)

Sick.

A (70:40)

I mean this is a sick workflow.

B (70:42)

So can I have another? Yeah, sorry.

A (70:44)

Oh, go ahead.

B (70:46)

I have another file like this for the survey analysis too, just to be clear.

A (70:50)

Cool. So if I were to summarize for you guys what we covered today, we gave you a masterclass in how to do discovery with AI. Discovery is the most important skill for building great products. And we've covered examples for how to do that both for surveys and for interviews. And we've showed you how to do that with Claude step by step using multi step proctoring or to use Claude code and build an agent to do it. Now you have all the tools. It's up to you to go apply this, see where you get stuck. Drop in a comment below on this video or find Kaitlyn online on LinkedIn or somewhere else, maybe take her course to learn deeper and really embrace this new way. Because the best products are built with the best user understanding. And this is your roadmap. Kaitlyn, thank you so much for sharing all these valuable insights for free.

B (71:41)

It was my pleasure. Thanks for having me. See you later everyone.

A (71:44)

I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube. All these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content in the production to get you better insights to stay ahead in your career. Finally, do check out my bundle@buildle.akashti.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode. Citizen.