← The Growth Podcast
Loading summary
Transcript95 lines
- [00:00]
A
Is there a risk that people jump too quickly into the solution space and don't adequately investigate the problem and solidify that they're solving the right problem with these tools?
- [00:10]
B
Before Vibe coding, you know, solution space thinking is very prevalent. People start talking about, we should build this feature, we should build this product. Sales teams are asking for features, stakeholders are asking for features, clients are asking for features. All that discussion is actually in the solution space. A lot of PMs are spending their time just managing the scrum process and they don't have time to do discovery. And now it's easier than ever to create solution, which is kind of interesting. If it's like, hey, if designing it and coding it is no longer the bottleneck, then it's like, well, what text you put in is the only thing that makes any difference.
- [00:41]
A
What specifically has AI changed? This feels pretty timeless.
- [00:44]
B
Yeah, it is pretty timeless. I mean, you know, at the end of the day, you still have to understand your customers and AI is not going to tell you, you know, about your customers.
- [00:52]
A
Dan Olson, I have been following your work for over 15 years. What has changed in the past 10 years?
- [00:58]
B
I think really two main things have changed. One is the other big thing.
- [01:02]
A
How has AI changed product management?
- [01:05]
B
AI has pretty much changed every step in the product management process. It can help you explore new opportunities that are out there in the market. It can help you identify segments, it can help you brainstorm feature ideas. It can help you do competitive analysis. But probably the most fundamental area it's changed the most is you are a
- [01:22]
A
legend in the product management community for your work on the Lean Product playbook. What was the thesis?
- [01:27]
B
Yeah, the thesis was basically there was this term
- [01:31]
A
really quickly. I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode. Dan Olson, I have been following your work for over 15 years. You are a legend in the product management community for your work on the Lean Product playbook. For those of us who have forgotten in the last 10 years what that was about. What was the thesis?
- [02:05]
B
Yeah, the thesis was basically that there was this term product market fit, you know, defined by Mark Andreessen back in 2007, but there was no, like, rigorous framework or systematic process for how to do it. And through my consulting and speaking, I realized that I could create that process and that's the Lean product process, which is a simple six step process that you know, thousands of companies have followed at this point to try to achieve product market fit. Yeah, so this is the six step process. Basically you start with defining your target customer. You figure out what are their underserved needs. Those two layers form the market part of product market fit. And then from there you define your product's value proposition. You figure out what your feature set should be. That's where MVP thinking comes in. So you don't over scope it, you figure out your ux, you design the ux, you prototype it and then you go and test it with customers to see how you're doing with product market fit. And you iterate through a learning loop to try to achieve product market fit or decide if you have to pivot or worst case, you know, you pull the plug. But that's basically the process at a high level for how does she pro market fit.
- [03:06]
A
What has changed in the past 10 years?
- [03:09]
B
I think really two main things have changed. One is when the book came out, product management wasn't as well understood as it is now or valued. And you know, in the last 10 years it's really exploded. People that are way more product manager jobs, people appreciate what product managers do. And so there's more appreciation for the techniques and frameworks in the book and more and more companies are actually applying it. The other big thing which we're going to go deep in today is AI. Obviously AI we've been fortunate to live through, I've been fortunate to live through a few different disruptive waves of the Internet mobile and now we've got AI and so it definitely is impacting how teams are creating products and I'm excited to get into that with you.
- [03:44]
A
What specifically has AI changed? This feels pretty timeless.
- [03:48]
B
Yeah, it is pretty timeless. I mean, at the end of the day you still have to understand your customers and AI is not going to tell you about your customers. You still have to figure out how to segment your market. You have to figure out your Persona. It can help you brainstorm and come up with potential ideas and segmentation attributes. But at the end of the day, you still have to get out of the building and talk to people and to do your discovery research basically. So it can help you synthesize discovery research, but you've got to come up with it. And then the next thing is you need to, once you've identified the customer problems, you've got to prioritize which ones are the biggest opportunities. That's where my importance versus satisfaction framework comes in and an AI is not going to be able to do that for you. You're going to have to use judgment. And, you know, I played around with Gen AI, it's really good for brainstorming and creating convergent, divergent ideas, but it's not good as good when it comes time to evaluate, kind of converge and evaluate and prioritize ideas. And so that's where you still need to help. You still need to do that. And then your value prop. You know, I've tried a couple times to get ChatGPT to create a product strategy. They kind of sound good on paper, but when you scratch the surface, the substance isn't really there as much. So I feel like that's the case. And then defining your feature set, that's another place it can help, it can help you brainstorm features, right? You say, hey, I want to solve these problems for customers. It can come up with some really cool feature ideas. And then the biggest place though, is in that creating the ux, creating the prototype, and we'll get into the details of that. But with AI today, you can do it a lot quicker. And if you don't have a designer, you can actually do it now, where before you might not have been able to even do it at all.
- [05:18]
A
I think that really is the unlock, right, is with AI prototyping tools and with vibe coding tools, what used to be PM sitting in a dock, writing things up and then begging their designer for some resources or some time to do some explorations off of a napkin sketch. PMs are now more empowered than ever to go ahead and come up with some explorations in the solution space.
- [05:44]
B
Yes, it's true. PMs really couldn't do a whole lot without designers. So let me walk you through the old flow. I learned early on in my product career how important UX design is. So in my book, I have a whole chapter on UX design, and I like to describe the different artifacts that people could use and like in a preferred order. So I like to categorize them by interactivity and fidelity. As you can see here, we usually start with a product brief or a prd, some textual document, and then we start doing hand sketches with our teams. We iterate those until we're happy with them, and then we can move on to clickable wireframes. If we want low fidelity, they're low fidelity. They're usually black and white. A tool like Balsamiq, you can test with those. But really the best place in the old workflow to test was when you had clickable or tappable mockups, right? These days you use a tool like Figma to do that. In the old days you might use InVision with whatever assets your designer gave you. And, and I, you know, for a lot of my clients this is where the rubber met the road. We would do waves and waves of user test at this stage and then once we worked things through and iterated to the point where we had confidence in product market fit, then we would go code the product and of course we test the line product. But basically in order to do that flow, you needed to have someone who could prototype on your team. And that brings up something I've been talking about for a long, long time, which is the design gap on many teams. If we just take the high level activities that a product team needs to do to create a product and we say it's basically defining the product, which is mainly PM's job, designing the product, which typically falls in the designer and the develop. Then basically there's different levels of UX maturity within a team and different org structures that you see. The most common one and level one is basically, hey, we've got developers and that's it. There are a lot of startups that are like that. Actually a lot of AI startups these days are like that too because at the end of the day if you don't have coders, you're not going to build anything. So that's like the least UX savvy level. The next level up is, hey, in addition to engineering we have pm. But neither one of them is really a UX expert. Right? That's the idea you're gonna see. You can see when you see products from these kind of companies, you see the lack of UX design come through in their product. Now maybe, you know, one of these two can lean in and help try to cover some or most of that gap. Maybe the product manager, you know, has picked up some front end skills, some design skills, or maybe you have a front end coder who's picked up design, right? They might be able to get by and maybe between the two of them they can kind of COVID it. Frankly, in the early days of Intuit, where I started my product management career, that's how they got by. Before design was a thing, that's how they got by. The PMs and the engineers thought enough about UI to get it right that our product was easy to use. But the real thing you want to have obviously is the triad, the trio that we talk about. You've got Product management, engineering and ux, you can do that. So, you know, unless you really had level five, it was really hard to like rapidly create good prototypes. The cool thing is with Vibe coding, it's completely changed that one. If you don't have. If you're living with a design gap, you as a PM now can actually create those prototypes and you can create them quickly. And even if you had a designer, we know that designers are busy, they get there on multiple projects, so your project might not be a priority for them. You can now take it on yourself and get it done. And so you can get to a prototype more quickly, even if you have a designer. And so the new workflow is, you know, pretty. It's pretty interesting is you can just, you know, you're starting with text again, you're starting with text. You know, whether it's a product brief or prd, that's going to be the input into your Vibe coding tool. And now you're going to get a live prototype. You know, before Vive coding, people would create what we call HTML prototypes. It would be full on HTML, CSS, maybe a little JavaScript, but no backend if they wanted to really kind of have it be really high fidelity. Nowadays with Vibe coding tools, you can totally have a front end and you can even have a backend tool. And so you can get quickly to that live prototype to test your idea. And that's one of the key things about the lean product process, Akash, is you want to get through all those bottom layers and get to the prototype as quickly as you can. Because it's by putting that prototype in front of customers and getting feedback where the real learning and iteration towards product market fit happens.
- [09:34]
A
Yeah, I think that the biggest unlock for people was just testing those clickable prototypes that we were building, whether they were in Balsamiq or Figma. But now what we're seeing with AI prototyping tools like V0, we just had the CPO on the podcast, Lovable Bolt, we had the CEO on the podcast with those tools. Is that any of the designer, the pm, the engineer? Heck, I've been hearing stories about salespeople and marketers can create live prototypes.
- [10:08]
B
Totally. Yeah, it's true. It empowers everyone to do it. You know, we can talk about, you know, basically, especially if there's a different use cases too. Right. If you're creating a brand new concept, then it's great for that because you're starting from scratch. You have no existing design system, you have no existing code base. So you can create It. So it's great for those kind of conceptual new concept ideas. You know, obviously if you have a designer, the design is probably going to be better if a designer's involved versus not. These things will generate kind of plain vanilla ui and they are getting better and better. You know, I've used several of these. I use Lovable and Bolt. I was just, you know, yesterday, two days ago, I was at this brainstorming session with a client and the team had come up with this idea and I just, I wasn't, didn't even say anything to them. I just pulled out my laptop and pulled up Lovable and next thing I was like, here's a prototype. And they're all like, whoa. You know, so it's a great way to get your ideas out quickly, which is super exciting. Now there is a separate use case where it's like, okay, I've got an existing product. We have an, you know, so one. If you have design systems you need to live with, then you can, you know, you need to figure that out. Although I was just playing around with some of these tools and one of them actually let you kind of import your existing design system if you want. Wanted to, which was pretty cool, you know, so it's, it's very interesting. So it's interesting because the, the unit of work before, like at a high level, it was like product managers created text, right? Designers created figmas and then developers created code from the figmas. So the biggest design work product was a Figma board, right? Well, now, you know, that doesn't have to be the design product anymore. You can go straight from Text to a live prototype. You can still go to figma. The funny thing is to see the, see the workflows, you can still go to figma or back from figma if you want to. You know, it's kind of interesting that everyone's still plugging and playing with Figma for the most part. And then at the back end, when it's time to integrate with your code, if it's, if you have existing code, then that's a whole other thing. But just to illustrate and get to a prototype and test a product id, it's really fast and it's great. This episode is brought to you by work OS. If you're building a SaaS app, at
- [12:07]
A
some point your customers will start asking
- [12:09]
B
for enterprise features like SAML authentication and skim provisioning. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand so that you can ship quickly and get back to building other features. Today, hundreds of companies are Already powered by WorkOS, including ones you probably know like Vercel, Webflow and Loom workos. Also recently acquired warrant the fine grained authentication service. Warren's product is based on a groundbreaking
- [12:42]
A
authorization system called Zanzibar, which was originally
- [12:46]
B
designed for Google to power Google Docs and YouTube. This enables fast authorization checks at enormous scale while maintaining a flexible model that can be adapted to even the most complex use cases. If you're currently looking to build role based access control or other enterprise features like single sign on, skim or user management, you should consider Work OS. It's a drop in replacement for Auth0 and supports up to 1 million monthly active users for free. Check it out at workos.comlenny to learn more. That's workos.comlenny Today's episode is brought to
- [13:24]
A
you by Jira Product Discovery if you're like most product managers, you're probably in Jira tracking tickets and managing the backlog. But what about everything that happens before delivery? JIRA Product Discovery helps you move your discovery, prioritization and even roadmapping work out of spreadsheets and into a purpose built tool designed for product teams, capture insights, prioritize what matters, and create roadmaps you can easily tailor for any audience. And because it's built to work with Jira, everything stays connected for from idea to delivery. Used by product teams at Canva, Deliveroo and even the Economist. Check out why and try it for free today at atlassian.com product-discovery that's a T L A S S I-A N.com product-discovery Jira product discovery Build the Right Thing Is there a risk that people jump too quickly into the solution space and don't adequately investigate the problem and solidify that they're solving the right problem with these tools?
- [14:29]
B
I'm so glad you mentioned problem space. My heart is very happy right now. That's one of the things I'm very passionate about. And yeah, you're right. I mean before vibe coding, you know, solution space thinking is very prevalent. You know, people start talking about we should build this feature, we should build this product. Sales teams are asking for features. You know, stakeholders are asking for features, clients are asking for features, features. All that discussion is actually in the solution space. And so if we go back to that high level, define design, discover or develop, a lot of PMs are spending their time just managing the scrum process and they don't have time to do discovery. And so they end up just kind of building what people ask for and then it's kind of like ready, fire, aim. You never validated that it was a true customer problem. So I do think one is our solution. Space thinking was already prevalent and now it's easier than ever to create a solution which is kind of interesting. If it's like, hey, if designing it and coding it is no longer the bottleneck, then it's like, well, what text you put in is the only thing that makes any difference. Right. And how well thought out is who the customer is and what their problems are and what features we need in order to support that. So I do think it will lead towards solutions based thinking. The other thing I thought a lot about too is I think it's going to lead to a higher challenge for differentiation. Right. If everybody's using the same tools, you know, it's kind of like raising the bar. You're not going to see as many bad UX websites if everyone's using this because it kind of follows general principles. You're not going to get some amazing ui, you're going to get some plain vanilla ui. You can ask it to copy the UI of another website. So you can copy people, you can create plain vanilla stuff. But if you really want to innovate in ux, which we can talk about, go a little deeper in next if we want, you're still going to have to design, but basically, if anybody can create any idea, then what's the differentiation at that point? Right. And the only real differentiations are you're solving a problem, a real problem, that's an important problem. You're solving it in a way that's better. You have a data advantage. Right. That's the other thing people are talking about as they realize AI kind of, these tools are kind of becoming commoditized and the proprietary data that you have becomes more important as well. So. So I think you're going to see increasing, you know, challenge in differentiating one product from the other when anybody can create a product pretty easily. Yeah.
- [16:42]
A
The way I like to think about it is that AI is really good at pulling people up to the average. You can even see this in a platform like LinkedIn. Like a lot of the bad content that used to be out there, those people are plugging their content into chat, GPT or Claude and they're bringing it up to average.
- [16:57]
B
Yep.
- [16:58]
A
But you don't see those people winning. Right. The people who are winning are the people who are, you know, crafting it on their Own. It doesn't sound like the. What I like to. I think of ChatGPT text as like brown. You know, it's like the average of everybody's writing style, right? There's, there's no clear like red or blue or pink or like different style there that you're getting that pointed voice. Right. That's why everybody loves Marty Kagan's articles, because it's his voice. He's not writing with AI. And I think that with design, that's the new bar that we're setting with these AI prototyping tools, is that you need to create that differentiated design. So what is the right sequencing? You know, when should a PM be experimenting with prototyping tools and when should they bring the designer into the process? How do they make sure that that solution phase, which is now totally transformed, is done in a way that both PMs and designers will be happy?
- [17:55]
B
Yeah. Well, I'd love to pick up on what you said because the way I think about how, how is AI helping people, what is it doing? I think of a bell curve like you said, right. And basically what it's doing is people that were at the bottom of the bell curve. The floor of the bell curve is now a lot higher. The floor is rising basically, right? So the floor is lava. The floor is rising as Gen AI gets better and better. Right. And I love the quotes are always like, hey, this is the worst GPT you're ever going to use in your life. It just keeps getting better. So the floor, you're right. So the people that were like the bottom decile, now they're performing in the second decile or third decile or something, right? So it's like there's, you know, it's getting rid of that bad low end of the curve, which is related also to the whole speculation about, hey, what's it going to do for jobs? I remember about nine months ago, people like, or a year ago, oh, it's going to kill PM jobs. And in my head I'm like, I don't think it is. If I think about the jobs that are most at risk, I think it's the developers, the junior front end developers because they're just at the end of the day, code is text and if they're in the bottom decile. Right. Of junior developer coders and this stuff can crank out third decile, fourth decile stuff, then it's gonna replace that. And so that's it. And I think to answer your question about when to bring in a designer, let's get into the. I think this is where it's important to double click on UX design. So let me show you my framework. Like I mentioned when I got into pm, I realized that UX design is super important. I mean it's just really, really important. And so I made it a point to learn a lot about it. I, I could write, to put it in the parlance of reasoning here, I could write the best text ever, I could write the best requirements, document the best user stories. But if we fall down on the ux, I think about it like an iceberg. Because like an iceberg, there's a part that's above the water, that's visible, that's what most people see and react to and that's the visual design. That's the part that's above the water. You know, we're all looking at. That's why people love high fidelity prototypes or live coding things. Because it's real pixels, it's high fidelity. It's got the colors, it's got the fonts, it's got, got the images. But when you use a product that's got product market fit and it's got a great user experience and is easy to use, it's because the team has like taken into account and done a really good job with these more foundational layers below the waterline that aren't as obvious. Starting at the bottom. It's conceptual design. Like what's the conceptual design overall design for this, for this product? Right. A lot of E commerce things. That conceptual design is just a catalog plus a shopping cart. It's a very common conceptual design. Uber's conceptual design wasn't necessarily that. It was just map centric. It was like, you know, the thing about a conceptual design, you can design it, you could, you could describe it verbally. It's like, hey, let's show the user in the middle of the map. Let's show the cars on the map, where they're driving around and when the user hails a car, then let's show the car going to them. Right? There's a thousand different ways to implement that visual design, but that's the conceptual design. The next layer up is information architecture. And so information architecture, interaction design. These are the things that these cool coding tools are probably not going to do as well as, you know, they're not going to do as well as a top notch designer. For sure. Right. Over time they're going to get better, like we're saying. But information architecture is, you know, is the way you're structuring the information and organizing it makes sense. Like when you go to look for something, is it where you expect it to be? Does the navigation make sense? Do the labels make sense? And then interaction design is basically anything you can click or tap and the flows, right? And so that's the one of the things that comes to mind to me when I vibe code. Like if you don't tell it the flow, then it's probably not going to get the flow right. If you've got a multi screen, multi page or if you're designing one screen, there's not really much flow. But if you've got a multi step workflow or multiple feature product where they've got to navigate between it, you got to think about this. And that brings up something that's really interesting is like I always feel like, yes, you should start in the problem space. When you get to the solution space, it's so funny, you see this with all gen AI, you make a request in your prompt and it gives you something you didn't expect, like, oh, I forgot to say make the grass green or whatever. And then you edit it, right? And then like, oh, I forgot to say make the sky blue. And so it's funny, by going back and forth, by going to the solution space, it helps you go, oh, I forgot to, I forgot this one problem space requirement. So it can really be helpful to actually help you explore jointly the solution in problem space to get better requirements and better specs, if you will. So but to answer your point, you know, if you're building some straightforward pattern, you know, like some E commerce shopping flow, there are best practices out there. You might try to innovate. If you want to innovate, then you should get a designer. But if it's pretty, pretty straightforward or you're following an existing design system, that's where you can kind of stay in the guardrails and just kind of replicate what you already have in your product. But if you're really trying to create a better user experience, a better information architecture, better interaction design, or even a novel visual design, then that's when you want to get the human touch of like a 10x designer involved in that. So I think that, you know, theoretically, timing wise, a PM could go prototype the general functionality and flow with vibe coding, understanding that, hey, this is not, this is not anywhere near the final UI per se. It's just getting us the flow and the functionality down perhaps.
- [22:50]
A
So what I'm hearing is that there might be a reduced need to bring a product designer into every feature that's shed.
- [22:59]
B
I agree with you. I mean, right now, you know, the good news is again, like, just like product management has risen the appreciation that to develop a great product, you need a developer, a designer and a product manager. And ideally they're all very skilled in their domains and they are ideally collaborating very well together. That's a high expectation. What a function of that now is most teams do have a designer and to your point, the designer is involved with any UI creation. Right. And therefore they are potentially a bottleneck for any UI creation. And so I think kind of combining what we're talking about, perhaps to your point, the more plain, you know, trivial stuff can be done without them and we can reserve their time for the stuff that really does require more creative thinking, like creating a new flow or creating, you know, some, some cool new feature.
- [23:49]
A
I also think that there will probably need to be a bit of an evolution within the product design field where I think that it has been little bit, like you said, a bottleneck for what a team can create. And I think that this next iteration of product design, we're gonna see more and more that product designers have to be able to work with a simple feature and maybe build that out using prompt. So, you know, we see things like Figma make, right, where it takes your existing design system, they just prompt it out so that they can do that really quick. And then when they actually need to more deeply apply the double design, double diamond design process, they can do that for bigger features.
- [24:27]
B
Yeah, yeah. Well, it's interesting.
- [24:29]
A
Yeah.
- [24:29]
B
So Figma launch Figma make, right? So you can go straight from your figma. The funny thing is everybody else, there's a lot of people, Figma is kind of, that was the dominant player in the old workflow, right? And there's, that's not going to change overnight. And so a lot of these tools plug and play very well with figma. And Figma themselves are trying to innovate and make it, you know, make it be the case that like, the last thing they would want to see probably is designers having reasons to not be in figma that would be bad for their business. And so I think, and they have a robust plugins and extensions platform and things like that. So to the extent they can support these, like I'm saying, like, you know, when you double click and get deep into it, it's like, okay, we need to build a new feature, it's relatively straightforward, but we need it to comply with our design system. You know, and certain companies have really, you know, robust, well, Thought out nice design systems. Well, if a, if a new PM comes in and says I'm just going to create a Vibe coding prototype and it looks nothing like it, that's going to be a problem. Right? So, but like I said, you're already seeing people, people are recognizing this, the industry is recognizing that. Hey, and this is a broader case. Take a step back. You know, everyone's excited about Gen AI and when you play around with Gen AI, like I did the whole action figure thing that everybody was doing and I decided to do a Star wars figure because I'm a Star wars fan. And what I found was it was good, but when you wanted to change one thing, it would like change the rest of it somewhat. And I view it like rolling dice. I call it a reroll. Every time you make an edit, it would like reroll the whole thing and it might fix the one thing, but it's like whackable. It would mess something else out. Right. And so kind of a general philosophical concept that I'm pushing these days is now that we have Genai that's generating new stuff, we need a good edit ui. Edit AI. We need a good edit AI. Right? That's what we need because, and that's the thing is, hey, cool, honor my design system, but just generate this stuff in it. So you're seeing tools like being able to support that use case, recognizing that. Yeah, Vibe coding is cool when you have starting from scratch if you're a startup or a brand new product. But for these companies that have thousands of lines of existing code, robust design systems, how do we use those tools there? I think the support is getting there and I think that's where the designers hopefully design tooling ecosystem will also support that, which I think it will.
- [26:37]
A
So what are the specific tools you'd recommend for Vibe Prototyping?
- [26:42]
B
Yeah, I mean, I think, you know, it's interesting. The space is changing very quickly, but as of right now, there are kind of like the more hardcore, like if you are a coder, know how to code, there are tools, you know, like cursor that are in that kind of camp and there are other ones, like if you're more like, well, I don't really know how to code, I want to care more about prototyping. You know, there's things like Lovable and Bolt that are kind of live in that camp more. Right. And then there's still. And there's another category and those are great. Like I said, if you want to build a new product from scratch, you can put In a prompt, it'll create a new product from scratch. Everything's cool. The other thing is like imagine you're a pm, you don't have a designer and you need to modify or enhance an existing product or feature. You know, in the old days you might take a screenshot of it and then do what I call Frankenstein it, like hack it up, you know, like put in some weird, try to make it like look as good as you can. But you're adding button janky buttons and dropdowns like it gets the job done. But the cool thing now is to support that specific use case where instead of starting with text, you can start with a screen grab of an existing product and then it creates basically a Figma like art board from that, right? And so then you can go, all right, I'm going to change this text, I'm going to add a button here, I'm going to do this. And then you can just like with Figma, you can create a interactive prototype. So when somebody taps here, it goes here and goes there. So the tools in that specific category are Visily wizard, which got acquired by Miro Magic Patterns and like UX Pilot. Those, those like, they're kind of, you know, and you know, like Magic Patterns will actually generate code for you. The other ones will generate the art board for you. You can then export them to figma. So again, there's multiple workflows depending on where you want to take what they create, but they're focusing more on that use case of hey, I want to. You want to create an editable, you know, you don't have the original figma. If you had the original Figma file, you'd go, and that would mean you probably had a designer, you'd go and edit that. So if you kind of live in a Figma centric world, your workflow is probably still going to rely on figma centric when you're trying to extend or enhance existing functionality. But if you're a PM that doesn't have a designer needs to do that, you can use these tools. So there's kind of those three categories of tools, the hardcore vibe coding tools where you're really going to get in the code. There's the lighter weight ones when you're just trying to prototype, and then there's these other ones that focus on this, what I call reverse prototyping use case. You basically suck in a screenshot and then you can make it really easy to modify and edit it and create a clickable prototype from that.
- [29:04]
A
So if I had to summarize, what we talked about in terms of product management in the age of AI, Genai is changing most of the steps. But the step that's most consequential is the change in prototyping. And PMs need to learn how to use these tools, but they also need to learn within their organization how to collaborate with designers in a way that it doesn't feel like a stepping on their turf. Is that a fair summary?
- [29:25]
B
I think so. Yeah, I think so. And I think that I'm glad you brought up the turf thing. I mean, you know, at different companies, sometimes PMs and designers do have minor, you know, some degree of clash or kind of uncertainty about responsibility or overlap and ownership and things like that. So you're right. When you've got tools that can suddenly generate designs, then you want to be careful about that. And you know, I think good PMs, when you do have a designer you're working with and in the interest of time because they're busy or you just want to get your ideas out there, you would create a wireframe or a mock up or nowadays a vibe coding prototype. It's always with the intent that if it needs to go through a design process, this is just kind of, you know, directional and illustrative. And of course we would love to get your design expertise to make this even better and add some, you know, some truly unique components to it. So I think that's kind of the spirit in which PMs could be approaching it if they're working with designers,
- [30:19]
A
I think. Yeah, when you're working with designers, a lot of it is also just about being clear and upfront about your intentions. Right. I'm not trying to say this is pixel for pixel, what we should build, but like Dan was saying earlier, to explore the problem space effectively, I wanted to also explore the solution space that was giving me good signals. And so I've put together an AI prototype. But depending on where we see the impact of this feature and your resource availability, we could go through more design cycles on this or we could just do a minimal making sure it conforms to our design system.
- [30:58]
B
Yeah, I think. Yeah, exactly. That's the intent. That's the intent of doing it. And it's funny because the way I think about it too, going back to that lean product process, like, of course you should start off with discovery research, you should talk to customers to understand their needs and preferences and what they're using today and things like that, there's kind of like that gets you to a Certain level, right. There's diminishing returns. After you talk to your 20th customer, you're probably done seeing new patterns. So there's diminishing returns. That's as far as you can go with customer interviews. The next way to kind of really de risk and increase your confidence is with a clickable interactive prototype. And so that's why they're so valuable. Once you get to clickable valuable prototype and the more, the higher fidelity, more realistic it is, which by definition these vibe coding tools are creating stuff that's actually in a browser or stuff that's actually in a mobile app, then that's when the customer goes, oh yeah, I know I asked for X in discovery, but now that I'm going through the workflow, I forgot that you also need to be able to export this to Excel. Right. So it really unlocks the next level of customer validation that you can do. And to your point, I think there's value in doing that. If the PM can drive a lot of that and de risk it and iterate it without taking designs time, that's great. And but then again, to your point, then it's like, hey, we validated these problems with the customers. We validated this high level flow and functionality set with the customers. Now we'd love to work with you to make it a great UX and a great UI.
- [32:21]
A
You know, AI evals are one of the most important skills for PMs and I know you know, they matter. The question is, are you doing them right? Most teams are winging it with basic metrics and hoping for the best. Meanwhile, the teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought to you by the Aievals for Engineers and PM's course by Hamal Hussain and Shreya Shankar. This live Maven course will teach you the battle tested frameworks from Hamal and Shreya who are the engineers behind GitHub Copilot's evaluation system and 25 plus production AI implementations. Four weeks live instruction. Next cohort starts July 21st. Start shipping AI that actually works. Enroll@maven.com with my code AG Product Growth for over $800 off that's AG Pro duct gr o w t H Today's episode is brought to you by the AIPM certification on Maven run by Mikdad Jaffer who is a product leader at OpenAI. This is not your typical course. It's eight weeks of live cohort based learning with a leader at one of the top companies in tech OpenAI just doesn't stop shipping and this is your chance to learn how. Run along with product faculty and Mo Ali. The course has a 4.9 rating with 133 reviews. Former students come from companies like OpenAI, Shopify, Stripe, Google and Meta. The best part, your company can probably cover the cost. So if you want to get $500 off, use my code a a k a s h 25 and head to maven.comproduct-faculty that's m a v e n.com p r o-u c t dash f a c u l t Y and ideally you're bringing design in early in the process where they can help feedback whether they need to be involved in putting those AI prototypes in front of customers and if you have a UX research team, whether they can decide. So I think it's almost like before the PM goes off and does all these things, making sure that they have the right checkpoint to say, I'm going to be going, doing these things. Do you want to be involved or even lead this process potentially?
- [34:31]
B
Yeah, that's a great point. I mean we're kind of, we're kind of painting a picture where there's like limited design bandwidth and that's why we're kind of going down this path. If there, if that's not a good assumption, if there's adequate design bandwidth, then yeah, it'd be great to partner with UX on this. You know, I think what you'll see is the forward thinking designers are going to lean into all these tools and processes. So it might flip back where it's like we're going to go back to. Yeah, you know, who's better at creating these prototypes than a pm? A designer who's really good at vibe coding. So, you know, like if they embrace it, right. Some people are sticking their head in the sand, other people, they embrace it. You know, they could theoretically. And I, you know, now that we talk about it like this, I am confident in the next year, definitely two years there will be like power AI design tools for designers, you know what I mean, that are going to have like power features and stuff. The PMs are going to be like, well, I don't even want to deal with that. What do you mean? Padding and margin. I don't want to deal with all that, you know, you know, so, so I think that, that you might see a bounce back with, with forward thinking designers where the tools have emerged enough where people, you know, people can do that. The funny thing is, you know, we're kind of skipping over the other thing. Another trying to talk about is just like, I've always been a big fan and believer in wireframes because for me that was the quickest way to kind of get your ideas into a solution ui, like a low fidelity one, One, for understanding with your team and stakeholders. But two, so you could have initial high level discussion with customers with tools like Sketch and Figma, because that's the tool and most designers, they go straight to high fidelity. And so we're kind of missing a lot of times you miss that high level discussion about the flow and things like that. The funny thing is, some of these design tools I was talking about, not the lovables and things like that, but the other category of tools like the Visily and the wizard, some of them actually offer a wireframe remote where they actually deliberately stay low fidelity so that you can kind of work through those issues if you want.
- [36:27]
A
When should people stay deliberately low fidelity?
- [36:31]
B
I think that, you know, it's, it's interesting now because with vibe coding, it changes the calculus that it used to be before. It's like, hey, the presumption was it would take a lot of effort. And the cool thing is when you do a vibe coding, it's usually a pretty plain vanilla ui. It's not getting crazy with some crazy fonts and colors and stuff that are going to distract, right? So in a sense, they're kind of low fidelity from a visual design standpoint, but they have to be high fidelity kind of. They have. If you're going to put a flow in there, they have to be a flow. But what I tell people usually is like, if you're having the number one situation is one of the top situations is this happens a lot. Say your team is trying to figure out the scope of your MVP and there's one feature that's on the bubble. You know, half your team thinks it needs to be in there for the mvp. This is going to be a horrible MVP without it. The other half of your team thinks, now we can get by without it. And usually it's not the team. Usually the team's cool without it. It's some senior stakeholder pounding on the table saying, I can't believe you're not going to have this feature or sales. I can't believe you're going to ship without. You know, it's got to be horrible. So in those situations, that's when I like to, in the interest of time, quickly create a wireframe without the feature and then say to the stakeholders, Pounding on the table, Hey, I. You're so convinced this is a critical feature, I bet you that if we talk to 10 customers, you're going to, you're going to expect that like eight or more are going to complain about where the heck is this feature, right? And they'll usually say, yeah, you're right. I'm like, okay, cool. Tell you what, we'll do a quick wireframe, we'll go talk to 10 customers and if, if like say five or more complain, we'll be the first ones to say, you're right and we'll put it in. But can you agree that if like, you know, you know, not like only one or two complain that we'll be okay without it? And that's how you set up the rules to try to get progress without taking the hit on that. So wireframing in the past was a great solution to that. And then you prototype it without the feature and you see how many people go, hey, where's this feature? You know, you don't ask them if they want it, you see if they complain if it's not there. So that's the idea. Theoretically with Vibe coding, you could kind of get to that same point as well the other use cases. If you have like a very complicated flow, like a multi step workflow with many branches and conditionalities and things like that, you might want to kind of do some UI exercises with wireframes to kind of validate the flow and things like that. Right? The interesting thing, as I said that out loud, is if you have a lot of conditionality, a live prototype then might be able to handle that much better than, you know, a clickable prototype. Because a clickable prototype, you've got to kind of, you can't put like an if statement in there, right? You've got to click on this, it's going to go here, you're going to click on this, it's going to go there. But so, so I think that's typical uses of wireframe is. And also just to get the team on board. So I do think that live, live Vibe coding prototypes can replace some of that stuff for teams. But I do think it's important to kind of at least think through and that's, you know, we didn't get into. When you're inputting a Vibe coding tool, if you want to get a good UI or UX out of it, you need to specify those things. You need to be like, hey, this is going to be a five page flow. Page one should have these things, page two Three, four, five. Right. You need to tell it to flow. If it's going to guess the flow, it's probably not going to be great.
- [39:32]
A
Yeah. So I think that starting low fidelity with your thinking is still important. Still thinking about, okay, well, these are the screens that we want. These are the conditional F statements that we want. And potentially using a Vibe prototyping tool to exercise those low fidelity, you know, before we have to draw those boxes on napkins or draw them in Balsamiq. Vibe coding can get that to you pretty quick.
- [39:55]
B
Yeah.
- [39:56]
A
So it sounds like don't skip the low fidelity step just because we can go high fidelity. And then potentially, what I think people miss with the Vibe coding tools is always input a screenshot of your product. Always give it your design system so it looks like your product, but you don't need to jump to that right away.
- [40:14]
B
Right. Right. Yeah. I mean, the thing is, it's tough. So back to that iceberg. If you tell it to Vibe code something like, when I did it in like five minutes for that thing two days ago with that team, it of course, put in some, like, alert icons, notification icons that were red and this and that. So it kind of can't help itself from doing some kind of visual design in high fidelity. And then people can fix on it, fixate on that and say, oh, that's a cool feature. And so, you know, another general principle, it's not so much low fidelity as when you're designing something. Like if your page has different components on it or widgets on it, like, what is the goal or point of each of those widgets? Right. That's the key. Right. So if you're creating some, like, dashboard that's gonna have four widgets on it. If you were just to kind of leave that, like a slack variable for the Vibe coding, say, hey, I want a dashboard with four graphs. It's going to pick four random graphs.
- [41:07]
A
Right?
- [41:08]
B
But if you say, hey, I want a dashboard, four graphs, the graph one should really show user engagement over time. Graph two should show, you know, whatever, you know, so it understands the intent. This happens a lot where people just design things. And that can happen when there's not a good handoff between PM and design, where the designer is just like, kind of making stuff up for the design. So it's really important when you do UI design that you understand what problem is it's solving. What's the goal of each UI component? What. What's the goal of this page? What's the goal of this component on this page? You know, things like that. The other thing you can really help with is creating realistic copy. So like that's where it can really save you time as well. Like, you know, back in the old days, designers would put like lorem ipsum delorum, like just placeholder text. We got that. That ship sailed a long time ago. And, but you know, if you're a designer, you gotta put in some data. You know, those five coding tools are really good. Like I had to create a dashboard to track a VC startup portfolio. It created like one AI startup with a cool name. It created a health tech startup. It created, you know, created like multiple startups with good names in different spaces and, and just kind of auto generated some data that I didn't even tell it to do, which was pretty cool.
- [42:13]
A
Yeah, so you can use it for better placeholders as you're going higher fidelity over time. So one of the things we, we've basically narrowed in on here is that there are going to be a lot of PMs or even engineers at startups that are going to be getting more into the design and user research process. And so I want to educate them, talk to us a little bit about these timeless principles. How should people be doing user testing?
- [42:37]
B
Definitely. And that's what I'm on. Another trend I'm excited about, Akash, is I think in the last, you know, six, seven years, people have realized the importance of user research. You're seeing more UX research jobs, there's research conference, a lot more research conferences. The value of it, you know, companies like dovetail user testing is obviously huge, but it's really exciting. And for me, I was, I was very lucky. My first PM job at Intuit. Intuit valued market research so much that we actually had a PhD in market research on our Quicken team. And so I learned from her how to do best practices in qualitative research and survey design and all this jazz. And so it's cool to see a lot of people want to do it, but to your point, they may not know how to do it. The cool thing I will say at the beginning is it's one of the easiest things to do. It's just talking to other humans, you know. And I'll give you some tips on how to do it in a sec. But why are we doing it? So you can see here, this is my kind of learning or iteration loop that I like to use. A lot of people may be familiar with build, measure, learn from Lean Startup. That was cool because it got people realizing, hey, we need to iterate we're going to build, we're going to learn, and we're going to iterate. A couple of things I don't like about it, it starts with build. And at least until Vibe coding came along, that was a very expensive step, right? And if you got to go into production code, it's still an expensive step. So I like to start with hypothesize. That's what we do. We form hypotheses, we design ways to test those hypotheses. In the product world, they're often prototypes. And then we go out, we test it with customers. From the customer test, we gain learning, and then from that learning, we revise our hypotheses and then we go through the loop again, we revise our prototypes, you know, and we go around. And I've seen this work so well. If you do this methodically, you can really like, iterate your product and achieve higher and higher product market fit. And so part of the key component is once you do have the prototype, however you created it with your designer with one of these Vibe coding tools that once you have at the top of the process, it's time to go and close the loop with customers and go talk to customers. And so, you know, when you have your prototype and you want to test with people, there are fundamentally three high level different ways of testing it. The first is in person moderated, where you actually in person with the person next to them, you know, with the prototype on their device, the browser or their, you know, their mobile phone. That used to be very common, obviously with COVID and Zoom and things like that, and the convenience, it's much more common to do the second one, which is remote moderated, and this is remote moderated is totally fine, right? Back in the old days when the screens weren't as good, the tools weren't as good, the bandwidth wasn't as good as the technology can get in the way sometimes. But these days with Zoom, you know, and an online prototype, it's pretty easy to have the person share their screen and conduct the interview. And then finally there's remote unmoderated. What does this mean? This means that there's not an actual moderator there. So this is like a recorded session, like usertesting.com and the way this works is you get your prototype set up and then you ask people like to complete a series of steps or tasks and you might try to ask them for feedback after each step or things like that. But you kind of really need to put a lot of thought into that script because People may get stuck, they may get off script, they may, you know, they may follow it, they may not. So the advantage of remote unmoder is you can do it quickly. And I think that's another interesting area, Akash, where you're seeing AI may come in to try to like do that more and try to like automate insight gathering. You know, like if, hey, go take this web coding prototype, take this script, go, you know, here's a recruited set of users. Go do unmoderated with them, have them schedule it, have them run it. Synth, do text, you know, do video speech to text, do analysis on that and give me the data that we are not quite there on that. But that's a future thing I could see happening. But I will say that the advantage of that is kind of getting a lot of data quickly. The disadvantage is you can't ask why, you can't clarify. And so, you know, the moderator at the end of the day for me is much more valuable than unmoderated. But unmoderated can be good, especially for usability issues. If you know you've got some usability issue on a certain page but you're not sure why, you can point a bunch of people at that flow and try to figure out why. I like to talk to people in ways of five to eight. You know, there's a lot of old school usability research about that kind of a number. You know, even if you talk to 10 or 20, it's still not going to be like statistically significant. But after five or eight, you usually start to get diminishing returns. You've identified the main things that you're going to learn there. So that's how I like to do it. And then basically you take your prototype and you test it with people in ways of five to eight.
- [47:05]
A
So when should people be thinking about what's the flowchart for? If this then in person moderated, if this then remote moderated. How do you think about the pros and cons of these different options?
- [47:18]
B
Yeah, I think that the earlier in your process and the earlier you are in creating this new product, the more you should lean on moderated because you don't even know what questions to ask. You don't even know how to write the script. Right? This is a mistake. I see people making user research. It's this allure of, hey, instead of talking and doing a one hour interview with 10 users, that'll take me 10 hours, why don't I do a survey of 100 users? Sounds great, right? The math sounds great, but the problem is you're not going to get the same quality of data. You can't ask follow up questions, you can't see what's going on. Right. So I think in general when you're trying to achieve product market fit, moderated is much more valuable in the specific use cases of okay, we know we have a usability, we have a product that's live and we have a usability issue. Let's go do remote unmoderated to figure out what, how, where people are getting tripped up. That's one specific use case. You know, I guess you could theoretically, if you were late in the process, say you like work through a lot of stuff and for some reason you just wanted to get a lot more customer feedback quickly. Then you could do remote unmoderated on like a mature, you know, mature live prototype just to kind of get some, see if there's other feedback. Right. If you're worried, maybe if you're worried about browser compatibility, you wanted to test on a range of browsers or a range of devices that might be something like that. So the general theme there is kind of like, yeah, testing different conditions to understand what's up. So in general I'm a big fan of moderated and I would only use unmoderated in those kind of specific use cases.
- [48:52]
A
What do you lose in a remote unmoderated interview versus an in person?
- [48:57]
B
Well, you know, that's a great question Akash. And they used to, used to lose more because like I said before, like the tools weren't as good. So sometimes like the prototype was bigger than the screen so they had to scroll up and down, they couldn't see the button you'd get. Right. You want to enter technical issues like the connection, all those things are gone. Nowadays, you know, you just have a zoom and you're good to go. So the last thing you really lose is when I am sitting next to you. It's very subtle but when you get good at user research, when I'm sitting next to you above and beyond whatever I see on zoom, I can pick up on little subtle things like you might sigh or you might tap your hand or might be some other non verbal body language that's letting me know, oh, you really like this or oh, there's a problem or you're concerned or you're confused. I can kind of see where your eyeballs are going too. It's very subtle. But you know, it's like that extra little 10, 15% of information fidelity. It's not critical. It's not critical by any means, but if you're a talented researcher, that helps give you an edge and, you know, you can pick up on things a little bit.
- [49:58]
A
Yeah, I always go back to some of the best research I've done was when I was working on Fortnite. And we always brought people in to our campus to actually watch them play because you learn so much more, especially when it comes to a game like are they throwing the controller, are they rage quitting?
- [50:17]
B
Right.
- [50:17]
A
We don't want rage quitting. Or are they having so much fun they're at the bridge of their seat?
- [50:23]
B
Right, right.
- [50:23]
A
You might be able to understand that over zoom, but the amount of texture you get is like, I would say like 50%. Like you get a lot less texture there.
- [50:32]
B
I, yeah, texture is a good term for it. Two other thoughts I have. One is, I forgot to say, like, I, you know, I started my PM career, Intuit, which was like a pioneer in usability testing. We had like this, you know, back in the day, million dollar US it had like a one way mirror. It had like a camera on the person's face. They had a camera on the screen, you know, back in the day. So you like, you know, but it's funny because that's a little bit like a lab rat thing. So they're coming into your thing. The other thing that made me think about Akash, which we also did, which is also important, is are you watching them use the product in the natural environment, like in situ? Right? Yeah. And so the big thing that Intuit did was they would do Follow Me homes. Like back in the old days, it was like a box shrink wrap software you'd buy at the store. So Intuit employees like camp out at Fry's and Best Buy and they like, you know, when someone checked out, they're like, hey, I know this is kind of weird, but can I follow you home and watch you install Quicken and see and then you realize, you know, like, oh, their disk drives over here or the printer's over here and they have Internet issues. Like, you know what I mean? So that's the other thing you lose if you're. There's. It's funny, I guess with an in person moderator, there's in person moderated in the lab and there's in process in person moderated in the field.
- [51:35]
A
Yeah.
- [51:35]
B
And so I do think for certain product types in the field, observation is very relevant. I could totally say for Fortnite that getting closer to what it's like in the Real conditions would make sense.
- [51:45]
A
So really there's a hierarchy that we've evaluated. The best type of user research you can do is, you know, truly ethnographic. It's in person, in situation, in the field and there's in person moderated in your lab, then there's remote moderated and then there's remote unmoderated. And probably people are also talking about synthetic AI feedback at the very end. Right. So yeah, there's all of this feedback that you can get. And it sounds like the earlier your product is, the more you want to go up. So probably the earlier your product feature is, the earlier you are in the discovery process, the more you want to be higher up in that cycle and probably also the more important the feature. Right. If it's a big bet, then you want to get higher quality research.
- [52:27]
B
I would call it the way I qualify it is the degree of uncertainty or like the degree of confidence you have. You know, if you, if your product, if you're the market leader in a segment and you know the market super well and you know the feature set super well and you're just launching some new incremental enhancement and everybody on the team feels super confident about it, cool. You probably don't need much research, but if it's like, no, we're doing a new product for a new market that we don't really know, then that's where you want to do more discovery, you know, and more in person.
- [52:56]
A
So a lot of PMs, they'll be new to this. Like UXRs and designers have been doing this, they haven't been doing this. So can you walk us through what is the timeline of a user testing session?
- [53:05]
B
Yeah, I mean I. And that's, and that's the thing, my book's called a Playbook because it's intended to give people like follow these steps. Obviously the details are different depending on what your product or market area is. But I try to put a lot of practical tips that I, you know, learned along the way. And I mentioned I started, you know, doing user testing at Intuit where it was like this million dollar lab. I went straight from Intuit to like a early stage startup. They didn't have a lab or anything. And so I coined the term Ramen user testing because you don't need all the fancy stuff. It's basically like you, the user and their laptop or their device and then the prototype and that's basically it. Right. And so just to kind of give people a starter skeleton of what the timeline, how to run this thing, this is what I recommend. You know, I don't like to just jump right in the prototype for a few reasons. One is, you know, talking to someone to share your opinions about some new product you may or may not have used. Especially if it's a new product concept, it's not an interaction people have every day. It's kind of weird. And especially people that don't like to talk to strangers and stuff. You're like, hey, can you, you know, see, there is value in just getting, warming up the person, letting them know that you're a fellow human who's not gonna be, you know, rude to them and you're nice and you ask them about their day or talking to them about what's going on, just to build some rapport. The more rapport you can build, you're kind of investing in the gas tank for later. You're gonna get more returns out of it. Right? And then also, this is where I like to just start asking, hey, how do you get this need met today? What are you using? What have you for? Why do you use the one you use? What do you like about what you don't? I get hired by clients all the time to do to apply my lean product process. I almost always learn about some competitor they didn't know about in this discovery research at the beginning. Like, oh, we never heard of those people. Right. Or you also might find out that it's not like some other software competitor, but like, oh, yeah, I just do that with Excel or CSV Export or duct tape them. You know, it's like whatever the MVP substitute is. So then once we've done that, then I like to just spend, you know, whatever the appropriate amount of time is. Usually at least 30 minutes, maybe more if it's a big robust prototype where I show them the mock upper product. Right. And then the thing is, this is a mistake. I see, you know, some, some PMs do that aren't as familiar with this is they try to guide the user through the thing. It's like, okay, first I want you to register and then they register. Okay, now I want you to go try to book a hotel now. You know what I mean? Like, they're kind of like directing the user. And the thing that occurs to me like that is not the real world. The real world is the person's going to be sitting there with your product in the browser on their phone and they have to figure it out.
- [55:38]
A
Yeah.
- [55:39]
B
So I like to be as real. I just shut up. I basically like, hey, here's our new product. I'll pull up the prototype on the laptop. I'm like, you know, I'd love to. You know, I'd love to see you, you know, interact with it. And then what I do is I give guidance. I'm like, hey, you know, like most people, you know, I call it think aloud protocol. It's on the next slide. But basically, most people are not used to sharing their thoughts out loud with a stranger when they're looking at some weird new product concept, right? So you have to be like, hey, share your ideas. And so I'll be sitting there quietly, and sometimes it's very. It's like this awkward silence. I'm like, so do you want me to register? And then I just give my pat line. I'm like, do whatever you would do if you're home with your. With by yourself with this product on your computer. Okay, so. So I'll register that. And I'm like, do whatever you do. And they realize, okay, it's not going to be a test or I'm going to guide them through the rat maze. They got to figure it out. And so they'll keep going. So you may be wondering, well, shoot, Dan, you sit there quiet the whole test. No, what I do is I wait to observe something noteworthy to pounce on. And that's where being in person, that texture, you said that's where you might pick up some sigh or some little thing where you can tell they're, like, struggling. Obviously, you can tell if their mouse is looking around. They're looking around. They can't find where to go. So I let them, you know, I let them flail for a little bit. Then I'll be like, hey, can you tell me what's going on? They'll be like, well, you know, I'm trying to find where I can sort by blah, blah, blah. And then I go, oh, okay, you know, can you tell me about that? So I jump in and we talk. So I don't know a priori ahead of time what we're going to talk about, what's going to come up in the user session. And we'll share in a minute if we have time here, a framework for how to capture this information systematically. But when it comes up, we go deep dive on those things. And then we kind of go, now if at the end. And so then I don't break character. I want to be at like. Like I'm not even there, right? And I ask questions now at the end, the very end, the wrap, it's important to do a wrap up. You know, they might even be like, so I was trying to click here. It doesn't work. Is this broken? You know, I'll be like, sorry, I can't answer that. Hopefully the engineers are watching it. If it's a live product, you want the engineers there too, so they feel the pain. But at the end I'll be like, hey, you know when you're trying to do that thing? Yeah, that's our bad, that's a bug. Sorry about that. And then if for some reason, like there was a key screen or page they didn't get to, I'd be like, hey, I want to just let me orient you to this page. Here you go. Can you please give me your feedback? Right? Just to make sure you get that. And then at the end, I found it really important to basically ask people, you know, how likely would you be to use this product? Because I learned the hard way that like usability, feedback, usability, poor usability can prevent you from having product market fit. But just because you have good usability does not mean you have product market fit. They are orthogonal at the end of the day, one can block the other, but they're orthogonal. Because I was working on a product and, you know, I tried to do my best on UX design. We did the V1, we launched it, ran some user tests. There were some UX issues, there were some bugs, there were some like, we need more instructions, we need examples, some copy needed to be fixed. So after. And we were really fast startup. So after about the 15th Test, we had fixed all the questions and complaints that had been brought up. So Starting about the 15th user test, people were just getting through the new user flow. They were getting to the main page, no issues, no complaints. So I was feeling pretty good. I'm like, all right, cool, so how likely would you be to use this? And to my surprise, like about 20% of people said I would never use this product, even though they had not complained, issued a single complaint, right? And what it turns out is there were different mental models for how to solve this problem that I didn't know about. And so once I discovered that, then now in my discovery research going forward, ask people, hey, can you tell me how you like to get your news? And I realized, hey, there's three different ways people like to get their news. And we had kind of like subconsciously designed it for the way me and my co founder like to do the news, right? And then we realized it's not going to be the case. So it's a good reminder, you know, that you need the usability. If you know what I didn't say about the pyramid is any one of those layers, if it's off, really off, it's going to prevent you from achieving product market fit. They don't have to be perfect, but if you have a bad ux, it's going to do that. So that's the thing.
- [59:28]
A
What are the do's and don'ts of these interviews?
- [59:31]
B
Some do's and don'ts, you know. You know, the other thing is most people remember what their mom told them. If you can't say something nice, don't say anything at all. Like, they're very nice. And, and so they're very positive about your prototype. Like, yeah, this is great. Oh, this is great, great, great. So what you need to do is like, let them know and I can do this as a consultant, like, hey, the whole reason we're getting feedback is we want to know, figure out how can we make this product better for you and other customers. So if we get through this whole test and you haven't told us anything we can make better, that actually doesn't help us out. So you're helping us by telling us how we can make this better for you. And by the way, I don't even work at the company, I'm a consultant, so you're not going to hurt my feelings. Like, please don't worry about hurting our feelings by telling us constructive criticism. You're actually helping us. And then I mentioned the think aloud protocol. So most people, I like to prime them at the beginning and say, hey, I know this is not typical, but as you're going through the prototype, could you just try to verbalize the thoughts that are going through your mind? And you usually have to remind somebody because they start just quietly clicking and doing stuff like, hey, can you let me know what's going through your mind as you go? So that's the idea. And then it's really important to take notes because there's so much. If you were taking notes and paying attention, there's so much information you can extract even from one user interview. And different people are gonna see different things. And that's why it's also important to do like a review right away. Like, you might line up, like, I like to, like, line up a power day. Like three to five user interviews in a day. We go through them all and then we have a team meeting at the end and we debrief and different people see different things. Now if you're not A super experienced moderator. It might be very overwhelming to try to moderate the test and take your own notes. So it's perfectly fine to have a separate note taker if that's kind of the situation they're in.
- [61:09]
A
So those are the do's, what are the don'ts? What are the big mistakes people are making in user interviews?
- [61:15]
B
One is that they try to help the user get through the user test and have a good experience. Right. And I understand if you're the pm, this is your baby. You're kind of secretly rooting for the user, but you know, again, your product needs to stand on its own. So you want to be a scientist, an objective realist and try to detach yourself. I've seen some PMs that go as far as like, put their hand on the person's hand on the mouse, move the mouse and click the button for them like it doesn't count. Like, you know, if you're doing it for them, it doesn't count. You obviously don't want to get defensive or blame the user. That usually doesn't happen, but every once in a while it does. And then the biggest thing is actually how you ask your questions. And so like, you know, at the end of the day, if you're, if you're a PM or a developer designer and you're like, you know, I really want to do X design, but I'm just feeling nervous about it or not confident, I encourage you to try. It's one of the easiest things to do in the product world. It really involves active listening, right? You just let the person talk. You can reflect back what you heard to probe more deeply, ask why, you know, why is that, things like that. But the biggest thing is how you ask your questions and the difference between leading or closed ended questions. So if I say, hey, that was easy to use, wasn't it? That's an example of a leading question. It sounds like a question, but it's not really a question. I'm putting the, I'm really pressuring the person to give a desired answer, which is yes. So you definitely want to avoid leading questions. The other thing is not as bad as a closed ended question. If instead I said, did you like that feature, what are they going to say? They're probably going to say yes or no. I'm secretly hoping they please say yes, please say yes, you know, because I'm rooting for the thing. But say they say yes. What did I actually learn? I didn't learn why they liked it or if they said no, I Didn't know why they liked it right now in normal humor conversation, human conversation, yes or no questions are great, but you want to avoid those kind of questions. And basically it all comes down to how you start your questions. If you start your questions with do you, did you does that, you're going to get a yes or no answer, are you? Was that right? So it's just a little Jedi mind trick to catch yourself and start using more open ended things like how, what, why things like that. You'll get a lot more richer data from your customer interviews.
- [63:15]
A
Brilliant. And how do you capture all these notes, how do you capture all this feedback in a systematic way?
- [63:21]
B
And that's, that's something that people struggle with. And so what I'll share here is how I it literally how I do it. When you do these tests, I like to categorize them in three high level buckets. You're gonna get feedback on your features header functionality. Like if they're missing some functionality or this functionality doesn't work the way they want it to, or it's missing some key aspect, you're gonna get feedback on the UX design, that iceberg that we talked about. And you're also gonna get feedback on messaging, right? If words don't make sense to people or it means one thing to you but something else to them, you're gonna get feedback. And so I start out with a blank slate like this and I don't know what in the rows each row is going to be kind of a material observation that I have. I don't know ahead of time what it's going to be. And then at the end, I like that at the end of the interviews I like to just ask, hey, on a scale of 1 to 10, how valuable do you find this product? And on a scale of 1 to 10, how easy to use was it? And you know that this is what I call semiquant or quant on qual. We don't have a huge sample size, but we can kind of get some indication and we can see if there are trends over time. And so I show, I run my first user test with my first user, right? And basically there this is what I learned. User 1 they said, hey, feature X, that was in your prototype, they thought that was valuable. Awesome. But they complained that we were missing key feature Y. So we note those things down. And by the way, I use a little plus or minus for positive bullets and negative bullets. They we noticed that they had trouble registering. So we note that as a UX deficiency and they happen to mention they like the hero image on our homepage. So we put that in the messaging and we asked them to score value and ease of use. We got seven and five right? We go to the next user, we move on to the next user and they bring up the same two feature set things that the first user did. So we're getting signal there. They didn't say anything. They didn't have as much problem with registration, but they didn't see the signup link and they happened to comment that they thought our design looked professional. So that's a positive. In the messaging they said they didn't understand our tagline. We got a 7 and 7 on value and ease of use, right? We go through that, we keep going through in this wave, I'll keep it at 5 to keep it simple. And as we talk to more users, we don't get new items coming up, we just get additional, you know, people mentioning or not mentioning the problems that we've seen to date. And then when we get to the end of the wave we can do this thing. I'm talking about semiquants. We can say, okay, what percentage of users brought up each issue? And now we can use this to take a step back and say, okay, how do we need to iterate our feature set, our problem space, our UX prototype to address this. So in this case, obviously we're going to have to add feature Y, right? Maybe that was the debate the team had, like, hey, do we need feature Y or not? Okay, looks like we need to, we need it after all. We need to fix reg. If 60% of people having trouble with it, we need to fix that sign up link is invisible, you know, and the tagline, you know you can prioritize based on this, right? And so that's what you do. So that's so we can collapse those five user tests into the feedback from a wave and then we go back and we basically say, okay, let's go fix all those things. And with vibe coding it'll be even quicker to have a new prototype that you can now go back. And then now I'm just gonna show you wave by wave. So in each of these waves we're talking to five date users, we're capturing the issues and what percentage of people have it. So this is how, this is the summary of the problems from wave one. We'd love to see all those good as zero percent in wave two. Let's see how we did. We won. We run wave two and cool feature Y now we added it. No One's complaining, it's missing. That's great. Makes sense. Means we nailed that check off. Good job, team. However, now we're getting new feedback that, hey, X and Y aren't working together the way they should. But we didn't think about that. We just added Y. We didn't think about how it works with X. And so that's a new issue that we need to address. It looks like we fixed the usability issues with registration. That's awesome. Sign up link. You know, we went from 60 to 40. Maybe that's some little progress. Maybe it's just noise. We still haven't really solved that. And now, yes, we did add feature Y, but now we're getting new feedback that it's actually hard to use. It's too hard to use. And it looks like we fixed our tagline issue. We got a slight increase in value and ease of use. Seven to eight and five to six, you know, we go to the next wave and then, cool, we made X and Y work together now we made progress there. We finally fixed the signup link. Looks like we might have made some progress on feature Y usability, but not enough. And, you know, now our value in easy view scores up 97. And then we do one last round and now we've kind of solved all those issues. That's kind of what that iteration through that loop looks like and how you kind of can systematically capture feedback and in a way that your team can say, cool, let's address these items. Let's see how we did going to the next round. As you're iterating rounds around and if you get like all greens like on wave four and you're getting high value EZV scores, you should be feeling really good that, hey, we've really validated product market fit and now we can either launch this thing or take it to production code.
- [67:55]
A
This is the work that PM's designers and user researchers really need to be doing right. This is how you ensure that your features aren't just bloating up your product with more unnecessary stuff that people don't need. This is how you ensure that you're building a minimum lovable product.
- [68:13]
B
It is, it is. And you know, what's interesting is a lot of places they value prototypes and they value getting user feedback, but the team struggles with, what do we do with this view? We've got pages and pages of notes, we've got tons and tons of videos of these sessions. How do we turn the corner, how do we synthesize it and make it actionable? How do we iterate from here? And that's why I kind of like this simple high level framework for teams to kind of be able to do that.
- [68:38]
A
So we've zoomed in on all of these changes that AI has created to product management. I want to reflect on product management a little bit. Have PMs become glorified Jira jockeys? Has the role gone off track?
- [68:53]
B
This is a complicated question. The short answer I'll say is mostly no. There are definitely some customer companies where that is the case. There are definitely some companies where that is the case. And PM is basically a waiter taking orders from stakeholders and customers and just turning around and feeding them to Dev and jira. So that's they're definitely Jira jockey places like that. But that is, you know, not what you're seeing in the best companies out there. The other thing that you can get into is like you spend a lot of time in scrum, right? You're spending a lot of time writing the stories, refining the stories, you know, talking to developers about the stories in all the scrum ceremonies, right? And you might be on more than one team. So you're going to two sets of ceremonies or three sets of ceremonies, right? You're working with design, you're probably doing some QA. QA is not sexy. So who does it? PMs, right? Status updates, updating stakeholders, all this stuff making slides, right? So you can get so sucked into that Scrum apparatus that you know, you're wholly consumed by develop and you don't have any time for discovery or defining and actually doing your job. So and I hate to say in the best orgs though, that doesn't happen, right? So again, if we look at the bell curve, right, the top quarter are not operating that way, the bottom quarter operating that way. And the main factor, I always like to kind of figure out why, diagnose why. And it's not always just this, but it's usually the staffing ratio. And if you just want to look at one metric, it's like how many on average, how many devs do you have? How many PMs do you have? Divide the two or even on a team, this PM has 10. I was just at a company where PM had like 12, 14, 12 or 14 devs. I'm like that poor PM. They're not going to have time to do any discovery or do anything but be a JIR jockey, a scrum jockey, because that's all they have time to do, right?
- [70:35]
A
Yes, I think we have been overburdening PMs where product leaders almost take it as a point of pride. I've kept my PM. Org lean. We have a 14 to 1 developer ratio. But we made developers so frickin productive with Cursor and Windsurf. Now they're turning out stuff so fast, if a PM is supporting 15 developers, they almost inevitably become a jira jockey.
- [71:01]
B
Yeah, and that's why the ratios are kind of messed up. And this actually, this predates my book. You know, I've been speaking to product managers since 2006. And so this is my original framework. I call it the four Ds. Discover, define, design, Develop. Right? And I won't go through the questions. You can see the questions that the key questions you're trying to answer and de risk in each of those phases. But what we're talking about here is a situation where because of staffing, ratios and other things, company culture, not understanding the role of PM, they are spending like over 90% of their time in that develop, that final column, maybe a little bit working with design, but they, as a result they are starving themselves and not spending time on Discover and Define, which at the end of the day is where product managers add the most value. Right? Developers are going to develop, designers are going to design. If you're not out there understanding the customer's a problem, then no one else on the team is going to be. And so it's a problem when PMs don't have time to do that. That's how you end up with a lot of products that are just feature factories that don't solve real customer problem.
- [71:56]
A
What's a product trend? Right now that's total bs.
- [72:01]
B
I think this happens anytime there's some hot new, hot new technology. And so right now I call it like, hey, sprinkle some AI on your product. I call it that. Right. It's like people and I think that, you know, a year ago everyone was like just trying to jam AI in their product, adding copilots all over the place, just without any rhyme reason. I think it's starting to settle. There's still some of that going on, it's slowing down a bit. But at the end of the day, going back to our discussion about problem space, solution space, AI is a solution, right? Just like, you know, crypto a solution, blockchain's a solution. These are all, you know, virtual reality, augmented reality, those are solutions. And so the problem that can happen, especially when there's a new big sexy one, is it's like a hammer looking for A nail. It's like, how can we AI this thing? And, and, and, and 1. A lot of product managers can't figure out how to AI it because you're trying to force a square peg in a round hole. The other thing is you're not starting with the customer problem. So even if you do manage to sprinkle some dust on there or integrate it somehow, it's not going to go. And I think we're seeing a lot of failure of those early attempts at trying to just bolt on AI to existing products. Whereas now you're starting to see people starting from what problems can it really solve? Like, a lot of these vibe coding tools are solving real problems, or vibe prototyping tools are solving real problems.
- [73:13]
A
So a lot of people will look at you, and a lot of the creators tend to have the highest retention on these podcasts and they'll be wondering, well, Dan, he's a really successful book author, but I'm not sure. Is that really the business of Dan? When you break down the pie chart of your revenue and how you're making money these days, what does that look like?
- [73:34]
B
Well, it's kind of funny when people think that a book makes you tons of money. So if you publish with a publisher, it's kind of like signing with Hollywood or being Taylor Swift with your records or something like that. You make some money. But that is not the main pie slice of my. Of my thing at all. I mainly spend time doing a few things. One is training workshops, right? So as PM has exploded and exploded in the last eight years, companies have said, okay, we understand the importance of PM. We hired a bunch of PMs. We want to train them now. And there aren't a lot of great resources. There are more these days than there used to be. But a lot of times they will hire someone like me to come in and I'll tailor the training for the team. Sometimes it's just product management. One of my favorites is when it's actually product plus design plus dev, and we go through all these slides and more that I went through with you. So they really understand because it's kind of funny. I like to use the analogy that product development is a team. So sport. It's like, you know, the Golden State Warriors, I'm in the Bay Area. They don't just show up at the court and go, hey, let's play some basketball. What position are you going to be? You know, they know their positions, they know their plays, they practice them. But when it comes to product development, we just hire A bunch of random people throw them in and building or zoom and go, hey, go make product. Right? We don't talk about positions, we don't talk about plays, we don't practice. So those workshops are very experiential, interactive, a lot of exercises, things like that. I also love speaking at product conferences and before COVID you know I speak a lot of conferences. I used to write an annual list and I'd keep track of like how are they growing, how many do we have, where the new countries that are having them? And then Covid hit and I stopped. So this year I was super psyched on my sub stack. Dan Olson substack.com I wrote my 2025 conference guide and I was excited to see there's more this year than there were before COVID and there's a lot of new conferences. So I just was in San Diego for a first time product conference. I'm heading to Calgary for a first time product conference. So a lot of first time conferences, so workshops and speaking. I also speak at private events if you know. The other cool thing is as products emerge more and more companies are actually getting like their product team together for a day or a half day or two days for an off site to talk about the craft of product management. Like that's the thing we're so busy that when do you have the time to actually sharpen your axe and sharp add to your toolkit? There's really kind of so. So a lot of companies recognize that. So I'll go and speak or give workshops at those events too. But then the other thing that I do that I love and how this all a lot of this content developed is hands on consulting, right? I used to be an interim VP of product for post series A startup. So I did that for box in 2007 I basically was their head of product. I did it for one medical group. In 2009 I did it for Medallion. I've done it for a lot of companies and that's. I kind of stumbled into being that interim VP of product back in the day. I did that in 2005 and a lot of people want to be fractional CPO's that day. Like I did it back then and that's accelerated my learning because instead of just being a one company I saw a lot of other companies. So I still do that today. These days it's more of an advisory or consultant capacity but I still like working with early stage startups. In fact if you're watching this, I'm super excited about AI So if there is an AI startup out there that doesn't have a better product that would be a great fit to kind of play around and help out and help them achieve, apply some of these concepts and then those are the main ways. And then I also, I'm just a big believer in best practices, promoting best practices, business and community. So for over 11 years now, I've been running a monthly meetup here in Silicon Valley where I live called Lean Product Meetup and I bring in top speakers. Marty Kagan's been there nine times. You know, I speak there. A lot of other people have spoken to Jeffrey Moore, Crossing the Chasm. A lot of all the people that you know that are probably on your podcast and blogging and have books have been on there. And so during COVID we had to take it virtual and when we came back from COVID we built up this great audience around the world. So I, we do hybrid events now. So but that's just like a little side project. And there's one other side project I do which is I run an annual product conference with three other folks, co organize it with them called the Product Leader Summit. We've done it nine times. It's a small 120 person invite only product conference for product leaders because there aren't a lot of events for product leaders. So yeah, and then I try to write on substack, I try to do stuff on LinkedIn, you know, try to play around with five coding tools, stay current and, and write thoughts about all that stuff. So it's, it's a pie chart with a lot of slices. But mainly workshops, speaking and consulting and advising are the main, main components.
- [77:48]
A
How lucrative can that be?
- [77:53]
B
I mean, it just depends. It can, it can be lucrative. It depends on how much you want to work. Right? That's the thing about being a solo person. I've been doing it for 20 years on my own. You can decide to dial up your time, dial down your time, you know, have flexibility. So it really, I don't think there necessarily has to be a limit on that. And you know, the other thing is you can decide do you want to like scale up and have a team to do some stuff, do you want to stay solo? Things like that. Those are some of the decisions that people get into. And some people that have like hired a team, they go, I don't want to do that, I want to be solo. So you know, there's different, it's interesting to see, you know, in the product world different people taking, you know, different approaches on how to, you know, create their own kind of product business.
- [78:31]
A
If people want to learn more, where should they go?
- [78:34]
B
Yeah, I have. My main website is dan hyphen olson.com o l s e n I'm on substack dan olson.subsect.com where I'm actually posting a series about product strategy if people are interested in that. And then I have a YouTube channel, just YouTube.com Dan Olson. And I'm on LinkedIn. Happy to interact mainly with people on LinkedIn. So hopefully also I'll see you at one of these conferences coming up.
- [78:57]
A
Awesome. Well, thank you so much for being here, Dan.
- [79:00]
B
Thanks a lot, Akash. Appreciate it.
- [79:01]
A
I really hope you guys enjoyed that episode. It would mean a ton to me and the team if you could please subscribe on YouTube, follow on Apple and Spotify podcasts and leave a rating and review. Those ratings and reviews really help grow the show and help other people discover the show and they help fund the production so that we can do bigger and better productions. Can't wait to share the next episode with you. Until then, see you later.