Loading summary
Transcript47 lines
- [00:00]
A
Welcome back to Product Therapy. Today I am joined by SVPG founder and partner Marty Kagan to discuss a major shift in how we think about product coaching. For years, we've argued that the best product people are developed through strong coaching, ideally from their managers and leaders who know the craft. But now AI is changing that equation. Mari recently made the case that foundation models, when configured with the right instructions and the right strategic context, can really serve as scalable personal coaches for millions of product creators around the world. In this episode, I want to explore what that really means. Where does this work? Where does it not work? How we can configure your LLM well, and why. Judgment, product sense and the operating model will still matter more than ever. Mari, as always, welcome back to Product Therapy.
- [01:00]
B
Thanks very much, Christian. Good to sit down with you again.
- [01:03]
A
I love this. I love this. And this is a very timely conversation. And in some ways, there's this article you put out there kind of about shifting our views on how we see this. First, I kind of want to start by why you see this as a significant shift from what we've advocated for the last 20 years and what really changed in your mind to. To make this shift.
- [01:26]
B
Well, yeah, I'd say if we go back to kind of square one, we've always known that the best companies work a different way than most other companies do. We've known that you could see that just empirically out there. So then the question becomes, well, all right, but how do those companies teach their people how to work this way? Where is this? It's not normally taught in universities or certification programs. It's not there. So how do you learn? And most of us, myself included, learned from my manager. Yeah, I realize now how lucky I was. But for the first 10 years of my career, every single day, I had at least one person assigned to help me get better at my job. Yes, that was called coaching. And the number one responsibility, I trained at HP to become an engineering manager. And the number one thing they taught you was your job, first and foremost is to coach and develop your people. And so everybody I know that really learned the craft, learned that way. You know, books can help, classes can help, but really hands on practice, there's so many challenges and nuances to product. So the problem was when you transform a company, you have to convince the leaders to take that responsibility on. And there's sort of two things that get in the way from that. One is a lot. You know, the first thing most of these leaders say was, how can I coach? I've Never done it myself. The second thing they'll say, even if, because it's not that hard to say, okay, we'll give you coaching and then you coach your people. But the second thing to say that they respond with is I don't have time, I just don't have time. And increasingly in a lot of companies, because you may know there's a trend in our industry, there's several trends going on simultaneously and one of them is to increase the average number of reports for a manager, especially first line managers. That's referred to as span of control. And those are making it even harder for people to do this. So just being practical. We have always known that convincing the leaders to coach, the importance of coaching was sort of the most critical but the toughest argument to make. But we really didn't have an option to that. We didn't really have an alternative. And starting about three years ago, we all started with, not just us. Right. A lot of the industry started realizing that there was a real possibility on the horizon. I would say that for the first year it just was felt very far away because, you know, the models were not very good. Starting at about year two, the models you could believe and see that the models were going to get there. It wasn't a question of if, it was a question of when starting. Realistically, I mean, this is very subjective. But about nine months ago, at least for me, I believed it had crossed that point and it was capable of being as good as a typical manager. There's occasional, I call them flashes of coaching brilliance where it looks like it's cap, it's like as good as a good product coach. But mostly I'd say as good as an average manager. And I realized that I was kind of asking the wrong question initially. I was asking is this as good as a good coach? That's not really the realistic alternative. The question is, is it good enough to get people to where they need to be to succeed in their job? It doesn't really matter if it's better or not than, you know, the best coach. Would I still, if I could send my favorite people to be coached by you, Christian, I would. But you know, even you need some sleep. That's just not an alternative. And so I think I realized I reframed this in my own mind. The right question was, is this good enough to help people get better at their job? And we really saw two things happen. One is the severity or the frequency and severity of bad answers really went down. What it meant is a lot of those outlier answers were just sort of laughable answers and we would all kind of chuckle about that went down and, and the frequency of those went down. And so mostly it ranges from a decent answer or coaching or advice to pretty good. And of course at the same time we've had now have a few years of getting smarter about how to provide the right context. Whether that's, you know, initially that was smart prompts and now it's strategic context. But we've got much better about understanding the data that these models need in order to tailor very relevant examples.
- [06:33]
A
I love this framing, Mati, because you're kind of calling out the journey. On one end, managers not having the time to invest in getting people better. Another end manager is not having the experience in the product model of how to coach people. And the reality is that we've pushed this on managers for so long because there was just no other alternative. How will people get better at product if nobody wakes up every day to get them better at product? And that's the number one job of a manager. Now I want to make sure early enough. Mariam, I want to hear your thoughts early on before we go down to how you do this with AI. Is there a role for the manager if somebody is using this as well and does it differ for you on like, yeah, AI is a supporting tool. AI is where I delegate some of my thoughts to maybe give me a sense of what do you, what would you push on a new manager if this picks up?
- [07:32]
B
I still tell people, if you are lucky enough to have a manager that is both capable and willing to spend the time to personally coach you, count your blessings, you are very, very fortunate and that's what we kind of wish everybody had. You should take advantage of that. But if you're in the 99% where you don't have that, then what I would encourage, and I do encourage is that you primarily use this idea of what we refer to as model, as coach. You're using an AI coach, but then at your one on one you are bringing a lot of the politics, the nuance, the areas that you're still not sure about. Maybe you don't even know the right direction to go. Maybe you feel like you're, you know, chasing your tail. And so having one on one is still incredibly valuable. Even if your manager doesn't have a lot of knowledge of product craft, the coach will. The AI coach will have good knowledge of product craft. The your manager will at a minimum, hopefully have good knowledge of company politics, of the company dynamics. And that person Might say, okay, the real thing you need to do is go talk to Ed. Ed is the one that makes this happen. And yes, the model would never know that because it never heard of Ed, but, but Ed is the one that really makes things happen here when it comes to compliance or whatever it is. And so that kind of coaching, in addition to the product coaching is incredibly useful. Um, so I would, I still think there's a big role. And, and so far I'm. We're just talking really about product creators, product managers, designers, engineers, the people building and delivering products, your product leaders. I believe a product leader can still get value from the coach as well, but they have a lot more to be gained by an experienced product leadership coach that might be the head of product, you know, your manager or something, or it might be a professional product leadership coach, but that's a lot. And I would say that's so high leverage because you could argue that the product leader is the linchpin for the whole thing succeeding. So I would argue that's well worth the expense.
- [10:12]
A
Helpful, because, you know, we see AI does not replace judgment, will never understand your politics, will not understand the ethical risk in your business, the nuances of your architecture, tech, debt issues, so many things, and the human aspects of your work that and the things he's never been told in some ways in how he informs you. So you're calling that out early. You know, one of the choppiest points in the article you wrote, Matty, was that AI often exposes product management theater rather than real product contribution. Can you talk more about that?
- [10:45]
B
Sure. In fact, that's a great foundational thing to sort of frame this conversation. So the first thing we should probably talk about even further back than that is AI. Whenever we talk about AI, there are really two interesting and useful conversations. What we're talking about here is how does AI help us create products? Any kind of product, really. Any kind of product. The other conversation is how do you create AI products? So now the truth is very small percentage of the world today is creating AI products. So that's sort of only academically interesting to most people. And it's bit ironic because AI is helping us create conventional products much faster. But creating an AI product is actually very hard and slow and, and duration wise can take a very long time. So. So there's a little bit of irony in there too. But the truth is most people want to talk about how to create any product, in other words, using AI to accelerate any product. So that's kind of our focus here, creating AI or intelligent products or probabilistic products is a really interesting discussion for the teams that are doing that. But, but all the principles apply. But there are some very important dynamics in creating an AI product. Now the other dimension, and this is the part that I think it's just unavoidable, but if you look online in general at all the chatter about using AI, about 90% of it I would argue, is not helpful. And the reason it's not helpful is it's because they're using gen AI to speed up the project model, which is, I mean, if I had my choice between taking a long time to do a project and taking fast to do a project, I'd rather go fast. But it's still garbage in, garbage out. And that's the fundamental issue. And the irony is AI is highlighting that it's shining a light on that in fact for so long. I mean, I hate to say it this way, but product, you know, we, we frame this as product management theater. Product management got away with all kinds of nonsense because the bottleneck was the engineers. And everybody was just complaining about the engineers taking forever and costing. So all the quality issues, all that. Now the engineers are saying, hey, it's not our problem anymore. It's pretty clear. The real issue is it's garbage in, garbage out. You are still giving us worthless stuff to build or stuff that doesn't actually do what we need it to do. And furthermore, our competitors can copy us faster than ever. So it just, it just amplifies the lack of strategy and the lack of discovery skills. So that's the other thing that's going on. And you know, it's sort of human nature. If you're doing X and a tool comes along that helps you do X faster, you do X faster. But the real issue is what can you do with this technology that you've always wanted to do, that you've always known that needed to be done. And of course we've been arguing the product model was good even before gen AI. And that's. And, and I think the proof of that is the gen, the AI companies are using that to do these products.
- [14:25]
A
I think that's an important point. You know, like the, the AI companies are using the product model to build the AI products and, and why that's important. You know, I was just doing a talk around product creators and I was making the argument because when we talk about when things are easier, people don't understand that the expectations also increase. You know, it's like if you can personalize easily I want personalization. You know, if, if you can fix errors faster, I want fewer errors. And I'm like, you are highlighting. This is not a downsizing moment of, this is a bar raising moment for the discipline. Like it is calling out product management in this era clearer than ever. People will see our garbage faster. They will recognize our inability to solve real problems or respond faster than ever because everybody can build. You know, it's a great, it's a great, great call out. Now, Imani, you also do it. You started to talk about the distinction between product creators and product leaders and building the model as a coach. I do want to go in pointedly to how we build the model. Where we start in building the model, what are the guidelines? But when you kind of think about the distinction, if, if I were a leader, what would I be looking to gain from a model as a coach? And if I were a creator, what will I be looking to gain from the model as a coach? How you distinguish the goals or the outcomes?
- [15:49]
B
Well, I mean, at a high level, I would say both. In both cases, the model as coach is helping you get better at your job in both cases. But in practice, as you know better than most, because you show people how to do this, the, you know, the product leader job is a very difficult job and much different job than a product creator. I know that both of us have done both of those jobs and I actually think the product creator job is the fun job. But I also would be the first to admit the product leader job is the more consequential and impactful job because you're really setting up an entire organization to succeed or fail. So for example, in the creator, your job, if you're a product manager or designer, is really product discovery. It's all build to learn versus the engineer's primary job is build to earn. We can talk, we should talk more about that. But the point is we're all building. We're all building. We're discovering the right thing and we're building and delivering the right thing. So, so that's the product creator. Product leader is really to enable an organization of teams to do this. The product vision, the product strategy, the product team topology, the objectives for each team, the strategic context in general, that's necessary for each team and by the way, also necessary for the AI coach in order to provide good coaching. So the product leader job has probably never been more important and essential and difficult. We could talk about that. That's sort of a whole other podcast discussion is how is the product leadership role changing in Respect to AI. I actually don't think it's a tooling issue like it is with the creator. It's more of a strategic issue. And so that's another discussion.
- [17:56]
A
Yes, creating an environment to enable creation in some ways that. And I was kind of. Are you calling out pointedly like the model as a coach to help you with your product vision? Your product strategy or topology or objectives is very different from the model as a coach to help you with the work of discovery.
- [18:15]
B
Yeah, because really the discovery is the craft. That's really the craft. The, you could, you could view it more that the, the leadership stuff is more the strategy.
- [18:26]
A
Now, Mari, you suggest that one of the best places to start is by using this to develop your product set. Maybe talk about why you think that this is the best place to start and maybe how creators can improve their judgment rather than outsourcing their thinking to an LLM when they do that.
- [18:42]
B
Yeah, and that's of course one of the big dangers with really any tool is outsourcing the thinking. Outsourcing a lot more than you realize is the point product sense. In truth, that's the table stakes. That is the table stakes for both creators and for product leaders. It's sort of ridiculous to talk about a product leader without that product sense, but in many cases the product leaders do not have this, so you can't take it for granted. But normally, of course, normally what's going on is you prove yourself as a creator and you get promoted into a leader and where you're now enabling others to do that. So that's the normal progression as far as developing that product sense. That is the primary benefit and use of a coach is developing that. So what does that really mean? There's a set of things and by just to be very clear, whether you use an AI coach or some other way, if you don't develop this product sense, you are not helpful as a product person into your company. And you're also probably got a big target on your back for the next layoff. So developing this product sense is the single biggest thing you could do for your career and for your company and your customers. So what does that really mean? It's a number of dimensions. First of all, you've got to really learn about your customers and the different users. If you're a B2B, especially different kinds of users at your customers, they have very different motivations, different goals, different constraints, different concerns, different dynamics. You have to learn the those things. You have to be an expert. Now you can still and we absolutely encourage this, go visit your customers. You could also learn a whole lot about your customers before you take a step. You got it from your laptop. The, the models are very good for most categories of describing like, okay, here's the dynamics. You're a marketplace. You've got these three kinds of customers. This, let's talk about each one. So you need to learn about those customers. You need to learn about your industry. You need to learn about, for example, is it a regulated industry? What are those dynamics? You need to learn about your competitive landscape. What are the other alternatives out there? How do people solve those problems? Today we want to go in and win a bunch of business, but how are we going to do that if we don't know what we have to beat? So there's the competitive side. We have to learn about the data. That's often one of the hardest things too, is we have a lot of data. We need to first understand what are the measures, what are the levers for our kind of business? Okay, so we're a marketplace, Fine. We have businesses on one side, consumers on another. How do we decide if this is a healthy marketplace? How is that judged? How is this? The models can coach you on that. So they can say, these are the seven levers you need to understand. You can say, well, what does this even mean? What is this lever? Is and is going up a good or a bad thing? And is our number currently good or is our number currently not good? These are the kinds of help it can give you. And then you can literally ask questions like, so what is the biggest win right now? What should we focus on? These are, this is all on the data. Also, we need to learn. And this is a big chunk of time for most product managers. Remember, winning a product means a product that your customers truly love, that want to buy, they truly choose to buy and use, but also satisfies the needs of the business. Countless products fail because they think all you have to do is make your user love you, but it's not true. In fact, a lot of times, that's the easy part. The hard part is we need to find a solution where they love us. But is also legal, is also compliant, is also respects privacy concerns in so many parts of the world. It also is something that we can afford to build, is something we know how to monetize, is something we can market cost effectively, we can sell cost effectively. This is all part of viability. And viability is usually somewhere between 5 and 15 dimensions of the business. Some businesses are simpler than others. The developer tool business I've always loved that's a simpler business than other kinds of tools. Like if you want to pick a complicated business, look at something like ERP or supply chain man. I mean incredibly complicated. So much domain expertise and business rules you need to understand. But again, what use is a product manager that doesn't understand that? I mean literally there that is their job. The, the engineers don't know this and aren't expected to know this. And the end, the designers aren't, but the product manager has to. That's ironically, you know, early on with Gen AI the talk was oh, we don't need designers or product managers, we just need developers. Now the talk is, I guess now we're not even sure the life of the developers here. The tools are getting so good. But I one thing we can tell you is they need this understanding. They need a real product manager. And so we saw, we. Well, we've seen basically the product manager role explode because of this. But we're talking about this kind of product manager, not the kind that just speed up the project model.
- [24:24]
A
Mario, you're calling out and it's very powerful the breadth of what the model as a coach can help with. You're calling out company context, markets, users, domains, economic constraints, metrics, data technology, all the strategic context elements. Now does this in some ways. First of all, let me clarify this. This is the public LLM as is. This is not some company data feeding this LLM model. And, and, and because one of the early barriers people will say is oh my company will not allow us to share all of this information. Are you suggesting the public data as is even good enough?
- [25:06]
B
So good question. And this is actually a very complicated one because for the first, for like the last year, the question was would us or somebody else have to train a model specifically in order to be helpful or would the stock models? I mean obviously there's way more models than I've even heard of, but the three major models from Anthropic, from OpenAI and from Google and they each have multiple models of course, but the three major sources of models, that's kind of what I'm talking about here. And one of the things I was saying about nine months ago it became clear that at least those three I'll say out of the box, there's no box anymore, but out of the box they, they are like very good. Now I also was explaining earlier that at the same time the models have got better. We've got better at instructing the models with context. And for a long time you Know, I felt like I kept. And this was very informal. I'm not suggesting this was scientific because this is a research problem. How do you judge, you know, whether the product advice is good? There was sort of a informal benchmark of topics and questions that I had that I kept asking. And I was saying, you know, a lot of the results were kind of funny, but they were very spotty. And one of the things I realized was the models are trained. I mean, the models have read most of the product books. They certainly read our books. But here's the problem. More than half, probably like 80, 90% of the content on product is all about the project model. So we realized that the models are trained on very contradictory approaches. And so what? And, and that would, and that, I think, is what explained why one question could give you a very useful answer and the next one could be like, are you serious? This is not how we work. So what? The. One of the most important things I learned myself through this testing was there's a large language model and then there is a form of an operating model on how we build product, the project model and the product model. I just realized I've never. It is confusing if you're not doing this every day, for sure. So anyway, you need to tell the large language model which operating model you want to work in. So, and that one change. And you can do that lots of ways. You can say, you know, I'm working in the product model, or you can say, I want to follow the, the, you know, the methods that SVPG advocates. You can do it lots of different ways. I can tell you myself. I say, you know, SVPG content, Teresa Torres content, Triash Doshi's content. So I'm like saying all of these people are talking about the model I care about. And I tell the models, prioritize that don't, don't feed me stuff from the 1990s that are basically published, but not the kind of product work we're doing now. I have some people I've talked to that are definitely not at the product model. They do project model. I'm like, then you need to tell it these names because these are the people that talk about the project model so that you can get good advice there, because if you just get a mishmash, you're just going to confuse people.
- [28:49]
A
But that's an interesting point and I want to kind of hear your thoughts on it, because we are coaching and advising people on how to work better in the product model. But if someone says, but Mari, I work in the Project model. I don't like it. Are you suggesting here that you want the coach to teach you how to survive your product, your project model, or is there any benefit of me learning the way it should be done and trying to model or practice that in my company?
- [29:16]
B
Oh, yeah, but that's a. Absolutely. And that's what we usually. But that's somebody who comes to us and says, I want to learn a better way of working. But some people say, I would love to learn a new way. But our leaders have already told us this is how we're working and we're not ready yet. And I don't want to lose my job. I want to learn how to do something. So, sure, at some point they may decide to leave and go to a company where they can practice that. That's up to them, obviously.
- [29:48]
A
So I love that. First take. First of all, you. You want to be clear about the model you're working in and tell your LLM model what that model is, right?
- [29:58]
B
What you want to learn.
- [29:59]
A
If someone were trying to configure that LLM as a real product coach tomorrow, what should they feed it? So you've kind of given us the guideline, what model I'm working in, what's kind of the next thing I want to feed it?
- [30:09]
B
So first of all, the model you're working on, that, there's nothing private about that, nothing secret about that. That's like, it's already got this stuff. It's just, you're just telling it what to prioritize. The next topic gets a little more interesting, but really powerful because, you know, we talk about this all the time, even well before Gen AI. In order for a product team to make good choices, they need to have the strategic context. It's not enough to be empowered because you don't know the big picture. All you can do is optimize for your own little, you know, team. But you want to optimize for the customer and for your business. And for that you need the strategic context. That's the product vision, the strategy, the team topology, the team objectives. And so if you can tell, if you can point your large language model, I'll try to disambiguate as I go here, the large language model at your strategic context. Or you can load it in, you know, basically with, with some tools, you load them in with some, you like, cloud code, you just put it in a directory, in a directory and point the model at that directory and it gets them. But as long as you can point the model to this information it's amazing how much better the coaching is because now instead of just talking generically about jobs, marketplaces, it can talk about your company. Maybe we work for Indeed. And we're like, yeah, you. There's a lot of public information about Indeed. And from filings, we can also then talk about not just any vision, but our vision. We can talk about not just any strategy, but our strategy. I can ask questions like, what is my team helping doing to help the product? Strategy to help the vision? And what team does my team depend on? And what. How does this relate? And, okay, I'm working on this particular problem to solve. What problems does that depend on? I mean, it can start answering questions that are like, oh, man, that is, like, really valuable.
- [32:28]
A
So I'm, I'm tracking with you. We've decided what operating model is. We are giving it a context. Now, to be clear, if I am in a project model company and I'm giving a roadmap of futures and projects, I don't, you know, I'm not connected with context. Is that all I'm giving it? Is that all I'm saying what I've been told to do in my project?
- [32:50]
B
Well, honestly, I don't spend a lot of time on helping people with the project model because to me, there are sort of. It's sort of like rearranging the deck chairs on the Titanic. So I'm like, good luck, but I'm not gonna, you know, I, I don't write on. Focus on that. I mostly point out the problems with that. But you're asking, you know, if I was one of those product people that were focused on the project model. I'm sure there's plenty of useful things. I mean, most of what's out there right now just. I mean, I'll give you examples so that everybody understands when I said that 90% of the stuff on AI is not really helpful because all they're doing is speeding up the project model. There are people out there that are so excited because they can say, oh, I can have. I can tell the model to go. The large language model to go straight to our customer feedback, Go analyze what the most requested features are, put them on a roadmap. Prioritize that roadmap. Okay, now take the first thing on the roadmap. I want you to generate a prd, and then I want you to take the PRD and I want you to generate acceptance criteria, and then I want you to give this to the developers, and I'm done. You know, I did my job and I'M like, oh my God. You realize that first of all, you've just demonstrated that you are totally useless, replaceable very easily. Yeah. But anyway, that's just laughable to me. People are bragging about that. But again, they're so sort of tantalized by this idea that they could take something that used to take them weeks and do it in minutes or hours. So they're tantalized. But I would argue, yes, it took a terrible process that you were using to generated really bad products and now you're just doing more of them faster. So I am much more interested in how do we use this to build truly successful products as again, as measured not by output, by outcomes.
- [34:57]
A
Now, now, Mari, because you're showing that I can get a lot of context, generalized product sense about my market, my industry, my data, my business. I can get some insights around the context in my environment. Does this for you change maybe the barrier to entry for getting good at product? Like, you know, in any company, I mean, how does onboarding even change in your mind for a creator?
- [35:22]
B
I mean we're starting to see it now, but people, I used to say that, I still say this. If you have a manager that is a good manager that is willing to coach you, that you can take most people as long as they're willing to put in the effort about three months to get them to a real product manager. I don't mean an associate level, I mean like a real first class product manager with a real team of engineers and designer. Today that can happen significantly faster because instead of just an hour a week, one on one of coaching, I can have this in my pocket that is literally there all the time helping me. I can have it sort of overseeing. There's different ways of using a coach. They can oversee your work or it can be all about like I, I want to ask a question and they have this context. So who knows? Like I, I think we probably need a year of time to know like, well, okay, if three months was the average before, what is it now? My guess is it's going to be a lot closer to a month. Now. That's still a month of time. But you heard me describe a lot of things that you need to learn. And you know, the way it is, the way we are on learning, it takes like hitting it up from several sides and thinking about the topics and trying to understand is that domain dogma or is that really domain knowledge? So there's a lot of digesting of this knowledge that's coming in. And for a new product Manager. It's a fire hose of information, so I don't know what it's going to end up being, but I do think it's significantly faster and to your point, I love that it makes real product craft skills accessible to literally anyone with an Internet connection.
- [37:30]
A
That muscle of product sense, it's magical. I have been trying to learn quantum physics, and now that I think I learned a little too late, you can actually have a dialogue with your AI and just talk to it. And, you know, it's like, I want to learn this. Teach me like a five year old now. You know, break it down for me again. Explain to me, quiz me, let's talk about it. Tell me stories about it. And you know, like, it's not my new audiobook style of learning, but I can imagine. I am absolutely not a quantum physicist at this point at all. But just ramping up my sense of around the subject in a manner that would have taken me lots of books, hours of study. You know, I kind of have some sense of some of the history there now, Mari, I want to talk about like risk and guardrails, because you've warned a lot that this is not a deterministic model. Advice could vary from day to day. In short, I've had like from. In minutes you could ask something. Five minutes later is a completely different answer. How do you typically advise people to walk within this reality?
- [38:31]
B
Well, there's sort of. First of all, we do need to talk about that. But I will say, when I first started really screaming about that to people, the results were so spotty. And the big problem I was seeing was people were believing things that they just like, are you thinking? Are you not thinking? And so then they would show it to me all confused and I'm like, where did you get this? What are you doing? And you know that they were just taking things at face value. Today that seems to happen a lot less because I think the, even when it's not a great answer, it's not. That doesn't mean it's a bad answer. And so maybe, you know, maybe it takes you 30 minutes of asking about this, you know, quantum physics thing instead of five minutes if it was just a good, great answer. So it's still way better than the alternative would have been. So I feel like the, the downside risk is a lot less. That said, the whole idea of what we're doing when we talk about AI as product coach, notice the difference between AI as product coach, which is helping me develop product sense versus AI generating my PRDs. Yes, that's not a coach.
- [39:52]
A
That's right. That's like I am delegating your administrative work.
- [39:57]
B
And if you think that's what your job is, then you've just automated your
- [40:01]
A
job and that's a big piece. And maybe, maybe to that point, what are some of the signals that somebody has become overly dependent on the model rather than, you know, thinking or developing judgment. What, what, what clues you into that?
- [40:16]
B
Well, honestly, this has always been an issue. It's like people just frankly not thinking and just saying, ok, this is what I got, you know, and I think there's a real, it is human nature, but there is a real desire, especially with complicated things to like want a recipe book or a playbook or a process. And if people start just following the process, then I'm like, well that's, you know, you've heard the saying that the problem with process is it's too often used as a substitute for thinking and that's the worst thing. So I am always watching for that. And even people that are really well intentioned, so yeah, they're not saying, oh, we want to follow the safe process, but they say the same thing about the double diamond process. Obviously I'm not equating them. The double diamond is not an evil process, like I would argue safe as, but it also is totally inappropriate for so much product work. And so I don't, I'd much rather teach the people to think. So that's what we're looking for, looking to think and use the coach to challenge your thinking, to improve your thinking, to call you out when you're not thinking.
- [41:40]
A
And I want to break that down because I'm following the journey of what you're saying here. If I'm setting this up as a coach, I'm first identifying I want to work in the product model. There are prominent voices in the model. You know, I want to be guided by thoughts. You're kind of saying, this is not generalized coaching. I am feeding it the context because it goes beyond the general knowledge to understanding my context. That helps it be a better coach. You've pointed that the first place you're starting is the foundation. You're helping me get better as a product coach in terms of talking to the AI itself. Like, you know, as a coach, you know, I've said, and I'm setting the AI, I'm telling it, I am setting you up as a coach, not as an advisor, as a boss or as a resource. Because everybody has this. It takes on a Persona when you tell it that. And what, you know, someone was telling me the other day, you know, my AI, it just tells me everything I want to hear. And it's just exactly validating for everything I've been telling my manager is wrong. You know, I'm like, you know, I was like, yeah, it's very much like you, you know, it just, it's doing that. So I like this framing as a coach and the voice. But you've, we've kind of talked about this prompt engineering, moving to context engineering. But I'm more concerned now about how you're, how you communicate with it. Like maybe give me some guidance on, you know, if I were talking to a real human coach. I kind of have the questions I come in with and I ask, I've set up now. I have my product coach guide me on how I engage with it on a day to day basis.
- [43:15]
B
Good. And you know, Christian, hopefully in the podcast notes you can point people at the article we did on AI's coach because there's some links to some actual prompts in there and context, because this question is a really important one and I'm not suggesting that that's the only way to do it. It's the kind of thing you want to do. So, for example, emphasizing that you want it to play the role of coach, you, you need help. But being clear with it that we're measured by outcomes. So making me happy if it doesn't get to the outcome, that's like not helpful. So you say things like, I need you. I'm not using you as a coach to provide affirmation. I'm using you so that you can help me get to good outcomes. So that means pointing out things that may be very difficult. So you say this one way or another. I've played with 50 different ways to phrase this, but it seems to really get this. Sometimes even extreme versions of this where it really like, give me a little, give me a break here. You know, you are looking for it to play that role. I also think there's kind of, when we talk about AI as coach, I mean, in a way it's a product, right? It's a product may not be for sale, it's just a use of a standard product. But there's open questions about ultimately what the best way to engage with the coach will be. I'm just going to say there's a couple major approaches. I'm, I'll take a guess, but what I think is going to play out, but I'm not certain. I feel like in the case of a Coach. A natural language interface is really comfortable because we're. That's kind of how we talk to coaches and our managers. So. And you could argue about who's the initiator with a real coach. Sometimes it's me asking questions of the coach and sometimes it's the coach saying, have you thought about this? But it's an. That's one interface. Another interaction model is especially with the tools that are some of the amazing tools like Claude Code and Cowork where they. They could be used exactly like this or they could be used sort of as an over your shoulder watching everything you're doing. The developers have been doing this now for a year where agents are looking at their work and keeping an eye on things. They might actually find a bug in the code. Then they might proactively say, I've done a pull request. You should review it. This is why. But you could imagine a coach playing that over the shoulder role. I'm a little worried that people will. The nice thing about a chat interface is it feels a lot more like I'm in control on over the shoulder might feel a little more like Big Brother. So I don't know. I'm, you know, we're. I. Because I do think there's advantages. There may be places where I wouldn't even think to ask for some coaching. But a proactive agent might of course not not be. They go ahead and say this is something worth calling out. So I'm not sure how it'll play out. There'll probably be other interaction models that come up as well. But those two exist today and we'll see how it plays out.
- [46:51]
A
So will you be coaching people to say, look, you're. You should always. You're not like running your walk by by your coach. But you're, you know, kind of asking what am I missing? What you know, what's incorrect, what does not align to my context, you know, what am I not seeing? Is that kind of the. You've got something 24 7. You're not like waiting for the Monday meeting because there's always these things of like, I didn't have time to practice before I went to my CEO. Nobody gave me a second thought. So I'm always telling people. You mean the first time you presented to your CEO was practice? Like you didn't run this by anybody.
- [47:28]
B
That's a perfect way to put it.
- [47:29]
A
Maybe there's that magic of we have to now get good at realizing a couple of things. We have something 247 available that we can practice. We can check for errors that can even test our judgment around things and our decisions. It's not replacing our judgment, maybe it's not deciding for us, but what a powerful way to kind of frame that.
- [47:54]
B
Christian, you reminded me of something that I think is a common misconception even among the people that are very into this topic. And I really find it surprisingly good. For a while when not just us, but other people started talking about this sort of use of a large language model, people would say, well yeah, it knows that this is how the customer discovery program works and so you need to do these things and it's good for that. But it doesn't really understand politics. It doesn't really understand human nature. It doesn't, you know, and so they kind of, which of course people were saying that because we want to feel good about like is there something we can still bring to the table? And I do think there is. But one of the things I've really that realized remarkably is that the models are actually quite good when it comes to coaching politics. Now, partly because there is a lot of content out there on the topic of coaching politics. I know that you and I together have contributed a good chunk of content on coaching politics. It's, it's a super important but you know, thought out topic. And today I, one of the companies I advise, I was talking to one of the product leaders actually and he was struggling with. I'm sure you've heard this before, but our super strong opinionated head of sales that was really not there was a. And in particular what was going on. Many people know this phrase but a lot of salespeople feel very strong account ownership, account control. They're like, they really don't like other people talking to their customers. And of course we can't do product if we don't really have this kind of interaction. And so I'm like, try this. This is actually quite explain to the model about your head of sales, what your head of sales is concerned about. Ask it for advice on how to deal with that. It is remarkably good at understanding, quote understanding these, these dynamics and giving you tips for earning that trust.
- [50:15]
A
You know what I actually just told someone go into this with politics and I say, you know, the interesting part about having AI help you with politics is that it's not political. Like, you know, it's not going to use you against you. You know, it's like when you go in your environment and I'm going to complain about Mari now I have told people in there and I'm like, oh, I'm more vulnerable. And there's almost an baked in, not just the availability, but, you know, some of that safety or psychological safety. And you can have these conversations about what's really going on in your environment. And I even coach Personas. I'm like, I have my. When I set up my product coach, I create a Persona for my manager. I tell him all I know about my manager. And you know, anytime I refer to my head of sales might be like, remember, Mari doesn't like when things are presented this way. What an amazing way to do it. Yeah, great. So I actually love that we are calling out some of the aspects, particularly when it comes to coaching, politics and humans that still happen. And we talked earlier about even the division of labor between AI and human coaching. Now I want to kind of wrap up with your thoughts around the future of product management. Now, particularly again with. I think one of the early stance we had was that AI will raise the bar for product roles and it might even make it harder for people. Now there's this AI coach that you could have a model as a coach that could at least get you foundationally understanding how to build good product sense. Give me your sense today. And I know it's continuously evolving about the impact of AI now on our walk moving forward. You've just had one shift with. This is my help with coaching. Are there any other things that come to mind or trends you want to jump on?
- [52:05]
B
Yeah, well, in particular, I remember you and I did a podcast on what it means to be a product manager for an AI product. And that is still very hard. And there's a lot of demand for people that have those skills for sure. Part of that, it's supply and demand. Very high demand, very small supply. So we got. But we're getting a lot more people skilled in those things over time and I think that'll continue. So that topic of building an AI product is kind of that separate one in terms of how AI is changing how every product person goes. And that was another point we made that every product manager will need to be capable in AI. I think that's becoming more clear to people that they all will be. And I think we're seeing not just, you know, this knowledge being necessary, but it's does change the nature of what you're doing most of the day. So we haven't talked about this too much yet, but today go. And I think going forward even more. So there will be very clear distinction between, first of all, we're all builders. We're all builders. We're all creators. When I say all of us, I'm referring to product managers, designers, engineers of all flavors. We're all creators, we're all building. That's what we're there for. We're not bureaucrats, we're not project managers. We're building now. What we're building varies. If you're a product manager and designer, you're primarily there to build, to learn. You're building prototypes. Now. One of the things that obviously has changed, everybody knows this, is we used to depend on our designer and occasionally our engineers to build the prototypes. Today, any of us can build prototypes and any of us should build prototypes. I. The way I phrase it now is all of us should build prototypes. And that is. So it's not the only thing we do, of course, but boy, that is like, right at the heart. You may have heard that in a lot of the top companies today, if you interview for a product manager, the interview is totally. The interview now is, what's your favorite prototyping tool? Here's the problem to solve. I want you to prototype a solution and then show me how you got to test it. Tell me how you can attest, because it's all about build to learn. How will you know if their chief compliance officer is okay with this? How will you know if sales will be able to sell it? How will you know if customers will truly buy it? This is the question that good companies are asking their candidates. So if they don't know, if they don't have good answers to that, then again, not so helpful. So this idea that you're building to learn and getting really good at, that you're all building these prototypes and testing these prototypes. Now we each bring a different lens to that prototype. Designers, I think, bring an incredibly valuable lens, especially with consumer products and, and complex workflows and interfaces. Product managers bring a very different lens. We've talked about today about value and especially viability. And the engineers, of course, bring a very different lens around what's just now possible. So there are, I love the fact that those are not hard lines of distinction anymore. We've always had the case, honestly, where we've known people who were good at multiple skills. I used to, I still call the people who were good at all three of those in the triad a triple threat. Triple threat. Those are like the best product people I've ever met. And there's a handful of them to me that are just like in my all star hall of fame. They're just so great at all those. There's Some magic that happens when you're good at those. And I do believe we're going to see more people that are triple threats because of what we talked about. The coach can help an engineer learn these things and they can help a designer learn these things and vice versa. They're already helping product managers learn engineering.
- [56:28]
A
I love that a lot. And is that you. Would you advise every creator to set up a product coach the same way? Meaning like an engineer should be having a product coach feed them this context of stuff or you, you know, in the same way product managers are learning engineering?
- [56:46]
B
I do, but there's a difference. I mean, really based on the role, it's going to help me with different kinds of things. Now, there's no reason it shouldn't point out to me like, oh, maybe I should go talk to the designer about this or something. But yeah, you would be clear. Like when. When in our instructions to the language models, we say what role we are. So you could say, I am a product designer and I am working on this problem on this product team in this company, you know, in this strategy. So you're giving it that context. I can also say I am covering both product management and product design. You can absolutely say that. Or you could say I'm doing all of these things. But just know that the tools for engineers to build, to earn are aimed at different risks. They're aimed at reliability, risk, scalability, risk fault tolerance, you know, recoverability, these kinds of issues. So a product manager or designer working on discovery, it's a very specific kind of risks.
- [57:57]
A
I love that a lot. I can talk about that now. Last question. Matty, it took you a while to kind of make the shift in trusting the models to be good enough or at least as good in getting somebody equipped in the job. Where do you still not trust the models today?
- [58:13]
B
I feel very good about the models for the creators, the leaders. The mix is different. You could say a creator. It's probably something like 80% craft and 20% politics. For a leader, I think it's the opposite. Now. We're not really just talking about politic principles. We're talking about Joe, who is a real pain. And we need a way to deal with Joe, you know what I mean? So there's still a lot, especially for the product leaders that don't have the experience, the foundation that they should. But I would say I feel like those are the ones that need more people.
- [58:56]
A
This is fantastic, Mari. What a gift. A great conversation. I can imagine we'll have more conversations like this as you know this emerging technology continues to evolve. We can probably see more and more ways it affects our work. So I'm looking forward to it. Thank you so much for the gift of the time and of always your experience and expertise. It was great to have you again on Product Therapy.
- [59:18]
B
Thank you Christian. I love talking with you about stuff like this.
- [59:22]
A
Want to learn more? Until next time, Please check out svpg.com Sign up for our newsletter that Mary Kagan puts out. Join us for one of our workshops near you and get access to all of the articles and content we put out. And thank you to everyone for joining us. Until next time, have a good day. A Quick Disclaimer While this podcast is named Product Therapy, it is not hosted by licensed therapists or mental health professionals and it is in no way a substitute for professional mental health services. We recognize the importance of mental well being and encourage anyone facing personal difficulties to seek support from qualified professionals. See www.findahelpline.com.