Loading summary
Transcript83 lines
- [00:00]
A
You created Gemini Gems, Lisa, what are they? How would you describe?
- [00:04]
B
So Gemini Gems are custom versions of Gemini that you can create for your specific use case. If you aren't using Gemini Gems, you aren't getting the most out of Gemini.
- [00:12]
A
What are the most important gems for product managers to create?
- [00:16]
B
Well, Kosh, I think there's three must have gems for every product manager. Let's call them the Writing Clone, the Product Strategy Advisor, and the User Research Synthesizer.
- [00:24]
A
This is Lisa Huang, the creator of Gemini Gems. She's been an AIPM at Apple Meta and Google. Now she's an SVP at the 18,000,000,000,000-g giant zero. She also created Gemini Gems. What is the process of creating one? Can you show us?
- [00:38]
B
Sure. It's actually really simple.
- [00:40]
A
So wow, these are insanely powerful. So if we were to build a mind map together, what would be the most important projects or gems every PM should be creating?
- [00:49]
B
Yeah, so when you think about what are the key skill sets of PMs, I might break it down to four big categories. You certainly need to own the overall product strategy and you'll need to own execution for that with your teams. But also very importantly, you need to communicate that to a lot of different stakeholders and then informing that product strategy. You've probably done a lot of research on competitors users and that those are key data signals to input as well.
- [01:10]
A
What are you looking for in an AI pm?
- [01:13]
B
So I think.
- [01:18]
A
Before we go any further, do me a favor and check that you are subscribed on YouTube and following on Apple and Spotify podcasts. And if you want to get access to amazing AI tools, check out my bundle, where if you become an annual subscriber to my newsletter, you get a full year free of the paid plans of Mobin, Arise, Relay App, Dovetail, Linear Magic Patterns, Deep Sky, Reforge, Build, Descript and Speechify. So be sure to check that out@buildle.akashg.com and now into today's episode. Lisa, welcome to the podcast.
- [01:51]
B
Thank you. Thanks for having me.
- [01:53]
A
As you all know, Google Gemini, since the launch of Gemini 3 Pro, has become one of the top AI models as good as ChatGPT and Claude. But if you want to get the most out of Gemini, you have to be using Gemini Gems. You created Gemini Gems, Lisa. What are they? How would you describe Gemini Gems?
- [02:11]
B
So Gemini Gems are custom versions of Gemini that you can create for your specific use case. Everybody knows that LLMs today are really powerful, but the key problem that they all have is that they lack context. So what ends up happening is that as people use these LLMs repeatedly, over and over, they're just entering in the information that's needed to make sure that the LLM gives out the right answer. Whether it's your role, your company's strategy, your writing style, the history of your product. You're constantly having to type this in again and again, and that's a little bit of a pain in the ass. So you can think of Gemini Gems as the difference between a general contractor versus a master craftsman. Standard Gemini or standard LLMs. They're kind of like your general contractor. They're super powerful, really capable, but you're going to have to give them detailed instructions every time you hire them, and you have to give them your project's blueprints. Compare this to the Gemini gem. It's kind of like a specialist, your master craftsman. It already knows everything about your house. It understands your preferences, understands where all those little pieces of your furniture are. So when you ask it for something, they can craft the exact thing for you without having to be retold all of the basics. So Gemini Gems, it's a little tool that you can create inside Gemini. It holds all of your context, it keeps your personalized tone and style, and it's really tailored to your specific use case.
- [03:34]
A
I think that's the important thing is you can create so many different ones. You can create. People have been talking about a personal cto. You can create a coding gem. You could create a writing style gem. What are the most important gems for product managers to create?
- [03:52]
B
Well, Kash, I think there's three. Must have gems for every product manager. Let's call them the Writing Clone, the Product Strategy Advisor, and the User Research Synthesizer. So on the Writing Clone, one of the key jobs of PMs is to communicate all day, every day, to lots and lots of people with different contexts. So what you can do here is create a gem that sounds like you to help you accelerate those communication tasks. What you could do is upload your key files, like your PRDs, maybe some of your emails, some of your team slacks, and then you can have a version of Gemini that sounds like you to create those nice first drafts. The second gem is something that can help partner with you to be a strategic partner in your thinking process. So think of this as your product strategy Advisor. You can include things in your company's strategy documents, Market position, go to market and give it some competitor analysis. Then it can be a little thought partner for you as you're thinking through different decisions and helps you think through maybe some angles that you hadn't thought about before. And then lastly, on the user research synthesizer, obviously as a pm, you have to stay really close to the source. So what are those user needs? What are users saying? But there's quite a lot that's going out there and you can't always be in all those conversations, all the interviews. So instead what you could do is upload all of those user interview transcripts, the raw survey data, maybe some customer support tickets, and just ask the Gemini gem to synthesize all of that for you. You can ask Q and A, ask it for key insights to help you sort through all of that data.
- [05:26]
A
Okay, these are epic. I understand why people should have these. What is the process of creating one? Can you show us?
- [05:34]
B
Sure. It's actually really simple. So when you go to Gemini, there's a little tab you can click on, which is gems. Then you click on create a new gem and then three key things you want to do. Number one, give it some clear instructions. Number two, add knowledge. So upload your key context, documents, et cetera. And then thirdly, make sure to test and iterate. It may not be perfect the first time, but as you use it more and you kind of iterate on those instructions and that knowledge, it'll get closer to what you're looking for.
- [06:01]
A
Sweet. And do you want to just show us?
- [06:04]
B
Sure. Let me switch screens here. And here we go. So here I'm in Gemini and what you can go to here on the left you'll see there's a little link called gems and there's some pre built gems for you if you want to try out. If it's kind of your first time and you want to get a feel for what these things do, you can click on new gem. And let's say we want to go ahead and build that, you know, product strategy buddy, someone to help you think through your product strategies. So someone to bounce product. Oops. Bounce product ideas off of. All right. And in. In instructions, you do want to create very detailed instructions. I'm sure you guys have seen this with how LLMs perform, the more detailed your instructions are, the better output you'll get. So I've prepared some here actually for this. So you'll see a lot of detailed context, core responsibilities, how I want them to approach their thinking. And then as you scroll down, for those of you who want, depending on your gem, you can select a default tool if there's something that it'll often use. In this case, I'm not going to select any because it's more of A general kind of brainstorm. And then for knowledge, I'm going to add some key company files. So here I've selected four key files I think are most important. Some competitive teardowns, our strategy and then a recent roadmap. And then what you could do is actually just take it out for a test here on the right. So let's say, I say give me some product, let's say maybe new product ideas for this year and it'll go off. Use all that context.
- [07:55]
A
Yeah, I like how because you're a public company, you can just put the earnings release in to give it the lens.
- [07:59]
B
Well, actually this is not real data, but yeah. So here you go. You can see does a nice job here. You can continue to tweak it. If you don't quite like the output, just change the instructions, change the knowledge here and just kind of iterate on it until you're happy. And once you're good, you can go ahead and save it. And what you'll see here is you can start a chat with anytime and it's listed here under your gems category that you can refer to anytime you want. So there you go. Please do check it out.
- [08:29]
A
Is this something you can share with your team? So like as a product leader, could I create a gem and then say, hey team, here's a gem resource you could use or maybe.
- [08:36]
B
Oh, totally. Yeah. So one of the key use cases we do see is for personal productivity and that extends beyond you yourself to your broader team. So, you know, especially for a given function or inside a given company, you all have very shared context and so create them, share them around. You can be that power user in your own company to build it for your teams. And I'm sure your, your colleagues would be really happy to use your gems.
- [09:02]
A
So what's the story behind this feature? Because a lot of people, they're being asked to build AI features, even if they haven't before, it could be really useful for them to get the AIPM sort of zero to one process of building a big feature on a big AI product like this that is so successful. How did you guys do that?
- [09:22]
B
Yeah, you know what happened is I worked on this back in 2023, which at this point feels like, you know, the age, like a long time ago. But at that time, even then we realized alums are so powerful, there's so many things they can do, but the average user was having a hard time discovering all of the capabilities that were possible. So we brainstorm as a team and we're thinking about how do we unlock people to discover more of what it can do? Perhaps could we give Gemini different Personas, different instructions, and maybe let users customize that for themselves so that they can save it in a way that better fits their mental models? So instead of just big Gemini for everything you want to ask it? Maybe I have my doctor gem when I want to ask for medical advice, and then maybe I have my writing tutor when it's like I need to write something really nice. So anyway, we want to offer up these tools to help people better discover the different capabilities of Gemini and also create a way for people to share, just like the example you mentioned of maybe sharing with colleagues or others in a similar industry as a way for people to discover more of the things that Gemini can do.
- [10:32]
A
For my money, the most important new skill for every product manager, whether they're an AI product manager or not, is AI prototyping. And Lisa even touches on it in this episode. The PRD used to be a 10 page document. Now we're sending an AI prototype along with a one page spec aligned with it. The problem with most AI prototyping tools is they're not purpose built for product managers. They're built for founders or aspiring vibe coders. Realistically, you need a tool purpose built for product managers, and that is today's sponsor, Reforge Build. Reforge Build is purpose built by the team that knows product managers better than anyone. Reforge. Brian Balfour himself is going in there and making sure that it takes your design system. It helps you create divergent ideas and it helps you get the most out of your AI prototyping tool. So use Code Build to get a discount. Use Aakash's bundle to get a full year free and click on the link in description below and go visit Check it out. That helps support the podcast and supporting Reforge Build helps support the Reforge ecosystem with that. Back to today's episode with Lisa. Did this come before or after custom GPTs on ChatGPT?
- [11:48]
B
Yeah, you know, it's such an interesting story because we started working on this, so it kind of came up organically in the team. And then in the middle of it, chatgpt, you know, OpenAI the launch, custom GPTs. And so suddenly we were okay, I think we have to go faster with our launches, go faster with how we're developing this. Yeah. And we're off to the races.
- [12:10]
A
How, if at all, did you try to make it different from what they had launched?
- [12:15]
B
So when OpenAI launched GPTs, the framing of that Product launch was really around. This can become kind of like the new 3P app ecosystem. So, you know, if you remember, they launched custom GPTs, but then also the GPT store they talked about there's ways we will offer tools to monetize over time. So very much the same framework paradigm as an app store. When we looked at the capabilities of and how we thought people might use this, we took a bit of a different approach. I think our perspective at the time was there's not that much proprietary about each of these gems. The instructions are easily copied. Yes, you can give it custom data knowledge, but at the time, it wasn't clear that you couldn't just easily prompt a gem to kind of spit out what some of that custom knowledge was. So it wasn't clear that this would really foster that burgeoning ecosystem or app ecosystem. So instead we really focused on how can we support Gemini Gems as a personal productivity tool. So something you customize for yourself to really amplify your productivity, things you create for your own teams, your colleagues, that can really amplify all of your collective productivity.
- [13:28]
A
And I think that's an important takeaway for folks like a Me Too feature. It doesn't always do as well versus like coming up with the first principles of what are we trying to accomplish at our company and then working backwards from that. And it seems like it turned out pretty shame because not too many people are monetizing custom GPTs. They didn't really pursue that angle. So sometimes you can also really see ahead if you think from first principles.
- [13:53]
B
Yes, absolutely.
- [13:54]
A
So whether people are building a jam or they're building a ChatGPT project, or they're using my favorite AI these days, Claude and building a Claude project. So if we were to build a mind map together, what would be the most important projects or gems every PM should be creating?
- [14:09]
B
Yeah. So when you think about what are the key skill sets of PMs, I might break it down to four big categories. So you certainly need to own the overall product strategy, and you'll need to own execution for that with your teams. But also, very importantly, you need to communicate that to a lot of different stakeholders. So that's going to be key and then informing that product strategy. You've probably done a lot of research on competitors, on users, and that those are key data signals to input as well.
- [14:35]
A
Okay, makes sense. So really, if you're using this feature to its best, you probably have something like these 20 Gemini gems or projects created.
- [14:45]
B
Yep, yep, absolutely.
- [14:48]
A
What are the biggest mistakes people are making with these.
- [14:50]
B
Yeah, so I'll flag a few. As I mentioned, you gotta be really clear with your instructions. So not a vague thing like help me write better, but the more detail you can give it, the more examples, the better. So do spend time on those instructions related to this. You also want to make sure you have the key context uploaded. So what is it that makes it personalized to you and your circumstance and not just general instructions, for example, key company documents, example emails, whatever it is that you're looking for to help mimic and help follow, make sure those are uploaded. I do recommend specialized gems for different jobs. So just like we talked about in the last exercise, if you break down the different tasks you have, what is the specific gem you want for each of those? And then as with all good products, I'm sure the first version is never going to be perfect. So make sure that you are iterating, you're testing it, and as you use it, continue to add those tweaks along the way to make it even better over time.
- [15:46]
A
And does the gem have like persistent memory in it or do you need to make those changes to the system prompt and context files?
- [15:53]
B
It will, it will read off of what the instructions are there in the context files. So if things change on that over time, then you should update that as well.
- [16:02]
A
Okay, so I think that is one distinction maybe with Gemini Gems and cloud projects. For those of you who have been following my content on cloud projects where they, you can kind of train the project. So I think you may want to, maybe even if you're training it, you can probably even ask the gem, hey, how would you update your system prompt to accommodate the types of edits we made today? And then it can probably give you a system prompt back. Yes, iteration is key. I've been using these things like crazy and I have some that I just like religiously live with now. So treat it like an AI mini product that you're creating for yourself and iterate, I think is the key. So that's our first segment we wanted to cover for you guys. That's Gemini Gems or Projects Masterclass for you. Now I want to talk a little bit about the AIPM career because we have somebody who has crazy amount of experience. Right. In fact, Lisa's kind of been collecting these like infinity stones. She's got McKinsey Private Equity, Apple Meta, Google now, zero. So my question for you is, what are the key lessons? You have this vantage point that so many people want. Like you've seen it all. What are the key lessons you've learned from navigating through all these amazing companies in your career.
- [17:15]
B
Yeah, you know, it's interesting because every time I get that question, it makes me smile a little bit because I never planned any of this. I'm not someone that knew what I wanted to be when I grew up. I might say I still don't really know what I want to be when I grow up. And for me, the principle I've always used in my career is really curiosity. What's the thing I'm interested in learning next? How do I want to grow next? And that's led me to every opportunity. I mean, looking back on a nice slide like this, it looks like, oh, that was a nice plan. You kind of followed abc, but in reality, every step of the way, you're kind of figuring out the next step. Yeah, and I feel really lucky to have been able to work with some really amazing people on some really amazing projects over the time.
- [17:59]
A
All right, I think this really resonates with what you've talked about where you've said make your PM career your own. What do you mean by that? What is the practical takeaways for a more junior pm?
- [18:10]
B
Yeah, you know, I think what's really interesting about PM is that it's so different company to company, role to role, product to product. The responsibilities are quite varied. And so I like to think of there's different archetypes of PM. Some that are really good at 0 to 1, some that have that craftsmanship and really love consumer, some that really think about platform. So I think the important thing is to figure out what are your strengths, what are the things you enjoy about product, where do you want to play in that product space and then set out to find those roles, those opportunities, the people that will support you to really hone that part of your craft and then also figure out what are the tool, additional tools you want to add on top of that to round out the skill set. And that's how you can make product, you know, your own.
- [18:58]
A
So Apple meta, Google, I feel like we, they seem to have completely different product cultures from the outside looking in. Apple seems to be very design focused culture where they make design and executive decisions and then those decisions kind of filter down into how you might build the product meta. It seems to be very growth testing, lots of experimentation, type of culture. Google seems to be, you know, putting the user first, being very first principle. That's at least what it seems like from the outside. What can you confirm or deny about what the product culture is like at Apple versus Metaverse Google, yeah, they're quite different.
- [19:37]
B
But I think one thing I really appreciate is what I've taken from each place. I take what I like the best of each and kind of take it with me into my future, into my future roles. So Apple, the thing that I really like is product is just supreme there. And what I mean by that is everything bends to the will of having that amazing, amazing product. So we don't care which teams we have to drive crazy for it, we don't care that it's like just a sub millimeter off. We will fix it, we will fix it until it's perfect. And so that like love of craft that like details, that every little detail really matters and holding an extremely high bar on product, that's something I do take with me, I've taken with me to different roles. And then on Meta, yes it is very experimentation driven. The thing I love about Meta is very, very data driven. One of the best data cultures I've been in anywhere. Like I've been way too just spoiled with the kind of data access I had there and the available tools that were there, but also really, really impact driven. So lots of focus not just on like the perfect strategy, like we need a good strategy, but also like we gotta execute fast, we have to compete. So that part I take as well, that execution muscle and that you know, really focusing on metrics and data. And then at Google, Google is the most technical of all the companies I've worked in and that applies to PM as well. PMs are expected to be very, very technical. They're expected to partner very, very deeply with their engineering counterparts. And so I grew a ton of technically in my time at Google and that's something, you know, especially now as you look at AI and the future of where things are going, I think that's just going to be an expectation of every pm. You really have to get much more technically deep.
- [21:18]
A
So now you're in the unique position as an svp, I'm sure of hiring lots of AI pms. When you are looking at the backgrounds of people, what are the things that stand out or don't stand out? What are you looking for in an aipm?
- [21:32]
B
Yeah, so I think there are maybe I'll call it some core things and then some additions. So on the core it's your standard PM101 skill sets. Do they understand product strategy? Do they have a good vision on things? Can they understand metrics? Can they work well with different teams to execute upon that strategy? And then the cross functional skills can they work well with others? Influence without authority. These are what I'd call standard. And then things I always look for, which are a little bit extra, are. It is that grit, that growth mindset, the willingness to push beyond. So not just what's expected, but like, do you care deeply enough about this to solve it? And like, you'll let nothing stand in your way? Some people do have it. You can, you can tell pretty easily as you talk to them about their past projects. And I find that those are folks who, they learn really fast. You're able to put them in any environment. Things are constantly changing really rapidly. And so they're really good folks to have around because they're just able to solve problems and push through things.
- [22:33]
A
Makes sense. So we covered a lot on product development because I just needed to hear about that from you. Now I want to go deep on one of these experiences we have here, which is Meta. And actually I want to put something on the screen for everybody because if you guys don't know, like these meta glasses, they're really taking over as far as the most successful AR product out there. And you worked on one of the coolest AI features for AR out there. You worked on the first generation Meta AI Assistant for Ray Ban Stories Smart Glass. So what was the story behind this feature?
- [23:07]
B
Yeah, it's. As with all things, yeah, it's a really awesome product today. I will say there was definitely a long road in getting here. So I worked on this product back in 2019 through 2021, so a long, long time ago for the first version. And at the time I was a part of the Meta Assistant team. And back then, you know, I had a really strong vision that the AI assistant would become the singularly most important feature on these glasses. But is definitely not true that everybody agreed with me. So there were folks who thought, hey, the AI thing is just going to be a voice input, that's nice, but we also have other input mechanisms, you know, and, yeah, very convinced that we needed to invest a lot more here and that this would, even though the technology was not fully there at that time, that this would become that, you know, future intelligence and interface layer for this hardware. Anyway, long story short, we did manage to get it funded. We worked very closely with the Raybond Stories team, as with all 0 to 1 products, lots of learnings there. This was the first time that we were working so closely with a major eyewear company like Zotica. So lots of executive involvement on our side, on their side, a lot of joint collaboration, lots and lots of unknown. So from is this going to have product market fits? What are the real use cases? We put a camera on people's faces. What about privacy? How might bystanders feel about that? What's the user interaction on this thing? And then the engineering complexity was extreme. Right. You're talking about lots and lots of technology in a very, very small space and you still have to make it fashionable and you have to make comfortable to wear for long, many hours. So yeah, a lot of constraints, a lot of challenges, but like tons of fun. It was just learned a lot through that entire process and you know, made some, made some good friends along the way as well.
- [25:00]
A
So I've talked to a lot of AI PMs they might be building without those constraints. Like you are thinking about building an AI into a wearable. Like you said, it can't be too heavy. Like we already saw, the Vision bro flopped this hit 4 million sales in 2025 because it was light enough. How does that impact the type of AI features you can build? What are the ways to get around that? Are you going to host the AI in the cloud and do the processing there or how do you build AI for ar?
- [25:27]
B
Yeah, yes, a lot of today's models are through cloud, but a lot of models are getting much, much better. And I personally believe there's going to be a wave towards on device for a lot of reasons. Right. Privacy people don't really want to share their data out there. Once you do make it small enough that it can be on device, there's a lot more places you can put reduces kind of battery and other kind of technical constraints as well. So personally I think that the future of where AI is going to go for AR is that it's going to a lot more of it. Maybe the vast majority of what you want to use it for will be on device. And yeah, I think it'll make everyone feel more comfortable using it as well. That what's your private device and what it's capturing day to day as you're putting, you know, you have this thing on your face and you're going through all your different interactions that that stays local and secure.
- [26:15]
A
So we see kind of at least three big players investing heavily in this space. OpenAI, Apple and Meta. What would be your advice to PMs if you are one of those PMs working there in those roles, how do you truly build successful features in this space?
- [26:30]
B
I think it's probably a similar advice for really any product or any AI feature. Don't fall Too much in love with the technology, but also deeply, deeply understand it. I, you know, having worked in AI products for about a decade now, I think the best products are created when you mirror. You deeply understand kind of the user need the product value, but then also deeply understand the technology and find that like perfect little intersection in the middle that does both. It leverages the technology in the right ways that are actually really useful for people. So yeah, that's, I would say that's step one and then step two is just test and ship and learn. Right. Everything is developing so quickly. I don't think it's worth it personally to overthink too much because whatever your assumptions are today, in a month and two months, it's going to change. So just, just build, go fast, see what people are doing. User behavior is also changing really quickly. User expectations and what they're willing to do is changing rapidly. So yeah, have your hypotheses. Stay close to the product and user needs, stay close to technology and then just build and iterate.
- [27:34]
A
We've been following the story of your career. We dove deep into Gemini Gems AI assistant for Meta Ray Ban. I want to talk about the latest AI features you have been building. This is Jax at Xero. Can you walk us through what this feature is? Some people may not even know what Xero does, so maybe you can give us a teeny bit of background on that so that we can understand how this is a significant feature.
- [27:54]
B
Sure. So Xero is a finance platform for small businesses. We do accounting, payments, payroll and other financial services for small businesses. Globally we serve about 4 million businesses. We are about $18 billion market cap company. And yeah, we are in some sense for small businesses finance, this financial system is their lifeblood. You know, how they understand what's happening with their business, how they get cash in, how they make payments out. So it is a really, really core part of how they operate and that's why we are quite deeply embedded with a lot of these small businesses. So the key product I've been working on, I joined Zero to lead AI as a. The key products I've been building here. It's really around this umbrella of Jax as a financial super agent. And what it means is we map out all of the financial workflows that small businesses use. All of their key jobs we've done and we look in detail at which parts of those can we automate with agents, with kind of other AI. What can we do to offload a lot of that manual work for small businesses. But on top of that what can we do then to also understand the deep insights embedded in those businesses? So for example, like on a platform like Xero, we have transaction level data of each of these businesses. So every invoice, every bill, every payroll run that happened, we know a lot of detail about what's happening in that business. And this gives us an opportunity to really leverage that data, understand those deep insights and help our small businesses grow their businesses better, help them survive, thrive and really achieve what they what they want to do with their business.
- [29:30]
A
So everybody's been trying to build agents into their products. What have been some of the harder lessons learned that you've won from experience in building an agent? I know sometimes people talk about unreliability of MCP for tool calling. People talk about, you know, customers having higher expectations than what their early versions of the agent could do in terms of tool coverage. People talk about the unreliability and the non deterministic nature of some of these agents plugging in and then configuring a product wrong. And in accounting I imagine that's doubly an issue. How have you guys built around some of those key problems and what have been some of the hard lessons about building an agent into your product?
- [30:07]
B
Yeah, what's really interesting about working in this space is when you're talking about finance, the decimal, you know, the accuracy down to the decimal really matters. So accuracy is not a nice to have. It's a core part of our differentiation. And what we know is the LLMs out of the box are not that great at math, accounting, tax, et cetera. So the thing that we bring, it's two things really to make sure that our AI is really accurate and really reliable. Number one, the domain knowledge that we apply. So we are in a position to deeply understand what are these workflows that small businesses actually use, what matters to them, what doesn't. Every step along the way, what accuracy level is required and what is acceptable at different task and subtask levels. We understand which stakeholders need to be looking at that data and using it in which ways. And so we do fine craft our JAX experience to make sure that it is well suited and something that will plug into you users existing workflows and be really useful for them in what they're trying to achieve. The second thing is what I alluded to before, we have a lot of data on our system, so this definitely gives us a leg up in terms of one, we can personalize to every business much better and number two, we're able to use that data Collectively to understand what are some of the key trends, benchmarks, et cetera, happening across small businesses by sub region, sub industry, et cetera, and really offer much more helpful AI built on top of that. And you mentioned MCP and tool use, et cetera. We do use a hybrid system, so we leverage LLMs in kind of multi agent workflows in different ways. But then also importantly we also use programmatic code and really hybrid structures where we need more control over things to ensure that we have that high quality. I'm pretty blessed to work with some amazing engineers and AI scientists and the team. We have a bunch of X Google X meta X Amazon folks from these places and we've taken a lot of care in building out really robust quality measurement, eval and kind of flywheel systems as well as a team of expert annotators from the finance space that can really make sure that our AI is behaving the way it's supposed to when we're talking about financial data.
- [32:22]
A
So how do you measure success for this feature? And maybe you can talk both about kind of the evals because that's a really hot topic for AIPMs right now, but also just the basic success metrics, usage, engagement, whatever, sort of. How did you get over to ARR Impact on retention? Would love to understand what is the right framework to measure a new agent being launched in your product.
- [32:43]
B
Yeah, so I, I would break it down to three phases or three, three areas that need to build upon each other. Number one is just baseline quality. Is the AI doing what it's supposed to be doing? Number two is like regular product metrics around user engagement. So do people, are people adopting the product? Are they liking it? And then your third metric, which hopefully all this ladders up to, is the business impact. Are we making money? Are we driving the right revenue? Are we hitting those business metrics on the first one on quality? We have a pretty robust framework internally. It's something that we've developed but also will continue to evolve over time because for different use cases you are going to need different evaluation criteria. They're not always the most straightforward and you need to get clever with how you measure it. We definitely do leverage annotators to help us measure human evaluators along the way, but of course that doesn't scale. And so we also build LLM judges and kind of other eval metrics to help us understand what is the real quality of the product. And that's something we track very regularly across all of our use cases. And we, you know, we understand the current quality, what are some of the gaps and what are the initiatives the investments are going to make in different areas in order to improve the quality over time. So that's one on the user engagement metrics, very similar to, you know, regular products. So just usage MAU in our, well depending on the use case, it could be monthly active, it could be weekly. Some of them are daily. Yeah, just adoption, usage, retention and then the qualitative side. So csat, we do talk to our customers very regularly. So we get a lot of anecdotal feedback as well. So that's something we do monitor, you know, on social media channels or as we're having customer conversations, so that we talk as well. And then on monetization, this, this is all about looking at different companies may have different ways that they attribute revenue. For us, you know, we have a system internally to attribute the revenue we're making from AI and that's something we track as well.
- [34:38]
A
So you are at the forefront of building agents into your products. Sequoia had this really interesting thesis that they presented on tvpn, which I wrote about and ended up going pretty viral on Twitter where they said that you need to start thinking about agent led growth. I think your product is the perfect example where it seems like the JAX agent is doing a lot of the work now in the Xero platform for you. And so the, the thesis of agent led growth is that agents are going to be talking to agents, agents are going to be picking what tools, what software in the future. And so you need to think about how you build your project product to be attractive to agents so that maybe the JAX AI actually thinks about using your tool instead of another tool. Is that a valid thesis? What do you think about this concept of agent led growth?
- [35:27]
B
I do see it happening in the future, but I think we're pretty far away from it, to be honest. I know that, you know, in Silicon Valley people do tend to live in the future, but if you take a step out of that, and a lot of our customers are certainly not Silicon Valley, you know, homegrown businesses, they're mom and pop stores globally. You do get a view that people are very excited and they want to use AI, think they see a lot of potential in it, but the adoption curve is still going to take time because people have to trust AI, they have to adopt it over time. They need to train their staff on it. So I think before you get to agent led growth, you first need to, you need to get companies to actually delegate more and more to agents. Right now, even the age, at least from what I see, the where there are agents, it is still quite, there's still quite a lot of human monitoring. People are not necessarily. And especially in our space, right, in kind of the finance space, you're not going to let these agents go and pay your bills for you without someone looking at it, et cetera. So I think we'll get there over time, but I think it's still quite a ways away. I think we're still very squarely in the getting people to adopt and really leverage what AI can already do today.
- [36:41]
A
Okay, so you've done the hard work of building an agent into your product. Should most B2B SaaS, product leaders be thinking about building an agent into their product? Or what is the framework to understand whether you should, Oh, a hundred percent,
- [36:55]
B
like in some sense, everybody is already behind the technology, moving so quickly. I think user adoption and company adoption is still behind where the technology actually is. And so, yeah, my advice for every B2B SaaS is like, you should be deeply integrating agents, AI everywhere because the future software is not going to look like it does today. And that future is coming very, very fast.
- [37:21]
A
So we were talking about the future. I want to talk a little bit about the future of product management because you have a vantage point that pretty much no one else will have had working at all these big companies and now leading product at one of these big companies. What is the future of AI product management?
- [37:39]
B
Sure. So let me answer this with two questions. Number one, will AI replace product managers? I don't think so. The reason I don't think so is yes, AI is great at specific tasks and 100% the AI role is going to change. PMs will be expected to use AI for a lot of things that they still, some people still manually do today. But I don't pay PMs just to write PRDs or just to create mocks or to manually manage roadmaps, et cetera. I pay them for their product judgment. And that's not something I see going to an AI anytime soon. Because it's not that there, there's not a clear right answer. Right. A lot of these things, it's, there's a bunch of signals, they're not super clear. And you have to kind of give your product opinion. What should we do? What shouldn't we do? Why, why is this the right answer? What is the taste and the judgment in all that? So I don't think this is going away anytime soon. And that is a key skill that we're still going to need PMs for, but they'll just be supercharged by using AI in some of the more executional tasks that they do today.
- [38:40]
A
You know that feeling when you try to prototype something with AI and it spits out something completely generic? Then you spend hours tweaking colors, fonts, copy and features just to make it feel like your actual product. Here's the problem. Most AI app builders aren't built for product teams. They're built for those starting from scratch. But product teams aren't building from zero. You have an existing product, real customer design guidelines, a backlog full of ideas you need to explore and validate fast. That's what Reforge Build does. AI prototyping that starts from your product. Add your customer feedback, strategy docs and product features as context. Create reusable templates using your product design. Explore multiple variants side by side. Collaborate with your team in one place. Reforge Build generates prototypes that reflect your real pricing tiers, real features, real customer language, not generic placeholder Stop fighting tools Built for founders Start prototyping like a product team reforge build AI prototyping built for product teams try it free at reforge.com akash that's r e f o r g.com a a k dash a s dash and use the code build for one month free of premium. There's a lot of PMs I talk to who are feeling anxiety about this. You know, because there's that the layoffs. It seems like the layoffs might be disproportionately hitting the PMs. It seems like, you know, getting a junior PM role is harder than ever. Plus it seems like the director of PM role might be getting compressed or group product manager level where VPs are managing a bunch of ICs. What's your take on the future of this role? Is it going to continue to grow as fast as it has in the past?
- [40:13]
B
So what I will say is I do think the structure of PM and ENG and design is going to change. And what I mean by that is you don't need as big teams means to go do stuff. So the structure is definitely going to change. And historically we kind of look at PM to end ratios that will certainly compress. If you don't have as many engineers, you're not going to need as many PMs either. But I think what's interesting is I do think the PM role is going to evolve. PM UX design is going to become more of a hybrid. So I think it will become the expectation that PMs are also builders. Right. They got to go understand the product, propose it, champion it, but then also create the first design, create the first prototype. Like go build it, go code it. Like you can go do it with all the tools that are here now. So the roles can evolve. We're still going to need products in the future. I think there will still be needy roles. Yes. I think the change is it can cause anxiety because it's changing so rapidly right now and it is a more competitive market because there are fewer roles. There was kind of a glut over the last few years of hiring into the space. So it just means that there's a lot more competition. My advice would be if this is your passion, it's what you want to do. Just go learn the tools, reinvent yourself. Now is a time of transformation and everyone has the availability to go do that. So yeah, go ahead and lean in. And I would still encourage people to follow this career path if they're interested.
- [41:38]
A
Whenever I give this advice, I hear a bunch of buts. So I want to roleplay some of those buts. I think the biggest but is but I'm not working on AI features or products at my company and we don't have access to many AI tools. If that's the case, if you're that pm, how do you lean in? How do you kind of get to the part of product management that is growing right now?
- [42:00]
B
Yeah, I think that's. Personally, I think that's not an acceptable excuse. And the reason I say that is Everybody, not even PMs, just like people, everyone in the whole world mostly has access to AI tools now. It's not very expensive. You can get a little, you know, you can get a subscription, do cloud code, use OpenAI's tool, use Gemini's tool. A lot of stuff is free and then it's just a small amount if you want to pay, you know, for a little bit more of a premium experience. Experience. But you have the AI tools with you already. The vast majority of companies are not necessarily fine tuning models. They're using it off the shelf. And places like Claude, OpenAI, Gemini, they offer really good ways to access the latest. So you already know what these models are capable of just by using those consumer products. So I would 100% encourage people just go. If your company is not officially giving you a role to go build AI products, why do you need them to build AI products? Just go use the tools yourself. You can go build your own AI products. You can build Your own gits. Right. Just do all of that stuff yourself, there's no one stopping you. Yeah. I think it's never been more easy to access these AI tools than it is now. So for those that are really interested in hustlers, like, please do lean in and just go build stuff.
- [43:09]
A
And then you get to see the flip side. You get to see how people are marketing themselves. Having done this, what is the way as a job seeker to stand out to a hiring manager and showcase my side projects, showcase that I am using AI tools on my own. Own.
- [43:25]
B
Yeah. So I think here one is just doing the work. A lot of people are still talking about it, but not really doing it. So even that gives you a bit of a leg up, I would say. The second thing is the thing I mentioned before about like, you're going to have to check the box in the foundations, right. Product strategy, execution, cross functional, stakeholder management, et cetera. But the extras, not everybody has those extras. So if you are passionate about the space and you want to show that you're really leaned in, there's lots of ways to demonstrate that. I'll give you an example. In one of the recent people I hired that I interviewed, we were assessing them for AI role. They actually didn't have AI experience before. I was not sure that they'd be the right fit for our particular role. But this person really impressed me with in our first interview, what they said was, okay, looks like you're trying to build kind of financial tools for small businesses. Well, I went through and I watched three hours of TikTok videos from these coaches that coach small businesses. And here's a summary of all the things that they said about what these small business needs have and advice that they might want to give them from a finance standpoint. And I was just like, whoa. That was. I hadn't met a single other candidate who'd done that. And it just kind of showed the passion, the willingness to like roll up your sleeves and go deep. And that, you know, I deeply respect, by the way. This was a very senior person. So the fact that they were willing to just do like deep IC work themselves really did speak to me.
- [44:52]
A
So you're hiring these AIPMs. What are the vectors you're looking at to see that AIPM is staying on the bleeding edge? What are the things you're encouraging your AIPMs to do to stay on the bleeding edge of these tools? Because they're changing so fast.
- [45:06]
B
I think the best thing you can do as a PM is just to use Them you can apply them either in your job or it's actually totally fine to apply them in your personal life. That's typically what I've, what I've tended to do because there are less restrictions on using my personal data for a lot of these tools versus, you know, company data. But yeah, I think, you know, really just trying out obviously it's the horizontal LLMs. They provide a lot of new tools and capabilities all the time. They're really at the bleeding edge of a lot of things. So just using the latest that's been launched, but on top of that in your day to day, you know, are you using design prototyping tools to create your own designs? I have a PM who actually everything that we talk about, he always pulls up something in our code base, like our actual company code base and he will go prototype the idea he's talking about. So it looks like it's inside our actual product, which is pretty impressive. So yeah, whatever you can do to use AI tools for your product thinking, for research, competitive analysis, user research, use it for design prototyping, create your own designs and then build it yourself. I think the more of that you can do, you'll naturally use these tools. You'll naturally hear about them from people. You'll notice the little details of what's working, working well and not working well and why. So I think this will, this is plenty to keep you on the cutting edge. Beyond that you can always. There's like a million news articles about AI these days, so subscribe to your Just pick three high quality ones that you kind of follow week to week and you'll get all the key information you need from that.
- [46:32]
A
So don't overload yourself with unlimited content. Spend more time actually testing out the tools using your personal data so that you aren't restricted by what your company is doing. When it comes to advice that you have. We've gone over so much advice, but fundamentally the advice that people want the most is what is the roadmap for me to break into any one of these Apple Meta, Google zero, any of these big AI companies. So if you had to put together a roadmap for somebody, what would be the roadmap to becoming an AI pm?
- [47:04]
B
I would say if you have the opportunity to do it in your job, that's great because you're kind of doing it. You have to go through all the regular challenges of working with the team to go do it. There may be constraints from a resourcing standpoint, from a how do you go to market et Cetera, So if you, if you have the opportunity, definitely lean in there. That's always, is always great. If you don't have those opportunities, or even if you do, you could still continue to do things on the side because, yeah, this gives you the maximum flexibility to explore all kinds of things that you may not be doing specifically in your current role. Role. The other thing I would do is it is important to continue investing in building your network sometimes that it can give you leg up into some of these companies. So do continue to meet people, build relationships, keep in touch. And that relationship building, by the way, starts in your current role. So whoever you're working with, they often go off into do great things in the future. So do make sure you're showing up well in your own teams and keeping in touch with people that you've worked with in the past. And then the last thing I would say is if you are targeting one, targeting one of these big companies, they tend to have fairly standard interview processes. I will say interviews are themselves a skill. So you may be a great pm, but if you can't communicate that in an interview, in the structure of that interview, with the time constraints and the format that it has, it just may not come across. So when you are getting ready to actually interview, you just gotta practice interviewing. Go do you know mock interviews, Go ask your friends who are in those companies to help you do just practice over and over, over, just drill it and that will set you up for the best possible success in those interviews.
- [48:41]
A
That is one of the clearest roadmaps into breaking into FAANG that I've heard in a while. So if I had to summarize what I heard right, network with people, get referrals into these, obviously do a great job in your job, so you have direct AI experiences that you can then talk about in the interview and be ready for the game that is the interview at these companies. A lot of companies, especially Meta and Google, are not going to be just asking behavioral questions, they're going to be asking case interview questions. Product execution question and product sense being the most common dimensions. And so you really need to prepare for that specific beast that is those interviews. Do you have any tips for people when they're preparing for case interviews at faang?
- [49:19]
B
I think repetition would be my main advice. The more you do it, the more it'll just feel like second nature and you'll have seen a breadth of different types of questions. So I would just practice, practice, practice. If you can get folks who work at those companies or who used to Work at those companies to do mock interviews. That's great. And then for additional time, additional practice, you can actually just practice on your own. There's a bunch of sites that offer example questions. Just do them yourself. You know, do it as if it was a real interview. So speak out loud, write down what you want to, and then you can look at some of the example answers at the end to help you as well.
- [49:52]
A
Amazing. I tried to pick your brain. Of all the most important advice. Is there anything I shouldn't. I didn't ask you that. I should have.
- [49:59]
B
About landing a career in pm, I think, you know, I think we cover most of it. Maybe. The only other thing is I think passion and grit really shows through. So the more you are truly interested in that role, in that company, in that product, it will show. So do your research. I personally think you should always go for something that you're truly interested in. Not just because of a name, because of, you know, something on your resume. Yeah. The more passion you have for the product, both, it'll help you do better in the interview. But two, once you're in the role, you'll do a better job.
- [50:36]
A
Amazing. Lisa, thanks for taking the time to share all these amazing insights with my audience. If they want to find you online, where can they go?
- [50:44]
B
I'm on LinkedIn. That's probably my main thing. Although, yeah, you can. You can connect with me on LinkedIn and look forward to connecting.
- [50:52]
A
All right, everyone, we got so many different things we learned about today. We talked about how to create great AI features. We did deep dives into Gemini Gems and the AI Assistant for Meta Ray Ban. We talked about how to use Gemini Gems and we talked about how to become an AI Product manager and navigate your career. Hope you enjoyed this. There's more details in the newsletter and we'll see you in the next episode. I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify and commented on YouTube, YouTube. All these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content in the production to get you better insights to stay ahead in your career. Finally, do check out my bundle@buildle.akashtri.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.