Loading summary
Transcript90 lines
- [00:00]
Akash Goel
What is the most powerful AI workflow you've seen?
- [00:02]
Frank Lee
I think the most powerful thing is managing my product process in Claude code and cursor using a bunch of MCPs.
- [00:08]
Akash Goel
What does that mean in plain English?
- [00:09]
Frank Lee
So basically, you can service all of your product context into your agent or tool of choice.
- [00:13]
Akash Goel
What can you do with this?
- [00:14]
Frank Lee
Oh, man, I think you can do so many things. You can analyze charts, you can automate your weekly reporting, you can synthesize a huge amount of qualitative feedback, you can turn all of those insights into a bunch of specs, and then you can even push them to quad code or cursor to start prototyping directly in code.
- [00:29]
Akash Goel
So I've got one of the heads of AIPM at Amplitude, the principal PM in charge of their agents and MCP products here today. And we're going to walk you through everything you need to know to master analytics in cloud code.
- [00:40]
Frank Lee
I automate a lot of my weekly business reporting using a bunch of dashboard agents. I can easily analyze, run successive queries, pull in a bunch of data, and just run a bunch of analysis on my product data.
- [00:51]
Akash Goel
Wow.
- [00:51]
Frank Lee
They made this conscious decision on like, how do we make the most powerful agent or what are the most powerful features and set of contexts that can provide the model so that people can code in ways that they've never been able to do before.
- [01:01]
Akash Goel
What is the biggest mistake people make when using mcp? Before we go any further, do me a favor and check that you are subscribed on YouTube and following on Apple and Spotify podcasts. And if you want to get access to amazing AI tools, check out my bundle where if you become an anal subscriber to my newsletter, you get a full year free of the paid plans of Mobin, Arise, Relay App, Dovetail, Linear Magic Patterns, Deep Sky, Reforge, Build, Descript and Speechify. So be sure to check that out@buildle.akashg.com and now into today's episode. Frank, welcome to the podcast.
- [01:40]
Frank Lee
I'm stoked to be here. Thanks for having me.
- [01:42]
Akash Goel
My pleasure. What is the most powerful AIPM workflow you have seen people use these days?
- [01:48]
Frank Lee
It's got to be managing your whole product process out of cloud code cursor and hooking it up to some product analytics providers like Amplitude's mcp.
- [01:56]
Akash Goel
What does connecting Claude code via MCP really mean in plain English?
- [02:00]
Frank Lee
Okay, so there's two different tools. Claude Code. It's a terminal based coding agent with access to a bunch of different cloud models, but has some pretty cool stuff like MCPS skills, a bunch of other actions. Mcp. It stands for model Context Protocol. And the simplest way to put it, it's the easiest way to connect your AI models with any external tools, action and data. So we've used MCP a lot at amplitude to connect a bunch of tools and data that we have to a bunch of external AI clients. So big ones include Claude, Cursor, ChatGPT, and we use that to power a bunch of pretty complex coding and product workflows.
- [02:33]
Akash Goel
So what does this setup allow PMs to do?
- [02:35]
Frank Lee
So this setup lets you do a lot of things. So I automate a lot of my weekly business reporting using a bunch of dashboard agents. I can easily analyze, run successive queries, pull in a bunch of data, and just run a bunch of analysis on my product data. I can deeply investigate different trends that are happening in charts. So why I used to be creating a bunch of manual charts, the model is doing all of that for me. Now it's really easy for me to explain why my metrics are moving because I can pull in all of the underlying feedback data to try to hypothesize about what's what, why things are changing. I can drill into specific account health of our users and also some timelines about what people are doing within our product just to see like what their behaviors are like. I've been actually using these models to draft a bunch of PRDs based off of this analyzed product context that MCP makes so accessible in cloud code. And then if you're really venturous, you can actually take some of those insights that you're generating, convert them into PRDs. If you want to route them to a human, you can push them into JIRA or LINEAR via mcp. But if you're really daring, you can also push them directly into Claude code cursor and try to draft some prototypes on your own.
- [03:38]
Akash Goel
Okay, so it's really the gateway drug to all of the 10x AIPM workflows that people talk about these days. Before we really get into this, can you show us how to set all this up?
- [03:48]
Frank Lee
All right, sounds good. Let me walk you through how this is set up. So I'm actually going to jump the cursor if you see like this repo. I've spun up my own amplitude product repo. So on the left hand side I have a bunch of contexts that I've already actually aggregated about all the different product lines that I work on. So for example, we've been working on agents, AI feedback, ask AI. Like a lot of Our enterprise control features mcp. So I have specific folders and contexts and files detailing about some of the initiatives that we might have like within our, like our roadmap and some of the specs that might be thinking about. And also just a bunch of these different learnings and like verbiage that refer to like the specific parts of this product. So within Claude code or within Cursor I can easily refer to some of those pieces of context. Brainstorm about it Draft new spec. But I think the most interesting thing from the most recent changes in cloud code and Claude iterating on this like new protocol called Skills. So I'll just jump into a specific example. But I've created this specific skill and the way that I simply think about this is it's a clear like name about what the skill is. There's a little description about like what the skill is and like when to perform it. So that's some metadata that the models can use to determine when to use that function. But at the simplest nature, this is just a prompt. There's a prompt about what is this specific task that I'm accomplishing, what is the set of heuristics or steps that I might take. And then also you can configure what set of tools you might use that is connected to MCP in order to accomplish that objective. And so like we, we actually had this model like this tool release recently called like Auto Insights, which I can point it to a specific chart. We'll pull like anywhere from like 20 to 30 of charts that are similar to that based on like relevant properties. And then we'll do a bunch of analysis, try to find the specific segments or the specific like group buys and then try to figure out why, why things are changing, hypothesize about like what the biggest drivers are. And for a human that would have taken hours for me to spin up all these charts. We've actually replicated a lot of that using this cloud scale. So I taught like this model basically, hey, look at this chart. Pull this data from using Amputee's MCP on the URL. You can basically hear some heuristics about like how to identify and like look at specific patterns, so look for spikes, look for seasonality, looks for, look for anomalies and then basically taking some of that context, trying to look at other tools like within our feedback, within our releases, within our annotations, on like some amplitude data. Can I explain that analysis? And then I gave it a format like tell me what happened when this occurred. What is your primary hypothesis about what changed? Some plausible explanations, your evidence and then what this means for us as a business. And so I'm going to jump into Cloud Coded and Terminal real quick where I can kind of show you an example of like where, where this ran.
- [06:22]
Akash Goel
All right, so I have sort of two separate follow up questions here. Yep. Let's first start with the context. So what is through trial and error, what have you learned is the right context you should be including here for cloud code? Because I know there's always this concern about just using up your context context window also. So there is a concept of too much context. But what is the right amount of context and how do you organize that so that Claude code can ingest it?
- [06:47]
Frank Lee
Well, yeah, I actually don't worry about it too much because you basically can refresh a lot of the context every time you start a different session. Right. So within Claude code you can like within Terminal, if I feel like I'm running out of context, I can just exit out and then basically it'll refresh the context. Similarly with in cursor, if I wanted to spin up a new agent, I could just jump to the top like, Right. Spin up a new tab and it basically refreshes like it forgets what was happening in like a prior thread and then I have a fresh set of contexts in order to do that. That being said, there is a bit of context management that you do have to think about. One of the trade offs of MCP is that a lot of the tools that you might have loaded. So let me show you an example. I have amplitudes, MCP hooked up. So there's a bunch of tools that are here. Same with linear, I have that hooked up. So the context from what tools are available, the names, the descriptions, there's also some instructions tied to each MCP on like what's the specific format that this MCP uses? Or how do we want the data to be used? Or like some preferences on when or how the tool should get used. So that does fill up some of the context depending on what mcps you have loaded. That being said, most folks don't have a huge amount of MCPs loaded up. I think even with amplitude and linear loaded up, I might be using up anywhere from 5 to 10% of my context. So I haven't actually run into that many limitations when doing some of like this product specific analysis. That being said, if you are doing a really long analysis, so for example, pulling in a huge amount of chart data, doing a bunch of successive queries, Pulling in huge reams of feedback. Sometimes you might run into context issues. The one benefit on Claude code is when you start running out of context, they have this feature called Compaction where it starts trying to summarize some of what happened in that thread, consolidates that into some smaller summary, and then lets you move on given the fact that you had reached the previous set of contexts. And then there's also just like a pretty big like bitter lesson take on it. You can worry about like context management now. The reality is in like the next few months, a lot of the new models release. They'll probably expand their context windows or have some different mechanisms within agent client to handle some of that complexity for you. So just build for the future and like, think about pulling in like, what, what's most relevant to give your agent, like the best types of responses.
- [08:50]
Akash Goel
Yeah, you can use the slash compact command. They actually show you how much percentage of your context window you're using up in cloud code now. So I recommend for people, when you're hitting that 90% or something, don't just rely on the SL compact. Actually, I ran into this problem at about 1am last night where it said conversation too long, can't compact. And so my newest sort of hack is when I'm at like 90% or so, because I don't want it to just fail at compacting. I will say write a markdown file of your process and your progress and what you have left to do. That's a good way also to kind of transfer the context over if you need to. So you answered that we can put almost as much context as we want. So what context? Like, can you walk us through specifically? Like, what sort of methods are you using to get context, like meeting notes and stuff in there? And how have you structured those folders and files in here?
- [09:38]
Frank Lee
Yeah, sounds good at the simplest. Like, I have a product folder.
- [09:41]
Sponsor/Ad Voice
Today's episode is brought to you by Amplitude. Replays of mobile user engagement are critical to building better products and experiences. But many session replay tools don't capture the full picture. Some tools take screenshots every second, leading to choppy replays and high storage costs from enormous capture sizes. Others use wireframes, but key moments go missing, creating gaps in your understanding. Neither approach gives you a truly mobile experience. Amplitude does things differently. Their mobile replays capture the full experience every tap, every scroll, and every gesture with no lag and no performance hit. It's the most accurate way to understand mobile behavior. See the full story with Amplitude. Today's podcast is brought to you by Pendo, the leading software experience management platform. McKinsey found that 78% of companies are using Genai, but just as many have reported no bottom line improvements. So how do you know if your AI agents are actually working? Are they giving users the wrong answers? Creating more work instead of less, improving retention or hurting it? When your software data and AI data are disconnected, you can't answer these questions. But when you bring all your usage data together in one place, you can see what users do before, during and
- [10:50]
Akash Goel
after they use AI.
- [10:52]
Sponsor/Ad Voice
Showing you when agents work, how they help you grow, and when to prioritize on your roadmap. Pendo Agent analytics is the only solution built to do this for product teams. Start measuring your AI's performance with agent analytics at Pendo IO Akash that's P E N D O IO Aakash
- [11:10]
Frank Lee
so these are some different product lines I'm working on. So agents, AI feedback, ascii, enterprise features, mcp. So within each of them I might have some context about what is our Q1 plan. I won't show it because there's a bunch of stuff there. But like we're thinking about forms of memory like ambient agents, different types of agents that we're thinking about. So these are basically specs that I've converted into markdown files that I can refer to pretty easily using like the AT command if I wanted to brainstorm from a given like prd or just say, hey, we already have a bunch of planning within like our Q1 plan. Can you actually just fetch all of the specific items and then turn those into drafts of PRDs? And so as example, I have a few templates actually for some like pretty recurring types of tasks I might do. So like for example draft short prd. So give me like a sense of the problem context, talk through like in narrative form what a solution to workflow might look like. Here's how I want you to format or break down that response. Here's some heuristics on like how do you think and strategize about what to build. So there's some of like some light heuristics that I typically might have on selecting features emphasizing simplicity, like it's okay to say no. Here's how to think about some of the design like principles for some of these features and then taking some of those requirements that might be in the doc or might even live elsewhere and turn that into acceptance criteria here. So that's just a sample. But if for example in cursor it's like draft a Quick PRD using the PRD template. So it pulls this into context, immediately knows what the format I'm looking at. And then I could talk about, hey, talk about this specific topic or refer to this doc where we have a bunch of ideas and structure that into a better format. So that's one example, I guess the other thing you touched a little bit on Granola. I've been playing around with this. I use claude code actually to try to create some of these light little automations or connections that I previously wouldn't have. So my understanding as of now is Granola Dart actually does not have a dedicated MCP right now. So I try to hack cloud code to build some type of automation so I can adjust some of my calls. So there's, if you see here, there's some like notes, say like the year month and then there's some like meetings that I've had. And so it tries to inject some of like the summarized notes there. So this is, for example, this is like our MCP standup that happened. I basically could run a command. Can you pull in my recent Granola notes using the script rewrote? Let's hope this works. I haven't run this in the last two days, but basically that's also a part of my product repo where I use claude code to create the script and it's going to run this command terminal. And then yeah, you see there's a bunch of these meeting notes that were previously ingested that AutoCAD dumped in. So if you want you can refer to the whole folder. So I'm like, hey, at 2026. Or that's like a specific one, but like the 202601 folder here, you immediately pull all of those notes in context. If I wanted to say, summarize the main takeaways, give me the specific action items, I can put that into another folder. But basically all this stuff is immediately accessible if you needed to because it's all like consolidated in that one system.
- [14:07]
Akash Goel
Amazing. Love this use case. I think most people should be using this specific one. So how do you get around the mcp? He created this little function or skill or script in this case that he referred to it with Claude Cod so that he could access those Granula meeting notes. And how are you using Obsidian and Conductor in your workflow?
- [14:24]
Frank Lee
To be quite honest, not too much. I think those are the power tools if there's specific nuances on, for example Conductor. Right. So I have it pulled up. Conductor is like a great tool when, if you have a bunch of coding agents that you want to run in parallel. The problem is oftentimes if you have one branch of your GitHub and you're trying to push changes, if you kick off a bunch of cloud codes in the same branch, oftentimes they might have a bunch of changes that conflict with each other. They're not really talking to each other. So the thing that they nailed on Conductor was they made use of this feature called a git worktree which essentially copies your repo into a separate like work tree. So then you can run multiple cloud codes in parallel branches, they won't conflict and basically you can decide to change like to merge some of those changes like together. If you're like going into the huge amount of like agent orchestration workflows, if you're running, you're trying to run 5 to 10 of these coding agents in parallel. Conductor is probably the best in class tool that I've seen for that. I know there's a lot of excitement in the industry on what is this orchestration like framework looks like, but this is the prettiest and best that I've seen so far. But for just a call out for specifically product management, I don't really need too many tools. Like it's not a big deal for like this model to conflict. Like if I'm kicking off a task like usually I'm, I wanted to create one PRD here for this team or I want you to do an analysis here. So there's not as much of an issue where like the code conflicts within this branch. And so it's, it's honestly probably easier to just do it directly within different tabs in Terminal instead of having to spin up a bunch of git work trees. But if you're specifically these coding tasks, trying to do it all within the same branch is basically just going to lead a huge amount of messiness.
- [15:55]
Akash Goel
And where does Obsidian come into the workflow?
- [15:57]
Frank Lee
Yeah, so it comes with personal preference. Obsidian basically has access to your file system the same way that a cursor also does. If you hate staring at Markdown, then Obsidian is like a pretty wrapper to directly make edits. Like it's more akin to like a typical text editor. But me personally, I love playing around with Markdown specifically. I like spend a lot of time coding myself and it reads pretty well to me. So it's kind of just nicer to have the same tool where I can push code, push changes directly and use the agents natively using cursor or within cloud code.
- [16:25]
Akash Goel
Yeah, and you can also just double click or right click on any of these MD files and hit Open Preview or I do up Apple Shift V and you can look at these pretty so I feel like the Obsidian hack, it's not really necessary. So yeah we got people to set it up. What are the top use cases of cloud code and MCP with analytics for product managers and product analysts?
- [16:48]
Frank Lee
I think there's five things that I'm doing really heavily. So first one, anytime there's an anomaly or some type of big change in my metrics, I'm using some skills and tools within Claude code in our MCP to analyze and do some deep chart analysis. So I drop in a link and it basically the agent can figure out exactly what specific segment or what little group by or what specific related metric might hypothesize and lead to that change. So that saves me a bunch of time I think. The second thing is I've automated basically all of my like weekly business reviews and dashboard reporting at Amazon. I used to spend all all of my Sundays just coming up with our metrics and explaining why things are happening. Now I've pointed this to cloud code and some of these like Amplitude Dashboard agents and Monday morning I come all of the five to six dashboards I look at are automatically synthesized. I know exactly the three to five top insights for me and I know the one specific urgent issue to tackle with my team. So don't need to analyze like the dashboards anymore. I can just start like focusing on solutions with the team. And the third one is doing a bunch of deep qualitative customer feedback analysis with Amplitude's MCP and the AI feedback product. All of our tickets from Zendesk slack amplitude surveys that's all unified in one place now. So I can just point cloud code, tell it to analyze the feedback related to AI agents MCP and then the agent will navigate all of our tools, consolidate all the data, do its analysis and clustering and then give me a clean report of exactly what the urgent issues are, what the bugs are and then what's the single most thing that people love about our product that week? Fourth one spend a lot of time taking those insights coming in from those dashboard agents or like this cloud code chart analysis and then given this context, can you think about what's the right set of actions or plan to do to resolve this space. And so I've given this agent some heuristics on how I think about coming up with solutions and the agent can go at length explore a bunch of different ideas. I can drop in some images and then it can just turn that into a spec so that I don't have to write it anymore. So I OPUS is like the best thought partner that I've ever had on coming up with some of these like design and product ideas. And then lastly, I've kind of totally reshaped like how I think about what I route to the team to build versus what I just do myself. Now I can take these PRDs or take some of these problem statements that the agent has surfaced. If it's easy, I just drop it into cloud code or cursor directly and then it could pre draft some of the solutions for me while I'm in a meeting. But if it's a really tough problem, I can go ahead and hit the linear MCP and say, hey, I want to route this to the AI capabilities team or like the MCP team file it under like this specific project and like assigned it to this engineer and then assign it to cursor as like a courtesy to hopefully like soften the workflow on their end. So these are the five things I'm doing the most frequently.
- [19:22]
Akash Goel
I love these. This is the new Vibe PMing workflow. These five use cases. Can you demo each of these for us?
- [19:29]
Frank Lee
Just for context, this is one chart that I'm looking at on the NCP side we're curious about, like what is the average number of tool calls that might happen in a given week. And I'm seeing this anomaly where like the average number of tools being hit is 8.1 for this week, which is quite different. It's something that's worth investigating because I need to figure out how do we manage the right tools, the descriptions and make sure nothing's broken. Right. So I can take this link if I jump into cloud code. Like I said, I had a bunch of these different skills that I had. So there was the analyze chart skill which I'll just preview really quick for you here. Analyze chart. So remember there's a name description and a bunch of heuristics on what to do about it if I do the slash command, basically that can trigger that specific skill for you. So analyze this chart and explain why there's a jump this week. So stepping in here, so this might take a bunch of time. But now the agent is connected to the amplitude mcp. So you can see here it's parsing the URL, it's starting to pull in some data on the chart and then we have some tools that let you navigate the data. So there's A chart data. We can query the underlying datasets and then basically the model starts showing its work on how it's navigating your taxonomy, how it's navigating relevant charts. Maybe it's navigating the specific events and properties related to that. So this might take a few minutes and but basically you can see the agent reason about which charts it's looking at, which events it's looking at or what are some hypotheses about like where this might change. And so it's navigating the properties it's breaking it down by like what it thinks might be that spike. Now it's pulled in a different chart. And if you think about the human equivalent of this, it's the same as me having five different versions of this chart. I had to go like manually group and cut this by a specific property that I have in my head. Or I might have a create a separate chart where I look at a different event that's somewhat related to it. Or there's a bunch of this manual iteration on the GUI that I had to do like creating a bunch of different these charts. And now the agent is doing all of this just via terminal. So you can see here, this is the format that I specified for it. But it says here's the chart, here's the time range I looked at, here's the metric size analyze. And it walks you through like that a spike happened when it happened the underlying data that led to that and then some hypotheses about it. So detected that there's a change within the types of tools that we analyze enabled. It has access to our experiments and feature flags data. So it saw that there is some changes within what feature flags we had enabled which provides different tool access. And then so you see some flag activity, there's some differences in historical precedents and then hypothesized about something else that might have changed. Maybe it's data quality, a different power user. So basically like what I would have had to do manually to investig explain what's happening. The agent just, I think it literally did this in like a minute and a half. And this would have taken me probably one to like one to three hours depending on how complex the query was.
- [22:15]
Akash Goel
Yep. And that's with having a background as a data analyst, if you're just not having that background, it might take you half a day.
- [22:21]
Frank Lee
Yeah. And that's assuming you know which properties to look at. Like basically because at least on the amplitude MCP side we've done a huge amount of like configurations on how do you simplify the names of the tools and the descriptors? Or how do we improve our underlying search infrastructure? Or how do we capture the right context about when to use a specific chart or when to use a specific dashboard or when to use a specific event. And that context is all provided to the agent via these MCP tool halls so that it can more accurately navigate that data for you. And so if you compare this to an expert of the taxonomy, it's probably the difference of this running in 1 1/2 minutes versus them spending 10 minutes navigating the chart. If this was a person that did not know that taxonomy, they would have either gone the wrong answer or they would have had it spent like two days chasing down the right person in the org that could tell them what to look at. This agent basically was able to figure it out itself given a lot of this like underlying, like semantic infra very powerful.
- [23:13]
Akash Goel
So deep chart analysis. I think I understand this use case. Really the powerful thing here is you write yourself a skill and you teaching it how to navigate the MCP and make the right tool calls.
- [23:22]
Frank Lee
Yep, exactly.
- [23:23]
Akash Goel
Totally understand this workflow. What's the next workflow that was around automating dashboards, right?
- [23:28]
Frank Lee
Yeah, the next one's around automating dashboards. So similarly I have an analyzed dashboard skill. So here's when I use it Meeting prep dashboard investigations connecting quantaqual and I could show like I instructed the agent to look for the URL, query the underlying charts, fetching them like three at a time. Because there's some context limitations like when you're interacting with some of these tools, look for these specific things like if it's a KPI chart, like look at this and the percentage change. If it's a bar chart, look at concentrations of gaps. So teaching them some heuristics on like how I as a human would have thought about out like this data. And then what's cool is like now that the, this agent has analyzed a bunch of charts and figure out like what's happening in the dashboard, I pointed it to hey, go go pull the feedback data recently and see if there's anything relevant there. Or we don't have it here, but like we also have access to like session replays or experiments. So basically depending on what data you have and like whether it's relevant to a specific dashboard, you can instruct it to also query that as part of its agentic workflow. So this is. And then this here's like the takeaway of like how I wanted it presented just personal preference on what I think is helpful, some best practices, but in reality, like let me, let me show you. So I'll start over just given the context piece, but here's Claude. Analyze dashboards. Let's pull in. So we're talking about mcp. So can you give me a TLDR on my external MCP usage dashboard? And so this is different than last time because like last time I gave it an explicit URL, which is, and this time I'm kind of by giving it some like some natural language query. Right? So if you are an MC vpm, you have to spend a lot of time thinking about, hey, a lot of folks are going to give you pretty ambiguous natural language questions. How do you like take those queries and then like teach the model to figure out how to use your search infrastructure or navigate your data so that it pulls the right thing. So I have a dashboard that's called external MCP Usage Dashboard. So it's, it's quite, it's like kind of cheating. But if I told it said, hey, pull the MCB dashboard, you can teach it on the tools to be like, hey, here's how you think about using like the search endpoint or here's how to filter the data properly or run a few different searches and then figure out reason about what's the best response. So there's a lot of different little experiments you can do on optimizing the tool. But now you can see it started looking for external MCB usage MCV usage. These are different search queries. It was running on our search endpoint on the dashboard data. It pulled it, it identified the dashboard and then it has a different tool for querying those charts. So it's pulling in Those different chart IDs three at a time because that was the instruction I gave it. And then now once it's pulled all of that data in context, it gives me a quick overview about what some of the metrics look like. So won't spend too much time on the metrics because that's like, that's our metrics. But you kind of see like, oh, once again, I could have spent the 10 minutes scanning through like all of the charts in my dashboard. And then if I found anything that was interesting, I would have to like copy paste some of those individual data points, pull them into a specific like part of like the bullet points that I would summarize. I'd have to think about the narrative. I'd have to jump into my AI feedback and run a search on MCP and see if there's anything relevant. But. But the model given those instructions did it all in. What was that, like a minute and a half as well? Like, yeah, that. That saved my whole Sunday again.
- [26:30]
Akash Goel
So that was the promise, right? We were gonna save our whole Sunday. So if I recall, like, I used to do this too. Like, when I was at Epic Games, PMs used to have a pretty. Pretty much do the analyst hat as well. So I remember on Sundays for my Monday meeting, I'd spend like six hours going to various tableau dashboards, having them load, copy, pasting them, interpreting them, analyzing them. How do you put together that whole workflow of like, like, These are my 25 typical WBR slides. Go pull those all for me.
- [26:54]
Frank Lee
So we don't currently do WBSR anymore. Like, I have agents that are automatically pushing these reports into every single product team's channel. So, like, instead of having to, like, sit now, Monday and like, review the metrics, all of the main takeaways, all of, like, the goals, all of the main deviations, there's already a draft of that there. And then basically, like, each pm, like myself included, we can have our own planning meetings with, with like, our specific. Whether that's our stakeholders or within just my engineering team. If we want to talk about the metrics, we can. Or we just spend the time more focused on like, the tactical as opposed to like the reporting.
- [27:27]
Akash Goel
How do you schedule things here?
- [27:28]
Frank Lee
I do this via the dashboard agent that's in amplitude. I don't know if you want me to show that real quick.
- [27:33]
Akash Goel
Yeah, let's see it.
- [27:33]
Frank Lee
Yeah, so this is like the. We've had this product for a few months now, but this is the amplitude agents, like, kind of inbox. And so you can see here I have a bunch of these different scheduled. These different scheduled agents pointed at different dashboards across the company. So automated insights, the overall, like, AI native amplitude dashboard, like analytics foundations, like website optimization. Like, there's a bunch of these specific agents that I've spun up based off of dashboards that I've created. So there's a dashboard agent. I pointed it here, I gave some context about, like, you can give it some context optionally. Like, we have a pretty robust system prompt on how to think about this. But if you want to do weekly, monthly business reviews, you can. If you just want to report on a new feature, there's a bunch of these little prompts that like, basically we want to guide people on how to do it. And surprisingly to me, like, I came from, like, A startup background. Most recently, when you're dealing with enterprises like most folks don't like, they're not super familiar with agents, they're, they're not sure like what knobs they can turn when they're creating an agent. So giving them really easy understanding of how to create it, what system prompts, what the agent is steered by is extremely important. We had a pretty low agent creation flow until we brought in the starter prompts and then we saw the relevance and like the agent creation conversion pretty dramatically improved once we did that. So this is just an example of like this within the agent's page. These get pushed all like into Slack, into email. There's a bunch of folks that are consuming it there. People actually don't spend that much time on these asynchronous agents within the product. Like the idea for at least these specialized agents is let's get them into all the places where people do their work. So Slack, if we wanted to push that into email, if we wanted to push that into teams, or if we wanted these, push these insights to GitHub. Like there's a lot of work that we do on the agents and MCP front, like whether that's via partnerships, integrations, et cetera, to get these insights to like where they're most helpful at makes sense.
- [29:15]
Akash Goel
So I think I understand the second workflow. What's the third one?
- [29:18]
Frank Lee
The third thing that I do religiously on those Mondays. So we've pulled in feedback from a bunch of different places. So this is the new AI favorite product launch product that we launched in November. We're pulling data from Zendesk, Intercom, Salesforce, Gong. There's a bunch of stuff coming in from public channels, G2 app stores, et cetera. And then we're like, it's not here. But we also have an integration with Slack that we're, we're prototyping right now. So all that data gets piped into AI feedback, it gets auto analyzed. But sometimes like it's not exactly what I want. Like this is a specific tag or like insight that I have. But I might have like a specific pointed question I might have for it. So let's start the instance of Claude. So once again I have a specific skill that I've created in the template for that analyze feedback, pull in my agents, MCP and other AI related feedback. Give me tldr. So one, when I was working on Anari, teams would basically do all of this stuff in like spreadsheets. There'd be huge amount of these like customer surveys or they would be copying pasting messages from Slack or they'd have to send like this crazy integration with Zapier to like point like pulling data from one place into another. Our product pulls all this in one place for you and exposes it via like an MCP tool. So we have two different tools we we've tuned for. This one is for those pre process insights that I showed you. We can provide access to that and then we also provide access to the underlying feedback mentioned data. So you can see here that the MCP is looking at what is the set of insights that are already created but also what is like some of the underlying data that's most recent and then it's going to use that to generate a quick response.
- [30:48]
Sponsor/Ad Voice
AI is writing code faster than ever, but can your testing keep up? Testcube is the Kubernetes native platform that scales testing at the pace of AI accelerated development. One dashboard, all your tools, full oversight. Run functional and load tests in minutes not hours across any framework, any environment. No vendor lock in, no bottlenecks. Just confidence that your AI driven releases are tested, reliable and ready to ship. Testcube Scale testing for the AI era See more at testcube IO Akash that's T E S T K U B
- [31:23]
Akash Goel
e I O/ A A K A S I hope you're enjoying today's episode. Are you interested in becoming an AI product manager? Making hundreds of thousands of dollars more joining OpenAI and Anthropic then you might want to do a course that I've taken myself. The AIPM certificate ran by OpenAI product leader McDad Jaffer. If you use my code and my link, you get a special discount on this course. It is a course that I highly recommend. We have done a lot of collaborations together on things like like AI product strategy, so check out our newsletter articles if you want to see the quality of the type of thinking you'll get. One of my frequent collaborators, Pavel Hearn is the Build Labs leader. So you're going to live build an AI product with Pavel's feedback if you take this AIPM certificate, so be sure to check that out. Be sure to use my code and my link in order to get a special discount. And now back into today's episode. Here's the dirty secret about prototyping. You spend two weeks building a prototype. You validate your assumptions. Engineering loves the direction. Then what happens? You throw the whole thing away. Bolt changes this completely. When you prototype in Bolt, you're not building throwaway mockup, you're building Real front end code that integrates with your existing design system.
- [32:34]
Sponsor/Ad Voice
So when you hand it to engineering,
- [32:35]
Akash Goel
they don't throw it away, they ship on top of what you've built. I use Bolt every single day. I host my lan PM job cohort on it. And honestly I'm up till 2am some days just vibing in the tool, having fun and building, building. That's when you know a product is good. When you're using it past midnight, not because you need to, but because you want to. Check out Bold at Bolt New Akash. That's b o l t.net w a a K A S H link in the show notes.
- [33:01]
Frank Lee
Basically there's a lot of people that have to do this like the aggregation like and analysis manually. So you can see this is a format that I specifically like. But you can also provide the Asian freeform context of. I actually don't want to look at like urgent issues and feature requests. I only want to look at praises or I only want to look at references for competitors. So just explain that in natural language and then the agent can like kind of figure out it's like it's right response for that. This is actually what people are saying about some of the Asian products. They want more context. There's some like inaccuracies, like sometimes there's degradation in quality. Like these are all things that like we hear sporadically here or there in different channels. But having it all in one place, having it in a specific report, if I need to like change it in a specific format, it's so easy via like cloud code or like the next thing I'll demo for you is like, okay, can you convert all of these recommendations into specs and place them in the agents folder. So let's see if this one rips correctly. But.
- [33:57]
Akash Goel
So this is that fourth workflow. Once we've gotten an insight from our feedback customer feedback themes, let's convert it into actions or specs.
- [34:04]
Frank Lee
Yeah, so this is a light example. Like typically I actually do this in cursor because I want to use my specific draft PRD template. But you can also do this in cloud code. Because in cloud code it's all tied to the same thing. Right? I have this whole product repo in my GitHub. So you can navigate there, you can navigate it on cursor. I typically find that cloud code is a little bit more powerful. It takes a bit longer. But like the response quality is marginally better in cursor, but the cursor, like there is some benefit of having like A dedicated IDE experience doing the app references like being able to swap between different models and testing around seeing my specific file system and making edits directly. So I always get hesitant when people are, are saying it's one or the other. Like I use both extremely heavily.
- [34:45]
Akash Goel
Yeah, I pretty much use cursor because I can use. They're really fast models if I want to. And then I use Claude in cursor, which you can also do.
- [34:54]
Frank Lee
Yeah, so you can see I think they're starting to add a few within this folder. But yeah, you basically can create a bunch of specs using one or the other based off the templates that you have. And then like, so once again I have a few MTVs connected. There's the amplitude one, there's context seven, which makes it pretty easy for you to fetch docs from, from other repos to explain what's happening. And I have our linear connected if I wanted to push tickets. So for like some of these long running tasks that I need to coordinate a lot with the team, I'll push them to linear so that people can track it. We're also kind of in a weird era where there's a lot of tickets I don't even push into linear anymore. Like I might just have take this context and just message the engineer directly in Slack just removes one part of the steps or overhead there. So.
- [35:35]
Akash Goel
And it can even. Slack is an mcp, so you can even do that through here.
- [35:39]
Frank Lee
Yeah. So this one takes time. There's like six different things, but basically it's, it's drafting these specs, it's putting them into markdown files. I didn't specify the specific PRD template, but like, basically like the first draft of it is being generated based off of this analysis in this context that I pulled in via the feedback.
- [35:54]
Akash Goel
Can we look at one of those and see like realistically, do you think those are quality? Would you use those? And right now we're using Sonic guys.
- [36:00]
Sponsor/Ad Voice
So it's a little bit faster, but
- [36:02]
Akash Goel
it might be a little bit lower quality. But let's put it to the test. All right, so it's generating these specs. Can we take a look at some of these and see if they really hit our quality bar? And by the way, can you explain that branching GitHub toggle that you're doing in the top left for people who don't know right here?
- [36:16]
Frank Lee
Yeah, basically all of these changes that are happening in cloud code or in cursor, they're happening in a local branch on GitHub. So there's a copy of this whole repo that's on my local desktop. When I feel like I'm happy about the set of changes, I can just push it directly into like my, the cloud, like my GitHub Cloud instance of this repo. The same way you'd think about a code base. Like I kind of just have my own product system that's in GitHub so that I can access it wherever. Because the other thing that this unlocks is if you're on cloud code on mobile, you can Refer to your GitHub repo that's in the cloud and then also come up with ideas and kick off like asynchronous tasks as well. Oh, that's super powerful because you're not always going to have access to cursor on your phone or like cloud code and terminal on your phone. For example, like all this stuff about cloud bot, like I'd be hesitant to put that onto my work laptop and try to steer cloud code via like a remote branch. But if I have all of my repo and all my contexts in the cloud, I can just kick off cloud code directly within the cloud app.
- [37:08]
Akash Goel
Nice.
- [37:08]
Frank Lee
Yeah.
- [37:09]
Akash Goel
What's the command for somebody to set this up the first time? Duplicate my repo into GitHub or something. Just simple like that.
- [37:15]
Frank Lee
I had just gone into GitHub and created like a new repository there. So when I set this up, I just follow the typical GitHub instructions that they have. Like instructions on how to clone a repo to your local branch. Yeah, I think you can actually just do it via like cursor. Now like if you wanted to be like. For example, if I were to go on GitHub, if I was to go on GitHub, I have this like amp product repo. So there's like a bunch of different ways that you can clone a repo to your local. But instead of following those instructions, you can either use cloud code or cursor and just say, let's say this was a fresh repo, I could say, can you copy this to my local branch and sync this to GitHub? And now since it has web and terminal, local terminal access, it can manage that whole set of instructions that previously you would have to do by hand, which is pretty tough typically for like a non developer. But now like the agent, now that it has local like terminal access, it can, it can do it for you.
- [38:08]
Akash Goel
Nice. So how are these PRDs looking? You're the actual PM on this. Is this a good PRD?
- [38:13]
Frank Lee
These ones are correct. There are two verbos. So I can go back to my prompt and say, hey, dramatically cut down the words or the narrative solutions is like way too concise. Like it's way too long. Condenses to like the bottom like one or two paragraphs or hey, these, these might be too much like too many acceptance criteria prioritize to think about like the top three. So if I wanted to like give this. There's like command L, this is too long. Can you make this way more concise? So this is this total bypass how big those instructions are. But.
- [38:43]
Akash Goel
And I think we probably could have made it better by using Opus and pointing it to your PRD template. So this wasn't necessarily the thing that's
- [38:50]
Frank Lee
the way to do it like for the overall system. But this is kind of the reasons why I like sometimes doing it in cursor because you can also highlight a specific section and then if you do Command K, you can give like quick feedback directly there. Or if you wanted to do command L, I can like basically the context in the agent is pointed to those specific lines in that file. So within. Within Terminal it's a little bit harder to specify like what exactly like the change should be. But like within, directly within the file I can reference it here.
- [39:17]
Akash Goel
Very cool.
- [39:18]
Frank Lee
This one is like much more concise and more relevant. So yeah, it goes back and forth.
- [39:22]
Akash Goel
So let's go ahead and say you've iterated the spec is good enough. I think we said the fifth workflow is either pushing these into linear or doing them yourself. So can you kind of show us both of those?
- [39:30]
Frank Lee
I'd be nervous to kick it off. But like one example of something we have have, we have this channel where we are basically calling out to cursor cloud code within an open channel about some instructions that we want to push in our code base. So I might take this PRD or the specific acceptance criteria and I can say at cursor the JavaScript repo, make this change to the AI feedback page or make this prototype this change there and then you can spin up a background agent there. Or depending on if you have clock code set up on your terminal, this is me pointing to our JavaScript repo. So if I wanted to like pull Claude, I can basically pull up Claude and push a change there. Or lastly, like I have the job script repo as well. Like assuming I'm on like the. The relevant branch, I would just dump that into here and say like create a branch for that. So I don't want to kick it off like basically, I don't want to handhold like the actual code being drafted. So that's. That would be one route.
- [40:19]
Akash Goel
Yep. So basically, guys, that's the end to end. Workflow starts with the deep chart analysis. You got your automated reporting, you've investigated customer feedback themes, you've converted that into an idea, and then you take action. You might even code it or a prototyp type of it yourself. What is the biggest mistake people make when using mcp?
- [40:35]
Frank Lee
I think there's two big ones. One is just kind of the wrong expectation settings for mcp. Sometimes people think that mcps can do everything. Just the simplest way to think about it is MCPs are easy ways for your AI to interact with external systems. They're not complex ways to do workflows, they're not complex tasks. Already there's a bunch of other tools more relevant for that, but just one setting the expectation of just being able to pull in data is the first step. I think the second problem is sometimes people get excited and they connect a bunch of MCP servers. Some have a lot of tools, some that are just totally irrelevant to a given workflow or in that repo. And so this comes back to the problem with MCP, where if you have a huge amount of MCPs connected, too many tools that are not being used, all of those descriptions about what tools are available, what descriptions they are, what formats are needed, they start being provided as context to the model on a bunch of different queries that might not be relevant. And so that might lead to higher latency on a given response or slightly skewed or like inconsistent responses because the agent is thinking about these tools even though it's not relevant to the task at hand. So just be thoughtful about when you decide to incorporate an MCP tool for stuff that's not as irrelevant, you can hide or remove those tools, typically within parser cloud code, et cetera. Like you can remove stuff. And then if you're actively developing yourself, you also need to optimize on what is the right set of tools that you want to make available in your MCP server. What are those things named? And then making sure that they're. There's not ambiguity for the model on which tool to use. And there's like accurate descriptors and accurate like instructions on what formatting is required, depending on what you're using it for.
- [42:04]
Akash Goel
What you just said there, these breaking the fourth wall behind the curtain insights of what it's like to be an MCPM have been one of my favorite parts of this conversation. So I think you're uniquely qualified to answer this question, which is that people say MCP is an unreliable or a bad standard. What do you say to that?
- [42:20]
Frank Lee
All right, I came prepared. It says a slap. So I see a bunch of things related to mcp. One, people just have been overhyped about what MCP can do. The second is MCPS waste a lot of context. The third is there's a lot of painful experiences on how people set up or have to deal with authentication when interacting with mcps. And then lastly, and probably the most important thing is pretty complicated on how to actually configure an mcp. So those are the four criticisms I see most frequently. And so I think the main response is is set the right expectations for yourself on what SAP is used for. It is by far the easiest way to connect external systems with most of the AI clients that you're using. And instead of as a pm, I have to pull forward this huge roadmap of integrations and spend a lot of engineering resources building them out. Most AI native tools at this point are exposing a lot of their tools, data and actions via mcp. By far the easiest way to get started. Second piece is MCPS required some amount of optimization. You need to go back and forth on what is the right set of tools you want to make available in your server. You need to optimize the names of them like datasets or actions you make available. And then whenever you're finding edge cases, it's similar to some of the eval driven development that you see for agents. Once you find those edge cases, figure out how you can optimize the tool names, the descriptors. And once you come to the right set of tools, those servers and those integrations work very seamlessly and kind of power a huge amount of these complex workflows. I think the last thing is most of these AI clients, Claude chatgpt Cursor everybody's all converged around the same standard on MCP. Now like it's the standard. Most people have some type of managed connector flow where instead of having to deal with this like manual URL setting that is in like the first version of these tools you could just click one button and like those MCPs are set up for you assuming you're logged into the right tool that you're trying to connect to, auth is handled. So now that it's the native standard and everyone's converging on like the same like standard, it's actually pretty seamless to get started now and so and then lastly, there's been a lot of this criticism about like, context rot when there's a huge amount of tools within the last month and a half that's actually dramatically changed depending on the AI client. There's like pioneers like Cursor, they've done some experiments around dynamic tool calling. So instead of having the tools all called every single time, they actually use RAG to find the right set of tools to call given a natural language question. So you're not using all of your context all the time. Same within Claude. Like they've adopted some type of similar standard for not using all of your contexts across these tools. And then that's also one of the reasons why they built Skills. Skills gives you some instruction on here's a specific task, here's the instructions and the prompts that only gets loaded in context when the model thinks it's relevant to pull that skill. So you can kind of circumvent a lot of these problems with MCP by pairing it with like the right changes on the client side or with a skill to instruct when to use that specific tool.
- [45:10]
Akash Goel
Yeah, for my money, Skills is one of the most important releases. So we started with it, we're ending with it. Make sure you guys are learning those skills.
- [45:17]
Sponsor/Ad Voice
All right, so we've talked a lot
- [45:18]
Akash Goel
about MCP and Claude code. What else is Amplitude cooking up? What are the most powerful agents you guys are building and release?
- [45:24]
Frank Lee
So we're super excited about February 17th because we're bringing agents across the whole platform. I wanted to give some context about Amplitude and its journey. And internally we actually think about this launch as kind of our cursor moment. And the reason why this matters is when we think about Cursor, they made coding accessible to so many folks that had never coded before. It was the first foray for a lot of folks on how do you spin up your own GitHub PRs, or how do you pull in AI and H agents into the coding workflow? And it brought so many people and lowered like raised the floor for a bunch of non technical folks that were never able to code before. But what's interesting on like the cursor side is when they think about product development, they're always trying to raise the ceiling of what's possible within software engineering. They made this conscious decision on like, how do we make the most powerful agent or what are the most powerful features and set of contexts that can provide the model so that people can code in ways that they've never basically been able to do before. So with our agents launch, we genuinely think this is our cursor moment. There are, there has historically been problems where folks, they're not super familiar with the data taxonomy or how to navigate like the amplitude UI or what actions or like ways that they can improve their product are available on the platform. We've built an agent that's embedded across the whole platform. It has access to every single data point, every single action, and like, improvement that can be taken in amplitude. And we make it possible for people to navigate the whole product just using chat now and using this agent. So really like basically raising the floor of what's possible for them. But what's even more powerful is given how powerful these tools are, these actions are, we're actually seeing these like 10x, like analysts, 10x marketers, 10x product managers that can do so much more than they used to, like, given some of those workflows that I showed you in cloud code. And so more people are creating like so many new guides and surveys or like running a bunch of new experiments, or like being able to like deeply analyze and like understand trends that are happening across the data, all without the same amount of manual time, and also like navigating that data in ways that they were never able to do before. So there's these power PMs, these power engineers that have access to way more data than they've ever had previously and can accomplish and orchestrate some of those actions within amplitude, but also within external places like cursor, cloud code, figma, et cetera. So just jumping directly into what we're shipping, there's four parts of this launch. One is this embedded amplitude agent. So we call this the global agent. It lives across the platform. You can pull it in a side panel in chat. You can also just navigate on any single page. It has us similarly to cursor of whatever's on the page and all of the underlying tools and data that we have available on the platform. We also have the second part of the launch, which we're calling subagents. So these are more of these specialized asynchronous agents. So there's a dashboard agent that auto monitors your dashboards for anomalies, automates your reporting. We have a session replay agent, analyzes hundreds to thousands of your session replays, understands qualitatively what people are doing in your platform, what bugs people are running into, what issues they're stumbling on, what conversion blockers and telling you exactly, like what people are doing. And we built a feedback agent that pulls in all this qualitative data cuts it in different heuristics and tells you exactly what customers are feeling and saying. And then we have a website optimization agent that scrapes your website and understands ways to optimize your copy, optimize different components on your site and then recommends you a bunch of different experiments you can run. What's underpinning all of this is mcp. Like we have mcps provided to these, these internal like this Amplitude agent, these sub agents. But we've also made all of these tools available externally. So if you want to spin up your own bespoke workflow, workflows and agents, you have access to all of those creation those, read those edit types of tools across all of the data and actions available in the Amplitude platform. And so we've seen these power users build super complex internal workflows. They're mixing and matching data from like one specific digital analytics platform with their tableau data and then they're autopiping like these insights and anomalies into like like Cursor similarly to how I I showed you and building a lot of these bespoke agents using your mcp. And then lastly we want these agents everywhere in slot like everywhere that you work. The first thing that we releasing like in February. This Amplitude embedded agent is available in Slack. You can ping it Like I think about this as like the big short. You know this, this is my quant. Like anytime I have a question about what data needs to be pulled, what analysis I want to run. I can just hit Amplitude Agent directly in Slack. It's going to run for a few minutes and it's going to tell me exactly like that analysis without me doing any of the work, I can link back to exactly what charts it looked at and it can also change anything within the platform. So if I need to edit a feature flag, if I want to run a different experiment, if I want to like watch a survey like a session replay, you can kick it all off and if see Slack and then have follow on conversations with it.
- [49:54]
Akash Goel
Super powerful. If people want to learn more, contact you. Where can they find you?
- [49:58]
Frank Lee
If people want to get started, you should jump to amplitude.com if you want to see what's happening within this launch, you can go to amplitude.com AI and once you're jumping into the product, regardless of what plan you are, you can interact with our agents, you can set them up in Slack, you could connect to all of our Data within Claude ChatGPT, cursor, Claude code using NCP. And then if you want to hit me up Directly I'm Frank Lee on Twitter, but you could also hit me up on LinkedIn.
- [50:22]
Akash Goel
Frank, thank you so much for this masterclass in using Claude code, using analytics and the future of agency analytics.
- [50:29]
Frank Lee
Thank you.
- [50:29]
Akash Goel
All right guys, so we've walked you through what I think is the future of PMing. We were joking off air. This is Vibe PMing. Just like there's Vibe engineering, Vibe coding, Vibe designing all these different trends. This is Vibe pimming. We walked you through five workflows using cloud code and MCP connected to your analytics tool. And by the way, if you don't have an analytics tool that is pulling in your gong data and your Salesforce data and the other data, use those MCP piece to pull in that data and your Zendesk data, et cetera and inform your insights, create specs, even potentially code something like he showed you using a cursor or cloud code agent or create the relevant tickets in linear to get your engineering team to do it. This is the future way PMs are going to be working. If you are not in an org that is giving you this access that is allowing you to do this stuff, you should be making the request. And if you're a product leader, you should be using this video as an example to get everything through through your IT departments because this makes PMs much more effective. When we talk about all PMing PMs becoming AI PMs. This is exactly why everybody should be getting cloud code access. Everybody should be getting cursor access, everybody should be getting GitHub access to the code base. This is what not just teams like Amplitude will be doing, but the Toyotas and the Fords and the United Health Groups and the Mount Sinai Systems in two or three years they'll be doing that. That too. And there's going to be a huge advantage if you are one of the teams that adopts this sooner. So if you need more help on this, be sure to check out my newsletter where we're going to have a deep walkthrough of everything he presented today. Check out my other tutorials on MCP which is with the CTO of Zapier with Claude Code with the full stack PM on this channel to learn more to make sure you're comfortable with it and send me what you learn, what you built, share the insights so that you can continue to create better content for you and I'll see you in the next episode. I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube. All these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career. Finally, do check out my bundle@bundle.akashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you, as an AI product manager or builder, succeed. I'll see you in the next episode.