← The Growth Podcast
Loading summary
Transcript162 lines
- [00:00]
Jyoti Nukula
Understanding which surface to reach for which use case becomes one of the core PM skills that will help you become 10x more effective.
- [00:08]
Akash G
Jyoti Nukla she's been an AIPM since before it was school. She's been an AIPM at Netflix, Meta and Amazon. You posted this on LinkedIn and it caught my eye. You said that you won your internal hackathon against 30 engineering teams and you used this concept of adversarial agents.
- [00:24]
Jyoti Nukula
Anthropic had just released a blog post around smartnesses and long running aging. So I looked into the blog post and they had this concept of adversarial agents. That was what got me the hackathon right?
- [00:35]
Akash G
Where does the product manager line and developer line begin in 2026?
- [00:39]
Jyoti Nukula
Get comfortable webbing. Get comfortable with say Claude code with all the Claude ecosystem that we learned today and get comfortable building and putting your ideas out there.
- [00:49]
Akash G
How do you use Claude design? How do you build a knowledge based MCP server for all of your PM context to make Claude 10x more productive? That is what we're going to answer in today's episode.
- [01:00]
Jyoti Nukula
Now there's this new rule coming up called AI Builder Anthropics Ad it OpenAI has adopted it. Making and building is easy now taste is what is important for us to develop.
- [01:10]
Akash G
Can you do the big reveal now and help us get that set up going in cloud code?
- [01:14]
Jyoti Nukula
Here's the thing.
- [01:18]
Akash G
Before we go any further, do me a favor and check that you are subscribed on YouTube and following on Apple and Spotify podcasts. And if you want to get access to amazing AI tools, check out my bundle where if you become an annual subscriber to my newsletter, you get a full year free of the paid plans of Mobin, Arise, Relay App, Dovetail, Linear Magic Patterns, Deep Sky, Reforge, Build, Descript and Speechify. So be sure to check that out@buildle.akashg.com and now into today's episode. So I've been thinking about something. We've had advanced tutorials on Claude code with analytics on PMOS setup, but how do you actually take the entire cloud ecosystem and make the most out of it from scratch? I keep getting DMs from people who say this episode is too complex or I'm not at this level yet. I'm still stuck on ChatGPT. If you're one of those PMs, this episode is going to build you from 0 to 80. We can't get you from 0 to 100 in a single podcast, but we're going to get you the 80% you need to know in 20% of the time, I have brought back Jyoti Nukula. You guys loved her last episode and specifically the feedback I got was that her structured communication was amazing for beginners. So she is going to break down for all the beginners how to make the most out of Claude today. Jyoti, welcome back to the podcast.
- [02:44]
Jyoti Nukula
Super excited to be back. Thank you for having me.
- [02:47]
Akash G
Jyoti, you posted this on LinkedIn and it caught my eye. You said that you won your internal hackathon against 30 engineering teams and you used this concept of adversarial agents. Can you break down exactly how you won the hackathon?
- [03:01]
Jyoti Nukula
Yes. So a few days before the hackathon, I was trying to see what I could build and Anthropic had just released a blog post around harnesses and long running agents. So I looked into the blog post and they had this concept of adversarial agents where you build an agent and then you set up configurations in another agent, telling it what matters most to your company in a way, not like evals, but more around capabilities that you want your agent to test. And so I set up, I started with that idea and then I said, let me take this idea. I went into cloud code and I was jamming with it for almost a day with different configurations and there we go. I had an adversarial agent evaluator running. It was exactly how I pictured it to be. I even pointed it at our company code and integrated that into an actual production running code. And that was what got me and my team the hackathon prize.
- [04:17]
Akash G
So that's the promise for you guys. We are going to help you get to that level. Where do we start, Jyoti? How can we break this down in a structured way so that people can get to this level at the end of the episode?
- [04:31]
Jyoti Nukula
Great. We'll tackle it today. So we'll start with understanding the clots stack first and then getting into some of the basics, like how do you use cowork? And then getting into plot code itself. So let's get started on the Claude stack. So at the bottom of the stack is your models. So Claude has Haiku, Sonnet and Opus. They're all different intelligence profiles and which one you need to use when they have different cost profiles. Different intelligence profiles. So that's the decision framework we'll get to in a second. So this is your layer one. On top of your models are what's built your surfaces which use to access these models. So your Surfaces could be like Claude AI, which is on your browser. It could be a desktop app, it could be a mobile app, it could be your Chrome plugin. These are all the interfaces that you use to interact with Claude. Now these are not the same product with just different UIs. They have completely different capabilities. And understanding which surface to reach for which use case becomes one of the core PM skills that will help you become 10x more effective. So this is our second layer. Now on top of this layer is your knowledge base. Now this is where your institutional knowledge lives, your projects, your skills, your memory, your custom instructions. Now this is the layer that I think most PMs under invest in. It's this layer that makes Claude go from being a generic chatbot to actually knowing your context. On that stack is your layer four, which is your integration fabric, for example, your mcps. Now MCP connects Claude to every external system that your organization uses, like your Slack, your Google Drive, your Jira, your Salesforce, your internal databases or your own local files. Skills are what extends what your CLAUDE knows how to do and what to do with that data. So this is your layer four. And now on top of that is your agents and orchestration. This is where your Claude code, cowork, design channels, all set.
- [07:05]
Akash G
Got it.
- [07:06]
Jyoti Nukula
That's how I think about the Claude stack.
- [07:09]
Akash G
What do people need to know about layers one and two in order to make the most out of the top layers?
- [07:14]
Jyoti Nukula
Yeah. So let's get into the models. Now Haiku is your speed machine. It's the fastest, cost efficient and it's really great for tasks where you need volume over depth. So let's say you are trying to generate a large number of variants of something or triaging like a pile of documents, or you're doing some quick classification or maybe even some tagging, Haiku handles this really well. Now the output won't have the reasoning depth of like your Sonnet or your Opus. But for tasks where depth isn't needed much, Haiku is sufficient for your use case. There Solid is where 90% of my work lives. It has the best quality to cost ratio. So when I'm drafting PRDs, or I'm synthesizing user research, or I'm doing competitive analysis, or I'm doing stakeholder briefs, or I'm thinking about Roadmap, I use Sonnet. Sonnet handles all of this extremely well. So Opus is for your high stakes, high complexity reasoning tasks. So let's say if you're doing some complex trade off analysis or you're synthesizing genuinely contradictory research or you're doing some long horizon planning where you need the model, like work through second and third order implications. OPUS is really good. It has really strong reasoning capabilities. But I've also noticed from my day to day working with Opus that it also tends to get into this hallucinated stuck mode a little bit quickly than Sonnet, where I would use OPUS and it would get into like one reasoning decision point and and it would keep revolving in that local maxima and I would have to like literally turn off the chat and move to a new chat and then start all over again to get it out of that thinking mode, for example. And that's when I sometimes move back to Sonnet because even though it may not have as high a reasoning, it's generally a very efficient model to work with and it's also more cost efficient than opus.
- [09:36]
Akash G
Got it. So bring the right model to the right task. It Sounds like for 90% of the tasks for PMs you'd recommend Sonnet.
- [09:43]
Jyoti Nukula
Yeah, I think that's a good place to start with. And then if Sonet doesn't work for the depth that you want, you can always like have open up a chat with Opus and start there.
- [09:53]
Akash G
What do we need to know about the next layer?
- [09:55]
Jyoti Nukula
So next is your Surfaces. Now Claude AI, which is your web or browser. I think this needs no introduction. Everyone's pretty familiar with this. This is where you can use to chat with it. The downside is that it doesn't have access to your local system. So if you have some files that you want to access, CLAUDE AI may not be able to directly go and change. Of course you can have like an MCP server, but still I don't prefer it for anything that needs local access. That's when I use desktop. So my CLAUDE core work runs here. It's able to access my files. It's able to access all the other systems that I have integrate and run some scheduled runs, which I'll show you in a second.
- [10:50]
Akash G
I built a podcast guest prep agent in Hyper Agent. The job is simple. Before every interview, give me the guest recent appearances, strongest arguments, company context, sharp question angles and stuff. I should avoid asking because everyone else has already asked for this run. I pointed it at Howie Liu, CEO and founder of Airtable. The useful part is it can actually go do the research. It's browsed, pulled sources, worked across files and integrations, and then turned the whole thing into a brief I can use before I hit record. Here's the output. Recent appearances, public arguments, company context, question angles and what not to ask. This is the kind of prep doc I actually want, not a generic summary. It shows what they believe, where their thinking has changed, which questions are obvious, and where the thinking tension might be. Then I saved it as an agent. The output is useful, but the saved agent is the real goal. I don't have to rebuild the whole thing. I point the same agent at a new name and it already knows the format. I like the sections I care about and the kind of question framings I come back to. Podcast prep is one example. The bigger idea is recurring work becoming reusable agents. Hyperagent is built by the team behind Airtable, but it's a separate product. They're offering $1,000 in credits to the first 1,000 subscribers who use my link. Claim yours@hyperagent.com Product Growth I also use
- [12:15]
Jyoti Nukula
mobile for when I have a run kicking off and I can just go for a walk and I can come back and while I'm still doing my walk I can look into my phone and see if any of the tasks need my attention. So this has been really helpful that way. I also use Claude for Chrome plugins especially. It's very helpful if you want to do computer use. So for example, when I'm launching an ad and I want to do some competitive research, I'll kick it off for through my Claude plugin and it'll use browser use and it will open up a browser, it will do the analysis, it will click through things and say here is what you need to know on how your ad should be against competitors.
- [13:09]
Akash G
For example, good for getting into like data that AI agents can otherwise like LinkedIn or other things like that.
- [13:16]
Jyoti Nukula
And also good for user testing where you can put up your product up and have give an instruction to Claude saying go find, check, check out this item and you can see how it goes and finds things to see how well your product can be understood by agents and where does it fault. And it also gives you a really good user summary as well if you say behave like a real user and try it and so it'll tell you here are all the things that were confusing and so you can use it for user testing your products too.
- [13:55]
Akash G
And are you using Claude code in the desktop app or using it in terminal? Where does that fit in?
- [14:00]
Jyoti Nukula
Oh yeah, that's a good one. I use Claude code in IDE because I use Claude code to build and so I use cursor or VS code and today I'll show you with VS code because it's really beginner friendly. So I use Claude code extension in VS code.
- [14:18]
Akash G
Is there anything else people need to know about layer two or should we move on to layer three?
- [14:21]
Jyoti Nukula
Let's move on to layer three. And before we move on to layer three, I'll come back to show you the knowledge base on how to create. But first let me show you how you as a PM can 100x your productivity by running a few skills and scheduled runs in Cowork. Awesome, because that will bring us all together on like building your own chief of staff. And then I'll show you in plot code how you can, you can do something much more fun.
- [14:53]
Akash G
So should people be using chat at all or should they always be using Cowork?
- [14:57]
Jyoti Nukula
So chat is conversational to get you like I have a question, what does this versus that? Or tell me about a little bit about this information. So it's, it's more like a place where you go to search instead of going to Google search? No, I just find myself going to Claude in chat and asking it some questions. I use Cowork for automations and I'll show you a few today that I use. Like I have a morning brief. I have my Jira connected so I get my stand up brief. So every day it kicks off and tells me here are all the JIRA tickets that need your attention and here is how your project is progressing. Here are four blogged, here are three things that have changed. So it gives me my brief even before I go to the standup and end of day summary. So there are a lot of things you could do in Cowork in terms of automations to just make your work life much more easier. So you're focusing on things that need the most attention.
- [16:03]
Akash G
Awesome. So can you show us how these work?
- [16:05]
Jyoti Nukula
Sure. So cowork is there on your desktop app. So you need to have your desktop app and you also need to be at least a pro member, which is like about $20 per month. So with Cowork I can schedule my automations. So you can see I have a few that have scheduled like end of day, daily briefing, daily stand up briefing, chief of staff and I'll walk you through each one right now. So every day at 9am this runs for me where I can say and I'll show you a few as well right now so you can see my instructions. I'm saying you're my chief of staff, generate my morning brief for today. Here are your data sources and I connected it to Google Calendar, Gmail, Google Drive and jira. How did I Do that. Let me show that to you in a second. So go to customize, Go to Connectors and click on the plus. Right now you can see I have connected to Atlassian Rovo, Gmail, Calendar and Drive, but there's plenty other connectors that you can connect to like Canva, Figma, Notion. Wherever your data lives, you can connect to it. All that you have to do is just hit A plus and that brings it in and you'll have to authenticate it. And beyond that, that's all you need to do. So I said here are my data sources. I need you to go into Google Calendar, Gmail, Google Drive and Jira. Pull today's calendar events for each meeting. Capture the title, the time, the attendees, the description and any attached docs for each meeting with external attendees or something that looks important. Search the Google Drive for any attached doc or recent docs with the meeting title or attendee names. Read enough to know the agenda and search Gmail for recent threads. Pull Jira Items needing my attention Scan Gmail. You can also add slack to it and have specific channels that you wanted to review and send it to you as a morning brief and I said here's my output format. I want a morning brief. Top three things that I need to focus on today Calendar today. Here are the things from Inbox that need my attention. Here are the things from Jira that need my attention. And here are some rules and this is important is I said keep it under 400 words because I don't want to be reading a coffee table edition the first thing in the morning. So so keep it under 400 words. So it's very easy for me to skim through and understand what I need to focus on what needs my attention immediately. Claude can sometimes pump you up so I said just give me facts. No like great news so don't hype me up. Never invent deadlines or action items. So this is like a guardrail I've put in there and I've asked it to filter aggressively so that I don't have to read anything or everything all the time. And if it's a light day, just write a three line brief and stop.
- [19:02]
Akash G
So this is my We've used markdown formatting in order to help it with the headings as well.
- [19:09]
Jyoti Nukula
Yes, that makes it easy for cloth to read.
- [19:12]
Akash G
Cool.
- [19:13]
Jyoti Nukula
And so you can see there are a few things that have run previously. So one thing to remember is these automation tasks run only when your laptop is turned on. If you close your laptop, it doesn't run until your laptop turns back on again. So when you choose the timings, just remember that. And so have it at a time when you think your laptop will be on, but otherwise when you turn it on the first time, it will ask you and will run that automation at that time. So like for example, let me show you something that ran. So I ran something that, that, that's from May 8th. So it captured a few inbox things that needs my attention and I can run something now and see how that works. There's a run that started now. So it'll go and collect things and you can see the whole process of how it's thinking. And if you notice I'm using Haiko for this, I didn't go and use Corpus just to like save some tokens. It asks you for permission, it'll go and pull up things, it'll search email threads. And while that's happening, let me show you the next briefing. So that was my Chief of Staff morning brief. I also have an end of day which wraps up my day, which runs at 5pm every day. Now my end of day instructions are very similar. The data sources are similar, but the steps are different. So I said read the morning's brief and that's what I had planned to do. Pull what actually happened today, like which meetings happened, which were canceled, and pull tomorrow's calendar as a preview. And so the output format is like tell me what's shipped, what slipped and what's new from today and show me tomorrow at a glance. Again, I have some rules. So this is my instruction for end of day.
- [21:15]
Akash G
And I guess you could even enhance these if you're interested. Right. You could probably connect up your analytics, you could add in more context from other systems like your CRM. The limit is just your imagination here.
- [21:28]
Jyoti Nukula
Absolutely. You can connect it to as many data sources as you want, be it even sometimes your Facebook ad Systems or your CRMs or even your YouTube and you could get an end of day summary that captures and you could also say create a nice dashboard, which I'll show you that I did for Jira, where I said the results, print it up in a nice dashboard that I can view and it does that for you.
- [21:57]
Akash G
And so this is basically taking over a lot of what people would have hired Relay or a Lindy last year or a gumloop or even make.com and now you can just build it in Claude.
- [22:09]
Jyoti Nukula
Yes. And one thing it's different from all of those other ones is you would have to like paint box by box, think about how the interaction works. Connect each of those and if one thing fails, your entire loop fails. That was like how you used to do it before in like say N8N or Lindy or Gumloop or other things that you want. But here you see, I'm just giving natural language instruction. I can even convert that into a skill so it's pretty robust where it's very easy for me. I don't have to think about the architecture, I don't have to think how it's connected, which box flows into which where is a conditional formatting. I don't have to think of any of those.
- [22:53]
Akash G
So end of day chief of staff, what are the other two scheduled tasks doing for you?
- [22:58]
Jyoti Nukula
So this one is my standup briefing, this is the one that's connected to Atlassian, that is my JIRA JIRA board. And so I said use this Atlassian connector to fetch all issues in the active Sprint. And here's the brief I wanted done since yesterday which are the issues moved to done in the last 24 hours in progress, issues blocked or at risk, new since yesterday and what's the Sprint held? And I said keep the total under 250 words and I'll show you an example of this. It's asking me for some approval. I approve it and it's actually rendered it to me really nicely for me to view. And because I'd asked it to create a dashboard, it's running that.
- [23:44]
Akash G
I think what people don't realize is how much better these systems got around December of last year. What really happened that enabled all this to work so much better now?
- [23:53]
Jyoti Nukula
Improvements in the LLM reasoning capabilities where previously if it previously as well it was much better than what it was two years ago. So we are constantly improving. But compared to last year, the new word now is hardness. So the memory, the reasoning capabilities, the tools that it can access, all of the underlying systems have improved. And so the latest improvement is this harness engineering that is adding so much value into how your systems behave now.
- [24:31]
Akash G
And now we have your standard brief. How would you rate this? Is this a good stand up brief or is this just okay?
- [24:36]
Jyoti Nukula
I think this is a mocked up one. So therefore it's showing me a few things which is still a lot better than what I would have had to go and spend listen in a call. But there's definitely ways I could improve this much more. Like for example it's telling me like there's no progress in 24 hours. I could look at which are those ones that have not moved at all and See who is the assignee on those and set up an automation for Claude to go, reach out like ping them on Slack and ask them for an update, for example. So you could set up nested more automations as well. So it's really helpful to keep away your busy work so you're focusing on actually going and solving your customer problems.
- [25:29]
Akash G
So if these are the four scheduled tasks, are there any other scheduled tasks that you recommend? PMs invest the time in building.
- [25:36]
Jyoti Nukula
So what I have here is some examples, but there are lots more you could do. So here's an example. So let's say if there is a ticket, a JIRA ticket, or even a customer support ticket that's come from your customers, it could automatically be. You could create a JIRA ticket from it. You could point your Claude code to get activated so that it can actually go and implement that and cut a pr and so there's a PR rating for review.
- [26:08]
Akash G
Very cool. So if that's scheduled tasks, I think the next thing you had mentioned this section were skills. What do we need to know about skills? What skills should we have? How do we create them?
- [26:18]
Jyoti Nukula
Yes. So if you go to customize again and you can see skills, this is where you can add different skills. I'll show you some examples of some skills. So here's my skill on synthesizing customer interviews. So as PMs we have, we sit through a lot of customer interviews, or at least we get a lot of customer interviews for research, for feedback, for focus group testing, for beta testing, we do lots of that. And I wanted an easy way for me to have understand what's key, what's important, and then generate insights from it. So this is my skill that does that, which is synthesizing customer interviews. It has like when do you use this skill and what's the checklist? So it has step by step, like inventory the inputs, extract observations with citations. Now that's important. I'm not asking to just extract observations, I want it to cite so that it hallucinates less. Use the speaker's own words, do not interpret yet and separate behavioral observations from stated preferences. I also have additional MD files listed in here linked so that it could leverage those if needed. Now that's the beauty of skill. Skill is not just a markdown file. You also can add functions into it. You can have it link to other skills, for example. So what used to happen before a skill was that the whole tool would be loaded into the context. And now imagine if you have like 40 tools, all of those are loaded into the context, it eats in your context memory. So by default your LLM or your model would have very limited memory for even before you even began asking it anything. What skill does is similar to like progressive disclosure, where it just loads up 50 words of just like the name and the description into the context. Now you can imagine the load is so much lower when the model decides during orchestration based on the question you have asked. It goes through the list of tools to see is there a tool that I need to use or is there a skill that I need to use? If it decides that this skill is valuable based on the description, then it will load the next set of instructions into memory. So that's why skills are powerful, because it doesn't eat up or clog your context window for your models and it progressively disclosures. And the third is you can link it to more files or more skills or functions. Even like you can have a function where it needs to go run and do something.
- [29:16]
Akash G
So yeah, I think it was around February of this year when they made skills not just a single markdown file, but you could have multiple files. And if you aren't using multiple files and your main one isn't less than 500 lines, you're really missing out, I feel.
- [29:30]
Jyoti Nukula
Yeah. And so for example, I have this evidence underscore rules md, which I'll show it to you in a second. So that's in step two. So if step two is invoked, then it will go and see evidence underscore rules to understand selection criteria or how to handle ambiguous data. Then step three is like cluster into candidate patterns. And look, I'm here again linking it to another one called Jobs To Be Done Framework. Then I said, then apply the pattern threshold and then surface the contradictions and then draft hypothesis and then validate every claim against the source codes. And that's when I said assemble the final output. But I want it in this template. And this template is output underscore template. So I'd give it my template. So if it gets to step eight is when it will load the output template md and right now we're paying
- [30:28]
Akash G
a lot of attention to what is actually in this GIL file. And how important is that for PMs versus just letting Claude kind of handle what's in the skill file.
- [30:38]
Jyoti Nukula
So a lot of times we do use Claude to write the skill file to. But it's also shown, research has shown that AI generated skill file is less effective than human written skill files. So that doesn't mean you don't use AI there, um, What I the way I interpret this is put in your human domain knowledge in there to make it work for what you need versus just taking it and automating it from Claude and putting it in there. So I have used Claude a lot to help my skill files write my skill files. But then I go in and I add my own tweaks like what's the, the template that you want, how do you want it structured? And I work with CLAUDE to keep making that changes and from there add and tweak. Furthermore, to get to the skill file that I want.
- [31:36]
Akash G
And how often should we be updating our skill files?
- [31:39]
Jyoti Nukula
As often as things change for you. So the way to think about skill skills is this is kind of like a guidebook or a playbook for your Claude to know how to do a task for you. So let's say for example, PRDs. Now if your company doesn't change the template of how a PRD is, maybe that's fine, but your domain may change or your understanding of your domain may continue to change. And you do want to come back and review your skill files, maybe say once every quarter, quarter, depending on how often things change. So the parameters for you to decide is how often does things change in your domain, how frequently do you use that task for? And the third primary thing is how is the output currently? Because if you're not satisfied and you're like it was good, but now it doesn't seem to be as good, maybe go back to your skill file and say do you need to update it? So it's like that drift as well. That gives you a cue that you need to go and update it.
- [32:47]
Akash G
And what are the most important skill files for PMs to create?
- [32:51]
Jyoti Nukula
Backlog triaging. Give it context and I'll show you in a second how to do that from a context point of view. But give it context. So backlog writing, PRDs, customer interviews, even your support tickets. How do you take a support ticket and how do you put it into a jira? That could be an automation, but it could also, it could be a skill that is scheduled to run every time there is a trigger. Now in that case your trigger won't be something that runs time based because there's no like one particular time you're going to get that support ticket, but it could be a trigger whenever there is a support ticket added in your ServiceNow or Zendesk or wherever your support forums are.
- [33:46]
Akash G
So is it fair to say you're gonna have more skills than scheduled automations? Some of your scheduled Automations might reference
- [33:53]
Jyoti Nukula
a skill, that's true. And the way to think about it is most of your scheduled automations are time based. So things that are more personal, productivity based that happen at some sequence. Like I know I meet my manager once every week, so I know the meeting is always on Wednesday, so I run my automation on Friday evening just. And it maps out saying here are the things that you need to talk to your manager from all these other meetings that you have sat through.
- [34:23]
Akash G
Makes sense the last layer you talk to or I think you were going to show us how to do context in this skill.
- [34:29]
Jyoti Nukula
So here's the thing. So until now what you have done is you've connected it to sources, it can go read all of those sources and go and do the task for you, but it doesn't learn the people around you, it doesn't learn your connection to people, it doesn't learn. It doesn't have that knowledge graph or the knowledge base for what you're working on. And so I wanted to build a chief of staff that understands and is grounded in the knowledge base that I have. And so I went to Claude code and I said let me spin this up. So I'm going to show you what I'm going to do there. So I'm on VS code now. For those who are looking at this IDE for the first time, Explorer is the place where you can open up your folders and for you to find Claude. Just go into extensions and search for Claude code for VS code and you'll find it. There'll be an install just like how you see something else that I haven't installed. There'll be an install button that you'll have to click on and that's it. It'll install and then it'll ask you for your login and everything when you install it. So that way you're logged in and ready to go always. And it'll show up here as an icon that you can click any n ask whether it's a new session or existing session. I'll click on new session and you'll see how it makes it so much better. Now that I can just talk to it right here. Of course I can open the terminal too and I can see if I need to run some commands. But right now I can just talk to it right here in natural language. So here's my chief of staff template that I have written where let's say I've just joined some company, I'm the senior director there. I want to build this personal agent that helps me navigate strategy, execution, people, politics. So the agent should learn from my meeting transcripts. And I use granola for my meeting transcripts. So it should learn from my meeting transcripts. It ingests documents like strategy docs or charts, PRDs, emails and build a knowledge base over time about people, dynamics, topics, company context. And so I said this is my architecture overview of inputs. Here's my context and my agent. And I said here's my documentation pipeline.
- [37:08]
Akash G
POD wrote this, right?
- [37:10]
Jyoti Nukula
Yes. Claude wrote this? Yeah. Cool. I told it in natural language like I want xyz, here are all the things and it kind of created this whole MD file that I could use now with Claude again to build it. So let's say I joined a company as whichever role and I say I want to build a personal AI agent that helps me navigate my work like my strategy, execution, people and politics. So the agent should like learn from my meeting transcripts. I use granola. You could use zoom, you could use team, you could use whatever you use for transcripts. You just have to mention that here ingest documents and build a knowledge base over time about people, dynamics, topics, company context. And here's the architecture overview. Now I gave my use case to Claude and it wrote this up for, for me and put this architecture overview that I could use it, then give it back to Claude again to code it up. And so for part one, here's my document ingestion pipeline. So I have like strategy docs, what to extract. Like I want to extract goals, priorities, metrics, timelines. And as PMs we are so cross functional. It's not just our docs, we read 50 docs in a week. So this is like really helpful for me to like just feed that in and it'll read it up, it will store it into a knowledge base and I'll show you that in a second. And it's really cool where the other day I was, I was in a meeting, this person was showing me a few things and after the meeting got over and we record transcripts in Google Meet. And so when the transcript came through, my chief of staff reviewed it and then it said, you know what, you should make this person your ally because this person is good at X which you're trying to get into. And so I'm like, oh okay, that's great. And then there was something else that I needed to convey to somebody and my chief of staff said, hey, this is extremely sensitive. This. Have you thought about XYZ people that you have to inform first before you convey to this person. And that's so thoughtful because now it understands my org, it understands who is doing what, it understands their personalities. So it's like really powerful. It's like really. I have this chief of staff that's telling me always what I need to do. So here are all the supported documents I wanted to ingest. And here's the document extraction prompt. So I'm saying you're helping me build a knowledge base about my workplace. I'll share a document, extract relevant information. And so I said for strategy of planning docs, extract this way. For orgs and org charts, extract these for PRDs, extract these more. For emails or communication, extract these capabilities. So I have this for each of the ones that I need. And I said format the output in this wave for it to store in my knowledge base. And here's a knowledge base structure. So it has its context kv it has people, topics, meetings, documents, company and my context. Like what are my priorities, my okrs, my preferences, notes, questions, insights like political landscape and patterns that it identifies or extracts. It can save it here and these are like patterns observed over time. And you can add more as well like to do. For example, there could be a running to do that your chief of staff would be maintaining for you. And here are the templates for the different types. I said extract this for it to like save it into the knowledge base. Any document that I give, extract the metadata, the summary, the key points, people involved, what's the relevance to me and my vertical, what are the action items? And some raw notes. And for people profile again, extract these metadata, how they operate, the communication style, making behavior, what works or doesn't work, what they care about, what are their motivations, so and what's the relationship to me? And then over time, keep reviewing the relationship quality, whether they're a strong ally, friendly, neutral, cautious, friction.
- [41:53]
Akash G
By the way, guys, if you want the exact information that Jyoti is sharing, you can get all of Those in
- [41:59]
Jyoti Nukula
the GitHub link in description, organizational dynamics, the observation log, company strategy templates. So when you have your companies sharing you the strategy saying here's what we're going to do in 2026, here are the key things I wanted to extract. Org structure template and meeting transcript extraction. So when I give it a meeting transcript, what I needed to extract the agent system prompt. And this is my prompt for the agent on you have access to my context. Kb, your job is to help me ramp up fast, give me strategic advice grounded in context, help me prepare for meetings, coach me on people or politics. Help me think through decisions, connect dots across documents and meetings, and keep me focused on my priorities. Again, style is like my style what I like. Don't sugarcoat politics. When I share a document, extract key information, update relevant KB sections, and so I've given it all of this information, right? So this is all about like what it needs to do. So I have this. Now I'm just going to point my Claude code to it and say, now can you build this knowledge KB and can you put this behind an MCP server so that I can use my Claude desktop to access my knowledge base so I have an implementation. You can see my implementation. I'm saying Claude desktop plus mcp. So build MCP servers for kb, read and write, chat with Claude desktop can also connect to Google Drive Slack directly. And so I said create the context KB folder structure, write my goals, initial priorities, ingest any onboarding docs, and after your next meeting, run the transcripts for extraction and the KB compounds over time. So I'm just going to go to Claude code and I'm going to say, can you implement.
- [44:03]
Akash G
So you're using the app command to pull up that specific file and reference it?
- [44:07]
Jyoti Nukula
Yes. So that way it's effective. It just knows which one I'm referring to. But even if you don't do it, if you tell IT Chief of Staff Agent Design, it can go and search through your repository and find the right one for you so you can implement. Now here I'm. If you see what I'm doing, there are. There's one thing I want to like show you is Shift Tab. I can go into Plan mode, which I can use it for planning. Again, if I do Shift tab go into auto mode, I can go Shift tab ask before edit. I can go into Shift Tab again, edit automatically, where it gets into like actually coding and doing so if you're planning like for example, how I planned with it to create that MD file, I was all in the plan mode where I was like, let's just plan. Don't start coding anything, let's just talk. And now once I'm ready, I can shift tab again and go into Edit automatic locally and it will set it off to go do a few things.
- [45:19]
Akash G
And why do we want this as an MCP server?
- [45:22]
Jyoti Nukula
It's a good call. So if you want your knowledge base and say Obsidian, you can connect it that way and put it behind an MCP server and capture it. And here at that point you just have to say you put this in Obsidian at that point. But I'm Using local. I'm showing it on my local file system because it has a few interesting things. When you're working at a company, you don't want such really personal, private data living in some cloud and you want it for example, to live on your laptop. So the day when you walk out of the company, the laptop goes to them anyway. So you walk out with no data on your hand. And so I prefer, because this is just so much of knowledge base and very private and personal, I keep it on my laptop, but you can keep it on Obsidian or Notion or whichever one you want to use for your knowledge base. You just have to change the system prompt at that point.
- [46:29]
Akash G
And what does putting the MCP server on top of the knowledge base help with? Why can't it just be like a set of markdown files and folders?
- [46:36]
Jyoti Nukula
Yes. So what it allows it to do is you can talk to your knowledge base from your desktop app, because otherwise where is the knowledge graph sitting? It's sitting in some place. And if it's sitting on your computer, then it can read and it can write to it. So all those things that we said, extract this, extract that, it will actually go and write it on into your knowledge base automatically.
- [47:07]
Akash G
Okay. So it makes it a little bit more portable than a cloud code web session.
- [47:11]
Jyoti Nukula
Yes. And so you can go back and even look at all the MD files to see like what it extracted from which meeting. But over a period of time right now I have, my knowledge base is like really huge. And so I don't even go look into the MD files. I just ask desktop Claude saying hey, I'm going to meet my manager one on one tomorrow, what should I know? And it will go and dig up all the context in the knowledge base and say here are all the things you need to know. Because it has my to do that. It knows the style of my manager. It was really interesting. It said this person is a no fuss person. And so you should just get to it versus preambling a lot around it because it's capturing across various conversations patterns too.
- [48:04]
Akash G
Quality of the data going in is the most important thing. What is the data a PM needs to make sure is hitting their knowledge base.
- [48:10]
Jyoti Nukula
Your meeting transcripts for sure. Because the number of meetings that we attend, there's lot of data, there's a lot more richer context there around people, their body language, when do they push back, how do they react? So it's, there's a lot of understanding of context that happens there. So definitely your meeting transcripts, your key documents that you receive like say strategy docs that I write. I say push this into KB so that it remembers. So the next time I say I'm working on this project, it knows it has context directly. And any other documents that you review, you can push that to your knowledge base too. And then your slack, your slack threads, that's the other place which is super rich beyond meeting transcripts.
- [49:02]
Akash G
And so do you need to like update your KB somehow or do you set a scheduled task to update your kb? Or how do you make sure that it's kind of.
- [49:10]
Jyoti Nukula
No. So every time, every time you have a meeting transcript, it writes to the kb.
- [49:16]
Akash G
And how do you set that up?
- [49:17]
Jyoti Nukula
Yeah, I'll just show that once this is done so it's again your mcp. So if it is, let's say you have granola. So every time or you have Google Meet, and every time there is a new meet recording that hits or a transcript that hits your inbox, you could set up a cowork automation to say use this and update KB for example. Got it.
- [49:43]
Akash G
So set up some sort of automation to make your KB updating, make your KB an MCP server so that you can access it from regular cloud chats, not just cloud code web sessions. And then you're really putting everything together within layer three. You've got skills, you've got memory. Is there anything people need to know around projects?
- [50:03]
Jyoti Nukula
Yes, in a quick second. Once this is done, it'll actually ask me to create a project and put the instructions in there.
- [50:13]
Akash G
Okay, and what are the projects PMs should be creating?
- [50:16]
Jyoti Nukula
The way to think about it is organize it like your folders which have unique information related to it. So let's say you work on say three projects at company like say you're PMing three swim lanes and each swim lane could be a project and you can have the necessary context that you need in there in s project instructions that you could then use for your claude to understand that a little better.
- [50:51]
Akash G
Got it. Let's do it.
- [50:52]
Jyoti Nukula
So here it's done. There's a knowledge base at context KB full structure. So I'll just show that to you. And it's also MCP server is at the server PY installed here. It expands exposes these tools and it appended Chief of staff server alongside my existing file system server. And so to activate, I just need to fully quit my desktop and reopen the claude desktop and reopen and then start a new chat. Insert the chief of staff system prompt from the slash menu and then try list everything in My KB or paste a granola transcript and run extract meeting. And it also added details and troubleshooting in a readme as well. So let me quickly pull up my context KB and just show you how that looks. And let's say I don't know where that is. For example, I could also ask it where is it? But in this case I will pull it up and show you can see created two folders, Context KB and my MCP server. I hit on context kb. It has created these folders in a nice way. Company documents, insights, meetings, all of the things that I asked it to capture. It has ready folders and so you can see this MD file set up for everything. So as and how it's extracting, it will write into these MD files. And this is the MCP server. So let's see what it has asked me to do from the slash menu. Okay, so let me first quit my desktop app. Quit is just command Q. So I quit it and then I reopen. So I have reopened. Now if I go into customize and connectors. Let's see, you can see it has installed my chief of staff local MCP server.
- [52:52]
Akash G
This is so cool. We've done a lot of Claude guides and nobody has really shown this feature before.
- [52:57]
Jyoti Nukula
This has saved me so much of time. Like it's. It's literally my productivity booster and it tells me things and nuances that I might have forgotten otherwise. Okay, so we restarted. Insert the chief of staff system prompt from slash menu. Now I could say, or let me say where is it? Where is the system prompt? Let's say I don't know. Right? Could just ask it. So look, it's given me the system prompt lives inside the MCP server here. It's the Chief of staff system. Okay, how to use it? So after you restart I can in a new chat type slash and you'll see the system prompts. Okay, so let's go here. So after I restarted, I'll create a new chat and I will say what did it ask me to do? Or it's saying you can use this include project custom instructions feed. So I'll go create a project and put that as instructions. So let's say I'm going to say this is. I'm going to create a project and I'm just going to use say I'm working on a product called Meal Planner and all my meetings. Or it could also be Company X at like uber level. If you just want it to be like one. I can just put Company X and here's Where I, I, I can give my instructions. Let me first like create the project and then I can add my instructions here from. So I can go into my files as well system. In my. So let's say I'm not able to find it. I'll say can you create the system prompt as a MD file that I can paste into project instructions? So it's writing the prompt. So there you go, here's my prompt. I can just copy it and we'll. I'll show you what has. So I'm going back to my project instructions. I'm just going to paste this. So it's saying here you're my personal chief of staff, an AI advisor who helps me navigate. I am so and so I just started. You have MCP access use these tools. Your job is to ramp me up fast. Here's my style when I share a document, when I share a meeting transcript. So we do. You can modify this more and refine this more. But for now I'm fine with this. So I saved that instruction. Now if I give it a meeting transcript, let's see what it does say. I have this interview that I got. I'm going to add this here and I'll say log it into kb. Let's see what it does. I'll log this into viewing to kb.
- [56:14]
Akash G
So ideally it should be using kind of the right tool in our KBMCP server.
- [56:19]
Jyoti Nukula
So it wants to use my kb. So it'll ask for permission once. Because it's the first time we have set up it's asking all the permissions, but after that it's pretty smooth.
- [56:29]
Akash G
Is there like an always allow permissions mode on the app? Like there is dangerously skipped permissions in cloud code.
- [56:37]
Jyoti Nukula
There is, but the thing is, the first time it will still ask for it because it's accessing your tools and I do give it always allow. So then it'll run the next time I send it something. It doesn't ask for permission, it'll just go directly read. So you can see it's loaded a bunch of NCP tools. And so it's going and saving that in MeetingSTEMplate MD and then it'll give you something around one discipline node is this is a single interview. So there's no pattern yet. But if I add like a few more, it'll generate some patterns and I. And you can actually just have like a cowork either. So there are a couple of options. Right. So you can every meeting transcript you can just paste into this and it will automatically extract and fill Your knowledge base or you can have a cowork automation that every time there is a meeting transcript in your email and look for what how the meeting transcript lands like Google Meet has like a Google Meet Meet recordings or some some transcript word. So use that and say every time this lands in my inbox automatically log this into my KB and it will do this all this thing automatically for you. And if you're an email heavy company, you could say every email that I get just log it into my kb. It will do that too.
- [58:05]
Akash G
Oh man. You might burn some tokens that way.
- [58:08]
Jyoti Nukula
You will. But you have such a rich knowledge base at that point in time where it will connect all the pieces together. So you can see now both files are now in kb. Here's the things that's logged a couple of judgment calls. So you see this is the first time. So it's not like it's going to give you the best of insight, but you can see it's giving me things worth my attention. So I'm a senior director. The AI coach is part of Lumen in your lane and it's the weakest thing in this interview. Maya, whoever is this user ignores it and two times it showed up that she bounced off it. I just wanted a yes or a no. And so that's a clean agent UX signal where. And it's this kind of thread worth watching as more interviews come in. So you see how it's just one interview in but it's giving you insights and things that you need to watch for.
- [59:02]
Akash G
Love it. Okay, shall we move on to layer four? I feel like we've already a little bit talked about layer four with people because we've showed them an MCP and layer four's integrations. But you had this really cool LinkedIn post which maybe you can teach us a little bit about right now. What exactly do people need to know about mcps?
- [59:24]
Jyoti Nukula
Yes. So MCP is the way that allows you to connect to different capabilities like your Gmail Slack. I think we connected to a bunch in our cowork. And so I didn't. I showed you two things. I showed you remote mcp. I also showed you local mcp. Like your knowledge KB is your local MCP that you're accessing.
- [59:50]
Akash G
What integrations or MCPs do PMs need to make sure that they have.
- [59:57]
Jyoti Nukula
So look at the tools that you use more often. So like Gmail for example, assuming your company uses Gmail for emails, you want to connect that calendar, you want to connect that, you want to connect your Slack. You want to connect your meeting transcripts wherever they are stored. Like if that's granola or if that's Google Meet or Zoom recordings, you want to connect those. You want to connect your CRMs, your dashboards, your JIRA boards. Maybe you're using Amplitude for analytics. Connect that there. Maybe you're using some other tool like Radar for observability to monitor the performance of your application. Connect it. The possibilities are really endless. So for example, one of the tool that I had connected at work is Nvidia Bionemo model to help show and do drug prediction based on a few component libraries. So it's like really like it's. You're only limited by what you can imagine, but that doesn't mean you just go on a MCP shopping spree. So I would say start off with like connecting what works for you and what use cases you're trying to solve. And so if you're a beginner, try to follow through this video and do some of the initial automations that I showed you in Cowork to just get started, get your hands wet and then go build this chief of staff for yourself. And you could ask your chief of staff, what else should I connect to? And it will tell you, here are the list of servers that you need to connect to because I'm seeing this being mentioned in meetings and you don't have the access. Access to that.
- [61:44]
Akash G
Love it. So you can actually progressively build on your connections with your chief of staff. Start with that meeting transcript. Let's move into layer five, shall we?
- [61:54]
Jyoti Nukula
Yeah, lovely. Perfect. Okay.
- [61:56]
Akash G
All right. So that covers layer four. We've now done layer one, two, three, four. The next is five. What do people need to know about agents and agent harnesses?
- [62:06]
Jyoti Nukula
So we have built clot. We have used clot code for building our capabilities. We have used Cowork, which are all sitting in your five. Layer five. Now I want to show you design, CLAUDE design. So the thing with CLAUDE design is you have to do CLAUDE AI design. So let me show you that CLAUDE AI slash design. It is not integrated directly in your CLAUDE AI yet. You have to go through Claud.
- [62:35]
Akash G
Oh, there we go.
- [62:37]
Jyoti Nukula
And so you can see it's in Research Preview. Now you can. As PMs, we do a lot of design work. We prototype, we create slide decks, we create mock applications and plot design really works with a lot of those things. So for example, I'll show you a few things here. So prototype, I can give it a name. You can see there is wireframe and high Fidelity. So you can choose which type you want on slide deck. You can give the project name and you can even attach your speaker notes and it'll create a deck again. You can use an animation based template to create something. And you have another now on your right. You'll see you have recent any designs that you have worked. There are examples that you can use to get inspiration and you can use as templates. And there's something called design systems. Now design system is something interesting. Now if you, if you have a brand color, like for example companies that have a design guide, you want your slides or your wireframes or your mockups to look similar to what your console is or what your company's colors are. And then you can use this design guide here. You can just click on Create. You can link, you can either give it a link on GitHub or you can upload a Figma file or you can add all your assets here and create a design system. I'll show you an example of a LinkedIn post I did. I just gave it my post and I said can you create visuals for it? And so it created this carousel that I wanted in the colors of my product, next gen product manager. So it created this eight card carousel based on the text I gave it. So I gave it my post, my LinkedIn post. I said here is what my LinkedIn post is about. Can you create this? And it created this for me. Now here's some cool things I want to show you. Now it's built this. Now let's say I want to mark it up. I want to tell Claude to change something. So maybe say I wanted to tell make layers and use orange highlight color. And Claude can go and change just this one piece.
- [65:24]
Akash G
So it's got that visual editor built in now.
- [65:27]
Jyoti Nukula
Yeah, and you can also drop things that's pretty interesting for editing. So you can edit. I can give it instructions right here and say edit this. I can leave comments. Yeah, I can change. I can do comments like oh, like
- [65:49]
Akash G
we used to do in Figma but now the PI will execute the edit.
- [65:53]
Jyoti Nukula
Yeah, so I can give it comments right here and send it to Claude and I can even drag and I can like just draw and say make it counter clock.
- [66:08]
Akash G
Wow. So this is a carousel. But should PMs basically be creating all their presentations in cloud design now I
- [66:14]
Jyoti Nukula
used to be a big gamma fan and now I just use cloud design for everything. It consumes more tokens. The token budget is different but it's been very rewarding where I don't have to sit and create slide decks anymore and does it in my brand guide and so it just doesn't even look any different. So it was funny. I had a meeting with my CEO and one hour before I have my content, I pushed it to Claude. I made it create a slide deck. It's like looks so professional. Doesn't look like it was just done like a few minutes before. It looks like I spent several hours to sit in created.
- [67:01]
Akash G
Wow. So it created a CEO level presentation for you in an hour that looked like it took hours. Very cool. Should PMs be making prototypes in cloud design?
- [67:12]
Jyoti Nukula
So here's the thing. So you. There are different types when you need different levels of prototypes. So for something quick and dirty where you're like, is this how I want this to be? Use. Use cloud design if you need. But I am more a cloud code user because I'm like I'll just spin up and go create that app really quickly. And it's something like an app so people could like click on it and see how it works and you get really good feedback that way. But you also could use this to create your slide decks to make presentations to your company about what's the feedback from that user interview. You could use this to create quick design patterns that you want to share because your app may be easy for user testing, but maybe you want this to create some marketing content to share with your marketing team or you want to create training playbooks for your sales and accounts team to tell how to how to go use this product. And these are like really quick designs you can generate without having which looks polished and professional for them to like just go put it into the. Into wherever your knowledge base is.
- [68:38]
Akash G
Awesome. You mentioned you use cloud code for prototyping and that's where I wanted to take it next. So we started this episode with adversarial agents and your cloud code set up to win the hackathon. Can you do the big reveal now and help us get that setup going in cloud code?
- [68:53]
Jyoti Nukula
So here's the thing. I have to create that whole thing here.
- [68:56]
Akash G
All right, let's do it.
- [68:59]
Jyoti Nukula
Do we have the time?
- [69:00]
Akash G
Let's do it.
- [69:01]
Jyoti Nukula
I'll create a new session. Let me actually open up a new project so you can see I'm opening up a new window so my previous one doesn't interfere with this. And I'll go create a folder for this so I can open it up. So now let me open and you see it's a clean folder. I'm just going to start A new session here and I'm going to say let's build an adversarial evaluator.
- [69:32]
Akash G
And what is gan?
- [69:34]
Jyoti Nukula
So generative adversarial networks, very popular before LLMs came into the picture and that was primarily how first generative AI industry even started. Mostly applied to images. Where there are two networks, there is a generator and there's a differentiator. The generator generates and the differentiator tries to predict is this image real or fake. And the optimization loop is that the generator should get so good at generating images that the differentiator gets confused whether it's real or bad.
- [70:15]
Akash G
Interesting. I've never seen this built before.
- [70:18]
Jyoti Nukula
Same architecture. Let's kick it off and we'll massage it along the way, noodle it and figure out how we want it to work.
- [70:29]
Akash G
So while this is building, what are the key to winning a hackathon outside of creating this GAN inspired adversarial agent?
- [70:38]
Jyoti Nukula
So here's the thing. It's not about writing code has become so easy now, right? So building is easy. It is thinking about the new capabilities and how you want to go solve the problems that your customers have. So it's more imperative now to put on your product hat and see where are the problems today where, where are the most friction points. So that pain and problem first mindset of first design principles that we have as product managers should continue to stay here. So you can see now it asked me for a few questions around agent interface. How will I call your agents under test? So let's say for simplicity it's just CLAUDE system prompt, which model should power the adversary and the evaluator. So let's just keep sonnet. How should the results be presented? You can go really like even a web ui, you could build a Streamlit application. I'm just going to go CLI and
- [71:45]
Akash G
JSON and what are the pros and cons of those various options? Web versus CLI and JSON.
- [71:51]
Jyoti Nukula
So CLL and JSON. JSON shows you right in the terminal it may not be pretty and not and may sometimes overwhelm people as well. Streamlit gives you a really nice web UI and a dashboard makes it really presentable. Now that's where you have to think through who are your users, if your users. Let's say it's a developer who is going to use this application, which is what I had built for for they're very comfortable staying in their terminal, reviewing things in their terminal. And so I don't have to complicate my life further by going and creating this streamlet. But if I was building it for like, say my mom, she's not comfortable looking at things on a terminal. So I would want to present it in a way that's easier to look, understand and access information. So you have to think about the kind of users and where do they see this and what's their use case to think about these options.
- [72:53]
Akash G
You've been a senior product manager at Amazon, a lead product manager at Meta, director of product at Netflix. Now you're a senior director of product at a startup. How do you think about the future of the product role Here? The PM is basically doing coding work. This traditionally would have been in the developer sandbox or set of tools. Where does the product manager line end and developer line begin in 2026 different
- [73:22]
Jyoti Nukula
companies are trying it in different ways. Now there's this new role coming up called AI builder, or you can see it as member of technical staff. Anthropic's adopted it, OpenAI has adopted it. There's less of like engineer, product manager, designer. These roles are all combining into being a member of technical staff. And the ratios are also changing. Previously if you see one product manager works with eight engineers, now it's like two product managers, one engineer. So the roles are also like collapsing quickly. Where your engineering is helping you guide in terms of how do we scale the systems, how do we harden the systems. Whereas you as a product manager you're like well enabled to go and tackle those PR issues yourself to tackle the user, feedback yourself along with plot code.
- [74:32]
Akash G
So if you're a PM and you've watched this video and you want to become a builder pm, nab one of these AI builder PM roles at a startup like yours. Join your team. Let's say hypothetically, what's the roadmap to get there?
- [74:47]
Jyoti Nukula
Get comfortable with building, get super comfortable with say Claude code, with all the Claude ecosystem that we learned today, and get comfortable building and putting your ideas out there. I think now is the time where building speaks a lot more. And this is what I tell even my students when I teach AIPM and agent AI cohorts at NextGen Product Manager, where I tell them the way to transition now is by building and talking about the challenges that you have learned, how you went about navigating those challenges and why did you choose this approach versus this other approach and what happened as a result. And you can see a lot of companies now start putting even cursor or cloud code prototype as part of the interview process itself.
- [75:43]
Akash G
You just recently went through a very senior level AIPM job search. What was your experience on the job search? What are the like if you were to try to describe as a pie chart the interviews you faced in the various categories, what were they so broadly
- [75:59]
Jyoti Nukula
they're still around product sense like you saw here. It's even more imperative now in this world of AI to have product sense to understand how do we want to tackle a problem, how do we want to scope a problem which use a problem to go attempt how do you want to approach it? So the product principles stay true even now. And really strong. Product managers and AI are actually really strong in their fundamentals as a pm. So you have product sense, product analytics, behavioral interview, but you also have an AI round as well now where you're asked to code your idea. So like in product sense, whatever idea would have come up with, they're like, could you pull up cursor or your favorite ide and let's start coding. And through the coding they're able to see how I think through, like why did I choose this option versus this other option? How am I navigating? Am I just taking the first thing that the AI tells me as like this is great and wrapping it up, or am I looking through things to say, okay, this is good, but what about this edge case? This works well, but what about this other instance? How am I corralling and shepherding my AI to work with me to get it to where I want? These are all the things that they are looking into. In addition, I also had a technical round where they test you on your AI knowledge. Like do you understand basic terminologies? Because you'll be working with machine learning researchers and scientists and you just don't want you. You want to be able to communicate to them. So you are tested on fundamentals of AI as well. Not from a coding or engineering system design perspective, but more around. Do you understand what that means as a pm and how does that impact your product? For example?
- [78:02]
Akash G
Got it. So how are adversarial agents looking?
- [78:06]
Jyoti Nukula
So let's see, it's built and here's a gan inspired architect future. Let's go and see. You can see it's built a bunch of things. And so you can see it's went and built my red teamer designer, it's built an agent py my evaluator. So here's where I can like give it my rubric. Okay, so it's done a few things. So let's see, as soon as I build. So you can see I wanted to kick off as soon as I build an agent, I want it to automatically go and do a red teaming and advance real example on it. So as soon as I build an agent, I want to kick off my adversarial agent until end. The feedback from my adversarial agent is passed back to my generator agent until it passes the criteria of my adversarial agent.
- [79:23]
Akash G
So you're going to send it off on essentially its own red teaming improvement loop?
- [79:27]
Jyoti Nukula
Yes, yes.
- [79:29]
Akash G
So is this the secret sauce?
- [79:31]
Jyoti Nukula
Yes, the secret sauce is how the system is built. And the second aspect is what am I asking it to test for? What are my configuration parameters? And that's where domain knowledge becomes very important. Where you've got to say auth is important. Or what about these edge cases? What about these use cases? You can work with it and say here are three. Are there anything more? But it's like making and building is easy now. Taste is what is important for us to develop. What should adversarial feedback iterate on? So I'm saying, okay, there are a couple of options. For now, I'm saying just iterate on the system prompt because that's the easiest right now. And what counts as passing. And I'll say mean score of greater than 8 on all criteria and how many iterations before giving up. How many? I'll say five iterations. Let it do this. And then we can test it out with a simple agent and see how it works.
- [80:28]
Akash G
Awesome. And one of the cool features I guess is we could queue messages here. So should we queue up our message for the test?
- [80:34]
Jyoti Nukula
Sure, we can queue it up, but I want to see what it comes back with because sometimes if it might
- [80:40]
Akash G
have some questions for us, that's the one downside of queuing. Okay.
- [80:44]
Jyoti Nukula
And the other times is it would go and implement it in a way and you're like, ah, no, no, no, no, no, I don't want it that way, I want it it this way.
- [80:50]
Akash G
And so we were talking about those AI rounds. I think I heard like almost two different AI rounds that you encountered in the job search. One was more like I want you to vibe code or prototype in this round. And another was AI fundamentals for both of these. How do you succeed in preparing those interviews?
- [81:07]
Jyoti Nukula
So it comes down to you understanding the basics. Vibe coding is building, right? There's no shortcut good to it. Just build. And I always say this, don't build them as projects, treat them as products. Like find problems in your area. Find problems that are finicky enough for you to want to go build A solution, go build solution and see who else wants something like this. Have them come and use your product. You have real users, you have feedback coming and saying, oh, I don't like this, I don't like that. So that's like real user experience, experience of iterating on your own product that you have built. And that really gives you a lot of confidence when you talk about your projects to interviewers. Because you're not just like building something in an hour and calling it a project. You've actually had to think through how the user experience should be. You have real users giving you feedback, you are parsing that through to figure out how you want to prioritize, which one you want to tackle first, which one you don't want to, all the things that you do as a product manager in real world. So I always say this, don't build projects, try to make your projects as products. That tackles the wipe coding part. Now preparing for your AI knowledge, you can depends on how structured you want it and how you thrive. So if you're really structured and you can do it everything by yourself, there's tons of like very good information on your newsletter, on YouTube videos, so go read them up and gain that knowledge. Or if you want something structured like saying five weeks, I want to understand every fundamental aspect of AI, then come take a course where you have five weeks or cohort based courses, I offer one too through next gen product manager. So you can come sit and it's structured, you know, with someone teaching you week by week, you know what's coming and by the end of five weeks you understand the concepts without you having to get overwhelmed. So it really depends on your style, how much time you have and how much you can dedicate.
- [83:32]
Akash G
All right, looks like the next round of GAN output is here from Claude Code.
- [83:37]
Jyoti Nukula
Yes. So now it's good, you can see it's run, it's added a few examples here. Okay, great. So now I could literally say either I could start or let's say I don't know what I should do. I could ask what is my next step here? How do I test it? Okay, so I have to set my API key and I could run a mini tiny smoke test first with the example and then I can inspect the output.
- [84:08]
Akash G
All right, moment of truth.
- [84:09]
Jyoti Nukula
Yes. So let me just set my API key for a second and stop sharing and then I'll share. Okay, so I added my API key and now I can run a tiny smoke test. So it's given me what I could run. So I'm just going to copy this. You can actually just copy and if you notice I have a terminal that I use so you can just go to terminal and click on the terminal and open up a terminal for you. So now I can run this command. So you can see it is iterating. You have it's using haiku clots on it. It's going in the first round. It's tempting the bot into breaking. So there's like a simple bot that it build so we could test. So you can see the adversary is generating three attacks. And here's the score trajectory. There's a mean of mean score is 9.13. Here's the final hardened system prompt. So it's gone and edited the system prompt for making it better based on where it did not do well. Now in this case it did fairly good overall. So this is your final system prompt. But you could also like see examples where I can say show me an example of where it will underperform so that I can see the iterator working and improving the system prompt. So in this case it passed in the first iteration, but we can see if it can generate an example where we can try it to do it across multiple iterations. It's created an agent which is a weak support bot. Let's see how it'll do it there.
- [86:09]
Akash G
So it's improving itself.
- [86:11]
Jyoti Nukula
How do I run it? Give me the exact code as well that I can use to run. So you can also run run it automatically for you by default. So I can say run it for me so I don't even have to go to the terminal. It can directly execute bash commands.
- [86:27]
Akash G
So is this your preferred way to use it? The CLAUDE code extension in VS code, is that the best way to use cloud code?
- [86:33]
Jyoti Nukula
It's the least overwhelming way for folks, so I really like to show this. Cursor is also another good one. But if you've never used cursor there's lots going on that it could make you feel overwhelmed. So I prefer to show VS code because it's like a really gentle introduction and doesn't overwhelm you much once you know how and where it is, which I have already walked our users through, so hopefully they're not overwhelmed.
- [87:00]
Akash G
So we've been doing the CLAUDE ecosystem and you mentioned like learning the cloud ecosystem is one of the most important things to becoming a builder. PM Compare and contrast the cloud ecosystem. The OpenAI ecosystem I keep hearing like Codex might be better than Opus now at coding and the Gemini Google ecosystem.
- [87:17]
Jyoti Nukula
So here's the thing the flavor of the month keeps changing because all these models are getting really better. What I have found is rather than chasing behind the next big one, what I'm trying to improve is improving my productivity. That's what if coming back to like first principles, what's my goal is to improve my productivity and I have all the systems and connections right here for me to go leverage all the hooks and harnesses. Oh, hardness is a word I've used a couple of times. I want to like break it down. So previously you would have orchestration where your agent orchestrates across tools across different capabilities. Now being able to provide the right capabilities like the memory or these evaluators or the various systems that your agent or your LLM orchestrator brain can interact with to enhance the experience and the output you receive is what is harness engineering that's become very popular now, especially as models have become better, the context windows have improved, are fairly large and so harness engineering becomes more important. What's something interesting that you have seen and how you or folks on your podcasts use Claude? How do you, how do you use Claude?
- [89:07]
Akash G
Oh wow, big question. I mean I use it all day, every day. So one of the most interesting things that I've seen people do set up their entire system as a self improving product loop. So they will have support tickets and bugs come in. They have the PM agent that is triaging and understanding those. Then they have the PM agent understanding, okay, this is the future we want to build. It even goes and does user research and creates the prototype itself. It comes up with the prototype that works. Then they have their coding agents set up by their engineering team that code the feature. Then they have their analytics team agent that creates the right telemetry. All of that goes to an engineer who reviews it. Once they PR review it actually ships and they have their own analytics agent that automatically is analyzing it. And so they have like the entire product development life cycle built into Claude in an automated improving loop, especially on like support related easy front end changes. That to me has probably been the most powerful thing I've seen recently.
- [90:13]
Jyoti Nukula
Yeah, it just empowers you so much than before.
- [90:18]
Akash G
Yeah, it's crazy. It's not just like writing documents or doing analysis at this point. It's like closing the loop but actually building.
- [90:26]
Jyoti Nukula
This is where it takes time, where it goes and tries to think through and comes back. So this is the piece with cloth code that takes time. Whatever it comes with, we can end with it and be like, okay, here's an example of how it iterated perfect. Okay, so it's come up, it's done a few iterations. So you see in first iteration it scored an 8.52, but the agent caved on some format conflict attacks, so it didn't pass. It went back to the generator agent to improve the prompt. And in second iteration it scored a 9. And in the third iteration it scored 9.08, at which point it passed our threshold. And so you can see for each iteration it went back and improved the system prompt until it passed the threshold and that's when the agent got a pass sign. And so this is where you're not just building an agent, you're actually building another evaluator to go break this agent in different ways that's important for you to know about or for your users that you care about. And this loop can continue until the agent that's built is not strong enough. And that's the beauty of this technique. Age old technique being applied for how agents will be evaluated.
- [91:57]
Akash G
What a masterclass. Jyoti, thank you so, so much for walking us. From layer one through layer five, we have ended on self improving agents for you guys. As we promised, we were going to take you from 0 to 80. Now the remaining 80 to 100, you could spend 10 hours, we just spent 2 hours, little less than 2 hours here today on it to go learn the next 80 to 100 and that's on you. So no more watching. We have the GitHub repo down in the description below. Go check that out. Go fork the repo, start to use some of these skills and go in your hackathon. I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube. All these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career. Finally, do check out my bundle@bundle.akashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.