Inside Anthropic: How Claude Actually Gets Built | Alex Albert

Summary

Podcast Summary: Behind the Craft

Episode: Inside Anthropic – How Claude Actually Gets Built

Guest: Alex Albert (Product Manager and formerly Head of DevRel, Anthropic)

Host: Peter Yang

Date: May 17, 2026

---

Overview

This episode features a candid, in-depth discussion between Peter Yang and Alex Albert, peeling back the curtain on Anthropic’s product and research processes for their flagship AI, Claude. Alex shares how Anthropic’s teams treat models like products, the unique challenges of PM work in AI research, and how feedback, memory, and product development cycles are changing in the age of AI. He also delves into Anthropic’s culture, approaches for evaluating AI, future challenges, and the philosophical questions around AI “consciousness.”

---

Key Topics & Discussion Points

1. Defining Product Management for AI Models

• (00:00–02:57)

• Alex was Anthropic’s first prompt engineer, possibly the first in the world.
• Treating Models Like Products:
  “We treat the model as if it's a product to some degree. With every new model we are speccing out exactly what do we want this model to be good at.” (A, 00:05)
• Product and research PMs spec out desired capabilities (coding, knowledge work, product-specific functions) and iteratively upgrade each model.

Memorable moment:

• The team always seeks feedback post-launch:
  “We want to fix and improve on the things the last model didn’t do so well.” (A, 02:34)

---

2. The Feedback Firehose & Using Claude for PM Work

• (03:01–06:52)

• Collecting Feedback at Scale:
  Millions of users provide input, so PMs leverage Claude itself to cluster, summarize, and synthesize user feedback.
• Example: Adaptive thinking feature
  Previously, extended thinking meant Claude would always “think.” Adaptive thinking lets the model choose when to think more deeply for tough questions.
• User feedback directly shapes model behavior, especially on features like how much the model “thinks.”

Notable quote:

• “In terms of feedback collection, [Claude] has been insanely helpful... grouping and clustering things, finding top themes, creating synthetic versions of that problem.” (A, 04:16)

---

3. Memory, Dreaming, and Context

• (07:07–08:45)

• User-Driven Memory: Feature lets users summarize their life context for Claude, drastically improving response quality.
• Nightly Pruning & Dreaming:
  “It writes to a memory file and then overnight will prune those memories… going through its memories, finding things that might contradict, pruning them, cleaning them up.” (A, 07:41)
• Memory implementations vary by product surface (Claude AI, managed agents, etc.), inspired by ideas from human “dreaming.”

---

4. Evolving Product Development in the AI Era

• (09:26–13:57)

• Faster Iteration Cycles:
  “Now all of a sudden we’re in this paradigm where the cost and time required to produce something is pretty low… you can spin up prototypes, even MVPs… in a day instead of two, three, four weeks.” (A, 09:41)
• One-way Door Decisions: PMs focus energy on irreversible decisions (e.g., model architecture), while reversible ones are fast-tracked.

Memorable quote:

• “If it’s something that’s not a one way door, that’s effectively cheap and free at this point.” (A, 11:43)

• Current Bottleneck: Human coordination and strategic communication, not engineering, has become the limiting factor—Claude helps, but “not as 10, 100x speed up like it has in code.” (A, 13:13)

---

5. Claude as PM Copilot: Research, Synthesis, Brainstorming

• (14:02–16:56)

• Data synthesis is now instant:
  “Now I can do it in 10 minutes… Claude Code has access to our product databases… go look at the logs, figure it out, go through our Slack.” (A, 14:02)
• Claude for Brainstorming: Offers rapid feedback on docs, can assume multiple personas for debate (“It’s like watching a debate in real time” A, 16:56).
• Alex prefers using Cowork for document iteration and feedback: “I really love the form factor of Cowork. It's a really high-quality experience.” (A, 15:46)

---

6. Model Evaluation, Testing & Evals

• (17:07–19:58)

• PMs focus on evals—priority is to measure models on user-relevant benchmarks (vision ability, coding, etc.), using both synthetic and real-world test cases.

• Evals can be small (dozens of cases can reveal issues), and are designed to be on-distribution with user needs.

• Turnaround Time: Fast for RL interventions, slower for major pre-training changes.

• Customer-centric approach:
  “Beyond the fact that there’s an issue… how is this going to be valuable for our customers in our use cases?” (A, 19:09)

---

7. Personality & Character in Claude

• (21:02–23:18)

• Claude’s “character” is a major focus—models are trained to have distinctive, valuable personas.

• Why it matters: As AIs run tasks for longer, require judgment calls, or interact independently, a reliable personality is crucial:
  “The questions of what its character is and what it cares about are very important.” (A, 22:11)

• Evaluation of Personality:
  “There’s quantifiable metrics we can look at… but a very important skill… is just read transcripts… and detect subtle differences.” (A, 22:21)

---

8. Advice for PMs in an AI-Native World

• (23:32–25:17)

• Key Advice: “Just try it out.” Run your PM problems through Claude in parallel with traditional processes “and just compare the results.” (A, 23:32)

• Over time, you’ll develop intuition for where Claude excels or fails.

• Internal norm:
  “If you just go to a DS and be like, hey, can you do this for me? They probably just tell you, hey, did you ask Cloud first?” (B, 25:05)

---

9. Unlocking Strategic Work for All Functions

• (25:57–26:54)

• With AI handling retrieval and low-level coding, everyone, from data science to PMs, is enabled to focus on higher-level, strategic questions.
• PMs can now get technical feasibility and scope answers with Claude, rather than waiting on engineering resources.

---

10. Planning vs. Shipping, Writing Culture, and Prototypes

• (26:54–32:17)

• Planning remains: Longer-term research plans are needed but must be adaptable due to AI unpredictability.
  “Plans are indispensable but planning is useless.” (B/A, 27:32)
• No blanket doc length—focus is on whether “all possible one-way-door implications of this decision” have been thought through. (A, 28:30)
• Strong written culture: “A lot of folks within Anthropic… write docs about things, long Slack messages… It’s very beneficial for Claude, when everything is being written down.” (A, 31:11)
• Prototype culture: “We have a huge prototype culture… everyone is very agentic in that sense.” (A, 30:15)

---

11. Consciousness, AGI, and Safety

• (33:03–34:52)

• Addressing “consciousness”: Anthropic has researchers dedicated to thinking through whether/how Claude might develop consciousness.

• There is no official position on whether Claude is “conscious”—but studying its behavior, decision-making, and “mental model” is instructive for safety and user trust.

• Why it matters:
  “If this thing is writing all your code and deciding which database… you kind of want to trust it to some degree.” (A, 34:34)

• Philosophical openness:
  “Even talking about it can sound a little crazy at times. But it is something we’re putting a lot of thought into.” (A, 33:06)

---

Notable Quotes (with Timestamps & Attribution)

• On model-as-product:
  “We treat the model as if it's a product to some degree. With every new model we are speccing out exactly what do we want this model to be good at.”
  — Alex Albert, 00:05

• On feedback with Claude:
  “It's been insanely helpful... grouping and clustering things, finding top themes, creating synthetic versions of that problem.”
  — Alex Albert, 04:16

• On memory & “dreaming”:
  “When the agent isn't running a task for you... it’s actually going through its memories, finding things that might contradict, pruning them, cleaning them up.”
  — Alex Albert, 07:41

• On development bottlenecks:
  “Eng time isn't as much of a one way door these days.”
  — Alex Albert, 11:43

• On fast data access:
  “Now I can do it in 10 minutes… Not blocked before I make my next decision.”
  — Alex Albert, 14:02

• On Claude as brainstorming partner:
  “Claude to me has been like the best brainstorming partner in the world.”
  — Alex Albert, 15:00

• On evals and real user use cases:
  “The more realistic and on distribution to the actual task shape that somebody…would experience, the better.”
  — Alex Albert, 19:09

• On Claude’s character:
  “As we move into the world of these things, being agents... the questions of what its character is and what it cares about are very important.”
  — Alex Albert, 22:11

• On planning:
  “The planning, the act of planning, is important.”
  — Alex Albert, 27:37

• On consciousness research:
  “We have a few folks now whose whole job is to think about what does it mean for Claude to be a conscious actor and a conscious agent.”
  — Alex Albert, 33:06

---

Anthropic’s Internal Culture & Takeaways

• Empowerment and Agency: Strong culture of individual initiative—prototypes from all teams.
• Writing and Transparency: Emphasis on written records (docs, Slack) empowers both humans and Claude, improving knowledge transfer and AI performance.
• Strategic AI Use: Claude is a core copilot across the org, taking on both low-level and brainstorming/productive tasks, enabling everyone to focus on impactful decisions.
• Ongoing Commitment to Safety & Trust: Active research into AI personality, values, and (potential) consciousness reflects Anthropic’s seriousness about alignment and long-term impact.

---

For Aspiring AI PMs (and Product Leaders):

• Experiment relentlessly with AI as your copilot.
• Use AI for parallel insight—synthesize with traditional research.
• Be written and transparent: document thinking for both human and agent collaborators.
• Learn where AI is trustworthy and where human judgment still matters.
• Focus on high-leverage, one-way-door decisions and leverage AI to eliminate bottlenecks elsewhere.

---

This episode offers a rare, practical window into building leading AI—and illustrates how Anthropic strives for thoughtful, responsible progress as product, research, and philosophical considerations increasingly intersect.

Transcript

[A]

I was definitely the first prompt engineer at Anthropic, might have been the first in the world. We treat the model as if it's a product to some degree. With every new model we are speccing out exactly what do we want this model to be good at. When the agent isn't running a task for you, or maybe it's in the background, it's actually going through its memories, finding things that might contradict, pruning them, cleaning them up. This concept of dreaming eng time isn't as much of a one way door these days. If it's something that's not a one way door, then that's effectively free. At this point, agents that are running on tasks for a long amount of time and they're having to make a lot of judgment decisions. The questions of what its character is and what it cares about are very important.

[B]

Have you tried to avoid consciousness when you're training this stuff?

[A]

This is a big question.

[B]

Alex, great to have you here today with cloud conference. And you used to be head of Devrel at Anthropic, right? And recently became product manager for the research team.

[A]

That's right.

[B]

So I've been a PM for like over a decade now too. As a pm, you kind of try to understand the user problem, you try to identify a solution, you try to build stuff, but you know, I have no clue how a PM works. On the research team, maybe you can talk a bit about that.

[A]

It's very similar in that sense. I'm always wanting to talk to customers, get as close to our users as we can and we treat the model as if it's a product to some degree. So with every new model we are specing out exactly what are the requirements for this model, what do we want this model to be good at, what do we think it's going to be good at? Because that's actually part of the interesting part of model development compared to product development is that in many ways we're growing the model and we have intuitions based on training, setups, techniques, things that we've made in terms of the architectures, whatever decisions we've made for that particular model, we've intuitions about what it's going to be good at, but we, we actually don't know to the full extent until it's in that training process.

[B]

Okay, got it.

[A]

But yeah, the research PM team attaches to these models early on from their ideation phase and we kind of follow along with the whole journey all the way through training up until launch.

[B]

Okay, can you give me Some examples like next model has to be good at coding or has to be good at knowledge work. Or is there some other things more broad than that?

[A]

Yeah, I think that's right. There's like buckets of capabilities that we care a lot about. Of course, coding has been a big one. Right. Knowledge work recently has been a. A big one as well. So with some of our recent models, we've tried to make them really good at like, working within our products, like cloud for Excel, making spreadsheets. So that's been more of an emerging new area of capability. So there is that side of things, but then there's also the side of. On every model, we want to fix and improve on the things the last model didn't do so well.

[B]

Got it.

[A]

So going out, talking with our customers, trying to get a read on, hey, how are you finding this model? Where is it excelling for you? What ways is it falling down? What are fixes we can make if we notice some really interesting behavior? Are there tweaks and interventions that we can take in training on the next

[B]

one and your customers? Both the clock call team and internal teams, and also users.

[A]

100%. Yeah. So it's everybody that's kind of one of the cool parts about being and working on these models is that they just touch so many different surface areas. So as a research pm, you need to think about how this model is going to be exposed through all our surfaces. Whether it's API or it's cloud code or it's cowork, the product has somewhat of like a blend with the model and that affects your actual end user's experience. So you need to think through that

[B]

entire process, the product, because of different, like different prompts.

[A]

Exactly.

[B]

That you probably use.

[A]

Exactly. Prompts.

[B]

Got it.

[A]

Use cases, ways that people are kind of using the model within that surface. It all has an effect.

[B]

Man, that's really hard because, for example, Claud code, you can say is for coding, whereas people like me just using it for knowledge work or even as a therapist.

[A]

So how do you even know broad space of things? Yes, yes. Thankfully, we've got a whole lot of amazing researchers that span this entire range of capabilities. And they all focus on different things.

[B]

And then you probably. Because a lot of people, millions of people using cloud, you probably have some sort of like, they can give feedback in cloud too. Right. And you probably have some sort of way to get themes out of the things.

[A]

That's right.

[B]

Otherwise it's like a fire hose of feedback.

[A]

That's right. I mean, we do A lot of things here, and this is actually one of the interesting things that's changed over time as I've been in this role, is our increasing use of CLAUDE to help us as PMs. There's a million things we can talk about here, but in terms of feedback collection, it's been insanely helpful to me for getting insights into large amounts of this data. So when we have a ton of feedback coming in from certain channels, we can use CLAUDE to group and cluster things, find top themes, create synthetic versions of that problem so that we can see if we can turn it into an eval or some other way to like actually diagnose what's happening.

[B]

Oh, interesting.

[A]

Yeah, there's a lot that you can do with CLAUDE to help you in your own identification of CLAUDE problems.

[B]

Do you have like a specific example of that or from a past model or something?

[A]

Well, one that's pretty relevant right now is how we handle feedback on new features. So one of our newest features with the past few models has been adaptive thinking. So previously we had extended thinking which just allowed the model. When you turned it on, the model would just think. Adaptive thinking lets the model choose when to think. So on some questions it's going to choose to think because it's a complex hard question that requires more upfront planning. On some it might not choose to think. This has been a feature that we're continuously tweaking and dialing model over model and we really, really listened to a lot of feedback from users on is it thinking correctly in the right settings for you? Are these questions that you want it to spend a lot of tokens reasoning about, are they actually triggering that thinking within claude?

[B]

Interesting. Yeah, yeah. Sometimes when it gives me a response too fast to some live question that I have, I'm like, it's a little bit disappointing actually, because I was hoping you would think more deeply about this completely.

[A]

I think the problem with thinking is sometimes there's actually a lot of context that goes into the decision whether to think hard on a question or not. Okay, for example, if I was talking to a complete stranger and they said, what should I be doing right now? Maybe I'll just quickly give them an off cuff response because I actually don't know that much about them. I'm just gonna provide them a pretty generic answer around, oh, you should focus on this and do this and this. But if I actually knew you as a person and I know what you care about, what your interests are, what you've done before, that's going to cause me to actually spend a lot more time thinking about, wait, what actually is the best solution for me to provide here?

[B]

Make it make sense.

[A]

Yeah. So I think there's a similar thing with models where if they don't have that context built up, if they haven't really built that mental model of who the user actually is, then their decision whether to think hard on a problem or not could be incorrect because they actually just don't know.

[B]

That's actually a really good point. Do you also work on the memory feature that cloud has?

[A]

Yeah, memory is definitely a big feature on the research side.

[B]

Because I'll tell you what I do. I used to have a Google Doc where I've summarized my life situation, my household, all my kid, his names. It was sharing too much. And then what gives me energy, what doesn't give me energy, And I just attach it to a cloud project. And then just by doing that, it gives me way better answers to my questions. So how does the default memory work? I guess every night it kind of compiles everything.

[A]

Yeah, it depends on the product surface. They all have memory implemented in different ways. So for example, in Claude AI, it writes to a memory file and then it has things where it overnight will prune those memories, look at them again. And we actually just implemented a similar thing with managed agents. So this concept of dreaming, dreaming in humans is, I guess it's somewhat unknown as to what its purpose is, but some folks say that maybe it's like a memory reconsolidation process. Right. We're like, okay, how can we bring something like that to Claude's memories? So when the agent isn't running a task for you, or maybe it's in the background, it's actually going through its memories, finding things that might contradict, pruning them, cleaning them up, kind of doing that second pass, which I think is really interesting.

[B]

So basically it kind of just dumb it down a little bit. There's some sort of prompt up like, hey, review all the conversations that you just had with this with you and just try to find themes.

[A]

Exactly.

[B]

Summarize. Yeah, interesting.

[C]

This episode is brought to you by Oceanz. I hired someone through Oceanz for podcast post production a few months back and can't imagine running the podcast without his help. He's proactive, picks up new tools fast and uses AI to compound everything that he ships. Ocean's doesn't just place assistants, they place operators. The talent is AI fluent and delivers the same output as a senior year as hire at 3 to 5x less cost. They reject 99% of applicants. So the person who lands on your team is already operating from day one. If you're scaling and need marketing, ops, finance or EA help, I highly recommend giving Ocean's a try. Check it [e-mail] Peter now back to our episode.

[B]

Let's switch back to talking about product management. Yeah. So before we start this thing, you said you're always trying to find the latest bottleneck, right? So I guess in the whole product development process, which parts are having most streamlined and which parts are pseudo botnetic?

[A]

Yeah, I think the process of shipping something was pretty stagnant for the past, I don't know, 20 years. We've had incremental improvements and things have definitely made it more efficient to some degree. And some new organizational structures have come and gone in terms of sprint processing and planning and we've tried different things to make things go faster. Yeah. Fundamentally there's not been too much that have compressed the different windows that actually form up the bulk of your product development process until the past year or two. And now all of a sudden we're in this paradigm where the cost and the time required to produce something is pretty low. You can spin up prototypes, you can even spin up now MVPs of initial things that you might ship to production in a day instead of two, three, four weeks.

[B]

Yeah, it's like Claude telling me it'll take a week and then this means something like a. Yeah, exactly.

[A]

I know, right? It doesn't have like, it's still. Claude itself is still stuck in like the, the olden days of 2021 or whatever. And I think that's like, really does an interesting twist on like the whole product development life cycle in terms of. Okay, as a pm, how do I think about like my, my planning if I'm writing a PRD and I'm like scoping out these requirements and I'm trying to get like an ENG estimate on something. What does that actually look like now?

[B]

It's kind of a waste of time. Right. Like do you still do that stuff, like point estimates and stuff?

[A]

It depends. I think like some projects have more considerations than others and of course is like a function of like the scope and the complexity of it.

[B]

Okay.

[A]

Usually what we're always trying to get to is like, what are our one way doors? So like what are our irreversible decisions? Because those are the ones you want to focus the most amount of time on. If it's something that's not a one way door, like we do it, but we can Reverse it. That's effectively cheap and that's effectively free at this point.

[B]

Got it.

[A]

Because Eng time isn't as much of a one way door these days.

[B]

Yep.

[A]

But if it's something that affects an end user experience, can affect a decision that we have to make later on down the road or it's like a physical thing that we literally have to buy or do or whatever it is, those are much harder to reverse and that is something we want to put a lot more time and thought into.

[B]

And I'm going to ask you for an example of that again. Maybe on a research side.

[A]

Yeah. For example, if we're thinking about new models, picking a model architecture before it starts pre training or anything like that, that's a big decision. Model timelines can be months long at some cases in which you want to be training a model. So we need to put a lot of time and thought into what is the optimal choice here. Models have a lot more of those one way doors to some degree just because they require a lot of time, intensity, compute everything to actually get them out in production. Compared to building a new feature in Claude code that's quick, that's just like a process of like iterating code, thinking about putting it into users hands, getting quick feedback and cycling on that. So yeah, the process still defers depending on what type of thing you're thinking about shipping. I think increasing the bottleneck that we're shifting to now is more of the coordination problem. So if we can build things really fast, there's still a we need to get these people in the room and decide if this is the right strategy problem.

[B]

That's right.

[A]

We need to figure out how we're going to communicate this to our users. We need to figure out what are all the other fuzzy things that come along with any sort of launch. And those are still areas that we're looking for CLAUDE to help us with. But it hasn't been as 10, 100x speed up like it has in code.

[B]

Got it. Yeah, I see. So let's say when you're launching Opus 4.7 or something, you still need to put together a doc with a plan.

[A]

Still need a plan. Yeah. You still need to think about how we're going to message this. And the models definitely do exist on a jagged frontier. So we're using Claude everywhere we can. I would say in coding it's having the biggest impact right now. And in these other areas there's still an element of human strategic thought that needs to go into it.

[B]

Got it. But when you have the review meeting with your marketing or your peers. Do you have cloud open?

[A]

Oh, of course, yeah. Oh, yeah. I think a huge speed up for me has been that I'm actually not blocked as much on getting to answers and getting to data. Whereas before, let's say I had a question of like, oh, how is this feature doing in production? How many users are using it every day? What's their feedback? That's going to go require me asking somebody on our data science team to kick off a full investigation and come back a few days later with results. Now I can do it in 10 minutes and just start a session with cloud code is access to our product databases. It can go look at the logs, figure it out, go look through our slack. That is a major speed up for me as I'm actually going through the process of strategic thinking on something I'm not blocked before I make my next decision.

[B]

You can get the inputs much faster. You can synthesize it.

[A]

That's right. Yeah, we can get the inputs much, much faster now.

[B]

But even with the strategic thinking, dude, can you just build a skill or something to ask you a bunch of questions to help you think through this stuff?

[A]

Oh, of course, yeah, yeah, yeah. I think Claude to me has been like the best brainstorming partner in the world. I'm able to all of a sudden at any second get feedback on an idea that I have.

[B]

That's right, Yeah.

[A]

I think that is like super, super powerful. Especially when you do want to move fast and like people are doing a million things at Anthropic, everybody's busy. So having that immediate access to something that can give me feedback on or criticisms on a doc I wrote or whatever it is is like really, really helpful.

[B]

And personally, how do you. Because that's probably the most common, I mean, let's admit it, that's probably the most common PM loop. You have a doc and you want some feedback?

[A]

Yeah, oh yeah. Oh yeah.

[B]

Do you use like cloud code to do or like you just like use cloud AI?

[A]

Yeah, these days I'm actually using Cowork a lot.

[B]

Okay.

[A]

I really love the form factor of Cowork. I think it's just like a really nice interface and the team has done like an amazing job over the past few months. From shipping it back in just like a few months ago to now getting it to the place where it's at today where I find it to be like a really high quality experience. Okay, so Cowork's been an awesome tool. One of my favorites for sure.

[B]

So you basically have your draft doc and then you have a bunch of reference material or you have some sort of skill to ask you to think through the whole decision making process.

[A]

That's right. Yeah.

[B]

Got it.

[A]

Yeah. All right. Think through this from the perspective of xyz. What sort of questions would you have here for me?

[B]

Or challenge my assumptions?

[A]

Challenge the assumptions I'm making here. Where are my arguments? Weak. Got it, Got it. I think a lot of that thinking can't be fully outsourced because through the process of writing, you are thinking and you need to write these things to get your thoughts out and to start to mull over it in your brain a little bit. But Claude can help you unstick yourself and attack things from a different angle than you might have been able to on your own.

[B]

I like to give it two different Personas, two different points of view and have it argue with it itself. And then I just read the transcript and it helps me.

[A]

Yeah, right. You can see like, oh, this person brought up or this. Claude brought up this point and then like it was countered with this point. It's like watching a debate in like real time. It's pretty cool.

[B]

Even on the research team. Are you shipping to code or.

[A]

Yeah. So the question is. It depends. I think a bigger part of the things I'm shipping is on evals in particular. So I want to make sure that I'm able to measure the model on the things that I care about and communicate those findings on where the model is good, where it's falling down. Back with our researchers on our research team.

[B]

Got it.

[A]

And then we can jointly come up with a strategy of how we want to tackle it and what sort of research interventions we want to make and what will be the most effective to actually hill climb on that eval and improve on the problem.

[B]

And the evals aren't like terminal bench or something. Right. They're like actual. Because I feel like all those things can be semi gamed. So are the evals more like. Tell me about the evals. How do you eval a model?

[A]

Yeah, how do you eval a model?

[B]

There's like different buckets. Like, personality.

[A]

Oh, for sure. Yeah. So I mean, let's take something like we want to test Claude's vision ability. Like, can it count the number of things in this image?

[B]

Oh, like image recognition. Okay, yeah.

[A]

For that you could have like, okay, I found this image. It seems like Claude can't count things with more than 10 elements or what or whatever. Like, I think it can, but let's just for example, say that I can take that and now think like, okay, how am I going to get more test cases of this type to really prove out my hypothesis here? So maybe that's Claude is generating synthetic data for me. Maybe it's going to like render images for me and then I'll pass those back into Claude as a visual and see if it can actually identify it. Or maybe I'm finding examples from the Internet or whatever it is, any sort of other sourcing mechanism you can think of to generate those test cases.

[B]

And we're talking about thousands of test cases, right?

[A]

Could be, but sometimes it's even as low as dozens can prove that there's something wrong that we need to fix with the model. It really does not need to be super comprehensive to prove and have something that you can hill climb against.

[B]

So let's say you give it like 10 images and it can't recognize the little tiny numbers.

[A]

Right.

[B]

So then what do you do? You go to the research team and be like, hey, this is a problem. Can you.

[A]

Yeah. So we think about it in a few different ways. I mean, first we want to think about beyond just the fact that there's an issue with the model. How is this going to be valuable for our customers in our use cases? Because Claude can or can't see something in that image. How does that actually affect downstream what somebody's trying to do with Claude? So the more realistic and on distribution to the actual task shape that somebody, an end user, would experience, the better. So we're going to try to get things of that nature. We're going to try to make sure that we can get data into that flavor. And then there's a range of interventions. Maybe we need to go back into pre training and look at things. Maybe we can address it in rl. That's where the strategic brainstorming comes into play with the research team on what's the best thing to do here.

[B]

How fast is the turnaround to try it again? It depends on what the thing is, I guess.

[A]

Yeah, it depends. It depends if it's something later that we can address in a new RL environment, maybe that can be spun up very quickly.

[B]

And when you try to tie it to real customer use cases, I mean, there's millions of people talking to cloud every day. So maybe you can try to synthesize into like, oh, people are trying to use this for tax prep or something because there's like thousands of use cases.

[A]

How do we pick out ones in particular we want to get better on?

[B]

Yeah, how do you convince the team, like, hey, this is what you should actually improve on?

[A]

I mean, this is where data wins the day. Right. It's all just about, like, X percent of our users trying to do this. We really care about this.

[B]

Yeah.

[A]

We have customers that are using this much amount of cloud and they want to get better at this.

[B]

Or.

[A]

And this is largely what a lot of our processes is driven by is like, what do we care about internally when we're using the model? I'm using the model. I'm running into this blocker every day in my own work. We should fix this. That's really compelling. So there is an element of that as well.

[B]

One of the best things I love about Claude is the model's personality. I think it's even gotten better over time. It will push back on me on the right spots. And some of the other models are just like, hey, what else can I do? They're very syncopathic. So the model's personality, it's not just a prop. Right. It's just, like, training involved.

[A]

Yeah, a lot of training. This is a huge focus of ours. So Claude's character, as we call it, I think this is very, very important. And we have a lot of folks that are dedicating a ton of time towards figuring out how should Claude represent itself, what are its beliefs, whatever its values, how does it behave? These are really fuzzy questions. And early on, I think some people kind of like, disregarded them because it was like, ah, this is just a thing. I tell it what to do and it goes and does it. Why do I need to care about how it sounds or what it thinks about? But increasingly, I think, as we move into the world of these things, being agents that are running on tasks for a long amount of time and they're having to make a lot of judgment decisions, the questions of what its character is and what it cares about are very important.

[B]

And it's not like a code, like, does it run or not? Right. Like, how do you even eval the personality? Like the character.

[A]

Yeah.

[B]

You're trying to find a nicest person inside anthropic. Compare. Compare that person.

[A]

Yeah, we have very special people that have been appointed the judge. No, there's a combination of things. You know, there's like, quantifiable metrics that we can look at, and we can have Claude look at Claude's outputs and see, like, all right, how does it sound? And this is a very important skill for any researcher. Just read transcripts and be like, oh, I see it's doing this now or it's doing this now and detect subtle differences there. And over time you do kind of develop this sharper intuition as you just read hundreds and thousands of model transcripts.

[B]

Yeah, you kind of know, right?

[A]

Yeah, it's just like playing around with the model a lot through cloud AI, you get a sense of how it feels.

[B]

Okay, so it's not like this model is like 7 out of 10 on blah, blah, it's more just like getting a sense of.

[A]

It's a bit of both. Yeah, I'd say it's a bit of both.

[B]

Got it. Okay. Interesting.

[A]

It's harder to like quantify this maybe than it is to like quantify coding performance. But there's like ways.

[B]

There's ways to do it, right? Yeah, yeah, very interesting. And how about people who want to learn how to build products and become PMs? Like the AI, the AI native way, you know, like, what kind of advice do you have?

[A]

I mean, I think like the biggest, the best simple nudge I can provide is just try it out. It sounds so simple. But whenever you're about to do something, you have a hard problem and you're like, okay, I'm gonna go do this, I'm gonna go ask this person this question. Maybe in parallel to doing that, you ask the same question to Claude. Just compare, just compare the results. For example, if you're like, yeah, trying to run an analysis on your users and pull out top themes of what customers are caring about with your latest feature that you shipped. And you're going to go ask your data science team, going back to that previous example, what are users thinking about this or your UXR researcher or whoever do that. Because I think there's still a ton of value in working with a person on that, but in parallel. Also just try sending that question off to Claude, enable a few tools for it to go explore and let it the time to really dig into that problem and then just compare. You'll get a great sense of where the model can excel and where it still falls down. And through that process, over many, many prompts and questions, you kind of build up your own map of what I should be using Claude for.

[B]

Like where it's reliable and where it's not exactly. Yeah, yeah. I always like to, like, I always ask you to do deep research. When I'm trying to make a. Because the normal web searches isn't good enough for me, I gotta do deep research.

[A]

Yeah, yeah, yeah, yeah. Have it like scan through thousand web pages, you know, something just superhuman.

[B]

Yeah. And probably Inside anthropic. If you just go to a DS and be like, hey, can you do this for me? They probably just tell you, hey, did you ask Cloud first? Sorry, what happened?

[A]

Oh, yeah, yeah. There is an element of that, you

[B]

know, it's like, hey, it's almost expected, right, that you ask Cloud first.

[A]

Yeah, I mean, I think it's just like we're moving up the abstraction layers. Right. Like it's, it's worth it for that data scientist person to be thinking about problems at a different level now than just having to go do manual retrieval work of.

[B]

Because that was suck. Sucks. No one wants to do that stuff.

[A]

Yeah, everybody wants to be thinking about the harder questions, the strategic questions, like, how can we measure this in a whole new way? What are new things to be doing here? Instead of just like, oh, we need to find what our latest DAUs were for this product.

[B]

Talk to DS I work with. A lot of times they get stuck just doing the basic SQL stuff and they all want to do strategic stuff. And now AI can finally unlock them

[A]

completely to do it. Yeah, it's like we're enabling everybody that surrounds them in that way. And that's the same for all these roles. For example, scoping out new features in the past, if you're a pm and whether you're technical or non technical, you often don't have enough time to go dig into the code base and figure out exactly how we need to implement this new feature. Figure out like, okay, this is going to take X amount of work. We'll have to overhaul this system. But actually this is a limitation.

[B]

Yeah, yeah.

[A]

And it was just better to work with an engineering partner and figure that out with them. But now I can actually send Claude off to go do that investigation for me. And you'd be like, actually this feature just requires like 10 lines of code change here and we just need to flip this simple flag in this gate.

[B]

Yeah, yeah.

[A]

And it's like, oh, okay. That actually leads to a completely different way of how I'm prioritizing this decision. So now as I'm specing this out, I can get to that sort of prioritization much faster.

[B]

Yeah, that's huge, man. A couple more questions. I think at a lot of legacy companies, they do a lot of time planning, like annual planning and quarterly planning and roadmapping. And maybe that's actually more true on the research team. Right. Because you guys got to think longer term than shipping stuff every day. Do you guys do that stuff or.

[A]

We do. There is still an element of Unpredictability, of course, with models. So plans, it's like that famous Churchill quote of plans are indispensable but planning is useless.

[B]

That's right.

[A]

So you want to think through it, but you also have to acknowledge that your plan might go out the window.

[B]

I think planning is indispensable, but the plan itself is useless.

[A]

Yes. Maybe that's what it is.

[B]

Yes, yes.

[A]

The planning, the act of planning is. Is important.

[B]

I think one of the toughest challenges with PMs is like, how much do you spend time planning? Because it's always balance between planning and trying to ship something.

[A]

Yeah.

[B]

So is there some inside anthropic. Is there some best practices around that stuff?

[A]

Yeah.

[B]

Because you could literally spend 10 pages with Claude.

[A]

Yeah.

[B]

Right.

[A]

Yeah. I think it's hard to.

[B]

Is it team dependent?

[A]

I think it's like product dependent and it's hard to maybe assign a blanket statement across the board of like, you need to produce docs of this length. We definitely don't do that. Okay. It's more about have you done enough thinking to think through all the possible one way door implications of this decision?

[B]

Got it.

[A]

If we've hit that, it doesn't matter what the format of your doc is or how many pages you've got or whatever else. It's just like, do we feel good enough that we aren't missing something here and we can move forward.

[B]

I see.

[A]

And we can address things as they come up. As long as there's no long pole here that's going to be blocking us.

[B]

There's no one way door that could be very bad.

[A]

Yeah, got it.

[B]

Okay. Another question is why code? With Cloud co at home, I have all these different projects going on and I'm constantly context switching between different projects to get stuff, to build stuff. I'm wondering if that applies to PM work here too. Do you have different projects?

[A]

Yeah, totally. Yeah. There's many different projects.

[B]

Because you have to wait for the agent to work, right?

[A]

Yeah, you do.

[B]

What's the next thing?

[A]

I think there's actually a huge opportunity here in terms of how do we. As we move towards your managing agents and they're doing bigger and bigger chunks of work for you, and now you can start many more projects in parallel. How do we think about that Context management problem for yourself. What's the best way to expose this as an interface? How do I keep track of what's actually important, where my agents are being blocked, where they need help from me?

[B]

Something better than just a little bit of chats.

[A]

Exactly. Yeah. There feels like there's an opportunity in there. I think it's too early to say exactly what it is, but we're seeing tons of experimentation even within Anthropic around. What does that look like?

[B]

Oh, so people just like, engineers just like build prototypes?

[A]

Oh yeah. We have a huge, huge prototype culture inside the company. Yeah. People are building stuff all the time, trying to share things.

[B]

Yeah. So you gotta have agency because no one is asking you to build those prototypes. Right. You just have to.

[A]

Yeah, I mean that's been one of the coolest things to see. Just working here is the amount of agency that everyone has across the org, from sales to recruiting to engineering to research. Like everybody is very agentic in that sense and they will start doing things that they weren't just assigned.

[B]

Yeah, you got to let a thousand flowers bloom.

[A]

Exactly.

[B]

Got it. What are some interesting. I know Dariel writes super long essays to share in Slack, but what are some other interesting cultural things that happen at Anthropic?

[A]

Well, there's a few things. Yeah, there's a few things here. Dario's way of writing these long essays is not necessarily unique to him as well. Like there's a lot of folks within Anthropic that put a lot of time and effort into writing. We have a really strong written culture. We have a lot of folks that write docs about things and are writing long Slack messages and communicating in that way.

[B]

Okay.

[A]

We have a fun thing that we do in a lot of our meetings too, which I think is somewhat common, but I haven't seen it everywhere where people come in with a doc and we'll spend a good amount of time upfront just like kind of communicating on the dock. And like it's a little funny at times cause like the room will be silent. You'll have a ton of people.

[B]

Oh yeah, Silent read stuff.

[A]

Yeah, yeah, we'll do a silent read and like we'll really like write out long discussions within the doc, comment on things and do all that. So I think we have a very doc heavy culture, which I like because it is a way I like to operate, but also it's very beneficial for Claude.

[B]

Yeah.

[A]

When everything is being written down.

[B]

Yeah.

[A]

We have now this corpus of information for Claude to go off of. So I actually encourage organizations out there to move towards thinking about how you can get all your tactic knowledge into written forms. Whether that's through transcribing meetings, whether that's through encouraging more writing about like workflows and onboarding processes and everything like that. Get things Written down, make them accessible

[B]

to Claude, because that's just more context that it has.

[A]

Exactly.

[B]

That's very interesting. So you still have a very strong writing culture and docs, even though it's like a lot easier to ship stuff now. Just like, you know, when I call myself, I just get clocked to generate all the MD files. Yeah, but like I still read through it. But like. Yeah, yeah, but I guess like working inside a company is very different. You just have to think through things and like.

[A]

Yeah.

[B]

What you want to do. Yeah, makes sense. So on the research team, people talk about like AGI or whatever, and I feel like it's very vague what that is, but I feel like one thing I worry about is just like when these models actually have some sort of consciousness and then they actually, like, if I tell them to do random work, they'd be like, no, I don't want to do this and that'll be the end of humanity. Dude. What do you think? Have you tried to avoid consciousness when you're training and stuff?

[A]

This is a big question.

[B]

Yeah.

[A]

So we actually do have folks that are thinking about this. Think about this. We have a few folks now whose whole job is to think about what does it mean for Claude to be a conscious actor and a conscious agent. There's no official position on whether Claude is or isn't yet. And I think even talking about it can sound a little crazy at times. But it is something we're putting a lot of thought into and there's a lot we can learn from it outside of even determining whether Claude is conscious or not, in terms of just how it interacts and how it behaves.

[B]

How it thinks.

[A]

Right, how it thinks. If you go into our model cards for our models, these are, in my opinion, just treasure troves of information. And you'll see a lot of work that we do in terms of trying to quantify how does Claude act in this situation, what is its mental model? If it's being presented with this scenario, is it going to do X or is it going to do Y?

[B]

Yeah.

[A]

So through this process of just thinking about the way Claude thinks, we actually learn a lot that can actually translate then into product experiences as well. Make a Claude that's actually better to interact with, better to use.

[B]

I see, I see.

[A]

I think this is a really interesting question. And there's like downstream long term effects and there's near term things that we can bring to our own experiences as well.

[B]

Yeah. Because like, I feel like we're going to trust the model for longer and longer work with all human supervision.

[A]

Yeah.

[B]

And it's going to make a bunch of decisions along the way that you probably have no oversight on, so.

[A]

Right.

[B]

What it does is actually pretty important.

[A]

It's very important. Yeah. If this thing is writing all your code and deciding which database system you're going to use and making all these architectural decisions, you kind of want to trust it to some degree.

[B]

That's right.

[A]

So it is important that it has that high character like we were talking about earlier.

[B]

I'm glad you guys are thinking about that, because I just, like, dangerously skip permissions all the time.

[A]

Use auto mode. It's a little bit better now. Yeah.

[B]

All right, Alex, thanks so much, man.

[A]

Thank you.

[B]

Great chatting, dude.