← The Growth Podcast
The Growth Podcast
How PMs Ship 100K Lines of Code at OpenAI with Ryan Lopopolo, Member of Technical Staff
01:14:45
Loading summary
Transcript51 lines
- [00:00]
A
One of the craziest statements I heard from you was code is a liability. What does that mean exactly?
- [00:06]
B
Yeah, there's a bunch of different ways that this expresses, but primarily, all of our human software engineering organizations have been tooled around this idea that code is the most expensive thing to produce. You have synchronous limited throughput of your human engineer operators on the keyboard.
- [00:23]
A
Ryan Lopopolo led the team building it. He works on the OpenAI Frontier team and today he is letting you inside how you can transform your code base, transform how you work to operate at the frontier. You've said that you're not using a billion tokens a day. You're basically negligent.
- [00:39]
B
The more tokens that you're running through these models, the smarter they get, the more intelligence you are able to extract from them. This is sort of like the realization behind why we have reasoning models in the first place, right? If we give the model more effort to do inference, we're actually able to get greater results out of this.
- [00:56]
A
The engineer owns the code, the designer owns how it looks. The PM owns the problems we're trying to solve. What do the new lines look like in a new epd.org? Before we get into today's episode, I wanted to share that you can get a free year of my favorite AI tools, including Bolt, new Maubin, Arise, RelayApp, Dovetail, Linear Magic Patterns, Reforge, Build, Descript, and Speechify. If you join my bundle@bundle.akashg.com on top of that, I wanted to quickly ask you to please double check that you are subscribed on YouTube, Apple and Spotify podcasts. It's a free thing you can do that really helps support the show. And now into today's episode. Every time I talk to people from OpenAI, I get the feeling that they are operating how the rest of us are going to be operating in two years. Whether that's PMs and designers actually having access to the code base. How many of us still don't have access to the code base, right? Or actually building the right scaffolding so that PMs and designers can feel confident operating in the code base. That is the next level. And today we have on one of the people who gave the best talks on harness engineering that I have ever seen, Ryan Lopoplo. He leads one of the teams at OpenAI on the frontier, OpenAI frontier. And today he's going to break down how you can go from the old way of working to the new way of working. Ryan, thanks so much for being on the podcast.
- [02:35]
B
Thanks, Akash, for having me. I'm super excited to talk a little bit about today around what it means to enter this new wave of working where we can empower the machine to do much of the work for us, which frees up the humans in our organizations and teams to focus on the things that the true value that they bring to the table. Right? Scheduling the work, bringing their taste and expertise, and making sure that we're shipping actual delightful, high quality products to the world that solve real user needs. Right? That's the job. And it's been fun to be able to experiment with how we can do that more effectively and at higher velocity.
- [03:10]
A
I'm so glad you said that, because I think there's sort of a definitional phase we need to help people with. At the front, one of the craziest statements I heard from you was code is a liability. What does that mean exactly? And what has been recently enabled with the latest versions of GPT 5 and 5.5 that allows you to say that,
- [03:33]
B
yeah, there's a bunch of different ways that this expresses, but primarily all of our human software engineering organizations have been tooled around this idea that code is the most expensive thing to produce. You have synchronous limited throughput of your human engineer operators on the keyboard, right? So there has been a high sense of like, protectedness around that code. It is expensive to produce, it is expensive to validate, it is expensive to deploy. So these things have been treated as requiring the bulk of the organizational apparatus to manage in this world we are today, where the token prediction machine, these lovely, highly advanced models that we have, Codex, is fantastic at taking all the best parts of GPT5 and putting those codes and words into the world. The code is trivial to generate. It can be generated at arbitrarily parallelism, depending on how confident you are that the code is able to solve those problems, which we will get into in a little bit more detail, right? But you kind of have to change the way you think about scheduling work and exploring when you no longer have what was the overriding constraint in your teams. And because of this, you can sort of remove humans from the loop almost entirely on that production aspect of the code. And then you have a whole host of other problems on what it means to ensure quality. Right? You have a less tight feedback loop for those human engineers, right? Because they're not physically typing on the keyboard, they can't attest to every line of code as it is produced. But this is sort of the same spot that the rest of the organization has traditionally been in. Right. Like there is an indirection from my designer putting a mock in Figma to me actually realizing that in HTML, SVG and CSS there's also this lossy translation between doing customer interviews, going to a PRD and turning that into code and deployed artifacts. So it's kind of nice that we have this equalizing force here where we can potentially end up in this world where every member of these software delivery teams kind of is on equal footing in terms of what it means to actually produce that executable artifact which we're delivering.
- [05:56]
A
When everybody's on equal footing though, what even is the purpose of roles? Traditionally we've had this triangle. Product manager, designer, engineering. We've been able to cling to what we own. The engineer owns the code, the designer owns how it looks, the PM owns the problems we're trying to solve. What do the new lines look like in a new epd. Org that's built to take advantage of the latest this technology is offering?
- [06:24]
B
Now, this is really interesting. It's a question I've gotten a lot actually from executives and VP Director shape folks that I'm talking to here. And everyone comes to this question with a slightly different set of priors, basically asking which roles do I need to hire more of or less of sort of thing. But I think that's the wrong way to answer this question. Every one of these individuals on your team brings their unique lens of viewing the world. And these coding agents have all the best parts and all the worst parts of how to build software in them. So having a diverse team with a bunch of different perspectives on what successful delivery looks like means that you're actually able to take the best parts of everybody's worldview and compile them into your single code producing agent to do the full job. This is sort of what it means, I think for GPT5 series of models to be like fully agentic. They're able to do the full job. It's not just fancy autocomplete, it's not pair programming. It's being able to go from understanding of the user challenges, understanding the actual code base. We have the design system that we're building on top of what our typography primitives are, how we actually are able to deploy these vehicles in order to get them into the hands of users, what it means to observe whether or not they are effective, and then feeding bug reports and user feedback back into the work. This is like a fully closed loop that I would traditionally expect a team of four Five, six, seven, to have to do. But if instead all of these different people with diverse perspectives are empowering the same agent, that one agent can kind of like with full context, do the full job.
- [08:04]
A
So if you were to frame what the old and the new roles are, how would you do that?
- [08:11]
B
So sort of in the old way of working, you've got your engineering team who is responsible for compiling the user visible artifacts that we want to see into code that is then deployed into the world. And to do that, because code has been very expensive to produce and loops have been long on feedback, this has been something that has been bounded by throughput because the human engineers need to spend a lot of synchronous time and attention maintaining the quality bar, maintaining the long term viability of the code base as a long lived artifact sort of thing. And because their time has been largely focused on the production of code within this system, there has been, at least in my career, comparatively little effort around empowering others to do that same sort of work. This is why very often in organizations you see elbows out around not permitting folks outside of the team or outside of the engineering organization to contribute directly because the code that is produced is potentially very expensive to unwind or very expensive to lift to the quality bar versus having had your expert code authors producing it from scratch. This has led to each role on the team operating within their own lane with these very expensive coordination points and, and handoff points between folks. And because you have these sort of bottlenecks between handing off work, they're highly contended and low signal, there's lossiness there. It's very difficult for the designer to in the weeds observe how difficult it is to compile some of their mocks into code. Given the constraints of the design system that the engineering team is working with. And because the designer doesn't feel that pain directly, there can be sort of either resentment or missing feedback channels from the engineers to design. Which means like on the whole, the system is not working together as effectively as it could be. Sort of. When I think about, you know, collaborating with my infrastructure teams and developer productivity teams, we often want to shift left as early as possible, right? We want to feel the breakage in a system before it gets to customers, before it gets to a PR early in the developer loop because it's easier to correct them. Similarly, if it is the case that we are adopting product explorations that are difficult to prototype or experiment with because we don't have the right experimentation infrastructure here, right. As a hypothetical example, it may be difficult to push back on product priorities in favor of this sort of product platform work. But this collection of three roles as a whole would very much benefit from being able to pull that work above the line in order to inflect their throughput going forward and basically being able to in this new world get all three of these roles directly. Collaborating in the same code based artifact, feeling each other's challenges means there's much more empathy between them. Folks are able to help each other more often and in the happy case where everyone has contributed their expertise here we get higher throughput, higher velocity, more experimentation, more high quality UI and more product delivered into end user hands.
- [11:43]
A
I hope you're enjoying today's episode. Are you interested in becoming an AI Product Manager? Making hundreds of thousands of dollars more joining OpenAI and Anthropic then you might want to do a course that I've taken myself. The AI P certificate ran by OpenAI product leader Mikdad Jaffer. If you use my code and my link, you get a special discount on this course. It is a course that I highly recommend. We have done a lot of collaborations together on things like AI product strategy, so check out our newsletter articles if you want to see the quality of the type of thinking you'll get. One of my frequent collaborators, Pavel Hearn is the Build Labs leader. So you're going to live build an AI product with Pavel's first feedback if you take this AIPM certificate, so be sure to check that out. Be sure to use my code and my link in order to get a special discount. Here's the dirty secret about prototyping. You spend two weeks building a prototype. You validate your assumptions. Engineering loves the direction. Then what happens? You throw the whole thing away. Bolt changes this completely. When you prototype in Bolt, you're not building throwaway mock up. You're building real front end code that integrates with your existing design system. So when you hand it to engineering they don't throw it away, they they ship on top of what you've built. I use Bolt every single day. I host my land PM job cohort on it and honestly I'm up till 2am some days just vibing in the toll, having fun and building. That's when you know a product is good. When you're using it past midnight. Not because you need to, but because you want to. Check out Bold at Bolt New link in the show notes. That's what I want to see. How what are. What are the receipts? What actually happens? How can we quantify whether we're getting more Product velocity more throughput at quality.
- [13:22]
B
So I can talk a little bit about sort of the project that I worked on here, this experiment that I was running. And back in June of last year these coding models had just started to become viable, right? We were working with Codex Mini and internally the early versions of GPT5. We had just launched codec CLI and for historical happenstance and accident I set about to try and build an agent utilizing these capabilities. That was for non coding knowledge work. The thesis was that all agents were coding agents and we were going to prototype an internal agent that we were going to deploy at OpenAI. And in order to do that we started from zero empty code base and had Codex do the full software engineering job to produce this artifact. One interesting thing here from a project product development perspective is that having this process of codecs can only do the job. To produce something that was not an engineering native agent was super super useful because it forced us into delivering value to actual end customers that were not the builders. Which helped with prioritization and externalizing the learnings that we were due in this new product category to folks that could give feedback about the product, rather feedback about the process of developing the product. But in doing so we produce a repository with about a million lines of code. Funny story about producing agents. There's a lot of prompts embedded in agents, about 250,000 lines of markdown in this repository. Because when you're shipping an agent, what you're really shipping is a multi agent system. You have agents running around all over the place in what was an electron app for us, things that garden the individual git repositories that are used to bootstrap all the tasks that the agents are working on. Things for summarization, things for distilling skills, things for actually executing the work that we're doing on users behalf. Pretty, pretty cool way of building products. And an interesting way for non engineering users to collaborate. Because if many of the product surfaces are driven by agents, improving their behavior or changing their behavior is prompt engineering is tweaking text. And everybody knows how to tweak text. Everybody can tweak text and run the agent to get a vibe check to do an eval on whether or not the change was good. And that in and of itself is super, super empowering. But the important thing is not just that we produced an app with a million lines of code, it's that when doing so, literally zero of those lines of code were written by humans. And this was like Ryan as dictator, a constraint that I imposed on my team here, which I thought was very interesting because it forces the engineers right into the backseat of the Uber, right? You can't drive. The only thing you can do is like very politely give backseat driver directions. And that sort of mental model is I think, very useful because it's not great to yell at your Uber driver, right? Like you need to have them use the tools available to them in the car in order to get you to where you're going. So a big part of building in this way is making sure that the car, the gps, the test harness, the lints in the code base are all the knowledge corpus of all the features we've built, critical user journeys is there in the moment ready for the coding agent to drive. So when Codex was given a task and for whatever reason was not able to do the thing, the job of the engineers working on the code base was to basically stop, diagnose what went wrong, figure out creatively how to prevent the agent from making those mistakes going forward and try again. And every time you do that, you've taken feedback that normally you would have to give via synchronous steering and figured out a way for the repository itself to give that context to the agent. And this is why it was so easy for us to have non engineers contribute to the code base. Because every time the agent made a mistake, the engineers were forcing the agent to to learn how to self correct. Which means we basically did not have slop in the code base. It was very slow to get to that point. But you cannot have slop by simply not permitting the agent to write slop in the first place. And you do that with tests, review agents and other mechanisms that are LLM as judge driven to prevent that slop from being permitted to be committed in the first place.
- [18:14]
A
You mentioned it took a little while to get there. And I think elsewhere you also talked about how the first month was about 10x slower than it would have been doing it solo. Can you explain what did the first month and the evolution look like?
- [18:29]
B
Yeah. So the earliest versions of the product exploration, starting from zero, was to kind of have the agent triage pages on my behalf. Right. Going on call is a thing that I do. We're exploring what it means to have these agents do knowledge work and dealing with outages is a thing that I do. It's text heavy. I thought it was pretty amenable to having agents do this work. And from zero at the time without all these lovely apps we have in Codex today. If you said read my slack and figure out what inbound outages are happening. It's just not a thing the harness was able to do do. So it's obviously of course, much quicker for me to tab over to Slack and copy and paste into Codex, but that's not the goal, right? The goal is to get codecs to do the full job. So kind of had to stop say, okay, I can copy and paste, but I don't want to. What must I build first to give Codex access to Slack? And actually the thing I have to build first is like credential management in the harness. So Codex is able to request credentials on my behalf and in order to do that I need access to the keychain. And in order to do that, I need like secure coding practices that prevent me from making cryptography mistakes when we're dealing with this sensitive material. And you can kind of recurse into a task like that sometimes like six, seven, eight times before you bottom out at the core primitive capability that is missing. And then you kind of like pop back up the stack to accomplish the actual higher level primitive that you were trying to add to the product. One neat thing about that is as you're doing this breadth first search through the tools and quality gates that you need to give the agent. You knock those out one by one. You don't have to do them going forward. You are importing leverage into the code base for every subsequent change. It means that every bit of feedback I give, I at most have to give two or three times before we figure out a way for that misbehavior to just go away entirely.
- [20:35]
A
So one part that blew my mind, and I always like this PM angle, is that your PM just wrote a PRD in markdown and they went to a shipped pull request. Can you give us a couple concrete examples of what they shipped and how that flow works?
- [20:52]
B
Sure. Back in September of last year, we were experimenting with what were the precursor to skills in this agent. This idea that the human driver should be able to teach the model about the things they care about. We were doing some data analysis tasks, so understanding the data ontology of what it means to do business metrics in this particular part of the company was important. And we want the model, we want the harness to be able to pull this information out from the user. We have this lovely machine that is able to write a bunch of text. It should be easy for it to have a conversation with the user to figure out the things they care about versus the things they don't, so that all subsequent runs that are doing these Business metrics questions accrue that leverage context. The thing that we wanted to explore was what product shape does this look like? How can we give the user the ability to inject durable knowledge into the agent over and over again? So our PM set about to explore this, and the hypothesis was that a library with the ability to both interview the user and learn from all the tasks that the agent was already doing was the way we wanted to package up this capability. And we kind of reviewed the PRD as a team, sort of the beginning of the week, team meeting at the end of the week on our demos, without any of the engineers having talked at all to the pm. We had a demo of this feature and it rolled out to customers in the next week. And this is very, very cool. It was able to be done safely because we have architected our code in a very, very modular way where individual pieces of business logic are able to connect together in safe ways without kind of spaghetti dependencies. Everything takes an interface. The concrete implementations are hidden. It means every bit of business logic that we have, whether it was the skill wiki or the ability to manage connectors, always exports a high fidelity fake for tests, which means the tests that RPM was vibing into existence actually exercise its dependent services and the real business logic of the skill library without having to worry about monkey packaging or having tests that don't reflect reality. The eval here is whether or not the agent was able to learn from these skills, whether or not they were mounted into context, whether or not the UI presented in the way that it should. This is the same way I would think about evaling this slide deck we're looking at now, which was also created by Codex. Right. Codex is a coding agent. It's going to write code in order to make this slide deck. But I don't care about that code. I care that it produces a final artifact that I care about. And putting all this work into the harness, into the repo means that we can actually move to PRD as code input with the app as compiled output. But slow in the beginning. You can't do that in any brownfield code base today and expect to get good results. I think we'll talk a little bit around how we can give you a cookbook for what to do next week in your code bases. But I think the idea here is to have the role of the engineering team to encode leverage around what it means to do a good job. So the actual specifics around what you build and how it is realized in code are just kind of Artifacts of the repository itself.
- [24:41]
A
Encoding leverage is the name of the game. We've heard a success story on the PM side. In our pre call you had mentioned that it wasn't always successful. There were some things that for instance, you guys built with your designer early on it was like an open clause style that you had to trash. So what failed? What did you learn? And how can people learn from that failure so it doesn't happen to them?
- [25:06]
B
Yeah, so one thing we were trying to build was what you now see in the Codex app as automation, sort of this scheduled cron mechanism in order to kick off agent tasks on your behalf on a schedule. Very open claw like as you said. And this was a capability that our designer put into the product, but in doing so had to bite off a fairly chunky bit of app infrastructure work in order to put in a cron system which interfaced cleanly with our agentic core. We wanted the cron itself to create tasks, not necessarily be responsible for driving them to completion. We wanted a nice loose coupling between these core infrastructure primitives. The ultimate thing that our designer wanted to prove out was the UI interaction pattern. What it would mean to even surface scheduling to users in ways that they felt was intuitive. But in order to go from zero to putting that feature in front of users, felt that the back end needed to exist as well. And the reason we ended up unwinding that is because we ended up with spaghetti. Right. Like we were still halfway through realizing the true value of the harness here. Engineering had not yet fully specified all these non functional requirements of what it means to write high quality code that is scalable for future maintenance. This is really all the harness is, is figuring out ways to get those non functional requirements written down and injected into the agent just in time. This is what enables everyone to contribute to it safely. This is what empowers growth of the team to continue to accrue leverage back into the harness. But in the absence of that, we ended up with a spaghetti mess. I think the scheduler lived in the front end JavaScript. At some point it's just like I look at this and I'm like, oh, I'm going to regret this later. So we made the decision to revert the print. But we did want to enable the core objective here, which was to experiment with what this product surface could look like, to figure out which interaction patterns were most intuitive to users. And we do want our designer contributing to the code base because we need his expertise on how to use the design system effectively, how we should think about component architecture we wanted to continue to explore good information architecture so we're not overwhelming users, but also giving them the power tools that they need. So the norm we aligned on was we would kind of put a door between the UI as exposed to users and actually implemented functionality on the back end and ended up with what we're calling a painted door style where we can get the full product feature implemented with just a no op on the back end. With the addition of some creative product instrumentation, we're actually able to get most of the way there in terms of doing that experimentation and actually get higher signal for the engineering team on whether or not we should even prioritize this work. With code generation increasingly being automated, the bulk of my time is spent on feature triage and scheduling. So figuring out ways to get more information into that part of my work is an amazing outcome of this sort of norm we aligned on I used
- [28:42]
A
to think I had a retention problem. Turns out I had a messaging problem. I was sending the same onboarding emails to every new user, whether they activated on day one or never logged in again. I had no idea who was slipping or why. Customer IO changed that. Every message I send is now based on what users actually do in the product. Someone hits a key activation moment, they get nudged to the next one. Someone goes quiet. They get a different path entirely. Their AI agent makes it fast. I describe the campaign I want and it builds the full journey, form triggers, timing, copy, even branching logic. And when I want to know how something is performing, I just ask the agent directly and it tells me what to do next. They also have an MCP server, which means AI tools like Claude can see directly what's happening happening in your Customer IO workspace, your segments, your customer data, your attribution, all of it. So instead of explaining your business context every time you need help, Claude already knows it. Notion use Customer IO to personalize their onboarding and hit nearly 50% open rate, improved conversion by 6 to 7% with localized campaigns, and pushed open rates up another 20% through AB testing. The idea is simple. Customer IO helps you deliver more impact from every message you send. If you're a PM or founder and your onboarding is still one size fits all, try Customer IO at Customer IO. I'm notoriously bad at my inboxes. I guess there's a version of that where I seem cool and unavailable, but the reality is I miss sponsor emails, guest pitches and stuff that my team actually needs me for. So I got an AI assistant. The sponsor of today's episode, Ariso Ariso connects to my email, calendar and Slack. Then I just chat with it over Slack and it helps me with everything. It builds workflows to respond to emails, resolve customer issues, prep me for meetings. It actually comes to my meetings, updates its own knowledge, and remembers context from past conversations. So every time I talk to it, it already knows what I'm working on. I used to pay for Granola and Lindy separately. Ariso replaced both. One tool does more and it lives right in Slack where I already work. Check it out at Ariso AI Akash that's a R I S O a I A a K a Today's podcast is brought to you by Pendo, the leading software experience management platform. McKinsey found that 78% of companies are using Gen AI, but just as many have reported no bottom line improvements. So how do you know if your AI agents are actually working? Are they giving users the wrong answers? Creating more work instead of less, improving retention or hurting it? When your software data and AI data are disconnected, you can't answer these questions. But when you bring all your usage data together in one place, you can see what users do before, during and after they use AI, showing you when agents work, how they help you grow, and when to prioritize on your roadmap. Pendo Agent analytics is the only solution built to do this for product teams. Start measuring your AI's performance with agent analytics at Pendo IO Akash that's P E N D O IO Aakash okay, so PM writes PRD and Markdown designer paints a door. Then engineering team takes over the actual backend. That's the high level boiled down summary that I'm hearing. The thing that makes this all possible is the harness. Can you show us what's actually in it?
- [31:57]
B
The harness lives in the repository, but really the harness is a mechanism for getting the coding agent, getting Codex the right context it needs in order to do the job at all phases of the implementation lifecycle. I think there's kind of three phases you would think about here. One is I'm getting ready to go to work. I have a ticket, I have a PRD and I need to figure out what must be done in order to implement the thing and get it deployed. The next phase is kind of in the messy middle of implementation. I'm writing code, I'm exercising the ui, I'm seeing the bugs, I'm seeing logs, I'm making sure that everything is pixel perfect. And then there's the back half of things where I have produced a diff that I claim implements the feature and I need to convince the rest of my team to merge it, that it is acceptable, that it meets the actual user need that we're trying to solve with this change. And in each of those phases we have different mechanisms in order to surface context to codecs in order for it to do a good job in the beginning. Here we really have Agents md, which is this file that lives in the root of the repository that is forcibly injected into codecs. It's like what codec CLI does is take the contents of that Agents MD and give it into context for the agent. This is both good and bad, right? Things in agents MD we know are 100% reliably going to inform the agent's actions, but we're paying for that with context window usage and with these models. Context window is a hard constraint. You cannot fight against it. There is only so many hundreds of thousands of tokens you get. I will say that auto compaction in GPT 5.2535455 is so, so good. I never run out of context these days. Sort of like a crown jewel of research here to have good auto compaction. It is truly amazing to be able to go on any size of ticket completely end to end on a single session. It's magical. But still you do have to contend with things getting paged out of context, even with auto compaction. So Agents MD is a tool, but one that we want to be constantly trading off from this scarce resource of context. So what we do there is we put an operating model for codex in there. Basically a sort of like three step, four, five step plan of how you should think about doing a task. And what that means for us is you ground the decisions that you make in the documentation we have in the repository. There's a big docs tree. It includes everything from what it means to write performant react to writing reliable networking code, to all the design docs and critical user journey flows that we have in the app. And that's important to have as part of the operating loop because we don't inject that stuff all into the Agents MD up front. Instead we just tell codecs in the Agents md you have all these resources available to you. They are of X, Y and Z shape and they live here. You decide whether or not to look at them. Which is nice because we can deterministically store steer the agent toward the context we want it to discover. But we let the magic of these models reasoning decide what is important. What it actually thinks should pull into context. This is sort of the outset of the task. So ground yourself in the documentation that we have the app as it is written and then move into implementation. During implementation, the thing the agent is naturally going to do, the thing it is trained to do is write code and make tool calls. So we have to figure out ways to use those two mechanisms to inject our context into the agent. This is kind of a silly thing, but actually the code in the repository is prompts. Codex is going to pull code into its context window in order to figure out what business logic do we have? How do we do things here? What database adapter am I using? Am I using Jotai on the front end for state management? So actually having the repository enforce that the code is all the same or uses the same patterns or structured similarly allows you to efficiently and sort of like in parallel across the code base, apply context you've read in one file to other files. It means that the agent is able to more effectively cargo cult. I love to cargo cult stuff. This is how I have worked in big brownfield code bases when I'm new, right? I'm like, the team has probably done this before. We can poke around and copy and paste some stuff. We can let Codex be lazy too by making it do the hard work of making the code base homogenous. But another way we can inject context into the agent over the course of doing these runs is to take all that human judgment around what good looks like and basically make it fail tests if. If the code doesn't match. A stupid one that I do is to fail the build if there is any text in markdown files or HTML that does not use the curly quotes. Because our design team loves that proper typography in our user facing strings to have the smart curly quotes and apostrophes. And this is probably a thing that is unevenly adhered to, I think, in most code bases, if at all. But because this kind of like irked our designer a couple of times, had him vibe code some tests into existence, we don't really care about the quality of this code, right, so long as it actually does the job and prevents the code from becoming misaligned with the user intent. And we have a lot of tests like that that do probably weird things, maybe things you would consider low leverage, but collectively improve the quality of the stock software that we're delivering, which I think is pretty cool. And then on the back end, right, we've got sort of the agent executing grounded in the documentation we have in the repository, which is this accumulation of all of the team's historical work and knowledge, we've got taste refining the output as we go via these tests and lints and the code itself, showing the agent what good looks like. And on the back end, kind of take all of that same context and give it to a review agent. In that docs repository we have front end Architect md, reliabilityengineer md, appsecngineer md. And for each one of those we have this big matrixed CI job that says you review for front end architecture. Here's your guardrails, here's the diff. Identify all the P2 issues with this thing and leave some feedback on the pr. This feedback, more context. Just like those tool call error messages that we're giving the implementation agent to say, you must address this, right? We found some issues where you are misaligned with the way we expect things to be produced and with sort of all of this effort in prompt injecting the agent with the collective human judgment on the team. My synchronous involvement here can basically move all the way to the back of this process where the code is presumably already reviewed and approved and aligned by all this machinery that we've put in place over the course of executing producing the change. And then my job is to say, okay, we've got this change, it looks good, but I think it's misaligned in ways X, Y and Z. And then once those things are addressed in the moment, my job is to close the loop by taking all this feedback that I had to give and making it so I don't have to give it next time, and then back into the harness it goes.
- [39:55]
A
So we got the high level now of what it involves. Can you go really granular, show your actual codec setup, show what's on every engineer, PM designer's machine, and how to put this all together.
- [40:09]
B
Practically, I would say that the way I have steered my team has been intentionally frictionful so that we don't want them staring at the computer looking at their agent, which means they have largely been booting up codecs in the cli, going to multiple tabs where it's actually painful in order to pay attention to these things, because we don't want them to. If I were doing it again today, this is probably not what I would do. Right. The app is quite fantastic and is a thing that we are seeing meteoric growth in. I think it was just past 400 or 4 million weekly active users and all the good parts of the workflow that I'm describing are in the app. Right. I would package all that collective designer taste into a plugin that my team has a little shared marketplace of, that I can just suck all of that expertise into the app that's going to pre wire Figma, which means the place my designer naturally works is going to be a thing that is directly addressable by Codex in the same way that I have access to Figma and maybe I'm unreliable at refreshing my knowledge of the current mocks. Codex is just going to do that because these agents will follow instructions. Well, that is a core part of how they are built. So these plugins which bundle the skills, include those instructions on. These are the tools we use, these are how they are built. So I think the app today, as sort of the central coordination mechanisms of teams, gives a much more user friendly affordance for distributing these sorts of things. Plugins are definitely the future and what I would lean into today.
- [41:54]
A
All right, can you show us what this actually looks like inside of Codecs, how someone would set all this up?
- [41:59]
B
Sure. Let's get into the nitty gritty of it. This is my favorite part here. So we started with an empty repository, which means we had a high degree of freedom to start structuring the code in a way that Codex most natively understood. And this is how we kind of came to the pattern that all the things we want Codex to know need to live in the repository itself. This was its workspace, this was its brain. In order to mise en place get the agent set up, we wanted it to have the easiest time. And the thing that Codex loves to do is search code with ripgrip. So we put everything in one place. And what that looks like in practice is this big docs directory that has a ton of different content in there. We make heavy use of a thing called an exec plan, which you can find on the OpenAI Cookbook, which is essentially a skill you can give codecs in order to write a good implementation plan phase with milestones and deliverables so it can track its work and make sure that it is continuing to make progress against sort of high level and tricky implementation plans. And we kind of keep every single one of those that has ever been produced in the repository. They form a durable log of the implementation choices we have historically made. Similarly, every design doc ever written, rather than being buried in slack threads and Google Docs live in the code base, which means these things are actually continuously kept up to date to reflect the actual state of our app. They're sunset and archived when they become irrelevant or features get removed. And we have also built code and tests around this documentation tree. So all these markdown files cross link to each other. They embed git repository revisions of when they are current. As of if Codex ever touches one of them to update it, the build will fail unless all the crosslink documents are also updated to reflect the current state of the world. And this is what it means in order to have the harness self maintained, to have the test prompt injecting codecs with what it must do. We don't rely on codecs knowing that this is how it maintains the doc repository. A test will tell it what it needs to do, how it must proceed in order to get the build to green. The agents are super trained in order to get the test to pass. So if we can figure out ways to build context injection primitives into failing tests, we're not fighting the harness here. This is not a thing that will be obsoleted by the model capability improving. This is just a very neat way of aligning what must be done with the scarce resource of context with the actual thing the model will and always do, which is call tools and run tests. Similarly here you can see in this references section have a bunch of these LLMs txt. I don't know if everybody knows this, but a lot of programming projects will publish a plain text version of their documentation at LLMs text and this strips away all the tokens from the HTML and it's just like markdown. Stripe has these OpenAI docs, have these UV, which is a python runtime, has these. Our internal design system has these and we check these directly into the code base which basically give codecs a full reference manual for all of the big chunky libraries and components that we depend on. And we point to those. Our documentation on the design system highlights the key components that are probably going to be used most of the time and how to look for them in this LLMs Txt Here you can see these Persona oriented documentation that we talked about earlier, what it means to write secure code. This product sense MD is how we think about encoding, what it means to document a feature, how to turn a PRD into code in a QA plan. And this is the sort of thing that I doubt any teams are writing down today. It's just a thing they kind of, you know, talk about together and maybe whack each other on the nose when they don't do this when producing a pr, but figuring out a way to actually durably document what it means for the team to Operate is necessary for these agents, which every time they spin up have a blank slate in terms of context. Another neat thing here is this quality Score md, which is sort of a meta way for Codex to diagnose the state of the code base. We have it keep notes here every time it writes code, because over the process of making a change, it's going to pull a ton of code into context. Maybe code that hasn't been touched in a month, but our guardrails have changed. So this is a way for Codex to be continually assessing the quality of the code base, doing some gardening and spinning up some tasks for later on, how we can keep aligning the code base to baseline. Another neat thing we can sort of talk about here is what it means to close the loop. When we're working on this app, we split it into a back end and front end pattern, which means they could be tested independently. And we tried to house as much of the business logic as possible in our backend. And that means codec should be able to validate the behavior of our business logic headlessly. Right? You don't need to drive a UI. You can hit APIs and look at the database to observe the side effects that you were hoping to have. And normally for almost all of my career, when I'm working with services in local dev, I run them in my terminal and I get this never ending stream of logs in the console that I have no idea what to do with. And it's kind of impossible to use because they scroll away so quickly. What we've done instead is permit Codex to spin up something approximating a production observability stack. Think giving Datadog to Codex for every change that it makes. And this allows it to use normal observability tooling for metrics and logs, the same things your engineers would be using to diagnose issues in production. And we let codecs use it to diagnose issues in local dev. Right, shift left, validate the change and exercise it end to end. And we do the same thing with the UI itself. We mount the UI into a local browser shell that pre wires up Chrome DevTools. Today we would do this with computer use to allow codecs to actually inspect the app, drive it, click the buttons, observe that the loop is closed, that the feature is implemented implemented, and that it can attest that it works end to end. So I mentioned a little bit how being super overly architected in the application was a point of leverage for the engineering team. And the reason that is because we actually limited statically the plausible places that code can go. You see this kind of represented here as sort of a package layering stack. One common pattern in hypergrowth code bases that is a failure mode is that business logic kind of just gets spewed everywhere and it becomes a sort of code base that approximates a ball of mud. These are very difficult to deal with and evolve because there's a ton of non local side effects from touching any bit of code. But I mentioned that we have these nice isolated business logic domains that are able to take all their dependencies as fakes and make it safe for us to prototype new features. And the only reason we're able to do that is because we actually with code enforce that these things are hard separated with package boundaries between them. You cannot possibly create a ball of mud in the first place. Which basically means that for the engineers, we permit them to write packages at any layer of the stack. They can write database code, config parsing, react code in the front end business logic rpm, we kind of by norm and convention said you can write business logic and UI code, but try not to touch these lower level primitives in the app. And to get that painted door invariant for our designer had him focus largely in the UI layer of the app, with some affordances for stack dubbing out APIs on the back end so that we could get those usage signals that we cared about. Which is kind of neat to be able to think about reflecting sort of like ACLs on who is permitted to touch what code and the actual structure of the code itself. This is actually by folder paths that we're able to determine whether changes are allowed or not.
- [50:41]
A
This adds up. And I think you're. You have a really hot take on how it should add up. You've said that if you're not using a billion tokens a day, you're basically negligent. Now that's 2 to $3,000 per engineer. How does an engineering leader at a normal company build the case for that internally?
- [51:02]
B
The idea here is that the more tokens that you're running through these models, the smarter they get, the more intelligence you are able to extract. This is sort of like the realization behind why we have reasoning models in the first place, right? If we give the model more effort to do inference, we're actually able to get greater results out of this. And we see this time and time again with longer time horizons that agents are able to execute over. And I think it is the case that in order to have this sort of lofty call of being a Token billionaire. You kind of have to find ways to get the agent to do more work on your behalf, to execute in parallel more autonomously. And because I believe that GPT5 is able to do the full job, having that as a North Star constraint means that I am constantly looking for ways to get the agent to do the hard parts of my job that take a lot of my time. And constantly being surprised in ways. This frees me up. One sort of insane thing is not just a billion tokens a day, but sometimes it can be 350 million tokens on a single PR. There was a time a couple of months ago where I was buckling my laptop into the back of my car, going back and forth to the the office, tethered to my corporate phone in order to have the agent continue to cook while I was commuting to work. This is like a 60 hour codec CLI session that like I was like, please don't go to sleep, laptop. And this is a PR that probably would have taken me three weeks in order to do like one of the hardest refactors I have ever done in my career. And it took place over the course of three, four days and did as good a job as I would have here. Because of the investment we've put into letting the harness prompt inject the agent and manage its context efficiently. Over that 60 hours I only had to provide two prompts in addition to my first prompt that we gave the agent, which is truly magical. It means that it was actually approximating a human engineer on my team. I did not have to synchronously bash it on the head in order to make progress. It just, you know, went heads down and did the work the way I would.
- [53:26]
A
Okay, so 60 hour running long task. I think that points to something that if people have their conception of these models five, six months ago, it's really outdated. It's the what's recently become possible that's enabling this new way of working. How would you describe it? When did that happen?
- [53:46]
B
Yeah, so there was a huge uplift in capability when GPT 5.2 came came out. And the rate of progress with each additional point release of the model has been much bigger than what came before. GPT 5.2 came out while I was on holiday for a winter break. And when I came back without any additional investment, we were getting one or two PRs more per day per engineer on the team, which had this very strange effect where folks were looking for ways to do more work with the agent. It was kind of like adding Another lane to the highway. It was like, how can I put more cars on this thing right now? And because we had done so much work in order to get Codex to operate over those long horizons, the next sort of frontier was to go in parallel. We just blogged yesterday about sort of agent Orchestrator that came out of my team called Symphony, which once we saw the uplift in capability with 5.2 and invested in this additional Orchestrator, allowed us to further 10x the PRs per engineer per week. And that investment in increased parallelization is only a thing we were able to do do because we were confident in the code that was being produced already by the agent. Right. If I'm 10xing the number of PRs I am producing a week, I cannot possibly look at all of them. I couldn't possibly look at them all before. So the only way we could do this safely is with these techniques in order to eliminate classes of misbehavior and have confidence by default in the code that's being produced. But yeah, we continue to see these huge improvements in autonomy and confidence with each model revision. One of the most interesting things about 5.5 is with computer use, there's much less wrangling we have to do in order to get the agent to actually see and drive the app. This was a significant point of friction for us early on. We were having to figure out ways to wire up FFMPEG to Docker containers containers to give Codex a virtual X server so it could drive the app. Like this is all a ton of low level goo plumbing nonsense that was slow to develop and ultimately undifferentiated. But with codecs now shipping computer use, a tool that's natively post trained into the model, we kind of get to throw away all that bespoke code, keep the same process of having codecs drive the app and do it much more efficiently with better image understanding, ability to faithfully replicate wireframes and mocks, these sorts of things. Truly amazing.
- [56:42]
A
So you mentioned one of the hardest refactors in your career. You've worked at the regular companies, the Brex, the Stripe, these types of companies. If you are an engineer at this and you're looking at this new way of working, what is the new engineering job?
- [57:01]
B
The new engineering job, I think, is to have everybody be staff engineers. And the role of a staff engineer, the primary role, at least for me, is not to be producing all of the code, but to be empowering my team to produce the code. And today it is the case that every single engineer on this planet modulo token budget has access to 5, 50 or 5,000 concurrent hands on keyboards. And not being able to leverage that amount of parallelism today is largely a reflection of not having put in the work to harness those resources. One experiment we're running internally in order to drive toward this amount of parallelism is to tell teams that they have been forced to hire five interns named Codex. Part of their success criteria is judged on how effectively they incorporate these resources into their team. What does it mean to make use of this capacity that the organization has given them? And part of success is being able to use those resources. And if we think that with access to these lovely models and and compute, being available to use them, letting those GPUs idle is in a way not having hands on keyboards. So my job is to figure out ways to saturate these things, to have them doing productive work, to have them not waste cycles, to have fewer PRs that we throw away because they are high quality by default. To have fewer cycles back and forth in CI, fewer reviewer agents having to repeat repeatedly burn cycles for the same feedback, that sort of thing. It is very much a systems thinking sort of mindset where in a way you're playing factorio with your code base and figuring out ways to build the token factory.
- [58:56]
A
You're describing essentially being totally divorced from the code and really focused on using these interns. But I guess a lot of engineers, they prided themselves on being the best at writing code. That has been their output for so long. I imagine people on your team have struggled with this mental shift. What is the right way to think about it?
- [59:19]
B
This is the thing I have struggled with too for a long time. Because the primary function of the job was to produce code in order to achieve our product and business objectives. It's very easy to fixate on the production of code itself as the job. But ultimately the reason we work in tech is to deliver product into the world. And initially kind of like having this hard break in mental model of what it means to do work was a little bit like weird on the ego sort of thing. But where I have gotten to is latching onto the same creativity, the production aspect of things. Ultimately the thing that motivates me is to get product into the hands of users and see their delight and how they're able to improve their lives with the tools we built. And I'm still able to do that today. I'm able to do a lot more of that today. And I kind of like puzzles. You can have puzzles in how you Structure the code or you can have puzzles in terms of how you put teams together to produce it. To me, these skills and challenges are pretty transferable between each other. So I think they're interested. Is a new art to be found, a new craft in terms of how you effectively assemble a virtual team to go solve these things.
- [60:39]
A
Okay, so let's move into the roadmap. Your counterpart at Brex or Citadel or Stripe, they want their PM shipping 100,000 lines of code. What's the Monday morning roadmap that they need to take so that they can get started right now to move to this way of working over the next six months?
- [60:58]
B
So looking forward to the Monday morning roadmap? You have no idea. I was super excited to divide this slide into existence with Codex because this is the hook right here. This is why we're talking. So I think those map or those phases of the code production process that we talked about map pretty clearly to sort of what the Monday morning roadmap should be. You need the repository to be legible to the agent. It needs to know all the things that your team does right. The process of bringing product into the world, to producing code, to having good design, has a ton of implicit context around what good looks like. Hundreds or thousands of little decisions that we all make by default because we cut our teeth the hard way, gaining experience. But the agents don't have this, right? Well, they do, but they have every possible permutation of all those thousands of decisions because they have seen so much code and so much product over the course of their training. So it's our job to take the specific choices that we as a team would make, write them down, give them to the agent, and starting with this simple technique of putting a map in agents MD on where the agent should look. I really like having a documentation tree as sort of the entry point for making this happen for your code base because it's super easy for humans to think about just writing stuff down. And it's also really easy to use some of the tools that we have to do that. It's very often the case that agent will produce code we didn't like or there's a bug in production. We'll have a Slack thread as a team where we discuss what went wrong, how we think it should be done differently. And it's really easy at the end of that to just do mentioned codecs, you know, update docs X, Y and Z to reflect these new guardrails, this new taste we have discussed and that kind of removes us from the loop entirely from maintaining the legibility, we just kind of get to talk and we let Codex maintain it for us. And this is a way too that I think sort of by default, in larger teams, you can kind of sort of build a flywheel around improving agent behavior and build sort of cultural norms and processes around having this happen as the default way of doing the work. The other next step is to make validation cheap. Validation is happening at all parts of the code production part pipeline that I talked about. On the review side of looking at those guardrails over the course of writing the code, with all these lints and tests, being able to actually close the loop and exercise the app to see that yes, the search box that was in the PRD actually appears in the ui. And when I click on it, it shows a pending set of results that are our best guess. And it has very nice type ahead without any jank in the box. When we type, these are, you know, kind of the squishy ill specified parts of the task, but kind of providing proof of work to the reviewer agents and the humans. Looking at these PRs is necessary in order to build confidence. This is what it takes in order for me to not have to shoulder surf the Codex the same way I'm not going to shoulder surf my teammates in VS Code or cursor. And finally, it's important to have the users of this harness be more than just the engineers themselves. The exercise here is to empower all of the team in order to write code safely. So having a tight collaboration loop with the other members of the team, product design, user ops, your DevOps folks, allows for really high bandwidth feedback between these different roles and it allows you to do more work, different types of work, extract different domain expertise from the folks on your team and reflect it all back into the repository for the benefit of everyone. One sort of like emergent property we had here that I thought was really, really cool was we hired a new engineer onto the team who had formerly been a pm and he noticed that we had no documentation in the code base for what product features we had. And this is kind of an important thing because it grounds all the engineers, all the designers, all the product, the reviewer agents, everything and what we are actually trying to achieve. What are we shipping here? And Codex is very good at crawling through a code base and figuring out what the product features are and writing them down and surfacing them for human review. But what fell out of that was a set of QA agents that are constantly booting the app, running it through these Critical user journeys and asserting things work. Which means that when it comes time for deploy, we have to do less manual smoke testing because we actually have these just in time generated runbooks around what it means to have acceptance testing. And all of that came out of a single person with domain expertise and an idea of what good looks like. And sort of the rest of the work we had done made these higher level correctness verifiers kind of fall out of that one thing.
- [66:26]
A
So we're actually full circle all the way back to the beginning of the discussion. Now that people understand the harness, they understand the Monday morning roadmap. What does the product team of 2027 look like? How are these individual functions working together if everybody has access to the code?
- [66:49]
B
I think we're going to see a lot more prototyping in the world, a lot more diversity of products as a result. I think too, in this new wave of computing with AI and agents, we are currently at the early part of the technology curve where a lot of the products you see today are low fidelity text interfaces. And when I think about where I sit in OpenAI working on frontier and deploying to businesses in order to get the machine to do the work, there is a fractal nature of work, different surfaces, different ways of working to wedge these models into the nooks and crannies of the hard knowledge work that happens. So I'm really excited to be able to rapidly experiment with different modalities and different ways of delivering this technology. This is core part of the OpenAI mission here, right? Is to have wide access to do economically valuable work with these models, with these tools. So I'm excited to increase the rate of experimentation to to increase the diversity of product we see in the world and to see new and exciting ways we're able to use technology to do great things for users.
- [68:06]
A
All right, so we're into the home stretch here. I want to ask you a couple quick questions. Is that okay?
- [68:12]
B
Yeah, sure. Let's do it.
- [68:14]
A
What's one skill a tech lead who's trying to move to this way should build this weekend?
- [68:20]
B
I really think it's important that you figure out a way to get Codex to be able to see your app and drive the business logic in the absence of any other modularity or investment. The ability to actually close the loop and observe the behavior that you're changing is what it takes in order to know that you have completed the job. I see this all the time where folks will vibe something up and not be able to attest that it solves the problem they set out to do. And in order to convince the rest of the team to accept the code in the first place, you got to be able to prove somehow that the code meets the brief, that it does what it says on the tin. And the way to do that is to invest in closed verification loops, the highest leverage of which is the full end to end integration test. Click the buttons, observe the side effect, that sort of thing.
- [69:17]
A
It's funny how it comes back to some of the fundamentals, but now applied to AI. What's the biggest mistake engineering leaders are making with AI coding tools right now?
- [69:28]
B
I think having the tools in your head or in a box as these things that are tools that must be driven synchronously by a human, is limiting the amount of creativity in terms of deployment, deploying these things widely. Ever since GPT 5.2 came out, my belief is that it is as capable as me at doing the full job. And it's sort of my job to figure out ways to have it do as much of my job as possible. A lot of that means I can't be sitting in front of it because I don't sit in front of my teammates. So there's this whole metagame of building confidence in in the code that these agents are producing so that we can have it do more and more of the job, to execute on longer and longer time horizons, to be more and more independent. And this is necessary in order to unlock that true parallelism, right, that 5,000 engineers worth of compute that I talked about. I can't possibly do that if I'm locked behind two tabs in the app or three panes in my TMUX in my terminal.
- [70:35]
A
Final sort of meaty question here for you. We've been talking about GPT 5.2 at the same time came out, Opus is 4.6 roughly. And the whole discussion really changed in that December time period. If a team can only pick one, the anthropic stack or the OpenAI stack, what's the case for picking the OpenAI one?
- [70:57]
B
Our models are optimized around that full agentic behavior to being able to do the full job. I am super easily able to delegate entire swaths of work to the GPT5 series without having to constantly poke and prod it. And this is the fundamental unlock for me that has gotten to that parallelism symphony. This agent orchestrator that we have released is fundamentally very uncomplicated. It is a thing that advances linear tickets through states and gives the ticket text to codex and expects it to produce a pr. There's very little magic to it, which is why I think that sort of these GPT series of models that we have are the ones that are going to unlock the true power of agentix software engineering in your organizations. Additionally, the pace of product delivery on the model capabilities side in order to get more and more of that loop closing in place has just been fantastic. Things like parallel tool calling and background shells in GPT 5.3 to ever increasing auto compaction capability to computer use and the ability to have multimodal understanding at high fidelity with built in image generation capabilities in the latest 5.5 means that what would normally have been two or three people on the team handing off between them means that instead those two or three people can encode all their leverage into one tool which is then able to do the full job. There is this kind of neat story that I have been telling that in multi agent systems the correct amount of agents to to want to optimize toward is not multi, it is one. Because one agent is able to have full addressability over the entire task and its entire context. Which means being able to do design, backend and front end in a single agent means you're going to get higher quality results than having these lossy frictionful handoffs between them. And that experience comes from me having built agents for knowledge work with handoffs in them and seeing the massively accrued increasing capability of the GPT5 series over time. I want a single agent to cook and do the full job and we have done a fantastic job I think at increasing Codex's ability to address all parts of the SDLC in a single agent.
- [73:29]
A
Wow. There was so much information in there. If I want to get more if I want to get more of Ryan where can I find you online?
- [73:37]
B
I tweet a lot on X My handle is and me and the team should have more content coming on the OpenAI blog and I am doing a bunch of conference tour over the summer. I am next going to be at AI devcon in London. We'll see you there.
- [73:59]
A
All right guys, follow him on Twitter, check out his amazing blog post on the OpenAI blog and we'll see you guys in the next episode.
- [74:08]
B
Been a pleasure Akash. Thank you.
- [74:10]
A
Thank you. I hope you enjoyed that episode. Couple things you can do to support the show. 1 Comment 2 Review those ratings and reviews really help other people understand the value and the production that we are putting into this. Right? This wasn't an easy episode to produce. We put in a ton of pre work. We edited it for you. We brought in the best guests. If you don't mind sharing a rating and review, sharing the episode with others, making sure you are subscribed, that really helps the show do bigger and better productions. I'll see you in the next episode. Here's one of those that YouTube thinks would be a great fit for you.