Interviewer/Co-host (2:49)

That makes sense. So it sounds like basically there's like a few different changes between your core sort of GPT models where now you have something that takes a pause to think about something. So at inference time there's more compute happening. And then also it can do sequential steps because it can kind of infer what are those steps and then go act on them. How did you build or train this differently from just a core foundation model or when you all did GPT 3.5 and 4 and all the various models that have come over time, what is different in terms of how you actually construct one of these?

Brandon McKenzie (3:19)

I guess the short answer is reinforcement learning is the biggest one. So yeah, rather than just having to predict the next token in some large pre training corpus from everywhere, essentially now we have a more focused goal of the model. Solving very difficult tasks and taking as long as it needs to do to figure out the answers to those problems. Something that's kind of magical from a user experience for me was we've in the past for our reasoning models talked a lot about test time scaling. And I think for a lot of problems without tools, test time scaling might occasionally work. But at some point the model is just kind of ranting in its internal chain of thought. And especially for some visual perception ones, it knows that it doesn't. It's not able to see the thing that it needs and it just kind of like loses its mind and goes insane. And I think tool use is a really important component now to continuing this like test time scaling. And you can feel this when you're talking to O3. At least my impression when I first started using it was the longer it thinks like I really get the impression that like I'm going to get a better result and you can kind of watch it do really intuitive things and it's, it's a very different experience but being able to kind of trust that as you're waiting like it's worth the wait and you're going to get a better result because of it. And the model's not just off doing some totally irrelevant thing.

Interviewer/Co-host (4:41)

That's cool. I think in your original post about this too, you all had a graph which basically showed that you looked at how long it thought versus the accuracy of result and it was a really nice relationship. So clearly thinking more deeply about something really matters. And it seems like in the long run, do you think there's just going to be a world where we have sort of a split or bifurcation between models which are sort of fast, cheap, efficient, get certain basic tasks done and then there's another model which you upload a legal M and a folder and it takes a day to think and it's slow and expensive, but then it produces, you know, output that would take you, a team of people, you know, a month to produce. Or how do you think about the world in terms of how, how all this is evolving or where it's heading?

Eric Mitchell (5:24)

You know, I think for us, like unification of our models is something that, you know, Sam has talked about, talked about publicly that, you know, we have this big crazy model switcher in ChatGPT and there are a lot of choices and you know, we have a model that might be good at any particular thing, you know, that a user might want to do, but that's not that helpful if it's not easy for the user to figure out, well, which model should I use for that task. And so yeah, making the models better able, making this experience more intuitive is definitely something that is valuable and something we're interested in doing. And that applies to this question of are we going to have two models that people pick between or a zillion models that people pick between or do we put that decision inside the model? I think everyone is going to try stuff and figure out what works well for the problems they're interested in and the users that they have. But yeah, that question of how do you make that sort of decision, be as effective, accurate, intuitive as possible, is definitely top of mind.

Podcast Host (6:39)

Is there a reason from a research perspective to combine reasoning with pre training or try to have more control of this? Because if you just think about it from the product perspective of the, the end consumer dealing with ChatGPT, like, you know, we won't get into the, the naming nonsense here, but they don't care. They just want like the right answer and the amount of intelligence required to get there in as little time as possible. Right.

Brandon McKenzie (7:05)

The ideal situation is it's like intuitive that like how long should you have to wait? You should have to wait as long as it takes for the model to like, give you a correct answer or, and I hope we can get to a place where our models have a more precise understanding of their own level of uncertainty. Because, you know, if they already know the answer, they should just kind of tell you it. And if, if it takes them a day to actually figure it out, then they should, they should take a day, but you should always have a sense of like, it takes exactly as long as, as it, as it needs to for that current, like, model's intelligence. And I, I feel like we're on the, the right path for that.

Podcast Host (7:40)

Yeah. I wonder if there isn't a bifurcation though, between like an end user product and a developer product. Right. Because there are lots of companies that use, you know, the APIs to all of these different models then for very specific tasks. And then on some of them they might even use like open source models with really cheap inference with stuff that they control more.

Brandon McKenzie (8:00)

I hope you could just kind of tell the model like, hey, this is a API use case. And yeah, you really can't be over there thinking for like 10 minutes we got to get an answer to the user. It'd be great if their models kind of get to be more steerable like, like that as well.

Eric Mitchell (8:15)

Yeah, I think it's just a general steerability question. At the end of the day, if the model's smart, you should be able to specify the context of your problem and the model should do the right thing. There's going to be some limitations because maybe just figuring out, given your situation, what is the right thing to do might require thinking in and of itself to figure out. So it's not that you can obviously do this perfectly, but. But yeah, pushing all the right parts of this into the model to make things easier for the user seems is a very good goal.

Podcast Host (8:52)

Can I go back to something else you said? So the first guest we ever had on the podcast was actually Noam Brown.

Brandon McKenzie (8:59)

Oh, nice.

Eric Mitchell (9:00)

I've heard of him.

Podcast Host (9:01)

I know, two plus years ago. Yes. Hi Gnome. It'd be great to get some intuition from you guys for why tool use helps like test time scaling work much better.

Brandon McKenzie (9:10)

I can give maybe very concrete cases for like the visual reasoning side of things. There's a lot of cases where. And back to also the model being able to estimate its own uncertainty. You'll give it some kind of question about an image and the model will very transparently tell you. And it shouldn't have thought like, I don't know, I can't really see the thing you're talking about. Very well, or like it almost knows like that its vision is not very good. And well, what's kind of magical is like when you give it access to a tool, it's like, okay, well I gotta figure something out. Let's see if I can like manipulate the image or crop around here or something like this. And what that means is that it's like much more productive use of tokens as it's doing that. And so your test time scaling slope goes from something like this to something much deeper. And we've seen exactly that. The test time scaling slopes without tool use and with tool use for visual reasoning specifically are very noticeably different.

Eric Mitchell (10:04)

Yeah, I also say like, for like writing code for something. Like there are a lot of things that an LLM could try to figure out on its own, but would require a lot of attempts and self verification that you could write a very simple program to do in like a verifiable and much faster way. So I do some research on this company and use this type of valuation model to tell me what the valuation should be. You could have the model try to crank through that and fit those coefficients or whatever in its context, or you could literally just have it write the code to just do it the right way and just know what the actual answer is. And so, yeah, I think part of this is you can just allocate compute a lot more efficiently because you can defer stuff that the model doesn't have comparative advantage to doing to a tool that is really well suited to doing that thing.

Interviewer/Co-host (11:04)

One of the ways I've been using some form of O3 a lot is deep research. Right. I think that's basically a research analyst AI that you all have built that basically will go out, we'll look up things on the web, we'll synthesize information, we'll chart things for you. It's pretty amazing. In terms of its capability set, did you have to do anything special in terms of any form of specific reinforcement learning specifically for it to be better at that or other things that you'd built against it or how did you think about the data training for it, the data that was used for training it? I'm just curious how that product, if it all is a branch off of this and how you thought about building that specifically as part of this broader effort?

Eric Mitchell (11:46)

I think when we think about tool use, I think browsing is one of the most natural places where you think of as a starting point of like, okay, and it's not always easy. I mean, the initial kind of browsing that we included in GPT4 a few years back. It was hard to make it work in a way that felt reliable and useful. Um, but you know, in the sort of, you know, modern these days, last year, you know, two years ago is ancient history. I think it feels like a natural place to start because it's like so widely applicable to so many types of queries. Like anything that is, you know, requires up to date information like it should help to browse for. And so in terms of like a test bed for, hey, like does the way we're doing rl, does it really work? Can we really get the model to learn longer time horizon, kind of meaningful extended behaviors? It feels like kind of a natural place to start in some ways in that it also is fairly likely to be useful in a relatively short amount of time. So it's like, yeah, let's try that. I mean in rl, at the end of the day you're defining an objective and if you have an idea for like who is going to find this most useful, like, you know, you, you might like want to tailor your, the objective, you know, to who you expect to be using the thing, what you expect they're going to want, you know, what is their tolerance for? Do they want to sit through a 30 minute rollout of deep research? You know, do they, when they ask for a report, you know, do they want a page or five pages or a gazillion pages? So yeah, I mean you're, you're definitely, you know, you want to tailor things to like who you think is going to be using it.

Interviewer/Co-host (13:46)

I feel like there's a lot of almost like white collar behavior work that or knowledge work that you all are really capturing through this sort of tooling going forward. And you mentioned software engineering is one potential area. Deep research and sort of analytical jobs is another where there's all sorts of really interesting work to be done that's super helpful in terms of augmenting what people are doing. Are there two or three other areas that you think are the most near term interesting applications for this, whether OpenAI is doing it or others should do it aside, I'm just sort of curious how you think about the big application areas for this sort of technology.

Brandon McKenzie (14:15)

I guess my very biased one that I'm excited about is coding and also research in general. Being able to improve upon the velocity that we can DO research at OpenAI and others can do research when they're using our tools. I think our models are getting a lot of, a lot better very quickly at being actually Useful. And it seems like they're kind of reaching some kind of inflection point where they are useful enough to want to reach out to and use, like, multiple times a day, for me, at least, which wasn't the case. They're always, like, a little bit behind what I wanted them to be. Especially when it comes to navigating and using our internal code base, which is not simple and it's amazing to say more recent. Our models actually really spending a lot of time trying to understand the questions that we ask them and coming back with things that saved me many hours of my own time.

Interviewer/Co-host (15:12)

People say that's the fastest potential bootstrap. Right. In terms of each model, subsequently helping to make the next model better, faster, cheaper, et cetera. And so people often argue that that's almost like a inflection point on the exponent towards superintelligence is basically this ability to use AI to build the next version of AI.

Brandon McKenzie (15:33)

Yeah. And there's so many different components of research, too. It's not just sitting off in the ivory tower thinking about things, but there's hardware, there's various components of training and evaluation and stuff like this. And each of these can be turned to some kind of task that can be optimized and iterated over. So there's plenty of, you know, room to squeeze out improvements.

Podcast Host (15:55)

We talked about browsing the web, writing code, arguably the greatest tool of all. Right. Especially if you're trying to figure out how to spend your compute, write more efficient code, generating images, writing text. There are certainly, like, trajectories of action I think are not in there yet. Right. Like reliably using a sequence of business software.

Brandon McKenzie (16:14)

I'm really excited about the computer use stuff. It kind of drives me crazy in some sense that our models are not already just, like, on my computer all day, watching what I'm doing. Well, I know that could be creepy for some people, and I think you should be able to opt out of that or have that opted out by default. I hate typing. Also, I wish that I could just kind of be working on something on my computer. I hit some issue and I'm just like, what am I supposed to do with this? And I can just kind of ask. I think there's tons of space for being able to improve on how we interact with the models. And this goes back to them being able to use tools in a more intuitive way, I guess, using tools closer to how we use them. It's also surprising to me how intuitively our models do use the tools we give them Access to. It's like weirdly human. Like, but I guess that's not too surprising given the data they've seen before. But yeah, I think a lot of.

Podcast Host (17:00)

Things are weirdly human. Like, like, my intuition for, like, well, why is tool use so impactful to test time scale? Like, why is the combination so much better? Take any, any role. You can make a decision when you are trying to make progress against a task as to, like, do I get external validation or do I sit and think really hard? Right. And usually you want to do like, one is more efficient than the other. And it's not always just sit in a vacuum and think really hard with what you know.

Brandon McKenzie (17:25)

Yeah, absolutely.

Eric Mitchell (17:26)

You can seek out sort of new inputs. Like, it doesn't have to be this closed system anymore. And I do feel like the closed system ness of the models is still sort of a limitation in some ways. Like, you're not, you're not necessarily like turning this. I mean, like, I think it'd be great if the model could control my computer for sure. But in some sense there's a reason we don't go hog wild and say, oh, yes, here's the keys to the kingdom. Have at it. There are still asymmetric costs to the time you can save and the types of errors you can make. And so we're trying to iteratively kind of deploy these things and try them out and figure out where are they reliable and where are they not? Because, yeah, like, if you did just let the model control your computer, it could do some cool stuff. Like, I have no doubt, but, you know, do I trust it to, like, respond to all of the, you know, random emails that Brandon sends me? Actually, maybe for that task it doesn't require that much intelligence, but, you know, more generally, like, do I, you know, do I trust it to do everything I'm doing? Like, you know, some things. And I'm sure, like, that set of things will be bigger tomorrow than it was yesterday. But yeah, I think part of this is like, we limit the affordances and keep it a little bit in the sandbox just out of caution so that you don't send some crazy email to your boss or delete all your texts or delete your hard drive or something?

Podcast Host (18:57)

Is there some sort of organizing mental model for the tasks that one can do with increasing intelligence, test time scaling and improved tool use? Right. Because I look at this and I'm like, okay, well you have complexity of task and you have time scale. Then you have the ability to come up with these RL Rewards and environments, right? Then you have usefulness. Maybe, of course, you have some intuition about diversity and generalization across the different things you can be doing. But it seems like a very large space and scaling like new gen RL is not. It's just not obvious, like, how. To me, it's not obvious how you do it or how you choose the path. Is there some sort of organizing framework that, you know, you guys have that you can share?

Eric Mitchell (19:46)

I mean, I don't know if there's like one organizing framework. I think there are a few like factors at least that I think about in like, the very, very grand scheme of things is like, how much, like, in order to solve this task, like, how much uncertainty with the environment do I have to, like, wrestle with? Like, for some things where it's like, this is a purely fat, like, who was the first president of the United States? There's zero environment I need to interact with to reach the answer to this question correctly. I just need to remember the answer and say the answer if I want you to write some code that solves some problem. Well, now I have to deal with a little bit of not purely internal model stuff, but also, okay, I need to execute the code. And that code execution environment is maybe more complicated than my model can memorize internally. So I have to do a little bit of writing code and then executing it and making sure it does what I thought it did and then testing it and then giving it to the user and things get like, the amount of that sort of stuff outside the model that you have to, like, you know, you can't just recall the answer and give it to the user. You have to, like, test something and, you know, run an experiment in the world and then wait for the result of that experiment. Like, the more you have to do that, the more uncertain the results of those experiments. In some sense, that's one of the core attributes of what makes the tasks hard. And I think another is how simulatable they are. Stuff that is really bottlenecked by time. The physical world is also just harder than stuff that we can simulate really. Well, coincidence that, you know, so many people are interested in coding and, you know, coding agents and things and that like, you know, robotics is hard and it, you know, it's, it's slower and, you know, I used to work on robotics and like, it's frustrating in a lot of ways. I think both this, like, how much of the external environment do you have to deal with and then, like, how much do you have to wrestle with the unavoidable slowness of the real world are too like dimensions that I sort of think about.

Interviewer/Co-host (21:53)

It's super interesting because if you look at historically some of these models, one of the things that I think has continued to be really impressive is the degree to which they're generalizable. And so I think when GitHub Copilot launched, it was on Codex, which was like a specialized code model. And then eventually that just got subsumed into these more general purpose models in terms of what a lot of people are actually using for coding related applications. How do you think about that in the context of things like robotics? There's like probably a dozen different robotics foundation model companies now. Do you think that eventually just merges into the work you're doing in terms of there's just these big general purpose models that can do all sorts of things, or do you think there's a lot of room for these standalone other types of models over time?

Brandon McKenzie (22:31)

I will say the one thing that's always struck me as kind of funny about us doing RL is that we don't yet do it on the most canonical RL task of robotics. And I personally don't see any reason why we couldn't have these be the same model. I think there are certain challenges with, I don't know, do you want your RL model to be able to generate an hour long movie for you natively as opposed to a tool call? That's where it's probably tricky to have. You have more conflict between having everything in the same set of weights. But certainly the things you see O3 already doing in terms of, you know, exploring a picture and things like that are kind of like early signs of something like an agent exploring like an external environment. So I don't think it sounds too far fetched to me.

Eric Mitchell (23:21)

Yeah, I mean, I think the thing that came up earlier of also the intelligence per cost thing, the real world is an interesting litmus test because at the end of the day there is a frame rate in the real world you need to live on. And it doesn't matter if you get the right answer after you think for two minutes. Like you know, the ball is coming at you now and you have to catch it. Gravity's not going to wait for you. So you, you. That's an extra constraint that we get to at least softly ignore when we're talking about these purely disembodied things.

Interviewer/Co-host (23:56)

That's kind of, it's kind of interesting though because really small brains are very good at that, you know, so you look At a frog, you know, you start looking at different organisms and you look at sort of relative compute and very simple systems are very good at that, ants. So I think that's kind of a fascinating question in terms of what's the baseline amount of capability that's actually needed for some of these real world tasks that are reasonably responsive in nature.

Brandon McKenzie (24:23)

It's really tricky with vision too. So our models have some, I think maybe famous edge cases of where they don't do the right thing. I think Eric probably knows where I'm going with this. I don't know if you ever asked our models to tell you what time it is on a clock. They really like the time 10:10.

Interviewer/Co-host (24:41)

So yeah, it's my favorite time too. So that's usually what I tell people.

Brandon McKenzie (24:44)

It's like over 90% or something like that of all clocks on the Internet are 1010. And it's because it looks like, I guess like a happy face and it looks nice. But anyways, what I'm getting at is our visual system was developed by interacting with the external world and having to be good at navigating things, avoiding predators. And our models have learned vision in a very different type of way. And I think we'll see a lot of really interesting things if we can get them to be kind of closing the loop by reducing their uncertainty by taking actions in the real world just as opposed to thinking about stuff.

Podcast Host (25:21)

Hey Eric, you brought up the idea of how what in the environment can be simulated, right. As an input as to how difficult will it be to improve on this as you get to long running tasks. Let's just take software engineering. There is a lot of interaction that is not just me committing code continually. It's like I'm going to talk to other people about the project. In which case you then need to deal with the problem of can you reasonably simulate how other people are going to interact with you on the project in an environment that seems really tricky, right? I'm not saying that, you know, O3 or whatever set of foundation models now doesn't have the intelligence to respond reasonably, but like how do you think about that simulation being true to life, as true to life true to the real world as you involve human beings in an environment.

Brandon McKenzie (26:16)

In theory, my spicy, I guess, take on that is like, I don't know if it's spicy, but O3 in some sense is already kind of simulating what it'd be like for a single person to do something with like a browser or something like that. And I don't know Train two of them together so that you have, you know, you have two people interacting with each other. And yeah, there's no reason you can't scale this up so that models are trained to be really good at cooperating with each other. I mean there's a lot of already existing literature on multi agent RL and yeah, if you want the model to be good at something like collaborating with a bunch of people, like maybe a not too bad starting point is making it good with collaborating with other models.

Eric Mitchell (26:57)

Man, someone should do that.

Brandon McKenzie (26:58)

Yeah, yeah, yeah, we should really start thinking about that, Eric.

Podcast Host (27:01)

I think it is, I think it's a little bit spicy because yes, the work is going on. It is interesting to hear you think that is a useful direction. I think lots of people would still like to believe. Not me. Like my comment was extra good on this pull request or whatever it is. Right.

Brandon McKenzie (27:17)

And okay, I could, I could sympathize with that. Sometimes I see our models training and I'm like, oh, what are you doing? You know, like you're taking forever to figure, figure this out. And I actually think it would be really fun if you could actually train models in an interactive way. Forget about just like a test time, but I think it'd be really neat to train them to do something like that. Be able to intervene when it makes sense. Yeah, just more me being able to tell the model to cut it out in the middle of its kind of chain of thought and it being able to learn from that on the fly I think would be great.

Eric Mitchell (27:46)

Yeah. I do think this is like the intersection of these two things where it's both like an a point of contact with the external environment that is like, can be very high uncertainty. Like humans can be very unpredictable in some cases and it's sort of limited by the tick of time in the real world. If you want to like, you know, deal with actual humans. Like humans have a fixed, you know, clock cycle, you know, in their, in their head. So yeah, I mean this is. If you, you know, if you want to like do this in the literal sense, it's hard. And so, you know, scaling it up and you know, making it work well is, you know, it's not obvious how to do this.

Brandon McKenzie (28:25)

Yeah, we are a super expensive toolk. You know, if you're a model, you can either ask me, you know, meatbag over here to, you know, help with something and I'll try to think really slowly. In the meantime, it could have like used browser and read like 100 papers on the topic and something like that. So it's. Yeah, how do you model the trade off there?

Eric Mitchell (28:42)

But the human part's important. I mean, I think in any research project, like, my interactions with Brandon are the hardest part of the project. You know, like, writing the code is. That's the easy part.

Podcast Host (28:50)

Well, and there's, there's some analog from self driving. Lot's going to say the, you know, hanging out with me every week is the hardest part of doing this podcast.

Interviewer/Co-host (28:58)

But it's my favorite part.

Brandon McKenzie (28:59)

Look at how healthy the relationship is. Eric, we need to learn from this.

Eric Mitchell (29:02)

No, we're honest. It's okay. We got to work through it.

Podcast Host (29:07)

In self driving, one of the, like, classically hard things to do was like, predict the human and the child and the dog, like agents in the environment versus, like, what the environment was. And so I think there's like, some analogy to be drawn there going back to just like, how you progress the O series of models from here. Is it a reasonable, like, assessment that some people have that the capabilities of the models are likely to advance in a spikier way because you're relying to some degree more on the creativity of research teams and like, making these environments and deciding, you know, how to create these evals versus, like, we're scaling up on existing data set in pre training. Is that a fair contrast?

Brandon McKenzie (29:59)

Spiky or like, what's the plot here? What's like the X axis and the.

Eric Mitchell (30:02)

Y domain is the X axis and Y is capability.

Podcast Host (30:06)

Yes. Because you're like, choosing what domains you are really creating this RL loop in.

Eric Mitchell (30:12)

I mean, I think this is a very reasonable hypothesis to hold. I think there is some counter evidence that I think should be factored into people's intuitions. Like, Sam tweeted an example of some creative writing from one of our models that I think was. I'm not an expert and I'm not going to say this is like, you know, publishable or like, groundbreaking, but I think it probably updated some people's intuitions on, like, what, you know, you can train a model to do really well. And so I think there is some structural reasons why you'll have some spikiness. Just because, like, as an organization, you have to decide, like, hey, we're going to prioritize, you know, XYZ stuff. And as the models get better, the surface area of stuff you could do with them grows faster than you can potentially say, hey, this is the niche we're going to carve out. We're going to try to do this really well. So I think there's some reason for spikiness. But I think some people will probably go too far with this in saying, oh yes, these models will only be really good at math and code and everything else is you can't get better at them. And I think that is probably not the right intuition to have.

Brandon McKenzie (31:34)

Yeah. And I think probably all like major AI labs right now have some partitioning between. Let's just define a bunch of data distributions we want our models to be good at and then just like throw data at them. And then another set of people and this same companies are probably thinking about how can you kind of lift all boats at once with some like, algorithmic thing change? And I think, yeah, we definitely have both those types of efforts at OpenAI. And I think especially on the data side, there are going to naturally be things that we have a lot more data of than others. But ideally, yeah, we have plenty of efforts that will not be so reliant on the exact subset of data we did RL on. And it'll generalize better.

Podcast Host (32:22)

I get pitched every week and I bet a lot does too. A company that wants to generate data for the labs in some way or it's access to human experts or whatever it is, but there's infinite variations of this. If you could wave a magic wand and have a perfect set of data, what would it be that, you know, would advance model quality? Today?

Eric Mitchell (32:49)

This is a dodge, but like uncontaminated evals, always super valuable. And that's data. And I mean, yeah, like you want, you know, good data to train on and that's of course valuable for making the model better. But I think it is often neglected how also important it is to have high quality data, which is like a different definition of high quality when it comes to an evaluation. But yeah, the eval side is often just as important because you need to measure stuff. And as you know, from trying to hire people or whatever, evaluating the capabilities of a general capable agent is really hard to do in a rigorous way. So, yeah, I think evals are a.

Brandon McKenzie (33:37)

Little underappreciated, but it's true, evals are. I mean, especially with some of our recent models where we've kind of run out of reliable evals to track because they kind of just solved a few of those. But on the, on the training side, I think it's always valuable to have training data that is kind of at the next frontier of model capabilities. I mean, I think a lot of the things that O3 and 04 mini can already do those types of tasks like basic tool Use. We probably aren't, you know, super in the need for new data like that, but I think it'd be hard to say no to a data set that's like bunch of multi turn user interactions and some code base that's like a million lines of code that is like a two week research task of adding some new feature to it that requires multiple pull requests. I mean, I mean something that was super high quality and has a ton of supervision signals for us to learn from. Yeah, I think that would be awesome to have. I definitely wouldn't turn that down.

Podcast Host (34:39)

You play with the models all the time. I assume a lot more than average humans do. What do you do with reasoning models that you think other people don't do enough of yet?

Eric Mitchell (34:47)

Send the same prompt many, many, many times to the model and get an intuition for the distribution of responses you can get. I have seen. It drives me absolutely mad when people do these comparisons on Twitter or wherever and they're like, oh, I put the same prompt into blah blah and blah blah. And this one was so much better. It's like, dude, you like, like, I mean something we talked about a bit when we were launching is like, yeah, O3 can do really cool things. Like when it chains together a lot of tool calls and then like sometimes for the same prompt it won't have that, you know, moment of magic or it will, you know, just take a little. It'll do a little less work for you. And so yeah, though like the peak performance is really impressive. But there is a distribution of behavior and I think people often don't appreciate that. There is this distribution of outcomes when you put the same prompt in. And getting intuition about that is useful.

Podcast Host (35:51)

So as an end user I do this and I also have a feature request for your friends in the product. Org. I'll ask Oliver or something, but it's just I want a button, assuming my rate limits or whatever support it. I want to run the prompt automatically like 100 times every time, even if it's really expensive. And then I want the model to rank them and just give me the top one or two Interesting. And just let it be expensive or.

Interviewer/Co-host (36:16)

A synthesis across it. Right. You could also synthesize the output and just see if there's some. Although maybe you're then reverting to the mean in some sense relative to that distribution or something. But it seems kind of interesting.

Podcast Host (36:26)

Yeah, maybe there's a good infrastructure reason you guys aren't giving us that button.

Brandon McKenzie (36:30)

Well, it's expensive, but there are. I think it's a great suggestion.

Eric Mitchell (36:34)

Yeah, Yeah, I think it's a great suggestion.

Brandon McKenzie (36:36)

How much would you pay for that? A lot.

Podcast Host (36:38)

But I'm a price insensitive user of AI.

Brandon McKenzie (36:40)

Yeah, I see.

Eric Mitchell (36:41)

Perfect.

Interviewer/Co-host (36:45)

Should have Sarah Tier as one of your tiers.

Eric Mitchell (36:47)

Exactly, exactly.

Brandon McKenzie (36:51)

I really like sending prompts to our models that are kind of at the edge of what I expect them to be able to do. Just kind of for funsies. Like a lot of the times before I'm about to do some programming tasks, I will just kind of ask the model to go see if it can figure it out. A lot of times with no hope of it being able to do it. And indeed sometimes it comes back and I just am pretty. I'm a disappointed father. But other times it does it and it's amazing and it saves me like tons of time. So I kind of use our model as almost like a background queue of work where I just like shoot off tasks to them. And sometimes those will stick and sometimes they won't. But in either case, like it's always a good outcome if something good happens.

Interviewer/Co-host (37:31)

That's cool. Yeah, I do that just to feel better about myself when it doesn't work.

Brandon McKenzie (37:36)

Yeah, I'm still providing value.

Interviewer/Co-host (37:40)

When it works, I feel even worse about myself. So it's very hit or miss.

Eric Mitchell (37:44)

Yeah.

Interviewer/Co-host (37:44)

There are some differences in terms of how some of these models are trained or RL'd or effectively produced. What are some of the differences in terms of process, in terms of how you approach the O series of models versus other things that have been done at OpenAI in the past.

Brandon McKenzie (37:57)

The tools stuff was quite the experience working at a large scale setting. You can imagine if you're doing async RL with a bunch of tools that you're just adding more and more failure points to your infrastructure. And what you do when things that get w fail is pretty interesting engineering problem, but also an RL ML problem too. Because if you're, I don't know if your Python tool, it goes down in the middle of the run, what do you do? Do you stop the run? Probably not. That's probably not the most sane thing to do with that much compute. So the question is, how do you handle that gracefully and not hurt the capabilities of the model like as an unintended consequence. So there's been a lot of learnings like that of how you deal with like huge infrastructure that's asynchronous for RL.

Eric Mitchell (38:45)

V RL is hard.

Podcast Host (38:47)

This has been great, guys. Thank you.

Interviewer/Co-host (38:49)

Yeah, thanks so much for coming on yeah, thanks.

Brandon McKenzie (38:51)

It was fun.

Eric Mitchell (38:51)

Thanks for having us.

Podcast Host (38:54)

Find us on Twitter nopriorspod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week and sign up for emails or find transcripts for every episode@no-priors.com.