← Big Technology Podcast
Big Technology Podcast
OpenAI Chief Research Officer Mark Chen: GPT 4.5 is Live and Scaling Isn’t Dead
00:24:55
Loading summary
Transcript47 lines
- [00:00]
Big Technology Podcast Host
OpenAI Chief Research Officer Mark Chen is here to talk about the release of GPT 4.5, the company's largest and best model yet, which is coming out today. We'll dive in right after this.
- [00:13]
Mark Chen
Will AI improve our lives or exterminate the species? What would it take to abolish poverty? Are you eating enough fermented foods? These are some of the questions we tackled recently on the Next Big Idea. I'm Rufus Griscom and every week I sit down with the world's leading thinkers for in depth conversations that will help you live, work and play smarter. Follow the Next Big Idea wherever you get your podcasts.
- [00:43]
Big Technology Podcast Host
Welcome to Big Technology Podcast, a show for cool headed nuanced conversation of the tech world and beyond. We're joined today by Mark Chen, the Chief research officer at OpenAI, who's here to talk about the company's newest release, GPT 4.5. Yes, it's finally here and it is debuting today. Mark, great to see you. Welcome to the show.
- [01:02]
Mark Chen
Thank you so much for having me on.
- [01:04]
Big Technology Podcast Host
Thanks for being here. This is in four and a half years of the show, our first OpenAI interview, so hopefully the first of many. We appreciate you jumping into the water like this. And it's on big news with the release of GPT 4.5.
- [01:18]
Mark Chen
Yeah, so GPT 4.5, really, it signifies the latest milestone in our predictable scaling paradigm. So you know, previous models that have fit this Paradigm have been GPT3, 3.5, 4, and now this is the latest thing. It signifies an order of magnitude improvement over the last models, kind of commensurate with the jump from 3.5 to 4.
- [01:41]
Big Technology Podcast Host
I think the question that most of our listeners are going to be asking, and certainly we've asked on our show in the past couple months, is why isn't this GPT5? I mean, what is it going to take to get to GPT5?
- [01:54]
Mark Chen
Yeah, well, I think GPT5, whenever we make these naming decisions, we try to keep with a sense of what the trends are. So again, when it comes to predictable scaling going from 3 to 3.5, you can kind of predict out what an order of magnitude of improvements in amount of compute that you train the model with in terms of efficiency improvements will buy you. And we find this model kind of aligns with what 4.5 would be. So we want to name it what it is.
- [02:27]
Big Technology Podcast Host
Okay, but there's been so much talk about when GPT5 is going to come. Correct me if I'm wrong, but I think there's been a longer wait between GPT 4 and 4.5 than there has been between, let's say, GPT 3.5 and 4. And I don't know, is this like, because we're seeing a lot of hype from OpenAI folks on Twitter about what's coming next or maybe this is probably, it probably is the most impatient industry in the world and the most impatient users in the world, but it seems to me like the expectations for GPT5 are built up pretty high. And so I'm curious from like your perspective, do you think it's going to be hard to meet those expectations whenever that GPT5 model does come out?
- [03:09]
Mark Chen
Well, I don't think so. And one of the fundamental reasons is because we now have two different axes on which we can scale. Right. So GPT 4.5, this is our latest scaling experiment along the axis of unsupervised learning. But there's also reasoning. And when you ask about why there seems to be a little bit bigger of a gap in release time between 4 and 4.5, we've been really largely focused on developing the reasoning paradigm as well. So I think our research program is really an exploratory research program. We're looking into all avenues of how we can scale our models. And over the last one and a half, two years, we've really found a new, very exciting paradigm through reasoning, which we're also scaling. And So I think GPT5 really could be the culmination of a lot of these things coming together.
- [04:02]
Big Technology Podcast Host
Okay, so you talk about how there's been a lot of work toward reasoning. We of course have seen that with A1, there's a lot of buzz about Deep Seq and now we're talking about again like one of the more traditional scaled up large language models with GPT 4.5. So the big question here, I think that was on a lot of people's mind when it came to this upcoming release we thought was going to be 4.55. Anyway, it doesn't matter. The big question is can AI models continue to scale when you add more compute, more data and more power to them? It seems like you have an answer to this. So I'm curious to hear your point of view on whether what you've learned about the scaling wall, given your development of this model, and whether we're going to hit it, whether we're already seeing some diminishing returns from scaling.
- [04:53]
Mark Chen
Yeah, I really kind of have a different framing around scaling. So when it comes to unsupervised learning. Right. You Want to put more ingredients like compute algorithmic, algorithmic efficiencies and more data. GPT 4.5 really is proof that we can continue the scaling paradigm. And this paradigm is not the antithesis of reasoning as well. You need knowledge in order to build reasoning on top of a model. Can't go in blind and just learn reasoning from scratch. So we find these two paradigms to be fairly complementary and we think they have feedback loops on each other. So yeah, GPT 4.5, again, it is smart in different ways from the ways that reasoning models are smart. When you look at the model today, it has a lot more world knowledge. When we look at kind of comparisons against GPT 4.0, you see that everyday use cases, people prefer it by a margin of 60% for actually productivity and knowledge work against GPT 4.0, there's almost like a 70% preference rate. So people are really responding to this model and it's this knowledge that we can leverage for our reasoning models in the future.
- [06:08]
Big Technology Podcast Host
So what are the examples? Like you talk about everyday knowledge work, what are some of the examples that you would use GPT 4.5 for that you would prefer it over a reasoning model?
- [06:18]
Mark Chen
Yeah, so I wouldn't say like it's a different profile from a reasoning model. Right. So with a larger model, what you're doing is it takes more time to process and think through the query, but it's also giving you an immediate response back. So this is very similar to what a GPT4 would have done for you. Whereas I think with something like O, you get a model where you give a query and it can think for several minutes. I think these are fundamentally kind of different trade offs. You have a model that immediately comes back to you, doesn't do much thinking, but comes up with a better answer versus a model that thinks for a while and then comes up with an answer. We find that in a lot of areas, like creative writing for instance. Again, this is stuff that we want to test over the next one or two months, but we find that there are areas like creative writing where this model outshines reasoning models.
- [07:21]
Big Technology Podcast Host
Okay, so writing any other use cases?
- [07:24]
Mark Chen
Yeah, so there's writing, I think some coding use cases as well. We also find that kind of like there are some particular kind of scientific domains where this outshines in terms of the amount of knowledge that it can display.
- [07:40]
Big Technology Podcast Host
Okay, and I'm going to come back to benchmarks in a moment, but I want to keep on this scaling question because I think there's been a lot of conversation about it in public. And it's great to be speaking with you from OpenAI to sort of get to the bottom of, of what's happening. So the first is the question that folks have is, do you end up at this size? And you don't talk about the size of the models, which is, you know, which is fair, but they're big, right? This is the largest model that OpenAI has ever released, GPT 4.5. So I'm actually curious to hear, at this size, does adding, you know, similar amounts of compute, simpler amounts of data get you the same returns that you did, or are we already starting to see the returns of adding these resources tail off?
- [08:25]
Mark Chen
No, we are seeing the same returns. And I do want to stress that GPT 4.5 is that next point on this unsupervised learning paradigm. And we're very rigorous about how we do this. We make projections based on all the models we've trained before on what performance to expect. And in this case, we put together the scaling machinery and this is the point that lies at that next order of magnitude.
- [08:49]
Big Technology Podcast Host
So what's it been like getting here? I mean, again, we talked. Okay, so there was a period of time that was longer than the last interval, and part of that was focused on reasoning. But there's also been some reports that OpenAI's had to start and stop a couple times to get this to work. And it really had to fight through some thorny issues to get it to be this step change, as you're saying. So talk a little bit about the process and maybe you can confirm or deny some of the things that we've heard about having to start and stop again and retrain to get here, actually.
- [09:22]
Mark Chen
So I think it's interesting that this is a point that's attributed to this model because actually, in developing all of our foundation models, they are all experiments. I think running all the foundation models oftentimes does involve stopping at certain parts, kind of analyzing what's going on, and then restarting the runs. I don't think that this is a characteristic of GPT 4.5. It's something that we've done with GPT 4 with O series models, and they are largely experiments. We want to go in, diagnose them in the middle, and if we want to make some interventions, we should make interventions. But I wouldn't characterize this as kind of something that we do for GPT 4.5 that we don't do for other models.
- [10:07]
Big Technology Podcast Host
We've already talked a little bit about reasoning versus these traditional GPT models. But it makes me think of Deep seq and I think you already gave a pretty compelling answer as to what you would use one of these models for versus a reasoning model. But there's another thing that deepseek did that is worth discussing, which is that they made their models much more efficient. And it's kind of interesting when I talk to you about. All right, so you need data, you need computing, you need power, you're like, yeah, and you need model optimizations, which is something that people often overlook. And just going back to Deep SEQ for a moment, the model optimization, the fact that they went from basically querying the entire knowledge base to mixture of experts, where they were able to sort of route the queries to certain parts of the model instead of lighting it all up, is credited with helping them get more efficient. So I just want to turn it over to you without commenting on what they did, or if you can, if you want. But I'm actually More curious what OpenAI is doing on that front and what sort of whether you did similar optimizations with GPT 4.5 and are you able to run these large models more efficiently and if so, how?
- [11:14]
Mark Chen
Yeah, so I would say kind of the process of making a model efficient to serve, I often see as fairly decoupled from developing the core capability of the model. And we see a lot of work being done on the inference stack. I think that's something that Deep SEQ did very well and it's also something that we push on a lot. We care about serving these models at cheap cost to all users and we push on that quite a bit. So I think this is irrespective of GPT4 or reasoning models. We're always applying that pressure to be able to inference more cheaply. And. And I think we've done a good job of that over time. The costs have dropped many orders of magnitude since we first launched GPT4.
- [11:57]
Big Technology Podcast Host
And so are there like, I mean, maybe tell me if this is too elementary, but the move towards, for instance, mixture of experts, is that more of a reasoning thing or can you apply that in GPT 4.5?
- [12:09]
Mark Chen
So that is an architectural element of language models. I think pretty much all large language models today use utilized mixture of experts and it's something that applies equally to efficiency wins in foundation models like GPT 4.4.5, as it does to reasoning models.
- [12:29]
Big Technology Podcast Host
So you were able to use that here as well? Basically.
- [12:32]
Mark Chen
No, we've definitely explored mixture of experts as well as a number of other architectural improvements in GPU 4.5.
- [12:39]
Big Technology Podcast Host
Okay, great. So we have a discord with some members of the big technology listeners and reader group and you know, a theme that's come up recently. It's kind of interesting to be talking with you right now about an extremely large model, because a theme that they can't stop talking about the people in discord is just that how small and niche models to them are going to, you know, potentially be the future. I'll just read you one comment that we had over the past few days. For me, the future is very much aligned with niche models existing in workflows and less so of these general purpose God models. So clearly OpenAI has a different thesis here and I am curious to hear your perspective on what we get with the big models versus the niche models. And do you see them in competition or as compliments? Help us think through that.
- [13:23]
Mark Chen
Yeah, yeah. So I think one important thing is we also serve models that are smaller. Right. Like we serve our flagship frontier models, but we also serve mini models, right, which are cost efficient ways that you can access the capabilities or fairly close to frontier capabilities for much lower cost. Right. And we think that's an important part of this comprehensive portfolio here. Fundamentally at OpenAI, though, we're in the business of advancing the frontier of intelligence and that involves developing the best models that we can. And I think really kind of what we're motivated by is really pushing that out as much as possible. We think there's always going to be use cases at the frontiers of intelligence. We think that going from 99.9 percentile in mathematics to the best in the world in mathematics, that difference means something to us. I think what the best human scientists can discover is tangibly different from what you or I can discover. So we're motivated by pushing the intelligence frontier as far as possible. And at the same time, we want to make these capabilities cheaper and more cost effective for everyone. So we don't think the niche models will go away. We want to build these foundation models and also figure out how to deliver these capabilities at cost over time. So that's always been our philosophy. There's always going to be some juice there in those last bits of intelligence.
- [14:46]
Big Technology Podcast Host
Yeah. So let's talk about that because we have a debate on the show often what matters more, the products or the model? I'm on Team Model. We have Ranjan Roy who comes on on Fridays. He's team product. He's basically like, just take what you have now and prioritize it. And I say, well, you could probably do more with a better model. But I have to be honest, I'm kind of at a loss for words sometimes about what that getting from that 99th percentile in math to the best in world in math will do. So actually I'm curious to hear your answer on this one. What does building the best model in the world do that otherwise?
- [15:21]
Mark Chen
100%. And I think really it signals a shift. Right. I think if you just think about, hey, you take the current models and you build the best surface for them, that's certainly something you should always be doing and exploring that exercise. I think three years ago that looked like chat, right? We launched ChatGPT and today when you take the best models and the best capabilities, I think it looks a little bit more like agents. And I think reasoning and agents, they're very, very much coupled. When you think about what makes a good agent, it's something that you can kind of sit back, let it do its own thing, and you're fairly confident you'll come back with something that you want. And I think reasoning is the engine that powers that. You have the model go and try something out and if it can't succeed on the first try, it should be able to be like, oh well, why didn't I succeed and what's a better approach for me to do so? I think very much the capabilities are always changing and the surface is always changing as a response and we're always exploring what the best surface for the current capabilities looks like. But just to hammer home, I'm on your team here.
- [16:25]
Big Technology Podcast Host
Yeah, but again, just to hammer home on this, what does that improvement in model get you? What do you think that it will enable?
- [16:31]
Mark Chen
Yeah, yeah. So I mean, I think agents of all forms, when you look at stuff like deep research, for instance, Right. It gives you the ability to essentially kind of get a fully formed report on any single topic that you might be interested in. Right. I've used it to even put together hour long talks and it goes and really kind of like synthesizes all the information out there and really organizes it, comes up with lessons, allows you to do deep discovery, it allows you to dig into almost any topic that you're interested in. So I feel like just the amount of information and synthesis that's available to you now is just really rapidly evolving.
- [17:15]
Big Technology Podcast Host
So basically it's not as simple as just go make deep research better with the product, with the model you have now. Am I reading between the lines the right way, saying that what you're expressing here is that if you make the model better, then the product is going to get better inherently. Take deep research, for instance.
- [17:32]
Mark Chen
100%. 100%. Yeah. And that's something that is not enabled unless you have models of a certain level of capability, both in reasoning and in the foundational, unsupervised learning sense.
- [17:42]
Big Technology Podcast Host
Okay. You know, it's interesting, I guess like this one question I've had in the back of my mind is, and I'm just going to ask it to you again just so I'm sure I'm clear on it, is my view, maybe erroneously, was that we were just going to or your industry was just going to move from these massive models to the massive models with reasoning. But you're actually saying that there's a dual track here.
- [18:04]
Mark Chen
Yeah, yeah. So I think we're always pushing the frontier. Right. And we, I think even since five, six years ago, the prevailing way to do that was to up the scale. We've been upping the scale in unsupervised learning. We've been upping the scale and reasoning. But at the same time, you care about serving mini models, you care about serving models that are cost effective, that can deliver capabilities at a cheaper cost that will often be sufficient for a lot of use cases. The mission isn't just about pushing the biggest, most costly models. It's about having that and also a portfolio of models that people can use cheaply for their use cases.
- [18:43]
Big Technology Podcast Host
Okay, so let's quickly talk before we leave about the upgrades that you're seeing in 4.5 compared to 4. So I'm curious, like if you can just run us through very high level, the benchmarks it hits versus the benchmarks of the previous models and then I'll just throw a double question in here. Yeah, I've already read your blog post and so I have an idea of what's coming. By the way, we're going to release this just as the news is released. So it seems like you're also saying, making a statement in some ways saying like, yes, we have the traditional benchmarks, but we also need to measure how this model works with EQ as opposed to just pure intelligence. So yeah, just hit us with the benchmark improvements and then why you think that it's important for us to look at both of these in conjunction.
- [19:27]
Mark Chen
So I mean, along all traditional metrics, like things like gpqa, ami, the traditional kind of benchmarks that we track, this does signify an order of magnitude about at the same level of jump from 3.5 to 4. There's a kind of interesting focus here. Also on, I would say more vibe space benchmarks. I think that's actually important to highlight because every single time we've launched a model, there is a discovery process of what the interesting use cases out there are going to be. We notice here it's actually a much more emotionally intelligent model. You can kind of see examples in the blog post later today, but how it responds to queries about a hard situation or advice in a particular difficult situation, it responds more emotionally intelligent. I think there's also just kind of like you can kind of see. This may be a kind of silly example, but if you ask any of the previous models to create ASCII art for you, actually they mostly just fall down. This one can do it almost flawless pretty well. And so there's just so many kind of footprints of improved capabilities. And I think things like creative writing will showcase this.
- [20:44]
Big Technology Podcast Host
One of the things that I think I picked up in the examples that you've given so far is that it doesn't seem like it feels the need to write a thesis for every response. One user was like, I'm having a hard time. And it actually succinctly wrote as if a human would, as opposed to maybe the traditional here's three paragraphs of self care routine you can do for yourself.
- [21:06]
Mark Chen
Exactly, yeah. And that speaks to the emotional intelligence. It's not like, oh, I see that you're feeling bad. Here are five ways you could feel better. It just doesn't feel like a grounded kind of compassionate response. And here you just get something that's direct to the point and really invites the user to say more.
- [21:21]
Big Technology Podcast Host
So I think there's going to be a criticism. I'm anticipating it. And let's talk about it right now that people will say, okay, OpenAI was talking about these traditional benchmarks. Now it's talking about emotional intelligence. It's shifting the goalposts and it wants us to pay attention to something else. What's your response there?
- [21:38]
Mark Chen
Well, I really don't think that the accurate characterization is that it doesn't hit the benchmarks that we expect it to. So when you look at kind of the development of 3 to 3 point to 4 to 4.5, this does hit the benchmarks that we expect. And I think the main thing is it's all about use case discovery every time you put a new model out there. And in many senses GPT4 is already very smart. And when we were putting that this parallels kind of like when we were putting GPT4 out, it's like we saw it hit all the right benchmarks that we expected to. But what are users going to resonate with? That was the key question. And I think that's the question that we're asking today with GPT 4.5 as well. And we're inviting people to be like, hey, you know, we did some early explorations, we see that it's more emotionally intelligent, you know, we see that it's a better creative writer. But what do you see here?
- [22:30]
Big Technology Podcast Host
Yep. All right, Mark, so I've been seeing you and we mentioned this before we started recording. I've been seeing you in all the OpenAI videos about every release. So first of all, great to speak to you live. But also over the past year we've seen a lot of exodus out of OpenAI. Maybe the media plays it up too much. Probably we do. But I am kind of curious what it's like working within OpenAI and how you see the talent bench inside the company. You recently became Chief Research Officer just a few months ago. And now look, we have a new foundational model. So just give us a sense as to what the talent situation is inside the government.
- [23:03]
Mark Chen
It's still, I think, the most world class AI organization. I would say that there's a separation between the talent bar at OpenAI and any other firm out there. When it comes to people leaving the AI landscape, it changes a lot, probably more so than any other field out there. The field three months ago looks different from the field three months before that. And I think it's kind of just natural in the development of AI that some people will have their own theses about here's the way I want to develop AI and go try it their own way. I think that's healthy. And it also gives an opportunity for people internally to shine. And we've never had a shortage of people internally who are willing to step up. And we've seen that a lot. And I really just love the bench that we have here.
- [23:47]
Big Technology Podcast Host
Very cool. All right, folks, GPT 4.5 is out today for OpenAI Pro users. Next week it's coming out for Plus Team Enterprise and Eduardo. Mark, great to see you. Thank you again for spending time. You're about to go and do the live stream, so I'm very grateful that you spent the time with me today. Thank you so much.
- [24:06]
Mark Chen
I really appreciate your time too.
- [24:08]
Big Technology Podcast Host
Thanks for having me. Well, let's do it again soon. And folks, so we shouted out the Ranjan and I argument. We'll go into that and more. Everything we can share about GPT 4.5 coming up tomorrow on the Friday show. Thanks for listening. Thanks again to Mark and OpenAI for the interview. And we'll see you next time on Big Technology Podcast.