← Latent Space: The AI Engineer Podcast
Loading summary
Transcript148 lines
- [00:00]
Alex Lukaska
Okay, so I think we're at this special time now where at least in some directions, AI has become superhuman, at least on certain tasks. And that's what led to these recent papers that resolved a problem that was puzzling. Physicists, experts in the field for over a year and they weren't able to resolve it, and AI was able to do it very quickly. So I think that's a certain milestone that we've passed. Glad that you guys are bringing attention to this because I think maybe for the average person on the street who doesn't care about theoretical physics, this is not very noticeable. But I think it's a very profound change and we've really passed some kind of a threshold.
- [00:41]
Brandon
Welcome to the A for Science podcast, part of Lean Space Network. I'm Brandon. I develop RNA therapeutics using AI at Atomic AI. I'm joined by my co host, RJ Honicke, CTO and founder of Mirror Omics. Yeah, it's a pleasure to introduce Alex Lukaska, professor at Vanderbilt University and fellow at OpenAI. He has. For a young researcher, he has quite a storied background. Amongst other things, he's the winner of the 2024 new Verizon's Breakthrough Prize. It's the call it the Oscars for science. I asked ChatGPT is this the most prestigious award someone of his career could win? And it recommended a second one called the IUPAP Award, which turns out he had also won. Anyway, right now he's having fun at OpenAI, doing some really cool research of pushing the foundation of theoretical physics using GPT models.
- [01:39]
Alex Lukaska
A pleasure to be here. The one message I wanted to convey is that I think we're on this trajectory, which I personally find very surprising and kind of surreal, but also amazing. Where I would say a little over a year ago, AI was very useful for email, but not the kind of work that I do that I consider important theoretical physics calculations. I thought, oh, that's special, much harder than email and AI is not going to be able to do that. And then there were a series of developments that came in rapid succession that completely changed my mind. And I can walk you through some of these examples. Specifically in particular, ChatGPT03 was the first really strong reasoning model that could do actual math that was useful for my research and could save me a lot of time. That's when I started to really pay attention and use it a lot more. And I thought, wow, this is a great tool. I gotta get ahead of this and learn how to integrate it into my workflow. Then when GPT5 came out, it was able to reproduce one of my best papers that took me a very long time to come up with in like 30 minutes. And that's when I really became AI pill. I thought, oh, my God, this changes everything. It's the most important discovery in my lifetime. It's going to affect everything about how we do research. And frankly, a lot of my colleagues, I would go around telling them this, Pay attention. Yeah, I was getting lots of different reactions, but I think people weren't quite getting it. But I talked to OpenAI, they were also really excited. And I thought, I don't know that much about AI, but I have to get in on this and to understand that this is happening and not be a part of it is a huge mistake. So I have to go to OpenAI. So I was on sabbatical. It was very easy to come here and join the company. And then it just kept ramping up even beyond that and to the point where now I think most of my senior colleagues in physics are aware of where things are headed and they're all getting on board. So, yeah, I think that's an awesome story.
- [03:43]
Brandon
Sorry, I was going to say, I find it really funny, that story because it almost. It reminds me of a lot of different people who had the same realization with Codex, starting sometime last fall especially, it just took off and a bunch of people are like, even like Andrej Karfathy went from, oh, man, this is 20% of my work. It's kind of a nice assistant to, oh, crap. What just happened?
- [04:07]
Alex Lukaska
Well, yeah, in August, actually, I remember when GPT5 came out. At that point I was really following AI pretty closely. And I think on Twitter, the reception was lukewarm. A lot of people like, well, we expected a lot more. And it's not better to write an email. And I remember thinking, well, okay, GPT3 could write email. How much better can it get at writing email? That's not the point. But at the science frontier, the capabilities were really taking off.
- [04:33]
RJ Honicke
Yeah, there was a lot of attention, I think, paid even to O3, but then.
- [04:39]
Alex Lukaska
But presumably GPT5 was a huge jump. And I think 5.4 is also a huge jump. I don't know how noticeable it is on the outside, although I did hear some. I saw some chatter online. People are running these independent benchmarks which do show this. So I think people are realizing. And also, anyway, in practice, researchers are now all over AI using it. And I'm getting inbounds all the time because I'm the resident scientist doing physics at OpenAI. And so everybody is sending me papers, chats, like, oh, my God, this happened. I got one just this week. Somebody said Codex just wrote up a simulation of the SYK model. This is, like, a very technical thing in quantum mechanics and gravity. And, like, yeah, a lot of research groups have been trying to run this simulation, and it couldn't do it. And Codex did it in 10 minutes
- [05:28]
RJ Honicke
just because setting it up was so hard.
- [05:30]
Alex Lukaska
Well, I think partly it's because of the Venn diagram, where you look at the people who have the physics knowledge and the people who have the top coding skills, and maybe the overlap is not that large, although I think it's been growing. But I think in this example, there are a lot of really good people in physics with coding skills who've been trying to simulate these things. So I think Codex is just really good now.
- [05:53]
RJ Honicke
Okay.
- [05:53]
Alex Lukaska
Yeah.
- [05:54]
RJ Honicke
Nice.
- [05:55]
Alex Lukaska
Okay. So I think we're at this special time now where, at least in some directions, AI has become superhuman, at least on certain tasks. And that's what led to these recent papers, which maybe we should talk about, that resolved a problem that was puzzling physicists for experts in the field for over a year, and they weren't able to resolve it, and AI was able to do it very quickly. So I think that's a certain milestone that we've passed. And I'm glad that you guys are bringing attention to this, because I think maybe for the average person on the street who doesn't care about theoretical physics, this is not very noticeable. But I think it's a very profound change, and we've really passed some kind
- [06:39]
Brandon
of threshold, specifically focus on the Gluon paper and the physics part, and we can get to the AI part later.
- [06:46]
Alex Lukaska
Okay. So in physics, there are two basic principles of nature that we think every law should respect or every theory should respect. On the one hand, there's the principle of relativity, which at some very high level, declares there's an absolute law that cannot be broken, which is that you cannot transmit information faster than the speed of light. But then there's another principle, which is the uncertainty principle that underlies quantum mechanics, which says that everything's a little fuzzy. You know, position, velocity. There's a little fuzziness to that. And so you can see immediately at this level of description already there's a tension between these two principles, because one is an absolute law declaring, you cannot go fastness be light, and the other one is saying it's a little bit fuzzy. And this is just to give a sense of how when you try to write down these principles in mathematics, the equations don't really play nicely with each other. And so it's been a real struggle to come up with physical theories that can reconcile simultaneously both principles to describe the physical world around us. And I would say that the great achievement of 20th century physics, which is really one of the greatest triumphs in human thought, as far as I'm concerned, is the elaboration of this framework called quantum field theory, which is a general framework that can describe the physical forces of nature in a way that it accommodates both of these principles. And in quantum field theory, which is our best theory to date, obviously it gets a little bit technical, but again, try to keep it pretty high level. What you're trying to compute or describe are the probabilities for certain events to occur. Because you're in this quantum mechanical setting, you can't say with certainty what's going to happen when you have a certain experiment. But you want to predict probability distributions. And in quantum mechanics, probability distributions are obtained by squaring certain complex quantities. And by complex, I don't mean complicated. I mean they're not real numbers. They're real plus symmetry numbers, which we call quantum amplitudes. So the goal of a theory is to predict quantum amplitudes, which are these objects, complex objects, that square to quantum probabilities. And that's the most you can say about the outcome of an experiment. And these quantum amplitudes in particular, there's a variety of them called scattering amplitudes, which describe the following scenario. Suppose you have a bunch of particles that you throw at one another. This is what happens in particle colliders like the LHC at CERN in Geneva. You take a bunch of particles, you smash them together, stuff happens, they interact via the physical laws of nature. Various processes occur, and then other particles come out as a result, at the end of the interaction. And so scattering amplitude is the object that describes the probability for a particular type of interaction. We have some particles coming in with some energies and momenta, and some other particles coming out with other energies and momentum. And so these scattering amplitudes, they're functions of all the data describing the particles coming in and the particles coming out. So in general, you can have arbitrarily many particles involved in an interaction. And this is one of the hallmarks of quantum field theory, that particles can be destroyed. So you don't have the same number of particles at the end necessarily as you had in the beginning. Particles can be created. Lots of things can happen. And in general, you want to describe all the possibilities. And so you want to have an amplitude for an arbitrary number n of particles. So that's called an endpoint amplitude, because there's n particles coming in and out. And it turns out in quantum field theory that if you have a particular force and you're able to compute the endpoint amplitudes, these functions of the, and parameters of the functions that square to the probabilities, then you know everything about the theory, more or less. There's always an asterisk, but it's basically the entire content of the theory.
- [11:05]
RJ Honicke
So if you have a theory that tells you any number of particles come in and go out, then I can say I can declare anything about that system.
- [11:14]
Alex Lukaska
Exactly. Then you know everything. And importantly, these amplitudes, they're not just numbers, their functions, because the probabilities that they compute depend on how much energy do the particles have, what are their momenta. And also, a particle has something called a lot of particles. Like the photon, which is the particle of light, has a polarization. So when you look at the surface of a lake and you have polarized sunglasses, and you turn your head, you can see more or less sunlight reflected off of the lake. And that's because a photon, which you can think of as a little particle of light, as it propagates, it carries a little arrow perpendicular to the direction of propagation, which is called the polarization. And this polarization has a direction. And sunglasses can selectively let in light with one polarization and not the other. And this polarization actually is light travels. It can rotate, it can wind, it can do its own thing. And in general, if it winds in the right handed way, so as the particle travels, if the polarization winds to the right, we call that a positive helicity or a right handed polarization. And if it winds in the other direction, we call that left handed helicity or negative helicity. So in general, these amplitudes, which are the fundamental object in quantum field theory that you want to contain all the information there is to know about physical forces, these amplitudes depend on not just the energies and momenta, but also the polarizations. Now, I've told you about how there's two basic principles of nature, relativity and quantum mechanics. They come together in this framework, quantum field theory. And I keep talking about forces. So there's four fundamental forces of nature. There's electromagnetism, which is responsible for basically the properties of atomic elements in the periodic table, and therefore chemistry and biology. And everything that you see, touch, feel, pretty much is all due to electromagnetism. Textures, colors. And this force is mediated by the photon, which is the particle of light. That one is the most familiar to us. Then there's gravity, which is another force that we feel very much because it keeps us to the ground. And then there's two nuclear forces, the weak and the strong nuclear force, which we don't really notice directly in our daily lives. But the weak nuclear force is responsible for radioactive decay and other such processes. And the strong force, which is the strongest of them all is what binds the nucleus together. So you learn in high school that, like, charges repel. But if so, then why do protons stick together inside the nucleus of the atom? They should repeal one another. And indeed that's the case. But if you bring it really close, then the strong force kicks in and overwhelms the relatively weaker electromagnetic force. So the strong force is mediated by the exchange of the particles of the strong force, which are called gluons, because they're what glues together the nucleus of the atom. So gluons are the particles of the strong force. And gluing gravity is mediated by gravitons.
- [14:28]
Brandon
I think the glue on paper, I think, was sort of maybe the starting point for this, maybe not. But the glue on paper had a really specific result, right?
- [14:38]
Alex Lukaska
Yeah, absolutely. So maybe. Let me just flash the paper itself. So we put this on the archive a little over a month ago now, and here's the paper. Let me explain in a few sentences, now that I've given a lot of background what the title means. Yeah, so the title says single minus gluon tree amplitudes are non zero. This might sound forbidding, but I think we can unpack this for the audience. So gluons are the particles that carry the strong force. And gluon amplitudes are functions that describe the quantum probabilities for gluons to interact via the strong force. Now, the word tree here is a little bit of a technicality. It means we're only considering processes where no gluons are created or destroyed. If gluons are created or destroyed, then you get loops, which we can explain later. But this is just a technicality. So we're considering special interactions where the same gluons that come in also come out.
- [15:43]
Brandon
So for anyone who's ever fit a polynomial, you can think of tree as being like a linear term. And then loops can be higher order terms. Correct?
- [15:51]
Alex Lukaska
Exactly.
- [15:52]
Brandon
Way more complicated than that. But conceptually, it's like kind of the lowest order in a series.
- [15:57]
Alex Lukaska
And so single minus. Now I have to explain that. So remember I told you earlier how particles have polarizations. So when you try to study gluon amplitudes, this is like a whole industry of physics. This is a very complicated field. People have written thousands of papers over the decades. So you always want to try to understand the simplest examples first. That's why you start with the tree amplitude, the leading effects, and then you worry about the loop corrections. So you might think that the simplest example to start with is one in which all the particles have the same helicity. So say they're all right handed, or that is to say they're all plus helicity particles. It's been known for a long time that actually in that case the amplitude is just zero, which means the interaction is forbidden and cannot happen. That's one way.
- [16:51]
Brandon
Thumbnail asymmetry just explicitly forbids this. You don't even have to calculate anything, you just know.
- [16:57]
Alex Lukaska
Yeah, just dimensional analysis, a very general argument. Yeah, you don't need to do very much work. And so, yeah, it's true that it's the simplest example, but it's so simple that nothing happens. So, okay, the answer is trivial. You might ask, what about the next level up?
- [17:14]
RJ Honicke
What about if I want to understand this, you have like a bunch of gluons, they're coming into an interaction, they're all in the same holisticity.
- [17:23]
Alex Lukaska
Yeah.
- [17:23]
RJ Honicke
And then you're just saying that just can't happen. Yeah, okay, like, because like I can't. I take my Glen gun and shoot and he takes his glowing gun and shoot and they go there and then that just can't happen.
- [17:35]
Brandon
They'll just go right through each other.
- [17:37]
Alex Lukaska
Oh, oh.
- [17:37]
RJ Honicke
So they just won't interact.
- [17:38]
Alex Lukaska
They won't interact.
- [17:39]
RJ Honicke
Ah, okay.
- [17:39]
Alex Lukaska
Yeah, okay. Yeah, yeah, yeah, yeah. That's a good clarification.
- [17:42]
RJ Honicke
Yeah.
- [17:42]
Alex Lukaska
And now you might ask, what if one of them has the opposite helicity, but all the others have plus helicity, but one of them has a minus helicity. So that's where we would call single minus amplitude. And if you look at the lecture notes and textbooks that have been written on this, the same argument that rules out the all plus amplitudes also appears to rule out the single minus amplitudes. They're too simple. They can't really interact. Nothing to see here. Move on. So then you might ask, okay, well what about the next thing where there's two particles that are minus helicity and all the others? So if there's n of them, there's N minus two others that have positive helicity. So these would be double minus amplitudes. And people in the 80s studied and computed these amplitudes. They're not zero. And in particular, there were two physicists, park and Taylor, who found this beautiful result. They did a lot of really hard work and computed these amplitudes. Very technical, difficult calculation, but at the end, you get all these terms and you have to sum them all up, and almost all of them cancel. And at the end, you're left with this very simple formula that fits in half a line, which is now known as the Park Taylor formula for these amplitudes. And these amplitudes are now called MHV amplitudes, which stands for maximally helicity violating, because they have the largest or so we thought possible, asymmetry between the plus and the minus helicity particles, the most asymmetry. Now, let's get to this paper, which came out last month. So this is a paper written with Alfredo Guevara, who's a postdoc at the Institute for Advanced Study. David Skinner, a professor at Cambridge University. Andrew Strominger, a professor at Harvard, used to be my advisor, and also Kevin Weil, who studied as a particle physicist in a previous life. So how did this happen? Well, maybe we'll get into how I ended up at OpenAI a little bit later. But I ended up at OpenAI, started to improve the model's abilities to do physics. The models got really, really good at physics. And I thought, okay, it's so good now we should try to solve some actual research problems at the frontier. And I called up Andy, who used to be my advisor, and I said, hey, Andy, do you want to come here to the SF, visit OpenAI, and we can try to solve one of your problems in physics. And I thought, it's probably not going to work, but if it doesn't work, at least we'll figure out why it doesn't work. And I can do this with a different physicist every month, and eventually something will work. And in the meantime, we'll learn how to improve the models. So it's all fun and useful. And so Andy was the first one that I invited to do this, and he said, well, I have this perfect problem that I've been thinking about with Alfredo and David for the past year. I'll explain now the problem. But the amazing thing is that we decided to start working on it using AI a little bit before Andy was scheduled to come, like, the week before. And in fact, using ChatGPT, we solved the problem before he even got off the plane, which was a huge surprise to him. Yeah, I think. And to me, to be honest, I had not expected that. And it's a really cool story. So Andy, David and Alfredo understood a year ago that this statement that the single minus amplitudes, the statement that they're zero, is not exactly correct. Because the usual argument in the lecture notes and textbooks has a loophole. And the loophole is that it assumes that the particles are coming from generic directions. But in a certain regime where the particles are exactly aligned with one another, we say they're collinear. Then the usual argument has a loophole and it's possible for the amplitudes to not be zero. But then if they're not zero, what are they? So then suddenly these really simple amplitudes previously thought to be zero, if they're not zero, we should compute them and they should be something really nice and simple and special. Now, I'm sweeping a lot of details under the rug here. This has to work in some different signature space time. It connects to lots of other things they've been worrying about. We're not going to worry about this.
- [22:21]
Brandon
I mean, I was actually hoping at the end maybe we could talk about what it means to be two dimensions in space and two dimensions in time. But yeah, I mean, I think like part of this is doable. The loophole is one about, you know, the alignment of the particles, but it's also a loophole about the space time of physics, that universe we're living in. And this is not really mind bending stuff.
- [22:42]
Alex Lukaska
So they understood that they're not zero and they started to compute them. And Alfredo is really, I think, the unsung hero of this story because he did a lot of really hard work to compute these things by hand. And I'll just show you an example. So in the paper there's a lot of formalism. So here is the beginning of the definition of the general answer. 1. Yeah, it's very hard to unpack, but it starts here. Then you have to define these vertices objects, V. And they're complicated. They involve sine and theta functions of spinors. And then you have this recursive formula, okay, it's a whole mess. And concretely, if you try to unpack this definition, remember these amplitudes are a function of the number of particles involved. So there's a three point amplitude where there's only three gluons in the interaction. And you know, the answer is pretty simple. This is some function that we've defined here. Not that complicated. Then this is the four point amplitude where now there's four particles. And you can see that we go from One term to a sum of two terms here. But then once you get to five particles, you start to get a lot more terms. There's eight of them being summed here. And by the time you get to six terms, it flows in your face.
- [24:06]
Brandon
For those people not watching this on YouTube and listening, this equation takes up a quarter of the page, is 32 terms, each of which is a product of four terms, each of which is itself encapsulating a rather complicated formula.
- [24:20]
Alex Lukaska
Yeah, so this is super nasty. And that's as far as Alfredo got or anyone else.
- [24:25]
RJ Honicke
So, Alfredo, is this just an expansion of some sort of. How hard is it to do this expansion?
- [24:33]
Alex Lukaska
Very hard. Okay. Yeah. And there's a nice graphical way to understand this in terms of Feynman diagrams. I hadn't planned to explain this, but there's a visual. This is kind of a visual subject. So the math is very complicated. And already back in the 40s, Richard Feynman, who's one of the pioneers of quantum field theory, came up with this very visual way to organize our understanding of the subject. So you can doodle these little cartoons that represent possible interactions. And the rules of quantum mechanics actually say that in these amplitudes where you scatter a bunch of particles, you get to fix what comes in and what comes out. Because that's the question you're asking, what's the probability for a certain interaction? But then everything that happens in between, you don't get to choose that because the physical laws determine what happens. And actually, in quantum mechanics, you're supposed to consider all the possibilities, all the ways in which the incoming particles can interact and transform into the outgoing particles. And you're supposed to average or sum over all the possibilities to get the final amplitude for the process as a sum over the amplitudes for each individual possibility for how you could get there.
- [25:48]
RJ Honicke
So just to be clear, there's incoming particles, they interact, and then there's all these different. They each have their own amplitudes. And then it's sort of like I select for this one one possibility and this one one possibility, and then I get, like, one possible interaction, and then there's an infinite number of those for each. And then I sum those infinite.
- [26:09]
Alex Lukaska
I suppose so.
- [26:10]
RJ Honicke
And I get the.
- [26:11]
Alex Lukaska
So, in principle, there are infinitely many pictures to sum over, but that's why we organize them by how complex they are. And it turns out that every time you get an interaction, every time there's a vertex where lines meet that point, interaction comes with a power of the coupling constant, which controls the strength of the interaction. And it turns out that every additional interaction makes the the amplitude more suppressed, so it contributes less to the final answer. And so you want to first consider the diagrams with the fewest possible number of interactions, because they will give you most of the total final amplitude. And then if you're trying to get a more and more refined answer, you then consider the more and more complicated cartoons with more and more interactions. And in fact, this is one of the ways in which the diagrams can get complicated, is that they can have loops. So, for instance, here you have a particle that decays into two particles, creating this loop, because then they meet it up again and disappear. So in this interaction, you have intermediate particles being created and destroyed, but whenever that happens, you get two extra vertices in your graph. So these diagrams are suppressed because it's less likely to happen that you get these extra felicitous interactions. And so you don't need to worry about this as much. It's like a small correction. And of course, in principle, you can keep going, but you're never done, except in very special circumstances.
- [27:39]
Brandon
The higher order power is in a polynomial or something, or a Taylor series.
- [27:43]
Alex Lukaska
And so, to go back to the story back in the 80s with the MHV amplitudes, which I think now is a bit of a misnomer, I would call them double minus amplitude, because that's what we're going to get to in a second. There was this heroic calculation where a lot of Feynman diagrams were summed and they were considering more and more interactions with more and more particles, and every time there were more and more terms, but they all cancel to the end, always give a simple answer. And in fact, that's what this PT term, PT stands for. Park Taylor these formulas, they fit in the line. So it's not that complicated, but it's very surprising that such a messy calculation at the end would clean up into such a simple result. And so what Alfredo, Andy and David did was to understand that these single minus amplitudes, in the special case where some of the particles are aligned, they don't have to be zero. And then you can do this very complicated Feynman diagram expansion to get the answer, which is not zero. But the problem is, if you do it this way, well, you can represent the answer in some horrendous, horrendously messy, complicated way. But if you unpack it, it's extremely complicated. It's complex in the following sense when you consider the endpoint amplitude. So the probability of n particles interacting, the number of terms in your answer which correspond to the number of diagrams, roughly, that you have to add up. It grows factorially in N the number of particles. And factorial growth is really bad. It's super exponential. It grows faster than an exponential, so it blows up in your face. This is what you're seeing here. And that's because roughly, you have to draw all the possible cartoons and the possible combinations is a combinatorial problem. And that's where the factorial behavior comes from. But we know from the 80s that in the actually more complicated double minus case, park and Taylor found this miraculous simplification. And so Andy, Alfredo and David spent the last year chasing the analog of the Park Taylor formula. The very simple answer that was obtained in the 80s for the double minus amplitudes. But now for these single minus amplitudes, which they understood are not zero, but then what are they? And they were getting this really complicated answer. And okay, you never know in physics ahead of time. If something will simplify, you have to believe in it to find the simplification. But because the double minus one simplify, it felt like these should simplify too. And we think they're important for lots of things and that these are somehow really important objects that are very fundamental and so they should have a nice description. And so they spent a year looking for that.
- [30:18]
Brandon
There's a funny. The next line, if you scroll down is something like, we need a simpler formula. Right.
- [30:23]
Alex Lukaska
When we run the paper, we need
- [30:25]
RJ Honicke
a more concise formula. Is he.
- [30:27]
Alex Lukaska
Yeah, a more concise formula is needed. And this is where AI comes in. Because when I asked Andy, hey, do you have a problem in your pocket that we should use AI to target? He said, well, I have just the perfect thing for you. We've been puzzled about this. It's really important, it's really interesting. It connects to all these things and we don't know the answer.
- [30:50]
Brandon
Yeah, I mean, like when I was a grad student, if I had approached something like this, I would have probably plugged it into a computer algebra system, chugged along, tried a few limiting cases, see if there's any magical simplifications which happen. This type of thing is something that oftentimes you see this and you're like, we need a different approach.
- [31:10]
Alex Lukaska
Exactly. Then, before Eddie even got here, we started to play with ChatGPT and Alfredo. Andy and I were trying different things, lots of different chats happening, going back and forth, David as well. And the first thing that happened is that we fed the five point amplitude into ChatGPT and we're like, can you Simplify this. And it was like, you know, there's a special region, so there's an extra assumption that you can make in which this answer simplifies to this one.
- [31:47]
Brandon
So this assumption is equivalent to you have one particle coming in and it decays into N minus 1.
- [31:53]
Alex Lukaska
That's one way to think about it, roughly.
- [31:55]
Brandon
But we're in two time dimensions.
- [31:58]
Alex Lukaska
Yeah, it's complicated, but basically you can look at what we call phase space. It's the entire space of possibilities for all the energies of incoming particles and their momenta. And there's a special region in that phase space where one particle has one different sign of its frequency compared to the other. And in that region, there's a big simplification that happens that ChatGPT found. And I should say this was the public model, but the PRO version that thinks really hard.
- [32:23]
RJ Honicke
So was that a known fact that it just was able to relate to the problem, or was that something that it put together?
- [32:31]
Alex Lukaska
As far as I know, it put that together. It said, this five point function, which is a sum of eight terms, each one of which is a product of three terms. They're all pretty complicated. It said, hey, actually this simplifies to this product of only three terms. And we stared at this and we thought, wow, that's really nice. We didn't know this. It's actually in hindsight, once you know, you can rederive this, but it takes a while to understand where this comes from. So I think that was a leap of insight that the AI had. And I think what it did, I mean, at some point said, I wrote a Python code and I ran through all 5,000 possibilities and I, okay, I did use this.
- [33:11]
RJ Honicke
So it's the equivalent of running his computer algebra system.
- [33:14]
Alex Lukaska
But it. But it just decided to do it on its own and came up with a huge simplification. So great. Awesome.
- [33:20]
Brandon
This was after making the assumption. This is after the decay one particle decay assumption.
- [33:26]
Alex Lukaska
Yeah. So it figured out there was a lot of exchange. This is very experimental, but we were talking about it a lot to figure out there's some forp region in which things simplify. And that region, it said, okay, this thing simplifies.
- [33:38]
RJ Honicke
GBT came up with that simplification as well.
- [33:41]
Alex Lukaska
Of the album. Yeah, yeah. And then we were like, okay, well, let's give it the six point function, which Alfredo heroically computed. And by we didn't have the seven point function, I don't think anybody could use the identity to expand it. It would be disgusting. And then ChatGPT does its little thing. And then it's like, yep, simplifies to this. And we thought, whoa, okay, that is really nice because now instead of 32 terms, it reduces to just four terms. And it's not a sum of 32 terms. It's a product of only four terms. And then we asked ChatGPT, okay, well, can you guess the general formula for all N? And that step, by the way, I mean, you could imagine using some programming language or symbolic manipulation software to do these reductions in certain examples. But to tackle the general case, I don't know how to use a computer to do that. But chatgpt said, yeah, this is the answer in the general case. Boom. How long does that take? You know, it's like using Pro. It thinks for 20 minutes at a time. You go back.
- [34:42]
RJ Honicke
But it wasn't like six days or something.
- [34:43]
Alex Lukaska
No, no, no. It's just like over the course of several interactions. And the amazing thing is that the formula that it proposed, instead of having this factorial growth, which is super exponential, where the number of terms, as you consider a number N of an increasing number n of particles, the number of terms blows up. Here it's actually linear. So if you double the number of particles, you only double the number of terms. It's the nicest possible behavior you could imagine. This is the equivalent, I think, of the Park Taylor formula for the double minus amplitudes that was known back in the 80s, but now for the single minus amplitudes, and this was guessed by GPT, I think it was 5.2 at the time, GPT 5.2 Pro. But it couldn't quite derive it.
- [35:31]
RJ Honicke
So it said, looks like this, but I don't know how to prove that.
- [35:34]
Alex Lukaska
Yeah, I think the model was not quite strong enough. Okay, prove it. But part of my work at OpenAI has been to develop stronger physics capabilities in the models. And a lot of people have been adding lots of. It's not just my singular contributions. There's a lot of great research happening and all comes together. It takes a village. But we had this internal model that could think for a very long time and was extra strong at physics. So we gave it the whole problem from scratch without actually giving it this. We just formulated the problem in a very sharp way and asked the model to solve to find the answer for the amplitude in the general case in this region. Because now we'd identified that this was the special place to look. And it took 12 hours, which is a long time, but it came back with the same formula which we had not given it. So it rediscovered the correct formula, but this time it also found the proof that the formula is correct. It derived it and in fact the remainder of the paper after we state the equation is devoted to the proof. That is basically what came out of the AI. So we say the rest of this work is devoted to proving that the conjecture is correct. There's three steps. First you show this, second you show blah, and third you show blah. And then this is basically what the AI came up with. So now I can finally summarize the paper. The title is Single Minus Gluon Tree Amplitudes or non zero. So these special interactions between gluons where only one of them has a different helicity from the others, which were previously thought to never occur, actually these interactions can happen. So the amplitudes are non zero. That's the main claim of the paper. I think it's quite surprising. I think it's a really nice paper. And the final result, I guess there's two results. One is understanding that it's not zero. That came from the humans like a year ago. But they were trying really, really hard to find this simple answer for what the amplitude is. And they were kind of stumped for a year. They were able to get this indirect representation that's extremely complicated in terms of Feynman diagrams. But they were looking for the simple formula that is the analog of this park Taylor work from the 80s for the more complicated amplitudes. And that was done with the AI. And so I think that's a really interesting result.
- [37:52]
RJ Honicke
Yeah, it's amazing.
- [37:53]
Alex Lukaska
It totally changes the way you should think about where we are in physics and how AI is going to change that. It's not just hype. I mean, this is like a real thing that happened. It's a result that top researchers in this field were thinking about for a year and then the AI solved it.
- [38:07]
Brandon
So I find it interesting. There's several things about the story which I think people didn't understand on Twitter. If maybe scroll down to like equation 38 to.
- [38:14]
Alex Lukaska
What's it?
- [38:14]
Brandon
35 to 38. Yeah, like, so I would say most, even intro grad students would look at 35 to 38 and say 39 is actually a very natural extension of this. Like, yeah, that is, I don't think, you know, that surprising. I think it's interesting. I didn't know until just now that you can. That when you proved 39, that was a fresh session, that was without the limiting cases. You started from scratch.
- [38:43]
Alex Lukaska
Yes. Why did you do it that way? Because I guess it's an Extra way to be confident in the answer. If a different model independently comes up with it from scratch, then you're not just spoon feeding the answer that you think is correct. That's an extra confirmation. But yeah, I think we thought a lot about how to put this out into the world and there's no perfect way to do this. Clearly we could have done a better job of communicating it. One thing that was important to us is to not make this paper about AI, because I think this is a really interesting physics result. People will keep reading this paper, I hope, for a long time. We didn't put AI in the abstract because this is a physics result that stands on its own. There's one paragraph really about AI where we just say the final formula was first conjectured by GPT 5.2 Pro and then proved by an internal OpenAI model. Because that's what happened. It's true. But we didn't really want to get into it because I think that's not the point of the paper. I mean, it's really interesting how it happened, but the result stands on its own. And I think if you read a paper today that was written 20 years ago that used the computer to do some critical step in the argument and it had this whole discussion of how Well I loaded Ms. DOS 3.1 and had 5 floppy disks and I had to swap my floppy disk, you wouldn't care. You know, that's not why you're reading the physics paper today. So we didn't really want to go into that in the paper. And we talked a little bit about it in the blog post that we released with the OpenAI, which is this one. And then I guess on Twitter there were a lot of questions and I wrote some tweets that I think clarified it. And there was somebody who is a physicist who wrote a great blog post, like actually understanding the story. And the Economist also put out a great article about it, which they really understood what happened. And I thought it was a great, great coverage. Science magazine also read about it. Harvard, the Institute for Event Study put out a press release. So I think it got a lot of attention, but it's kind of a subtle thing to explain. It took us an hour to go through what happened and what was done. So, you know, it's. It's hard to explain. I think it would have been kind of a distraction from the physics point of the paper to go into that.
- [41:00]
RJ Honicke
Okay, let's talk about the physics end. Give us a sense. Because, you know, my theoretical physics on the frontier comes from pbs Space time.
- [41:07]
Alex Lukaska
Right.
- [41:07]
RJ Honicke
Like I'm, you know, it's a great channel.
- [41:09]
Brandon
Yeah.
- [41:10]
RJ Honicke
And fantastic channel, but. And gives you a great high level picture. But hard to know how this sits in the pantheon of papers that represent the cutting edge of theoretical.
- [41:22]
Alex Lukaska
You're asking me how good is the paper?
- [41:25]
RJ Honicke
Not exactly that. I want to just understand. It seems like you're comparing it to this previous result that is pretty significant and highly cited and very important. How does this compare to that?
- [41:38]
Alex Lukaska
Okay, you're putting me in a bit of a tough spot. I will say I think the result is surprising. That's why the title is what it is. Single minus, amplitudes are non zero. And if you're somebody who works in this field, that should catch your attention. Ultimately, it's very hard to know in science when you release something into the world how it's going to be received and how impactful it will be. I think the true value of a paper can only be assessed into the future based on how much future work it leads to and what developments it opens up.
- [42:13]
RJ Honicke
Maybe a better way of asking is. So my understanding is that that previous paper kind of opened up a whole line of thinking about, yeah, I think
- [42:23]
Alex Lukaska
this is a great segue to the second paper that came out just three weeks later. Perfect. Then let's talk about. So it got its own blog post. This was March 4th, so I guess two weeks ago now. So we were talking earlier about how there's four forces. Strong force mediated by gluons and then gravity that's mediated by gravitons. Except gluons we can produce at the LHC and we can measure their effects fairly directly. Gravitons, we think, are also around us being produced all the time, even as I move my hands. But we've never done an experiment that directly measures gravitons, but they're supposed to be the quantum of gravity. So they're really interesting from a theoretical standpoint.
- [43:07]
Brandon
So going back to RJ's question earlier, what is a graviton?
- [43:10]
Alex Lukaska
There's different answers we could give. Ultimately, the correct answer depends on what the theory of quantum gravity, which we don't know yet.
- [43:21]
Brandon
If you just naively try to take all of your tricks from field theory that we know from the standard model, apply it to gravity, things just break down. The theory is not self consistent in some definition.
- [43:33]
Alex Lukaska
Various problems. Yeah, just like if you took in this room, there's light flowing around, there's some indivisible bit of light that you at some point can't break up into. Smaller bits. That's the quantum of light. We call that the photon. And the gravitational force is mediated by the exchange of gravitational force or gravitational waves. If you try to take a. A gravitational wave and break it up into smaller and smaller pieces, at some point you get a quantum that you can't break up anymore. And that would be the graviton. That's how we understand them.
- [44:09]
RJ Honicke
So there's the idea being that, like, you can't. You get to a certain point, and you can't have less gravity than that. You either have some or none.
- [44:18]
Alex Lukaska
Right? That's one way to think. Yeah. And so we wrote this paper, which is called single minus Graviton Tree Amplitudes are non zero. So it's the same title almost, except with graviton instead of gluon. And that's on purpose, because we wanted to extend the result. And it's the same story in the sense that it was thought that all symbol minus amplitudes are zero, but actually it's not true. And also for gravity, but gravity is a lot more complicated. So now if you want to compute the graviton amplitudes, it's potentially a lot harder.
- [44:57]
RJ Honicke
Gravitons have phase the same way that gluons do.
- [45:02]
Alex Lukaska
So they actually have spin two rather than spin one is getting into the weeds. So the amount of the numbers you have to use to describe them are a little bit different. They're doubled in some sense. Okay, so their polarization is more complicated. I see. But this is really getting into the wings. But the special region in which the final answer simplifies has two labels because it's a spin 2 particle, whereas in the gluon case, there was only one label because it was a spin one particle. So this is like. So it's not the same math.
- [45:31]
Brandon
Gluons and Gravicons do have some spiritual similarities compared to other types of particles, in the sense of they're particles of force. Yeah. Yeah. But they're, like, sort of doubled.
- [45:40]
Alex Lukaska
Yeah, they're sort of doubled. Yeah. I mean. Okay. I guess the people watching this podcast probably like to geek out on this. So the modern definition of a particle in quantum field theory, which is our best verified framework for nature, is that particles are irreducible representations of the pore group.
- [46:00]
Brandon
We just lost 90% of our audience.
- [46:03]
Alex Lukaska
Yeah, okay, maybe we cut this. So there's mathematical representations, and they've all been classified. All the possibilities are known by Wigner, actually a brain physicist. And it turns out that the representations or possible particles are completely labeled by the mass and the spin and the charge of the particles. So these are the three quantum numbers. And particles of long range forces like gravity and electromagnetism have zero mass. They have to have integer spin and spin one is three of the four forces and spin two is gravity. And then that's it. But let's set that aside. The really cool thing about this paper is that, well, first of all, it came out three weeks after the first one, which is really fast. And I think this is a great example of AI accelerating science. And in fact we could have put this paper out three days after the first one because that's, that's how fast we got the answer out of ChatGPT. But it took us three weeks because we wanted to check very carefully if that is correct. But most of the time was spent verifying the answer, not writing it, which is insane. Actually, if you take a step back, if you told me a year ago, yeah, you're going to have this AI that just does really hard calculations for you and then most of the human effort goes to verifying the answer. I've thought that, you know, you're crazy. So it's very surreal. And then we also had to write it up as a nice paper which you know, put into citations and references. That takes some time. And also had a baby in the meantime, so it was time there. But we do this really fast, so I think it's an example of accelerated science. Another really cool thing is that for this paper we didn't have to use an internal OpenAI model that had to think for hours. This was all done using the publicly available GPT Pro. In fact, we shared one of the main prompts that we used. It's if you go to the blog post extending single minus amplitudes to gravitons and you scroll down to the text, there's a link to one of the chats that we used. So you can see we use ChatGPT 5.2 Pro. And the amazing thing about this is that we gave it the glue on paper as a seed and we said, read and understand the paper. Make sure you understand the manipulations and the appendices, because that's where most of the hardware goes. But it comes back and it says, yep, I understood the paper, let me focus on the appendices. Here's what happened. And basically the punchline is that GPT Pro, with the glue on paper as an anchor, was able to do the graviton calculation, which is really different mathematically, completely on its own from, well, I guess not from scratch from the Glue on paper. But it's just a different thing. And it was strong enough to do it completely with this paper, took the
- [48:55]
RJ Honicke
conceptual leap from the previous paper and just said, okay, what math do I need to make that same conceptual.
- [49:02]
Alex Lukaska
And it's different math. That's an important thing to emphasize. So in particular, there's a crucial application of something called the directed matrix tree theorem. And Alfredo and David, we've been thinking about these things for a very long time. We're like, oh, that's really cool. That's surprising. We hadn't thought of that or seen that before.
- [49:18]
RJ Honicke
That was like known math. But it applied because maybe it has such broad understanding of math and physics that it able to say, oh, this is what. This is a good thing to apply in this case.
- [49:30]
Alex Lukaska
Yeah, exactly. And so here it understood the paper, the gluon one. And then we said, okay, well, the task is to generalize this paper to the gravity case. Here are two key changes, but otherwise manipulation should be similar. So we tweaked some things at the get go, and then we said, good luck. You're a brilliant theoretical physicist. So it's like we gave it two paragraphs. So we gave it Google on paper a couple paragraphs and said, good luck. Thought for 20 minutes, and boom, it starts to think. So it starts at the beginning. It works through the implications, all really interesting stuff. And then it says, here's what I would do next to turn this into the gravity paper. If you want, I can do blah. And so we say, yeah, go ahead.
- [50:15]
Brandon
Thought for 31 minutes.
- [50:16]
Alex Lukaska
Thought, 31 minutes. Yeah. So this exchange is 110 pages, but I think it's hilarious. I would describe this as vibe physics, because you can see, scenario goes away. There's a lot of hard work, lots of equations. It's starting to do the. Okay, so now you have to use this different math. You have to use these tree calculations, LSE reduction formulas. Okay. There's a lot of stuff happening. Subwoofer trees, concrete checks. It's starting to. Yeah, well, this is one of the things I love, is that it's able to do the same things that a human would do, which is check some basic cases, a study check, and to get intuition. And so it comes back every 30 minutes, says, well, here's what remains to finish the full gravity paper. And there's a list. If you want, I can write the gravity analog. Yes, do that. This is the first step. Okay. It goes back, thinks for 34 minutes, half collinear support. Okay? These formulas actually made in the Paper in some form. This is all correct. There's a bunch of stuff at the end. It says, if you want, the next most useful thing I can do is do this. And we're like, yeah, verify this by performing the explicit check. And it goes on. Just cut to the end. Finally we say, okay, write up the paper. And you can see the paper that it writes. And it's very close to the final thing that we actually put on the archive.
- [51:41]
RJ Honicke
So did it make suggestions that were not what you would have suggested as the next steps?
- [51:49]
Alex Lukaska
It's very smart. It knows kind of where to go. It's useful to steer it. If you compare what it came up with with the actual paper that we put in the intro, the abstract and introduction were written by Andy, who's an amazing writer, and I think he gave this wider perspective on the problem and how it fits into physics and how it connects to other things that the AI didn't do. It just the intro it wrote was more generic. But okay, AI could write really well. We didn't really try to make it. And the other thing is, we added the section this. Section two, which was not part of that initial exchange, is about how these graviton amplitudes transform under certain symmetries of physics. And that's something that we're really, really interested in because we eventually want to understand quantum gravity, as I mentioned earlier. And typically, the first step to uncovering a new theory is to understand what are its symmetries. That's something that gives you some kind of ground to stand on. Andy, in particular, has been pushing this program of celestial holography, which is like a whole thing we could get into, but it's an exploration of the symmetries of quantum gravity. And he really wanted to understand this. And there's a separate chat, we didn't share that one, where we led the AI to explaining how these answers fit into the symmetries that we know the theory should have. And that's something that went in there. But actually, I think from section three onwards, it's pretty much very close to what the AI wrote. So I would say this is really remarkable. It's a real solid result in quantum gravity that was done pretty much completely by an AI with humans steering it and asking kind of the right questions. But all the math was derived by ChatGPT Pro, the public model you can access. And most of the time spent was by us. The humans was like, checking everything and writing it up. And that's really wild. I mean, we're really so.
- [53:58]
RJ Honicke
I mean, as A physicist, you find yourself where a lot of coders have found themselves, where there's a kind of a fundamental, maybe epistemological question here, that if now, as a physicist, like, I could have done that, right? Like, maybe, maybe, like, I needed a little more background. But, like, a lot of it was like, yeah, go ahead, right? Like, take this paper. Give it some prompt. You guys obviously prompt it very well. But there wasn't like, maybe an undergraduate in physics could have come up with a lot of that. And so the question is, how does the undergraduate in physics now learn when they don't have to do the hard calculation of sucks similar to the how does the undergraduate coder actually.
- [54:42]
Alex Lukaska
You're opening up many different strands of conversation, Charles. Super interesting. So let's try to unpack that a little bit. So the most direct thing you asked is how does the next generation learn? That's a really good question. I think about this a lot. And now that a lot of senior physicists in the field are coming to grips with these new capabilities, one of the questions that comes up very quickly is, how do we train the next generation? Because the way we were trained is by going through these difficult rites of passage where you have to do these really arduous calculations. And this is how you build confidence in your own abilities and test your knowledge. And it's not just about what you're capable of doing. It's about knowing that you're capable of doing it and proving it to yourself and building that self confidence is important. And we don't have a good answer. This is something that academia is going to have to grapple with. So one thing that is especially difficult is that as a professor, I have graduate students. And the, the gap between where classes take you, even graduate courses, they only go so far. They go very far, but only so far. And the gap between where that ends and research begins is actually huge, and it's growing wider. Usually as a professor, what you do is when you take on new students, you keep in your pocket a few easy problems in the sense that you know they're going to work. Some questions that, you know, in principle you could work out, not that difficult, but you give them to a student so that they go through the exercise of learning everything around the question, developing the technology, and then you know enough about the problem that you're sure there's an answer that you can get there and you can advise the student in the process of discovering it. And I think the issue is that many such problems, now, I would say these models can probably crush. These are problems that we usually take. Again, timescale for a theoretical physics paper is six months to a year. That's pretty typical. So if you tell a student, go away and think for six months about this one question, and you have to work really hard, learn a lot of stuff around it, and do lots of calculations, even the most determined students, would they not within the course of six months ever ask Tattoo easy. Yeah, that's a little bit weird. Now, it's also an opportunity because I remember that time in my graduate school career in my second year of grad school. I took all my graduate courses my first year, and then my second year was my first project. And it was actually the hardest time for me in graduate school to traverse the desert for more classes, take you to the research frontier. It's very hard. And there's a lot of time spent banging your head against the wall, like, all the time. You're confused. You don't understand things just because you need to absorb so much knowledge. And AI can totally help you with that. It's the best teacher. It knows everything. It can unpack any complicated fact to any desired level of detail. Actually, my experience as a trained professional physicist working on my own research using GPT now is that I would say there's two key ways in which my research has completely changed. One is that I spend much less time being confused. So I'll do a calculation, get an answer, and I think, huh, how does this fit in with this other fact that I know? Like, how do I reconcile these things in my mind? I'm confused. Yeah, I do that all the time. Yeah. In research, usually you take a step, then you're like, you hit a roadblock, an obstacle. You're confused. Then you have to think for a few days. Maybe you go for a walk or work on another project, come back, get a new idea, but you spend a lot of time confused. That's nature research with GPT. I'm like, hey, I just did this. I found this. How does this mesh with this other thing? And then it's like, oh, well, you forgot this thing. Or, oh, you didn't quite think about it correctly. Or does the standard factor. And so the amount of time you spend confused just dramatically shrinks and you move so much faster. That's one of the accelerating effects. The other accelerating effect is that I only have so much free time and energy. Especially when you become a professor, you have to teach. You have students, you have grants, administrators. There's a lot of things you have to do. So Your free time to think about research without distractions, shrinks. And also, you only have so much energy to do hard calculations. And so what you would usually do is if you have a problem, you're at point A and you want to get to point C, you think about the route, oh, I have to go through point B first. And actually maybe there are multiple points. And you try to plot in your mind the course that you're going to take before you actually go start, do the hard work. You try to think really hard about where you're going and to chart a course. With AI, actually, you can launch 10 instances of chat and have each one try a different route and send it as a scout that moves very fast into the unknown, pushing outwards. And you can just very quickly get some feedback to see, okay, these approaches are not promising. These are much more promising. And then if you follow them, there's a huge difference between being the first to push into the unknown versus following someone ahead of you. And even if ChatGPT doesn't always get everything right, just kind of having a scout that signposts some key steps along the way that you can used to anchor your own movement is extremely helpful. So these are two concrete ways that AI has changed the way I work. And I think if you're entering research, having an assistant that can help you find your way to where you're trying to go can be very good. So I think it's inevitably going to change how we work, how we operate, and how we train students. And part of what's exciting about my job is trying to figure how all of this works. But it's not just a job for OpenAI, it's actually a job for every researcher and professor more generally to think about this. I think the future is very bright because we have some challenges to overcome. But on balance, this is such an amazing tool. I think it's going to give human physicists AI superpowers. Because of what I just described, you can do so much more. And I think actually the kind of skill that is really useful to get great results out of AI is very similar to the kind of skill that you develop as an academic collaborating with other humans. This is like a collaborator. And if you're a professor who's been advising students and postdocs, a lot of what being a professor involves is knowing for each student postdoc that you're working with exactly what question to give them. So matching the problem to the person and knowing how to give them the question in what way, with what level of detail, not too much, not too little. And that's actually kind of what you have to think about when you interact with ChatGPT. So I think that it's a transferable skill and people who are good at this are about to get AI suit empowers.
- [61:44]
Brandon
What you just described there reminds me of several conversations we've had on the podcast thus far which keep coming up to this concept of taste. One of the things that especially you say theoretical physics, high energy physics has maybe had a problem with. I'm not sure if you want to describe it that way, but it can be very trendy that there are certain things which become in fashion because maybe right now we're in a world where we don't have the data to define new directions to really guide or constrain where we're going. I'm curious, how does essentially something which is superhuman in that it has basically all known physics and is interact with a field where at its core what oftentimes can be popular people start working on is more based upon general aesthetics or what the community collectively thinks is cool at the time. Because I can imagine IKU could vibe so so many different worlds. Like for example, just using KleinSpace, using this sort of two time, two spatial dimensions for this was already sort of an assumption that I think is actually kind of important in some ways and does provide feedback to our world. But in the concept of, you know, you could have asked ChatGPT to solve this problem in all sorts of number of ways and maybe it could come up with all sorts of things which don't really align with maybe the useful taste as a community. How do you actually deal with that? Like a proliferation of really interesting results, but it's actually not clear where the field should go.
- [63:28]
Alex Lukaska
You're getting at the heart of what does it mean to do progress in theoretical physics and research. This is a hard question and there is a simple answer. If there were, it would be research. Um, let me say a couple of things. The first one is when you go to graduate school in physics, you it's usually because you're really interested in the big questions. Why are there three dimensions of space? What happened at the Big Bang? What's inside a black hole? These are the things that, you know, I was thinking about because of sci fi movies and books that articles. What you realize is that actually these questions, even though they're really cool and exciting, they're not really the most fruitful scientific questions. Because at any given time there's an edge of knowledge and the role of Scientists is to extend the edge of knowledge, is to push into the unknown. And to do that, you want to find the questions that are right at the edge or just beyond the edge, but not so far that you can't grapple with them. So the question of why there are three dimensions of space, that's a really cool question, but I don't know of anyone who said anything really compelling about that. So it's just a question that's beyond the edge. So it's not. As a professional physicist, I don't spend my time thinking about this because I just don't know of any pathway to solving this question. It's not useful to think about. So really, the process of training as a physicist involves coming to grips with what the edge of knowledge is, because that's where the interesting, fruitful questions to make progress on as a scientist, that's where they live. Oftentimes when you go through graduate studies, you worry, oh my God, I have to learn about Feynman diagrams and all this math and all these calculational methods. And it's true, that's a really hard thing to learn. It takes a lot of work. But in some sense, once you become a professional physicist, you should feel like you can learn any tool. You can pick up any tool that is needed for the task at hand, and you should develop that confidence. And that's what makes a competent physicist. A competent physicist is one that can learn any new mathematical tool or piece of code or whatever that is needed to solve the problem at hand. And that makes you a good physicist or a competent one. If you pick up this skill and in graduate school, it's daunting, you have to learn a lot. But by the end, you should have a lot of skills in the toolkit and the confidence to pick up any new one as needed. The difference between a good physicist and a great physicist is knowing what is the right question to ask. That is actually the hardest part of being a scientist. It's knowing what is the next fruitful question to tackle. And I think AI right now is a very good physicist. In fact, maybe superhuman when it comes to certain computations. But it's like this extremely technically skilled graduate student that you can give a sharp, well posed question to and will do incredibly hard calculations correctly now and come back to you with the answer. So it's super competent. But one of the things that it doesn't quite have yet is knowing what is the right question to ask. And I think just like with humans, that is actually the hardest skill to pick up. That's the one that comes last.
- [67:13]
RJ Honicke
I know you're not working directly on the AI so much. I mean, I don't know exactly how much you. But do you get a sense for, you know, you can imagine a future where you just do better reinforcement learning? Maybe you change the architecture of the model completely so that it's some other, you know, like whatever, not transformer and the trajectory just keeps going like this because it's been very, very rapid increase since 01 of the recent capabilities? Or do you get a sense that we're getting near the edge of the frontier of knowledge now so that the sort of the ability of the model to recombine knowledge in somewhat novel ways is, you know, like, that's kind of in me. It seems like not, not to disc or like not to play down any of these results, but that. It seems like there was a lot of what it did and maybe there's some. Not like this, but a lot of what it did was recombination of known facts.
- [68:13]
Alex Lukaska
Okay. Yeah.
- [68:14]
RJ Honicke
But do you get a sense that that's. Do you have any reason to believe that will continue? Or if we're going to just like, sort of.
- [68:21]
Alex Lukaska
Okay, we've.
- [68:22]
RJ Honicke
We know how to recombine stuff really well and we can't push beyond that.
- [68:26]
Alex Lukaska
Without getting too philosophical, I'm not sure that any of us are anything more than recombination of tech machine.
- [68:33]
RJ Honicke
Fair enough.
- [68:34]
Alex Lukaska
Working with GPT Pro on this problem, to me feels like working with a creative collaborator. It did stuff that I didn't know that I found surprising. And so I think. I'm not sure there's a qualitative difference. I think it's just a matter of degree. Yeah.
- [68:54]
RJ Honicke
Okay.
- [68:54]
Alex Lukaska
That as we continue scaling the capabilities, which is certainly happening, I don't see why it's going to stop. Like, we definitely have a bunch of things in the pipeline that are going to keep coming this year. And you know, my horizon for seeing it to the future is like not, not that good beyond the year, but like, definitely we're going to keep scaling up this year and I don't see any reason why it's going to stop. And I think that's going to make these models display feats of insight that look to us like real creativity. I would say this already happened in this project, at least. You know, what is creative insight is a bit in the eye of the beholder. Right.
- [69:30]
RJ Honicke
I mean, AlphaGo. Right. Was able to come up with moves that were very.
- [69:35]
Alex Lukaska
I talked to Terry Tao a couple of weeks ago. At UCLA, we had an OpenAI event with iPam, which is this institute of Mathematics there. And I talked to Terry Tao and he said that in his view, all of the proofs that he's seen AI come up with in math, even the ones that at first seemed creative and surprising later on were tracked down and found to have really pulled facts out of some obscure reference. So I don't want to put words in his mouth, but my understanding was that Terry Tao has not yet been impressed by a creative move in math. But Terry Tao is a unique individual. I've been impressed. I consider myself my bar is lower and I think as we keep scaling this up, I can't go into the details, but there's a lot of effort at OpenAI. There's a lot of really smart, hard working people that are pushing very hard to take this next step. And I think it's going to come eventually. I mean, just look at the trajectory that we're on. So a year ago I was black hole physicist in academia, not really paying too much attention to AI. I thought AI is cool for emails, but I'm going to do what I do, which is special. O3, which was the really first really strong reasoning model, came out and was able to do a calculation for me that would have taken me days and did it 11 minutes. And I thought, wow, that was shocking to me. And we could go into the details if we have time. I can show you the example because it was really surprising to me. And then I thought, okay, I gotta really start using this tool. There's no other software that could do this kind of calculation as far as I know, was really surprising and really cool. And then GPT5 came out six months later and that was able to reproduce one of my hardest calculations, which I think the number of people in the world that could do that, you could count on your hands.
- [71:28]
Brandon
So when you say reproduce, seeing this has been published or not published, it was a secret or internal result.
- [71:32]
Alex Lukaska
So last summer in June, I put out this, this paper which I really like, in which I found.
- [71:38]
Brandon
It's called why is there no Love in Black Holes?
- [71:41]
Alex Lukaska
Yeah, and love is actually a technical term. It refers to Augustus Love, a British mathematician who studied the tides. So when you have an object like the moon going around the Earth, it exerts tidal forces on the oceans. And so you can measure the tidal response, say of the Earth and its oceans to the moon via some coefficients that encode the strength of the tidal response. And these are called love numbers in reference to Augustus Love. But famously, black holes do not experience tides, so they have no love. And there's been a resurgence of interest in this fact in the last five years because people understood that this can be connected to symmetry principle. So in physics, whenever something is zero, like, why should black holes never experience tides? That's surprising. Well, oftentimes the answer is because there's a symmetry principle at work that forbids the existence of tides, that protects the structure of the black hole. And so I found these new symmetries. So these are differential operators that act on solutions to this equation that describe perturbations of a black hole. And these generators are symmetries, because if you act on the solution to this equation, you get a new solution. You know, I thought this was very beautiful. I liked it very much. And it came out in June on the archive, and in August, GPT5 came out, and the cutoff date for its training set precedes the release of this paper. So GPT did not see this paper in training. And when it came out, I was like, okay, I'm going to got to meet Mark Chen, who's chief research officer at OpenAI. And he said, give GPT Pro a really hard problem. Let's see how good it is. And I was like, you want a hard problem?
- [73:26]
RJ Honicke
Yeah, I got you a hard problem.
- [73:28]
Alex Lukaska
I just solved this problem and I wrote a paper. I was very excited about it. I thought, this is really deep, but cool. And I gave GPT the equation here. And I said, what are the symmetries? I didn't tell it that there are symmetries, because by default, the goofy answer should be that there aren't any, and thought for five minutes and said, yeah, there are no symmetries, which is what usually happens. And that was wrong. And Mark Chen was visibly crestfallen. He's like, oh, well, okay, why don't you give it an easier question? And so then I gave it the same question, but not for a black hole space time, but for an empty flat spacetime, which is a simpler problem. But that's actually how I approach this problem myself. You warm up on the easier question first. So I gave you the flat space question, which is in this paper also. So it's this equation which looks much simpler. Then this also has three symmetry generators which are shown here. This is not new. These equations have been studied for 200 years. Everything in flat space has been known forever. And GPT5Pro thought for it was like nine minutes. And it came up with the answer, Very beautiful answer, perfectly structured, perfectly correct. Actually, at the time I also tried the other models from our competitors and none of them could get this at the time. So GPT Pro is really ahead and I think it continues to be the best model for this kind of mathematical and physics work. And Martin was like, okay, well this is great. But now that it's done, the warm up problem in the same instance of chat, tried the full problem again now that it's been primed, I thought okay, why not? And so I gave it the the same question as before. What are the symmetries of this equation? Now the full black hole problem. And this time I thought for 18 minutes, which I'd never seen before. And it came up with the answer. So basically in under 30 minutes with one hint, which is the obvious warm up problem to prime the model on first, it completely solved this problem which you know is one of the nicest calculations that I've ever done. And that really blew my mind. That was my move 37 moment. Yeah, that's how we call it in the AI world. And once I saw that I thought, okay, we're on this crazy trajectory where 18 months ago was not useful. A year ago it could do really hard calculations that would take me days. Eight months ago it could reproduce some of my best work in under 30 minutes. And then now in the last month it solved these questions that we've discussed at length which the world experts had spent a year thinking about without being able to get to the answer. So you know, I think it's just going to keep getting better. Where are we going to be in six months or a year? I don't see any reason why it would stop. And I think we're going to be having a very exciting year.
- [76:33]
Brandon
Okay, going back to these thoughts about scientific discovery and what can these models do versus just being very good at superhuman at solving physics. People keep asking this question hypothetically, could we train a version of ChatGPT where it's never seen anything post 1904 and could it rediscover relativity? I think that there's a very analogous question we could ask right here which is new conceptual result about single minus gluon amplitudes which was sparked by human insight and there were certain very specific assumptions which went in here. Like understanding working in Kerr spacetime is something that people have been thinking about and has some useful transferable insights. People have been thinking about maximally helicity violating amplitudes for quite some time. Have you ever tried using a model right before the cutoff date of this and asked given a cur Metric. Is there anything interesting with regards to helicity violation or maybe turning around saying it's long been thought or it's long been known that with the exception of some set of measure 0 due to Wythen, there's no single minus non 0amplitudes. Have you tried either of these directions and asked it to discover a new insight, push the boundary as you were just talking about and make a leap in addition to not just solve a problem like you can give it, but actually could you get that intuition?
- [78:09]
Alex Lukaska
Yes. Yeah.
- [78:12]
Brandon
You have tried this.
- [78:13]
Alex Lukaska
Not exactly the counterfactual version that you're describing. I personally haven't done that. But pushing the models at the frontier to try to make this type of leap is something that we're very focused on. Yeah. And I think, well, okay, I don't want to talk about the internal research we're doing, but I can say something publicly, I think, which is you can take this page of this paper and you can feed it to ChatGPT Pro, say I like the best model we have out right now and you can ask it, what should I do next? Give me the top three follow up questions to ask. Based on this paper, I've done this experiment and the top three questions it comes up with are like my top three questions for what I should do next. So I think the models are smart enough now and have enough background knowledge that, you know, for this paper I'd say GPT is about as good as me at finding the next day to ask. And so that's really interesting and it
- [79:18]
Brandon
opens up a lot of.
- [79:20]
RJ Honicke
So can you just, you know, do what is. What is the name of the loop that the agent loop that everyone is talking about where you just say like, okay, what's the next question? Go ahead and solve that. What's the next question going in? So like, I guess and this goes back to the question I was asking before, before, which is if you do that, and probably has been, you've tried it or someone open AI tried that, like I presume you get to some plateau right where you like you're not pushing the boundary of knowledge anymore. Or is it just like the plateau is money and if you had more money you could go further.
- [79:49]
Alex Lukaska
Yeah. Just to be very explicit because I haven't said this quite out loud, I think we now have models that can really churn out papers that are as good as human written papers. In fact, this is a bit of a problem because when a professional physicist uses this tool and they steer the model and they check the answer they can get amazing results. But there are also people that feed it kind of wrong questions that go off the deep end. Yeah, they submit that to arXiv. And this is a problem that the academic community is dealing, is trying to come to grips with now, which is this problem of AI slop. But for science, this is something we have to figure out. But I would say that with proper steering, you could probably churn out a paper a day. Now, I don't know. It's like, give the question to ChatGPT, it'll solve it. If it's not that hard of a question, or it's a similar calculation to stuff that's already been done. It can totally do it in 30 minutes. And then you could say, write it up as a paper and you can send it to archive. Okay. So I think that we're already in this moment. We've passed that threshold. This is the new reality. And more and more people are catching on to this all the time. And so some of them are doing this. And this is why the archive is now inundated with. With submissions. So what's the correct response to this? I think we put out these two papers in very fast succession. We could spend the rest of the year writing 30 more papers like this. I don't think that's what we should be doing. Instead, I think now that we have this new tool that gives us AI superpowers, I think we should just raise the bar for what it means to write a good paper. We should aim higher, basically. One thing that I'm excited about is that I think these single minus amplitudes papers, they open the way now to a whole direction of research, which I think is a line of attack on really interesting questions in quantum gravity. To go back to the start of the session, this is the missing piece of the puzzle, fundamental theoretical physics. And I think we have a pretty clear line of attack through a series of questions, all of which I think will be amenable to solution with AI. And so I think I'm excited to spend a good part of this year trying to follow this path, but really solve harder and harder problems. And this paper gave an answer to a question that had stomped Andy, Alfredo and David, who are experts in this for a year, but we haven't seen an AI yet solve a question that stomps an entire community of physicists for decades. That hasn't happened yet, but I think given the trajectory that we're on, at some point, hopefully not too far into the future, we should see that. And so I think that's the exciting thing to try to go towards, which is pushing the envelope of what can be done.
- [82:53]
RJ Honicke
We wanted to start asking all guests question, which is if you could remove one bottleneck for your domain. So in this case, maybe it's AI
- [83:03]
Alex Lukaska
for physics, or maybe it's physics, or
- [83:04]
RJ Honicke
maybe it's mostly a. But if you could remove one bottleneck for, for your domain, what would that be and why?
- [83:12]
Alex Lukaska
Well, off the top of my head, you know, I spent so much of my time writing papers and the way I think now is so far from papers, it just feels like not the right way somehow to store and communicate knowledge. I think an extreme version of this, which makes the problem more apparent is math, especially certain parts of math, where papers are very terse and they take four pages. I had this experience when I was learning algebraic geometry in graduate school, going to a mathematician and saying, what's going on in this four page paper? This is just very terse notation. And he said, oh, forget what's in the paper. And he took me to the blackboard and he started to draw pictures. He's like, this is how you should think about it. And I was like, oh, wow, this is amazing. But none of that is in the paper. And mathematicians, I think, have this cultural norm that they kind of hide the messy work and they will write these beautiful, short, pristine papers. It depends on the subfield, but oftentimes that's the case. And the way they actually think about the subject as a living, breathing entity is very different from the way in which it's recorded papers. Some of that is also true for physics. I love doing calculations, coming up with questions, finding the answer. And then I would say a huge bottleneck is writing it up. So somehow it feels like papers are not quite the way of the future, or at least the way that we currently operate with. I write it up, send it to a journal, it takes six months. I don't know, it's just like, why are we doing all of this? Feels like maybe there should be something better. I mean, you could, you know, if you want to understand this paper, one thing you can do is upload it into ChatGPT and ask it to explain it to you. And you can keep unfolding the complexity into more and more detailed explanations. And so if we move into a world where we use AI to do the calculation, get the result, then we have the step condensing into a paper and then, you know, I sum up the paper to Brandon and he puts it back into an AI that will have, why are we doing this right? That's a little bit funny. I feel like, if you ask me, would I be confident that in 20 years we'll have these sort of static documents in which we publish our results as papers? I would think not. That doesn't seem like the best thing we could be doing. Maybe some kind of interactive paper which lives in some LLM. Maybe your whole paper is some ChatGPT page and you know there's a chat bot attached to the paper and you can say, explain the big picture and like zoom into this fact. I think we're going to head in that direction. That would be a cool thing to see. Writing a paper, though, is a useful exercise because it forces you to condense your thoughts and make them really clear. So I'm not saying it's a bad thing to do in general, but just the way we do it is very slow. But that's the first thing that came to mind. Maybe another answer is in this project, the Graviton paper, we got to a draft of a paper extremely fast and then we spent most of our time checking the answer. So I think that will effectively be maybe the next big bottleneck. And that is one of the things that the models, I would say, if you ask me, what is missing in the models, what can we really improve for scientific research? I think we've kind of touched on the two big things already, but just to spell them out, one is creativity and the spark of invention and really taking the next step. I think that will come as we scale up the intelligence. Well, we'll see, but I don't know that there's something missing inherently. I think it's just, it's starting to make these leaps for me. But maybe we should encourage the models to try to make bigger leaps, because large language models, after all, they're trained to give you the middle of the road answer. If you ask an AI like ChatGPT, write me an email about blah. You want it to give you kind of the expected answer, not to sample from the tails like wacky email. You kind of wanted to give you a reasonable thing. So for most tasks you want that, but for science research, sometimes you want the idea that comes out of left field, the thinking outside the box or really sampling far out of the distribution. And that's something we could do in principle, but that's not how the models are. You know, we're not really favoring that. So we might have to do tweaks of this kind to make the models be able to take bigger leaps. And then the Second thing is verification, because we're now in this new regime where the models are so capable that for very hard computations at the frontier of knowledge, they can just do the whole thing. But is it correct in this case? It was correct. Sometimes I get emails from people saying, oh, we did this really long calculation, but there was a mistake somewhere. Disappointing.
- [88:17]
Brandon
Okay.
- [88:17]
Alex Lukaska
I mean, the calculations are getting more and more complicated, longer and longer, but yeah, sometimes they mess up. And so I think improving verification or even just having the model indicate more directly how confident it is in its answer, because I think they're smart enough to know when they're very confident in the answer versus when they're just kind of guessing in some step. And getting the AI to be more explicit about that is, I think, a way to improve them for research. And that verification step, I think, is going to become maybe a bigger bottleneck this year.
- [88:52]
RJ Honicke
Yeah, Karina Hung from Axiom would agree with you emphatically. Formal verification, is there a thing?
- [88:59]
Alex Lukaska
Right. Yeah, I think it's interesting. A year ago I would have said it's super important to have formal verification. Then the models got so smart that I thought, well, if Brandon and I talk about a mathematical proof and we go over it, we're not going to formalize it in set theoretic notation or we don't think the way lean, which is this language, for formal verification reasons, we reason through the proof in natural language. We use words. And so if a model is really smart enough, then it should be able to do the same thing. And we've been saying this huge increase in capability for mathematical reasoning and developments using natural language. So then maybe for a while it looked like that wasn't the thing to really focus on. But now that we're in this regime where you can just get chatgpt to tackle thousands of questions at the same time, and it will return proofs for a significant fraction of them. Now, actually, the onus is back on the humans to verify all the outputs. And so, yeah, as that becomes a bottleneck, I think formalizing math and automating verification will become more valuable, it looks like, to me. And that's something we're thinking a lot about as well. Thanks.
- [90:20]
Brandon
What do you want the audience to take away from today? Is there one message that you want them to leave with?
- [90:26]
Alex Lukaska
Yeah, I think it's important to get the word out. The models that we're developing at OpenAI are becoming really capable at scientific research. I myself was a bit of an AI skeptic a year plus ago because I thought the models are very good at writing tasks, but not mathematical tasks. That changed with O3, the first strong reasoning models. And then GPT5 was able to do some of the hardest calculations that I can do and reproduce them correctly. And now, recently, in the past month, we've seen models solve open questions in theoretical physics, and now they're solving problems in quantum gravity and quantum field theory. So if you just extrapolate that into the future, imagine where we're going to be in six months or a year. I think it's kind of surreal to live through this time, but it's really happening. It's really amazing. And I think we're going to see a lot of big changes happening in research, so that's. Yeah. Pay attention to the space. Stay tuned. Awesome.
- [91:32]
RJ Honicke
Thank you so much for taking the time. This is like, I learned a lot from our discussion and, and I'm going to definitely keep up with what you're,
- [91:41]
Alex Lukaska
what you're up to. Thank you. It's been great to be here.
- [91:43]
Brandon
Thank you. Thank you.