Host 2 (3:20)

So the thread that, when I look back, look at the different things you worked, some of them seem preconnected, like the physics to equivariance and graph neural networks maybe, and that. That seems to be somewhat related to cusp. Do you have a. A thread through there?

Max Welling (3:41)

Yeah, I think physics is the thread. So having done, you know, spent a lot of time in theoretical physics, I think there is first very fundamental and exciting questions, like things that haven't actually been figured out in quantum gravity. So there's really the frontier, but there's also a lot of mathematical tools that you can use right? In, for instance, in particle physics, but also in general relativity, sort of symmetries play an enormously important role. And this goes all the way to gauge symmetries as well. And so applying these kinds of symmetries to machine learning was actually, you know, I thought of it as a very deep and interesting mathematical problem. I did this with Tackle Cohen and Tack. Cohen was the main driver behind this, went all the way from just simple like rotational symmetries all the way to gauge symmetries on spheres and stuff like that. So. And Maurice Wyler, who's also here when he was a PhD student with me, you know, he wrote an entire book which I can really recommend, about the role of symmetries in AI and machine learning that I find is a very deep and interesting problem. So more recently, so I've taken a sort of different path, which is the relationship between diffusion models and a field called stochastic thermodynamics. This is basically the thermodynamics, which is A theory of equilibrium, but then formulated for out of equilibrium systems. And it turns out that the mathematics that we use for diffusion models, but even for reinforcement learning for Schrodinger bridges for MCMC sampling, has the same mathematics as this theory, this physical theory of non equilibrium systems. And that got me very excited. And actually when I taught a course in Mausenberg, it is South Africa, close to Cape Town, at the African Institute for Mathematical Sciences, Ames. And I turned that into a book. So two years later the book is finished. I've sent it to the publisher. And this is about the deep relationship between free energy diffusion models, basically generative AI and stochastic thermodynamics. So it's always some kind of. I don't know, I find physics very deep. I also think a lot about quantum mechanics and it's a completely weird theory that actually nobody really understands. And there's a very interesting story which may be good to tell to connect sort of my PhD back to where I am now. So I did my PhD with the Nobel laureate Gerard de Toft. He's just the most brilliant man I've ever met. He was never wrong about anything as long as I've seen him. And now he says quantum mechanics is wrong and he has a new theory of quantum mechanics. Nobody understands what he's saying, even though what he's writing down is not mathematically very complex. But he's trying to address this understandability, let's say, of quantum mechanics head on. I find it very courageous and I'm completely fascinated by it. So I'm also trying to think about, okay, can I actually understand quantum mechanics in a more mundane way, sort of, you know, without all the weird multiverses and collapses and stuff like that. So the physics is always been the threat. And trying to apply the physics to the machine learning to build better algorithms,

Host 1 (6:53)

you are still very involved in understanding and understanding physics and the worlds even beyond just applications to machine learning or introducing no normalisms. That's really cool.

Max Welling (7:06)

Yes. I would say I'm not contributing much to physics, but I'm contributing to the interface between physics and science. And that's called AI for science or science. Or AI is kind of a super. It's actually a new discipline that's emerging.

Host 1 (7:19)

Yeah, yeah.

Max Welling (7:19)

And it's not just emerging, it's exploding. I would say that's the better term because I know you go from investments into like in the hundreds of millions now in the billions. So there's now actually a startup by Jeff Bezos that you know that 6.2 billion seed round. Right. It's like insane. I guess it's the largest, you know, startup ever, I think. Right. And that's in this field, AI for science. Right. It tells you something that we are creating a new bubble here. Right.

Host 1 (7:47)

So why do you think it is what has changed that has motivated people to start working on AI for science type problems?

Max Welling (7:55)

So there's two reasons actually. One is that people have been applying sort of the tool, the new tools from AI to the sciences, which is quite natural. And there's of course, I think there's two big examples. Protein folding is a big one. And the other one is machine learning force fields or sometimes called a machine learning interface, atomic potentials. Both of them have been actually very successful. Both also had something to do with symmetries, which is also cool. And sort of people in the AI scientists saw an opportunity to apply the tools that they had developed beyond advertised placement. Right. Or you know, multimedia applications into something that could actually make a very positive impact in society, like health, drug development, materials for the energy transition, carbon capture. These are all really cool, you know, impactful applications. Beside that, the science and the kind of the. Is also very interesting and sort of, I would say the, the fact that these sort of, these two fields are coming together and that we're now at the point that we can actually model these things effectively and move the needle on some of these sort of science sort of methodologies is also a very unique moment, I would say. And people recognize that, okay, now we're at the cusp of something new, as we're also what the company is called after we're at the cusp of something new. And of course that always creates a lot of energy. It's like, okay, there's something, it's like sort of virgin field, right. It's like nobody's green field, nobody's been there. You know, I can rush in and I can sort of start harvesting there. Right. And I think that's also what's causing a lot of sort of enthusiasm in the fields.

Host 2 (9:43)

If you're an AI engineer basically, if the people that listen to this podcast will be. And you maybe don't have a strong science background powders but are excited. Most, I would say most AI practitioners be engineers or scientists would consider themselves scientists and they have some background, a little bit of physics, a little bit of industry, college, maybe even graduate school that have been working or are starting out. How does, how does somebody who is not a scientist on a day to day basis, how do they get involved?

Max Welling (10:14)

Well, they can read my book once it's out. But this is basically saying that there is more. We should create curricula that are on this interface. So I'm not sure there is possibly already some universities, actual courses you can take, maybe online courses you can take. Maybe these workshops where we are now are actually very good as well. And we should probably have more tutorials before the workshop starts. Actually, we've, I've kind of proposed this at some point. It's like maybe first have an hour of, of a tutorial so that people can get new into the field. But yeah, there's a lot out there. Most of it is of course inaccessible. But I would say we will create much more books and other content that is more accessible, including this podcast, I would say. Right. So I think, you know, it will come and you know, these days you can watch videos and things. There's a huge amount of content you can, you can go and see.

Host 1 (11:07)

So maybe a follow up to that. How do people learn and get involved? But why should they get involved? I mean that we have a lot of people who are of our audience will be interested in AI engineering, but they may be looking for bigger impacts in the world.

Max Welling (11:19)

Yeah.

Host 1 (11:20)

What opportunities does AI for science provide them to make an impact to, you know, change the world that working in this, the world of pure bits would not.

Max Welling (11:30)

So my view is that underlying almost everything is a material. So we're focusing a lot on LLMs now, which is kind of the software layer. But I would say if you think very hard, underlying everything is a material. So I was saying there's the LLM underlying the elements of GPU on which it runs. And then in order to make that gpu, you have to put materials down on a wafer and sort of shine on it with sort of EUV light in order to etch kind of the structures in. But that's now an actual material problem because more or less we've reached the limits of, you know, scaling things down and now we are trying to improve further by new materials. So that's the fundamental materials problem. We need to get through the energy transition fast if we don't want to kind of mess up this world. And so there is, for instance, batteries. That's a complete materials problem. Right. There's fuel cells, there are solar panels. So they can now make solar panels with new perovskite layers on top of the silicon layers that can capture, you know, theoretically up to 50% of the light. Where now we're at, I don't know, maybe 22 or something. Right. So. So these are huge changes all by material innovation. And yeah, I think wherever you go, you know, I can probably dig deep enough and then tell you, well, actually the very foundation of what you're doing is a material problem. And so I think it's just very nice to work on this very, very foundation. And also, because I think this is maybe also something that's happening now is we. We can start to search to this material space. This has never been the case. Right. It's like scientists, the normal way of working is you read papers and then you come up with hypothesis, you do an experiment and you learn, et cetera. So that's a very slow process. Now we can treat this as a search engine like we search the Internet. We now search the space of all possible molecules, not just the ones that people have made or that they're in the universe, but all of them. And we can make this kind of fully automated. That's the hope. We can just type. It becomes a tool where you type what you want and something starts spinning and some experiments get going and then outcome a list of materials. And then you look at it and say, maybe not. And then you refine your query a little bit and you kind of do research with this search engine where a huge amount of computation is and experimentation is happening, you know, somewhere far away in some lab or some data center or something like this. I find this a very, very promising view of how we can sort of come, you know, build a much better sort of materials layer underneath almost everything and also more sustainable materials where plastics are polluting the planet. If you can come up with a plastic that kind of destroys itself, know, after, I don't know, a few weeks. Right. And actually becomes a fertilizer. These are, these are things that are not impossible at all. These things can be done. Right. And we should do it.

Host 2 (14:24)

Can you tell us what. A little bit just generally about CUSP AI and then we. I have ton of questions.

Max Welling (14:30)

Yeah, so Cusp AI started about 20 months ago. And it was because I was worried about. I'm still worried about climate change. And so I realized that in order to get, you know, to stay within 2 degrees, let's say we would not only have to reduce our emissions to zero by 2050, but then, you know, another half century or even a century of removing carbon dioxide from the atmosphere, not by reducing your emissions, but actually removing it at a rate that's about half the rate that we now emit it. And that is a unsolved problem. But. And if we don't solve it. Two degrees is not going to happen. Right. It's going to be much more and I don't think people quite understand how bad that can be. Like 4 degrees, like very bad. So, so this technology needs to be developed. And so this was my and my co founder Chad Edwards motivation to start this startup. And also because, you know, we saw the technology was ready, which is also very good. So if you're, you know, the time is right to do it. And yeah, so we now, in the meanwhile we've grown to about 40 people. We've kind of collected 130 million. Investment into the company, which is for a European company, is quite a lot, I would say. It's interesting that right after that other startups got even more. So that kind of tells you how fast this is growing. But yeah, we are now at the. So we've built the platform, of course, but it's for a series of material classes and it needs to be constantly expanded to new material classes and it can be more automated because we're not putting LLMs in as a whole. Things gets more and more automated and now we're moving to sort of high throughput experimentation. So connecting the actual platform, which is computational, to the experiments so that you can get also get fast feedback from experiments. And I kind of think of experiments as something you do at the end, although that's what we've been doing so far. I want to think of it as what I would call a sort of a physics processing unit, like a ppu, right, which is you have digital processing units and then you have physics processing units. So it's basically nature doing computations for you. It's the fastest computer known possible. Even it's a bit hard to program because you have to do all these experiments. Also quite bulky. It's like a very large thing you have to do. But in a way it is a computation and that's the way I want to see it. So you can do computations in a data center and then you can ask nature to do some computations.

Host 1 (17:05)

Right.

Max Welling (17:06)

Your interface with nature is a bit more complicated, but then these things will have to seamlessly work together to get to a, you know, a new material that you're interested in. And that's, that's the vision we have. We don't say super intelligence because I don't quite know what it means and I don't want to oversell it, but I do want to automate this process and give a very powerful tool in the hands of the chemists and the material scientists.

Host 1 (17:32)

That's actually brings up a question I wanted to ask you. First of all, can you talk about your platform to whatever degree explain kind of how it works and what your thought process was in developing it?

Max Welling (17:43)

Yeah, actually it's been surprisingly. It's not rocket science. I would say it's not rocket science in the sense of the design. And basically the design that I wrote down at the very beginning is still more or less the design, although you add things like I, I wasn't thinking very much about multiscale models and I was common our radar that actually multiscale is very important. In the beginning I wasn't thinking very much about self driving labs but now I think, you know, we, that's. We are now at the stage we should be adding that. And so there is sort of bits and details that we're adding but more or less it's what you see in the slide decks here as well, which is there's a generative component that you have to train to generate candidates and then there is a digital twin, multi scale, multi fidelity digital twin which you walk through the steps of the ladder. You know that they do the cheap things first. You weed out everything that's obviously unuseful and then you go to more and more expensive things later. And so you narrow things down to a small number, those go into an experiment, you know, do the experiment, get feedback, etc. Now things that also have been more recently added is sort of more agentic sort of parts. You know, we have agents that search the literature and come up with, you know, actually the chemical literature and come up with, you know, chemical suggestions for doing experiments. We have agents which sort of autonomously orchestrate all of the computations and the experiments that need to be done. You know, they're in various stages of maturity and they can be continuously improved, I would say. And so that's basically. I don't, I think that part is rocket science. But you know, the design of that thing is not like surprising, but it's surprising hard to actually build it. Right. So that's the thing that is where the moat is in the data that you can get your hands on and actually building the platform. And I would say there's two people in particular I want to call out which is Felix Hunker who is actually building the scientific part of the platform and Alessandro De Maria who is building the sort of the skate of this, the mlops part of the platform.

Host 1 (19:50)

Yeah.

Max Welling (19:50)

And so. And recently we also added sort of Aaron Walsh to our team who is A very accomplished scientist from Imperial College. We're very happy about that. He's going to be our chief science officer and we also have a partnerships team that sort of seeks out all the customers. Because I think this is one thing I find very important. In principle, it's so complex to actually bring a material to the real world that you must do this in collaboration with sort of the domain experts, which are the companies typically. So we only start to invest in a direction if we find a good industrial partner to go on that journey with us.

Host 1 (20:30)

Makes a lot of sense. Over the evolution of the platform, did you find that human intervention. Human. I guess you could start out with a pure. You could. You can imagine two directions. One, you start out making everything purely automatic, automated, agentic, so on, and then later on you like find that you need to have more human input and feedback, different steps. Or maybe did you start out with having human feedback, lots of steps and then like kind of.

Max Welling (20:56)

Yeah.

Host 1 (20:57)

Figure out ways to remove. You know, that's.

Max Welling (21:00)

It is the second one. So you build tools. So you. So it's much more modular than you think. But it's like we need these tools for this application, we need these tools. So you build all these tools and then you go through a workflow actually in the beginning, just manually. So you put them first this tool, then run this tool, then run this one. So you put them in a workflow and then you figure out, oh, actually this porous material that we're trying to make actually collapses if you shake it a bit. Then you add a new tool that says test for stability. And so there's more and more tools and then you build the agent, which could be a Bayesian optimizer or it could be an actual LLM, you know, maybe trained to be a good chemist that will then start to use all these tools in the right way, in the right order. Yeah, right. But in the beginning it's like you as a chemist are putting the workflow together and then you think about, okay, how am I going to automate this? Right. One very easy question you can ask yourself is, you know, every time somebody who is not a super expert in dft.

Host 1 (21:58)

Yeah.

Max Welling (21:59)

And he wants to do a calculation, has to go to somebody who knows dft. And so could you start to automate that away, which is like, okay, make it so user friendly so that you actually do the right DFT for the right problem and for the right length of time and you can actually assess whether it's a good outcome, etc. So you start to Automate smaller, small pieces and more bigger pieces, etc. And the end, the whole thing is automated.

Host 1 (22:22)

So your philosophy is you want to provide a set of specific tools that make it so that the scientists making decisions are better informed and less so, trying to create an automated process.

Max Welling (22:36)

I think it's. This is sort of the same what you're saying, because, yes, we want to automate, but we don't see something very soon where the chemists and the domain expert is out of the loop. But it's a retreat. Right. It's like, okay, so first you need an expert to tell you precisely how to set the parameters of the DFT calculation. Okay, maybe we can take that out. We can maybe automate it. Right. And so increasingly more of these things are going to be removed.

Host 1 (23:04)

Yeah.

Max Welling (23:05)

In the end, the vision is it will be a search engine where you. Where somebody, a chemist will type things and we'll get candidates, but the chemist will still decide what is a good material and what is not a good material out of that list. Right. And so the vision of a completely dark lab where you can close the door and you just say, just find something interesting and then it will just figure out what's interesting and we'll figure out and say, oh, I found this new material too, blah, blah, blah, blah. Right. That's not the vision I have, at least not for a long time. So for me, it's really empowering the domain experts that are sitting in the companies and in the universities to be much faster in developing their materials. And I should say it's also good to be a little humble at times because it is very complicated, you know, to bring, to make it and to bring it into the real world. And there are people that are doing this for their entire lives. Yeah, right. And it's like, I wonder if they scratch their head and say, well, you know, how are, how are you going to completely automate that away, like in the next five years? I don't think that's going to happen at all. Yeah. So to me, it's a increasingly powerful tool in the hands of the chemist.

Host 2 (24:22)

I have a question. You've talked before about getting people interested based on having sort of a big breakthrough in materials. It's just incremental change. I'm curious what you think about the platform you have now and are sort of stepping towards and how are you chasing the big change, or is this incremental or is there. They're not mutually exclusive, obviously, but what do you think about that?

Max Welling (24:46)

We follow a mixed strategy, so we are definitely going after a big material. Again, we do this with a partner. I'm not going to disclose precisely what it is, but we have our own kind of long term goal. You could call a lighthouse or, you know, sort of moonshot or whatever. But it is going to be a, you know, really impactful material that we want to develop as a proof point that it can be done and that it will make it into the, into the real world. And that AI was essential in actually making it happen. At the same time, we also are quite happy to work with companies that have more modest goals. Like I would say, one is a very deep partnership where you go on a journey with a company and that's a long term commitment together. And the other one is like somebody says, I need a force field. Can you help me train this force field and then maybe analyze this particular problem for me? And I'll pay you a bunch of money for that and then maybe after that we'll see. And that's fine too. Right. But we prefer, you know, the deep partnerships where we can really change something for the good.

Host 1 (25:47)

Yeah.

Host 2 (25:47)

And do you feel like from a platform standpoint you're ready for that or what are the things that. And again, not asking you to disclose proprietary secret sauce, but what are the things generally speaking that need to happen from where we are to where to get those big breakthroughs like that?

Max Welling (26:04)

What I find interesting about this field is that every time you build something, it's actually immediately useful. Right. And so unlike quantum computing, which. Or nuclear fusion, so you work for, I don't know, 20, 30, 40 years and nothing, nothing, nothing, nothing. And then it has to happen. Yeah, right. And when it happens, it's huge. So it's quite different here because every time you introduce, so you go to a customer and you say, so what do you need? Right. So we work, let's say on, on a problem like water filtration. We want to remove PFAs from water.

Host 2 (26:39)

Yeah.

Max Welling (26:40)

Right. So we do this with the company Chimera. So they are deep partner.

Host 1 (26:44)

Yeah.

Max Welling (26:44)

For us. Right. So we on a journey together. I think that the breakthrough will happen with a lot of human in the loop because there is the chemists who have a whole lot more knowledge of their field and it's us who will, you know, help them with AI training. I have a new methods and in that kind of these interfaces, interactions, something beautiful will happen and that, that will have to happen first before this field will really take off, I think. And so in the sense that it's not a Bubble, let's put it that way. So that's people see that it's actual real what's happening. So in the beginning it will be very, you know, with a lot of humans in the loop, I would say, and I would, I would hope we will have this new sort of breakthrough material before, you know, everything is completely automated because that will take a while. And also it is very vertical specific. So it's like completely automating something. For problem A, you know, you can probably achieve it, but then you'll sort of have to start over again for problem B because you know, your experimental setup looks very different. You know, the machines that characterize your materials look very different. Even the models in your platform will have to be retrained and fine tuned to the new class. So every time, you know, you have a lot of learnings to transfer, but also, you know, the problems are actually different. And so, yeah, so I would want that breakthrough material before it's completely automated, which I think is kind of a long term vision. And I would say every time you move to something new, you'll have to start retraining and humans will have to come in again instead of okay, so what does this problem look like? And now sort of, you know, point the machine again, you know, in the new direction and then use it again.

Host 2 (28:23)

For the non scientists among us, me included a bit of a scientist, there's a lot of terminology you mentioned, DFT equivariance we've talked about. Can you sort of explain in engineering terms or the level of sophistication engineering, what is equivariance?

Max Welling (28:44)

So equivariance is the infusion of symmetry in neural networks. So if I build a neural network, let's say that needs to recognize this bottle, right? And then I rotate the bottle, it will then actually have to completely start again because it has no idea that the rotated bottle, well actually the input that represents a rotated bottle is actually rotated bottle. It just doesn't understand that where if you build equivariance in basically once you've trained it in one orientation, it will understand it in any other orientation. So that means you need a lot less data to train these models. And these are constraints on the weights of the model. So basically you have to constrain the weight such that it understands it and you can build it in, you can hard code it in. And yeah, the symmetry groups can be, you know, translations, rotations, but also permutations. Like in graph neural network, there are permutations and then physics, of course there's many more of these groups.

Host 2 (29:37)

To play devil's advocate, why not just use data augmentation by your model is in all the different orientations as an option.

Max Welling (29:44)

It's just not exact. It's like, why would you go through the work of doing all that where you would really need an infinite number of augmentations to get it completely right, where you can also hard code it in. Now I have to say, sometimes actually data augmentation works even better than hard coding the equivariance in. And this is something to do with the fact that if you constrain the optimization the weights before the optimization starts, the optimization surface or objective becomes more complicated and so it's harder to find grid minima. So there is also a complicated interplay, I think between the optimization process and these constraints you put in your network. And so yeah, you'll hear kind of contradicting claims in this field, like some people and for certain applications it works just better than not doing it. And sometimes you hear other people, if you have a lot of data and you can do data augmentation, then actually it's easier to optimize them and it actually works better than putting the aggregates in.

Host 1 (30:41)

Do you think there's kind of a bitter lesson for mathematically founded models and strategies for doing deep learning?

Max Welling (30:50)

Yeah, ultimately it's a trade off between data and inductive bias. So if your inductive bias is not perfectly correct, you have to be careful because you put a ceiling to what you can do. But if you know, you know, the symmetry is there, it's hard to imagine there, there isn't a way to actually leverage it. But yeah, so there is a bitter lesson. And one of the bitter lessons is you should always make sure your architecture scale, unless you have a tiny data set, in which case it doesn't matter. But if you, you know, the same bitter lessons or lessons that you can draw in LLM space are eventually going to be true in this space as well, I think. Yeah, yeah.

Host 2 (31:29)

Can you talk a little bit about your upcoming book and tell the listeners like what's exciting about it? They should read it.

Max Welling (31:37)

So this book is about, it's called Generative AI and Stochastic Thermodynamics. It basically lays bare the fact that the mathematics that goes into both generative AI, which is the technology to generate images and videos, and this field of non equilibrium statistical mechanics, which are systems of molecules that are just, you know, moving around and you know, relaxing to the ground state, or that you can control to have certain, you know, be in a certain state. The mathematics of these two is actually identical. And so that's fascinating. And in fact, what's interesting is that Geoff Hinton and Radford Neal already wrote down the variational free energy for machine learning long time ago. And there's also Carl Friston's work on free energy principle and active inference. But now we've related it to this very new field in physics which is called stochastic thermodynamics or non equilibrium thermodynamics, which has its own very interesting theorems like fluctuation theorems, which are, which we don't typically talk about but we can learn a lot from. And I think it's just, it can sort of now start to cross fertilize. When, when we see that these things are actually the same, we can, like we did for symmetries. We can now look at this new theory that's out there developed by these very smart physicists and say, okay, what can we take from here that will make our algorithms better? At the same time, we can use our models to now help the scientists, you know, do better science. Right. And so it becomes a beautiful cross fertilization between these two fields. You know, the book is rather technical, I would say, and it takes all sorts of things that have been done in stochastic thermodynamics and all sorts of models that have been done in the machine learning literature and it basically equates them to each other. And I think hopefully that sense of unification will be revealing to people.

Host 2 (33:25)

And when is it out?

Max Welling (33:26)

Well, it depends on the publisher now, but I hope in April I'm going to give a keynote at iclear and it would be very nice if I have this book in my hand. But it's hard to control these kind of timelines.

Host 2 (33:39)

I'm looking forward to.

Max Welling (33:40)

Great, thank you very much.