Loading summary
Transcript65 lines
- [00:00]
Lex Fridman
Let's say you have 10 agents and you give them the same problem and then you come up with solutions. What are you combining at that point? And is it being done in pairs? I mean, does each offspring have two parents or can you combine more than one?
- [00:16]
Kenji Doya
Our first paper on model merging does require access to the weights of the model. It's an impractical assumption because in reality most of the frontier models are closed. And, and even if we did have access to the waste, they probably do
- [00:29]
Lex Fridman
not Share architectures this episode is brought
- [00:33]
Sponsor/Announcer
to you by Tastytrade on Ionai.
- [00:36]
Lex Fridman
We talk a lot about how artificial
- [00:39]
Sponsor/Announcer
intelligence is changing, how people analyze information, spot patterns and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tastytrade is building. With Tastytrade you can trade stocks, options, futures and crypto all in one platform with low commissions, including zero commissions on stocks and crypto, so you keep more
- [01:17]
Lex Fridman
of what you earn.
- [01:19]
Sponsor/Announcer
The platform is packed with advanced charting tools, back testing, strategy selection and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI powered search feature that can help you discover symbols aligned with your interests, which is a
- [01:41]
Lex Fridman
smart way to explore markets more intentionally.
- [01:45]
Sponsor/Announcer
For active traders, there are tools like Active Trader Mode, One Click Trading and Smart Order tracking. And if you're still learning, tastytrade offers dozens of free educational courses plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out tastytrade. Visit tastytrade.com to start your trading journey Today I'm going to myself. TastyTrade Inc. Is a registered broker, dealer and member of FINRA, NFA and SIPC.
- [02:30]
Lex Fridman
I'm interested in Evolution Neuroevolution, which in your co author on a book that just came out. I've talked to Risto and Sebastian, but I'm.
- [02:45]
Sponsor/Announcer
I'm interested.
- [02:47]
Lex Fridman
I don't want to talk about the book so much. I want to talk about neuroevolution or evolutionary strategies and what you're doing at Sakana or Sakana in those realms and it seems like you've got a few layers. I mean you had the model merging which we can talk about and, and then you know, evolutionary AI generally. But then AI, a scientist takes those principles and applies it to research. So can we start by talking about evolutionary AI or evolutionary strategies, and your work in artificial life, which is part of all of that, and how you came to that area, particularly since you came from finance, which doesn't sound related.
- [03:54]
Kenji Doya
Yeah. Oh, it is quite related, I think. You know, I, I feel like having lived, went through the, the financial crisis in 2008, it really made me learn about extreme events in the world. Yeah, I think anything can happen. Like, I think people are usually underestimating these sort of a tail effects. And I think my experience at that time kind of shaped my, my views about how, how our life, you know, how our civilization has came to be, you know, how our intelligence has came to be. And that in turn has shaped my views on how artificial intelligence could be developed. So if, if we take a step back when I first. No, I. I studied neural networks when I was an undergraduate in the University of Toronto, some of the earliest I did my thesis back then on using neural nets to do computer vision, analyzing whether there's these steganography marks in images back then. But after entering the finance world, I did come back to the AI world when I first stumbled onto a few papers and articles about neuroevolution. This was in particular the, the works and the presentations by Kenneth Stanley and Risto, who is my co author of the book. So back then, you know, around more than 10 years ago, people were quite excited about image recognition. You have these GPUs now, they can recognize images and winning Alex nets and, and so on. But. So that was fine and dandy. People can recognize cat from a dog. But for me, I was more interested in more general AI. And then I was browsing and learning more about what's going on in AI, because it seems to be coming back. And you had these. I encountered a slide deck actually published by Risto. It was for a workshop, a tutorial. I think it was one of the earlier evolution conferences, like a gecko conference, where they're using neural networks for playing games, having characters move around, evolving the brains of these artificial creatures that can work together. So on one hand you have a bunch of neural nets that can recognize computer digits or images, and on the other hand you have this area where you're evolving the brains of these artificial creatures that can play in games. And I thought that was really cool. Right. So I said, how come the rest of the world are focused on training these neural network models for kind of these narrow classification tasks and you have this other field that no one talks about that are evolving these neural networks to create arts, to actually have multiplayer games. And to kind of evolved a morphology of neural networks as well. So that really got me in. And the more I studied about neuroevolution, the more I understand that's, you know, like the, it's a broader view of how of AI. You know, like we had machine learning back then where we're essentially training predefined neural networks to do a particular task. And I think in the field of neuroevolution or evolutionary computation, there are many ideas that are broader which appeal to me. There's the idea of not just sticking to systems where you can calculate the gradients to do something, because I think in our world many problems are such that there's actually no particular gradient signal. And the idea that if you just chase the gradient, sometimes you can get stuck, sometimes you have to wander around the solutions to find an even better solution. I like the idea in evolution of this open ended discovery or open ended search. So rather than having one particular objective in mind, your objective is to find new objectives, find new novelty. So I think this idea is not just it came out of the neuroevolution community, but it's not like implicitly an evolutionary thing. It's basically defining goals that haven't been done before. And this idea has actually moved beyond the evolution community into broader AI. And I also like the idea of using evolutionary computation to evolve new neural network architectures and morphologies. So this was actually an idea that was formed in the evolutionary area. And I think when I joined Google in 2016, some of these ideas has been incorporated into deep learning. So now, you know, when evolution actually the core concepts has been embraced within like Google DeepMind and Google Brain when I joined in 2016. So people then took these ideas to develop them further. Now we ended up into things like neural architecture search, using evolution to optimize how these connections of the transformers were connected. So I think right now, looking back, many of the ideas in evolutionary competition in the last decade has been embraced by the broader AI community and we're seeing some good progress because of this as well. Now there's many groups working on open ended discovery rather than just a fixed optimization. Many groups are using evolution to evolve architectures of neural networks evolving in the LLM world, evolving the prompts or the relationship between agents. And getting back to your earlier question, I became interested in AI because of evolution, but I'm not stuck just in evolution. I joined Google in their Google Brain research team. And I was also really excited about the deep learning revolution as well. So all of this fantastic Progress about large neural networks that can generate images, generate videos, and so on. So when I joined Google in 2016, I wasn't just working on evolution. I worked on some of the earlier works in generative AI with my collaborator Jurgen Schmidthuber. We actually extended some of his ideas from the early 90s and we brought the world models concept back to the modern era. So I conceived of some of the earlier video generation models. So at the time the goal was not just for creating video, that's more like specific application, but it's to build world models for general agents that can learn inside of their imagination so that they can be more sample efficient. They can, they can kind of have a model of how to plan actions without actually trying them in the actual world and to actually have a framework for building knowledge. So I was actually quite excited about deep learning in general, in particular world models, all this generative AI, now we call them generative AI, but before it was more like generating the data distribution of what you learn and how to combine generative AI or deep learning with concepts in artificial life and evolution. So I think as I became more of a professional researcher, even though I started getting interested in artificial life and evolution, I worked on the core deep learning for several years. And many of my works, I would say it's a bridge between neural evolution and deep learning. For example, when I was working on world models, I would use evolution components to help find adversarial weaknesses of the world models, how to actually evolve different strategies to trick the world model. I was using evolution to evolve the morphologies of these virtual robots, which would then use reinforcement learning to train the robots. So kind of I view evolution as an outer loop of how things progress and deep learning as an inner loop. And now fast forward many years when I started the Sakana AI company, we continue to combine the frontiers of generative AI and deep learning with evolution. One of the things that we worked on, as you mentioned earlier, is developing the technology to, to combine frontier models, whether they're open source or closed source. Because I think the, I think more fundamentally or philosophically, our intelligence as a species is not like a giant big brain. We are a collective intelligence of like, you know, 8 billion people walking around with 30 watts in this thing here. And we talk to each other and exchange ideas. So this is how our human artificial or whether it's an artificial or biological intelligence is formed. And I think this is also a good model of how artificial intelligence could be developed. So in the AI space, we have many. Our observation was we have a large ecosystem of open source models developed enthusiastically by many people working on AI models. And we have closed models developed by all these great Companies like Google, OpenAI, Cohere, Anthropic and in China, Deepseek and Alibaba and so on. So each of these models have their own unique strengths and weaknesses. And it kind of makes sense for me if we wanted to build a model with the very, very best performance on a particular thing, is to conceive of a scheme to, to combine different models and the different strengths of all these models. So one of the first experiments we did is we use an evolutionary algorithm and we use evolution strategies to merge different open models so that we can create custom capabilities. If we wanted to have an open Japanese language model that can do math a bit better or coding a bit better, we couldn't do that. And recently at, at Neurips this year we also presented another paper called abmcts where we use Monte Carlo tree search, which is kind of a form of a tree based approach of combining closed models where we can combine OpenAI's models, Google's models, deep SEQ models to get the very, very best performance on ARC AGI to some of these hard challenges. So one of our theme is to try to build systems to bring the best out of our foundation models. And I think evolution is a fairly strong candidate in that domain. The other parts that our company works on, I would say this collective intelligence part is more like developing a foundation layer, developing the very, very best foundation layer technology to get high performance on a particular task. The other observation that we've been seeing is with the advent of these frontier mod, I think large language models, they hallucinate a lot, but at the same time, just like humans hallucinate a lot, some of these hallucinations lead to great ideas. Sometimes we just hallucinate and some of the best inventions come out of serendipity. We didn't really plan for this stuff to happen and our company explored this, our lab explored this last year by using evolution combined with LLMs for the purpose of idea generation. So last year we presented a paper at NeurIPS in 2024 called DISCOP. And in that paper, I believe we're the first or one of the first published papers to show that LLMs can actually conceive of new LLM training algorithms. So the key idea behind DISCOP is you take a frontier model back then, it could be like ChatGPT, or it could be like Gemini, or it could be like llama and you would use that model to produce thousands of ideas on how to produce a better algorithm to train an LLM so that we can train LLMs more efficiently. We actually caught this idea internally called LLM Squared. It was a bit ambitious. We will use evolution to find some of the best solutions that were hallucinated. And this solution has to be tested on actual code. They actually have to run as a python code on Pytorch. And evolution would combine these ideas for the next iteration and next generation. Then we ended up with a, a pretty good state of the art algorithm for training and fine tuning LLMs. So that was one of the test cases of using evolutionary computation to select and breed the best algorithmic ideas generated using LLMs to train another LLM. And I think that was really neat because ultimately this technology is consuming so much resources in the world that it makes sense to use AI itself to, to make AI more efficient. So after that work we went a bit further. We thought that it was quite promising that LLMs can conceive of LLM training algorithms. Then we proceeded to propose the AI scientists, which is a general framework for a system of agents to conceive of new scientific ideas. For us, we limit ourselves to computational sciences because we may not have the resource to run a wet lab in our. And we would have this system of agents run experiments on our humble cluster. They have to actually collect and refine the experiments and perhaps even publish. Write up an academic paper using latex. And we would also train another LLM agent for the purpose of reviewing these papers. So you kind of have an adversarial thing going on. You have your scientists and then you have reviewers that were calibrated on scientific review. And then you can have an evolutionary system in the loop to kind of select the best ideas and iterate on. So I feel like using evolution combined with the search process of LLMs makes a lot of sense because rather than having just one LLM telling us what to do, it's much more reliable to have LLMs tell us a thousand different things and have an evolutionary algorithm or a search algorithm go through each of them to let us know what is the best thing and then iterate the ideas from there. So evolution for us has been a really good tool of, in summary, to combine the frontier models to get the very best performance. It's also been a remarkable tool for using this technology and repurposing them for the purpose of AI driven discovery. So we've been, I think our lab is known for these two themes and we've been able to grow as a company to build on these two R&D developments. And in the last six months or eight months using these tools, we've been actually able to grow a healthy and sustainable enterprise business in Japan. Japan of all places to deploy systems. So yeah, things are going well. So we're looking forward to next year and beyond as we grow this thing.
- [20:59]
Lex Fridman
Yeah. I just want to ask your work on world models. I just had Fei Fei Li on talking about Marble and her world labs. And in the past I've spoken a lot to Yan Lecun about Jeppa and his, you know, his world model work. And there's a fundamental difference between what Fei Fei is doing and what Yan is doing. Fei Fei is creating an internal representation that, that then outputs an explicit model of the world of whatever world you want to output. Yana's working on building an internal representation of the world as it exists in order to reason on that model. And which side is Sakana on and how do you build your world models?
- [22:02]
Kenji Doya
Yeah, I mean this is, I was actually quite proud of like the concept of world models being an actual thing in 2025, there's, there's many companies working on this and many startups and I think like both Fei Fei Li and Yann Lecun are great researchers on their own. Right. They're probably exploring things that are worth exploring. My impression, you know, as you mentioned, I think Faith 80's Startup World Labs focuses on more like developing a representation for realistic 3D environments. You can tell by many of the demos that they look like three 3D games. This is similar to Google's Genie efforts as well. You look at many of the generations and they look kind of a 3D generated. So I think the emphasis is to create a representation of a realistic 3D world where the principle behind that is if you're able to harness the compute and scale, you're able to create higher and higher fidelity the fidelity models where you can then do many things inside of the world models, even training entire agents inside the world models to learn new skills. So this is not just Fei Fei Li's work, but recently sema, which was released by Google, also export this concept of high fidelity world models that is hyper realistic. And I think Yann Lecun's view is more like developing more of the abstract representations for world models where these abstractions are required to learn reasoning concepts. You want to have these thought vectors inside the world models and this is how you would do things. So both are very actually required directions for this technology to be done. One is scaling and one is more like investigating the representation. You know, for me, when I started the world models project with Schmidt Huber around, you know, 2015, 16, I think the Schmidt Hooper's earlier concepts is actually more. I would say it preceded Yann Lecun's ideas. Usually that's the case. But a lot of the motivation behind the world models work that I worked on is on developing representation. Like I didn't really care whether the world models would output a really realistic, like a rendering of the real world. And in fact we showed that the more realistic it is, the higher fidelity it is, the easier for evolution or the agents to exploit some really weird bug in your simulation. So it's kind of a. I think this is more like a paradox. The larger your model is and the more detailed it is, it's actually easy to actually find some particular small thing for your agents. If your agent is to beat that game that's simulated in your role model, it'll figure out how to move in such a way to get unlimited scores right away. But what we found is if you make your model noisier, where there's actually a bigger gap between your simulated reality and the real game or the real environment, your agent actually is forced to learn real more important skills or it has to actually fend for itself in this really nasty game. So imagine your representation is not so perfect, but it's actually more tricky. It's more tricky to navigate in the dream than the real world. It actually presents some of a difficulty for your agents to develop the skill. So I think it's an open question. I think it's a balance. Of course you cannot have a pure latent model with no grounding in reality. Then that would be foolish and it will not learn anything useful. But I'm really interested in studying this balance. I think as humans, I'm sure that's we don't have the same type of capability as Fei Fei Li's role models or Genie model. We cannot explicitly, at least me, maybe not everyone on earth cannot draw out photorealistic version of a video of what I'm imagining in my head. But somehow we can reason in this latent representation. And I'm actually quite sympathetic to Yann Lecun's direction and maybe his future company would focus on more of this latent higher order representation of world models. And to be fair, this is also things that Jurgen Schmidt Huber has been conceiving since the early 90s. He has been working on like actually even getting agents to dissect directly into the hidden states of the world model. So you don't actually use the world model's hidden latent representation directly. The agents would basically examine directly into all the hidden connections of your world model to able to figure out intuitively what it should do, even skipping the planet planning part. So there's a lot of work to do and for me as a startup guy, I really am excited about the world models direction, but we have to really focus on what we're building as a startup. So we decided to focus mainly on collective intelligence, combining existing models and using these models for scientific discovery. Because although deep inside me I'm quite passionate about this model's direction and I think we do want to get back into it at some point.
- [28:19]
Lex Fridman
Yeah.
- [28:20]
Sponsor/Announcer
On the paper that was presented at
- [28:23]
Lex Fridman
Neurips, I understand model merging where you have essentially two code bases and the evolutionary system decides which you know how to combine the two code bases, what to keep, what to throw out, how to fit it all together. I, I didn't understand how you do that with proprietary models where you don't have access to the weights. And it's. You're talking about kind of a meta layer that, that explain to me what the paper that was presented that easier than me.
- [29:10]
Kenji Doya
It's getting really interesting because our first paper on model merging does require access to the weights of the model. I think it's an impractical assumption because in reality most of the frontier models are closed and even if we did have access to the waste, they probably do not share architectures with each other. So what we decided to do, and it's also, you know, as these models become bigger, the context length also becomes longer as well. So there is a duality between the prompt and being accessed, accessing the weights of the model, the more context you can put in, it's in a way influencing what the model is doing. So in this abmcts work that we presented at Neurips this year as a spotlight, we used this scheme to prompt or search through the reasoning techniques of multiple models. You would actually use this sort of bandit approach to give the variations of the same questions for multiple different models, get their responses and decide based on the responses how to actually ask the next iteration of responses in a tree like fashion. So rather than using evolution to combine the weights, we're using like an evolution type scheme, in this case Monte Carlo tree search to actually conceive of different conversations with the model. So how to drive the conversation with these batch of frontier models to get really great answers for a particular task.
- [31:04]
Lex Fridman
And so you're prompting a range of models and you're taking the outputs and then creating a new set of prompts from those outputs and then re prompting that same series of models.
- [31:24]
Kenji Doya
Yes, that's correct. And it's done away in a tree search method. So you would actually go down the rabbit hole of the most interesting dialogues. So it's like if you have a party with 100 people, you would find the five most interesting conversations and then you continue to only talk to those five people and then you keep on going and going. So that's one work we're doing in our company. We're actually exploring multiple different ways of combining frontier models as well. So we will have some exciting work coming up in the next few months where we actually have better results for this direction. So I think it's quite exciting for, for combining different models.
- [32:11]
Lex Fridman
When you say in the coming, you're talking about working on the context and you're not talking about using open weight models and combining code bases and creating new merged models, is that right? You're talking about this other strategy.
- [32:33]
Kenji Doya
Yes.
- [32:33]
Lex Fridman
When you're saying that.
- [32:34]
Kenji Doya
Yeah, yeah.
- [32:36]
Lex Fridman
And can you, you know, with AI scientists. I wrote a little article in that, when you're. That paper was written by the model and system and submitted, I think it was to icml. Was it? Or I, I clear. I can't remember, but submitted.
- [32:57]
Kenji Doya
I, I believe we submitted. It's. It's quite early work. I mean the, the paper is not like particular good or anything, but we, we.
- [33:05]
Lex Fridman
No, I know, yeah.
- [33:07]
Kenji Doya
We had to do a few things because right now, you know, the using generative AI to produce papers at an academic conference, it is like a, it, it's an issue that we have to guide very carefully. So when we ran this experiment, we obtained permission from the program chairs of iclr, iclear. We also obtained explicit permission from the workshop organizers for this experiment. And we had like an ethics committee at the University of British Columbia approve this experiment. So it's like because we're running a human experiment and what we did was we submitted three papers into a workshop at ICLR earlier this year and I think one of them obtained a score that would have passed the workshop. Right. But what we agreed with the organizers is by default, even if they were accepted, we would, we would actually reject them, withdraw them because it's part of the experiments. So that was kind of the blind test that we ran.
- [34:13]
Sponsor/Announcer
That's right.
- [34:14]
Lex Fridman
But what was happening in the background there is that.
- [34:17]
Sponsor/Announcer
Are you
- [34:20]
Lex Fridman
so to Back up a little bit. What's interesting about evolutionary strategies is that unlike gradient descent where you're following one path and trying to find a local or universal optimum or optima, right. You're searching, you have many models and so you're looking across a much larger space and so is how. Just describe how AI scientists worked in that experience experiment and how it's evolving.
- [35:05]
Kenji Doya
Yeah. Question. So there is a progression in the way we're building the AI Scientists program as well. The very first version AI scientists that we released last year, it was more of an AI co scientist where the user would actually present the AI scientists with an initial idea and an initial code base for the experiment. The AI scientist is more like a grad students where it would take that code base and it would find novel ideas to extend it, run experiments and to get more interesting results. For the ICLEAR workshop experiment it was a test for our AI scientist version two. So we actually worked on this variation where there's less of a requirement to provide it with a so called templates of the particular idea that it wants to work on. It could actually work on its own idea. We just have to guide it by an initial prompt or a general theme and it would. In the AI scientist v2 compared to v1, there's a tree type progression of evaluating these ideas. So inside the AI scientist it would actually branch out to different variations of ideas and decide what is the most interesting things it should pursue. So we ran this AI scientist a few times and used the reviewer to select the the three most highly rated papers according to the automated reviewer and we submitted those three to the workshop. I think there's still a lot more room to be done going forward right now. What the limitation is, the AI scientist is still one system. I think science as we know it is not done by one person, but like we're building on the shoulders of giants and we keep on exchanging ideas and building on top of each other's ideas. So one thing that we see in the future is not necessarily just one single AI scientist, but more like an AI community of multiple different AI scientists working together and incrementally building on top of each other's ideas and branching out to discover new ideas. So hopefully in the future we're able to make more progress on that front. Because I think right now one of the biggest limitations is you have this single agents, multi shot idea generation which is great for this, I would say graduate student descent, you have this grad student that would improve an idea. But our goal is to actually conceive of novel transformative ideas, which I think is a big limitation in the current AI scientists in general, for all labs and all companies. And this is one of the grand challenges that we want to work on.
- [38:12]
Lex Fridman
Yeah, and how does that work?
- [38:14]
Sponsor/Announcer
So you have a population of models
- [38:18]
Lex Fridman
and you give them all the same initial problem, right? And then they, they will end up with a solution or an output. And then you, you score those outputs and you take the, the best ones and combine them in pairs and then ask those child models, I mean this, the offspring, the same question, and then they go through another generation. Can you just describe how that works?
- [38:56]
Kenji Doya
Yeah, this is a very exciting open question that inside our lab we're working on several variations of this, but actually we're not the only ones. I think recently at Stanford they also are launching this so called AI conference or like where people would be submitting papers generated using AI agents. And then you have a conference type system that would vet these papers to see whether they're good or not. So I think there are many ways that one can go about. The simplest way, as you alluded to, is okay, you have 1,000 agents and all of them have to just figure out a new incremental improvement to this idea. And you would use evolutionary algorithm to combine what are the most promising ideas. And we would call these parents. And you would conceive of child ideas. And these child ideas, you would then branch out to the same 1000 agents and you keep on doing it and so forth. And in fact, this paradigm has already been done this year in Codespace and it was initially released by Google called Alpha Evolve. I'm sure you heard of the Alpha Evolve. Of course, our company extended it into something called Shinka Evolve, which we believe to be a more efficient sample, efficient version of Alpha Evolve, where we showed that scaling up the agents for hundreds of different agents, we can come up with different ideas. And you use parent child sampling and it will create better code ideas, create better algorithms to solve a particular problem. And this is great. We can now have state of the art task on circle packing optimization algorithms, these sort of different code algorithms. It's really useful when you have a particular task and you want your agents to design the best program for that. I think if we want to be a bit more ambitious. So this is great for software engineering or at least designing algorithms for a different task. If we want to have the grand challenge of coming up with new ideas is not good enough. I think for the new ideas we have to insert the concept of Interestingness and open endedness into the search concept, where these agents are not only optimizing for how well they're doing in a particular task, they're also looking at scoring themselves about whether they believe their solution is novel or interesting or it adds something to the table. So what we're trying is also to have these artificial communities build on top of each other's ideas. And many of these concepts are not novel. In the evolutionary computation community there's a concept called equality diversity. So we're not just optimizing for the quality or the score of a task, but we're optimizing for the diversity of the task as well. So if your approach is too similar to the other approach, you have to get the very top score of this batch of approaches. If your approach is completely weird and not really explored, then you get a bit of a leeway. We don't expect that much of a good quantitative score from you. The bar is better just because you're doing something new. So many experiments before deep learning has shown that this sort of approach leads to agents with better generalization, like the robot that can continue to walk even if you damage your legs, and so on. It was published by Jeff Klune and his group in Nature many years ago. And I think the same applies to the AI scientist type work as well. So you would want to see an agent produce. Interestingness.
- [42:57]
Lex Fridman
Yeah, I, I, I just have a, a naive question, so forgive me, but I'm just a journalist. When you, if you, let's say you have 10 agents and, and you give them the same problem and, and then you, you come up with solutions and you want to combine or combine those, what are you combining at that point? And is it being done in pairs? I mean, does each offspring have two parents or can you combine more than one?
- [43:37]
Kenji Doya
Yeah, this is a great question and I think, you know, you could do it in several different ways. The approach that works quite well is if you take two parents, two solutions, then you would feed those two solutions just as a context. You can concatenate them and say, hey LLM, these are two previous solutions that seems to work. Combine these two solutions and take the best and produce one more solution. For me, the LLMs will be able to do that. It would actually give you the solution. This is one of the magical things about the current foundation models because you can pull that random. If you're writing English, like before deep learning, we have to conceive of a particular way of combining things in weight space. Things have to fit together. But now if you're generating ASCII code, you can concatenate them into a longer context prompt and just tell it to combine these solutions. So it's quite, you know, we're in this sort of interesting stage in our technology when we can treat some of these LLM models as an employee where we can tell them in plain text what we want to do.
- [44:55]
Lex Fridman
Yeah. And you're not simply combining discrete. I mean, you have 10 models and you have 10 solutions. There are multiple pairs within those 10 solutions. So you're going to have more than 10 offspring. Right. Or more than five offspring, I guess, because you can, you know, A with B or A with C or A with D. B with A, B with C, B with D. Right, yeah. And is all of that done in one epoch or one go?
- [45:39]
Kenji Doya
Yeah. So the open question, and this is something that our company explores quite deeply, is in the parent selection process. So if you have a system of agents that comes up with like a thousand possible different solutions and each solution in the quality diversity framework, they can have a performance and they can have like, how weird are these solutions? So you can have a grid of these scores and the open question is how to select the next 10 solutions to breed. And I think studies before have shown that if you don't, you could do the, in the literature they call it the elitist combination. You would simply just take the 10 best performing ones and throw away everyone else and just combine the elitist solutions. But you end up with getting into some local minima quite earlier if you just do that. So what the quality diversity literature suggests if you would combine, sometimes you would combine the low performing but highly weird and innovative solutions with the top performing ones to actually get new innovations. And I think these are the ideas that we're exploring as a lab, how to best combine them. And I think we've showcased some of these algorithmic innovations and choosing in our open source project called Shinka Evolve, where we actually have explicitly these sort of heuristics of how to combine parent selection to generate offspring. And we experimentally showed that leads to a dramatic improvement in sample efficiency. Getting the best solutions much more earlier. But I think there's still a lot of work to be done in this idea. Maybe in the future it could be an LLM agent that's tasked with choosing which ones you should use. So it's quite interesting.
- [47:46]
Lex Fridman
Yeah. And I know you don't have a lot of time, so let me shift to sort of a more philosophical question. If this, if you get this working well, the, the the scope of human knowledge is finite obviously, but the, the universe of possible knowledge is potentially infinite. Right. And, and using gradient methods or, or just human creativity, it's very difficult if not impossible to think outside the scope of, of what's known. And you know, periodically you have someone who's extremely creative that, that comes up with a new idea. Is that ambition or the belief that an AI system could push beyond the, the boundaries of existing knowledge and really be creative or in that these models are ultimately all trained on existing knowledge. Is it, is it, you know, yes, there's, there's some new ideas in mixing existing knowledge, but getting out of the, the, the bubble of human knowledge
- [49:22]
Kenji Doya
may,
- [49:22]
Lex Fridman
may not be possible.
- [49:24]
Kenji Doya
What are your question? I, I, I think you know, with, with, you know, of course we don't know for sure, but I'm quite confident that eventually either us or other labs would be able to create systems of agents that would be able to extend the boundaries of human knowledge and creativity. You know, I don't think that the current systems are that capable. I think humans are still great at creativity, especially when there are 8 billion of us. And of course some of us are very creative. But that's why I believe in this collective intelligence. As we scale up the number of agents, giving them their own room to explore, due to law of large numbers, there will be many tail possibilities of new things being discovered. I think most of the technology would not discover new things, but as you scale up the number of agents, the number of artificial scientists to a larger number, of course some of them would discover new ideas and new possibility. And I think to your point about training within the bubble of human knowledge, and that's a valid one because you're constantly outputting things in the data distribution that you're trained on. But how I think we're going to get around that is these AI scientists don't operate in a vacuum. I think there are many AI scientist projects happening right now. Our company, we're focused on the algorithmic generation and other companies are focused on material synthesis, AI for real scientists or drug discovery. And ultimately these agents have to interact with the real world. So even in our case the agents actually have to write code and run the code on Python and run it on a cluster. So you are interacting in the real world and collecting feedback, new feedback from the real world which may not be there in the first place place. So I, I do think that these agents will collect new data as they experiments with the real world and that's going to lead to new innovation and New discovery. You know, whe whether real world interaction is a requirement or not to discovering an idea that is an, that is an open question that one like you know, there are proponents to say that maybe that maybe you actually don't need embodiments. But I think it certainly helps because you're getting feedback from the real environment
- [51:53]
Lex Fridman
embodiment or at least grounding in a world model. Right. And then one more question and I forgot to plug in my power. One other question is don't you need continual learning in order to get to. To true creativity? And, and there are blockers to that or I, I know your, your, your systems can adapt and decide to forget some things to learn new things. And so there. It's not a static knowledge base. It's sort of evolving as the model explores. But ideally you wouldn't be forgetting, you would be adding knowledge. And then the synthesis of, of old knowledge and new knowledge may create new ideas. Are you guys. Do you think that's right? And, and how important is continuous learning?
- [53:06]
Kenji Doya
I think that's a fair point. And at neurips this year, continual learning is mentioned many times at several keynotes as the next big AI challenge. Even Richard Sutton's keynotes as you mentioned. If the agent just doesn't remember what it has experienced, how could it come up with new knowledge? I think the way we're doing it right now, I think it definitely helps to have a breakthrough in continual learning. But absence of that, we can still continue to chug along as if we're synthesizing continual learning agents. One of the good things about these agents is they can use tools. They can actually have access to a set of knowledge in the past. So as we're building these agents with scientific discovery, you could actually log everything in a text file or in a markdown documents or in a PDF paper where it would actually have to know retrieve all these ideas. And this is a problem not just for agents, it's for real scientists as well. How many ideas gets rediscovered by young scientists? You cannot expect even human scientists to know every single publication out there. So there is a need for the agents capability to, to do research, to search archive of existing ideas. Whether those ideas are known ideas or conceived by itself or earlier versions of itself. You would still need to do that regardless of whether you have a continual learning breakthrough or not.
- [54:40]
Lex Fridman
Yeah, that's interesting. Okay, we're up to the hour. I want to ask one last question.
- [54:48]
Sponsor/Announcer
I asked Fei Fei Liquid, what's your guilty pleasure?
- [54:52]
Lex Fridman
I mean you work very hard on research is there. Do you play Jim Rummy in the evenings or do you.
- [55:01]
Kenji Doya
I. I like to watch retro anime.
- [55:06]
Lex Fridman
Oh, is that right? Well, in your Japan. Oh, that's right.
- [55:09]
Kenji Doya
Dragon Ball and. And so on.
- [55:10]
Lex Fridman
Or Astro Boy or.
- [55:12]
Kenji Doya
Yeah, bring back the anime at the time when you know it was purely hand drawn without any computer tool.
- [55:19]
Lex Fridman
Yeah. Yeah.
- [55:20]
Sponsor/Announcer
That's fascinating.
- [55:21]
Lex Fridman
Yeah.
- [55:21]
Sponsor/Announcer
This episode is brought to you by Tastytrade. On Ion AI we talk a lot about how artificial intelligence is changing, how people analyze information, spot patterns and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tastytrade is building. With Tastytrade, you can trade stocks, options, futures and crypto all in one platform with low commissions, including zero commissions on stocks and crypto, so you keep more
- [56:07]
Lex Fridman
of what you earn.
- [56:09]
Sponsor/Announcer
The platform is packed with advanced charting tools, backtesting, strategy selection and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI powered search feature that can help you discover symbols aligned with your interests, which is a
- [56:31]
Lex Fridman
smart way to explore markets more intentionally.
- [56:34]
Sponsor/Announcer
For active traders, there are tools like Active Trader Mode, One Click Trading and Smart Order Tracking. And if you're still learning, Tastytrade offers dozens of free educational courses, plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out tastytrade. Visit tastytrade.com to start your trading journey today. I'm going to myself. TastyTrade Inc. Is a registered broker, dealer and member of FINRA, NFA and SIPC.