Eye On A.I. – Episode #323: David Ha: Why Model Merging Could Be the Next AI Breakthrough

Host: Craig S. Smith | Guest: David Ha
Date: February 24, 2026

Episode Overview

In this rich and wide-ranging conversation, Craig S. Smith talks to David Ha—AI researcher, entrepreneur, and CEO of Sakana AI—about "model merging" as a promising new paradigm in AI development. David outlines the historical context and conceptual underpinnings of evolutionary strategies in AI, describing how they contrast with gradient-based approaches and how they’ve influenced recent developments in artificial intelligence, especially in the context of large language models (LLMs), model merging, and the emerging concept of AI scientists. The discussion is grounded in detailed technical explanations but is interlaced with philosophical reflections on creativity, collective intelligence, and the frontier of machine-led scientific discovery.

Key Discussion Points & Insights

1. David Ha’s Journey: From Finance to Evolutionary AI

• Background: David shares his unconventional path from undergraduate neural network experimentation to finance and back to AI, influenced by the unpredictability of events like the 2008 financial crisis.
• Early Neuroevolution Influences: Inspired by Kenneth Stanley and Risto Miikkulainen, David moved away from narrow image classification to embrace open-ended discovery in neuroevolution.
• Quote:

  “My experience at that time [2008 crash] kind of shaped my views about how our civilization has come to be, how our intelligence has come to be. That in turn has shaped my views on how artificial intelligence could be developed.” (03:54, David Ha)

2. Evolutionary Strategies vs. Gradient Descent

• Key Ideas: Neuroevolution doesn’t depend on gradient signals and is suited for problems without clear optimization directions. Evolution emphasizes open-ended search—finding new objectives, not just optimizing fixed ones.
• Bridging Worlds: David explains how evolutionary ideas have merged with deep learning, influencing neural architecture search and optimization in LLMs and other domains.
• Quote:

  “Rather than having one particular objective in mind, your objective is to find new objectives...this idea has actually moved beyond the evolution community into broader AI.” (10:20, David Ha)

3. The Rise and Promise of Model Merging

• The Rationale: Each large foundation model (from OpenAI, Google, DeepSeek, etc.) has unique strengths—so why not combine them?

• Technical Challenge: Early model merging required access to model weights (impractical for closed models). Recent advances—such as Sakana AI’s abmcts method—allow combining models via their text interfaces using search-based prompting strategies.

• Notable Paper: abmcts presented at NeurIPS 2025.

• Quote:

  “We use an evolutionary algorithm and evolution strategies to merge different open models so that we can create custom capabilities...and recently...we use Monte Carlo tree search, a tree-based approach of combining closed models...” (18:25, David Ha)

• Explanation of Model Merging Without Weights:

  • Instead of combining model weights, output prompts are run through multiple closed models. Using a tree search, the best responses are selected and inform the next round, focusing the search as in an evolutionary process.
  • Quote:

    “Rather than using evolution to combine the weights, we're using like an evolution type scheme, in this case Monte Carlo tree search to actually conceive of different conversations with the model.” (29:10, David Ha)

4. Sakana AI’s Experiments with LLMs and AI Scientists

• ‘LLM Squared’ & Ideas: Using frontier LLMs to generate thousands of novel ideas for training LLMs, and then using evolutionary algorithms to identify and refine the most promising ones. This led to state-of-the-art algorithms for LLM training.
• ‘AI Scientists’ Project: Moving towards systems where autonomous LLM agents generate, test, submit, and review novel scientific papers with minimal human intervention.
• Quote:

  “LLMs can actually conceive of new LLM training algorithms…they actually have to run as python code…and evolution would combine these ideas for the next iteration…” (17:30, David Ha)

  • ICLR Experiment: Three AI-generated papers were submitted to a workshop following comprehensive ethics approvals; at least one scored high enough for acceptance.

5. World Models: Competing Visions and Sakana's Direction

• Fei-Fei Li vs. Yann LeCun: World models either focus on high-fidelity, 3D-like ‘real’ environments (Li/World Labs), or on abstract latent representations enabling reasoning (LeCun/Jeppa).
• David’s Position: Prefers a middle ground, valuing latent, abstract representations, and embracing the paradox that more realistic world models can be more easily ‘gamed’ by agents.
• Quote:

  “The more realistic [the world model] is, … the easier for agents to exploit some weird bug in your simulation.” (23:55, David Ha)

6. The Mechanics of Evolutionary AI Discovery

• Population Approaches: Describes how a population of agents are each tasked with the same problem, solutions are scored, combined, and iteratively improved—mirroring evolutionary parent-child selection.

• Innovation Beyond Optimization: It’s vital not just to seek high-performing solutions, but diversity and novelty, similar to how biological evolution prizes survival in different niches, not just maximum fitness.

• Quote:

  “We’re not just optimizing for the quality or the score of a task, but for the diversity of the task as well...[so] if your approach is completely weird and not really explored, then you get a bit of a leeway.” (41:15, David Ha)

• Technical Note: With LLMs, ideas can be ‘combined’ via context concatenation and prompting—no need for complex hand-designed mechanisms.

7. Creativity, Novelty, and the Limits of Pre-Training

• Can AI Break Out of Known Knowledge? David expresses confidence that, with enough agents, some will discover genuinely new insights beyond the training distribution, especially as they interact with the real world (e.g., by running code or conducting physical experiments).
• Quote:

  “As we scale up the number of agents...due to the law of large numbers, there will be tail possibilities of new things being discovered.” (49:22, David Ha)

8. The Grand Challenge of Continual Learning

• Open Problems: One key AI challenge is continual learning—the ability for models to retain, build upon, and synthesize knowledge without catastrophic forgetting.
• Workarounds: Even absent true continual learning, AI systems can reference archives of past results, but both human and machine scientists must rediscover forgotten ideas.
• Community Vision: The future lies in multi-agent, community-based AI science, not just single, superpowered “genius” agents.

Memorable Moments & Notable Quotes

• On Model Merging and Prompt-Based Evolution:

  “You can concatenate [two solutions] and say, hey LLM, these are two previous solutions that seem to work. Combine these two solutions and take the best and produce one more solution.” (43:36, David Ha)

• On Open-Ended Discovery:

  “The objective is to find new objectives.” (10:20, David Ha)

• On Future Innovation:

  “Our goal is to actually conceive of novel transformative ideas, which I think is a big limitation in the current AI scientists in general, for all labs and all companies.” (36:27, David Ha)

Important Timestamps

| Timestamp | Segment Description | |-----------|--------------------| | 03:54 | David's journey from finance to neuroevolution; AI influences | | 10:20 | Open-ended discovery's importance in AI | | 18:25 | Model merging rationale; methods and motivation | | 23:55 | Why higher fidelity world models are easier to exploit | | 29:10 | Details of abmcts method for merging closed models | | 43:36 | How LLMs can combine solutions through concatenated prompts | | 49:22 | Scaling up artificial scientists for novel discovery | | 53:06 | Continual learning as AI’s next frontier |

Tone & Style Notes

• The conversation is technically deep but engaging and full of curiosity.
• David Ha is candid about limitations and ambitions, punctuating technical points with philosophical commentary.
• Lex Fridman, standing in as a journalist guest host, asks both foundational and “naive” questions, surfacing points important for both experts and lay listeners.

Conclusion

This episode dives deep into the cutting edge of AI research: evolutionary strategies, the challenges and opportunities for model merging, and the ambitious vision of AI as not just a tool for discovery, but potentially as an autonomous scientific community. David Ha makes the case for combining collective intelligence and algorithmic diversity to push AI beyond today’s limitations, exploring the frontiers where human and artificial creativity might meet—and someday, surpass—the boundaries of known knowledge.

Final Lighthearted Moment:

• David’s guilty pleasure: watching retro hand-drawn anime like Astro Boy and Dragon Ball—“bring back the anime at the time when it was purely hand-drawn without any computer tool.” (55:11, David Ha)