AI + a16z Podcast: Beyond Leaderboards – LMArena’s Mission to Make AI Reliable

Date: May 30, 2025
Guests: Anastasios N. Angelopoulos, Weyland Chang, Jan Stoica (LMArena founders)
Host: Anjane Mittha (a16z General Partner)

Episode Overview

This episode dives deep into the journey and mission of LMArena—a platform that began as a UC Berkeley project and is now a company seeking to transform how AI models are evaluated. Rather than relying on static benchmarks, LMArena champions large-scale, real-time, community-driven testing to make AI systems more reliable, accountable, and suited to real-world applications. The founders share insights on the evolution of evaluation, the importance of subjective human preferences, new technical methods, and their commitment to neutrality and open source.

Key Discussion Points & Insights

1. The Case for Real-Time, Community-Driven AI Evaluation

[01:26–04:54]

Traditional benchmarks (like MMLU) are necessary but insufficient for today’s fast-evolving, mission-critical AI systems.
LMArena aims to scale to millions of users to tap into diverse, representative feedback across industries, enabling targeted, robust evaluations—for everything from healthcare to shipping to defense.
The majority of real-world questions are subjective, even in science and enterprise. LMArena embraces this reality rather than ignoring it.

“The future is about real-time evaluation... in the wild.” — Anastasios N. Angelopoulos [01:26]

2. Scaling: From Open Source Roots to Broad Industry Impact

[02:26–06:54]

LMArena aspires to support both global communities and private, in-house evaluation by labs or enterprises, offering “private arenas” for specialized use.
The platform assists both big and small labs, providing tools for pre-release and ongoing model testing to inform which models are production-ready.

“In fact, one of the things that we help everybody do is pre-release testing of their models.” — Anastasios N. Angelopoulos [05:18]

3. The Value and Challenge of Subjectivity in Evaluation

[06:54–14:52]

LMArena leverages the collective wisdom of its users, countering the idea that only "experts" should define evaluation standards.
The team acknowledges (and technically addresses) criticism regarding biases (e.g., toward longer, more emoji-laden answers) through “style control”—a method to decompose and adjust for stylistic influences in model evaluation.

“What we’re building is an ever richer evaluation that can tell us all the factors that go into response, how you can optimize people's preferences, keeping style fixed.” — Anastasios N. Angelopoulos [13:44]

4. Fresh Data vs. Static Exam Overfitting

[19:29–21:24]

Unlike static benchmarks prone to “overfeeding” (where models are implicitly trained on test sets), LMArena constantly generates new, fresh prompts and votes, insulating against overfitting.

“Chatbot arena is immune from overfitting by design. Means you're always getting fresh questions.” — Anastasios N. Angelopoulos [21:24]

5. Specialist Arenas: Going Beyond Chat into New Domains

[15:00–24:54]

The expansion into domains like coding (Web Dev Arena) exemplifies how LMArena accommodates more “objective” tasks while maintaining user-driven evaluation.
The inherent difficulty in tasks like real-time code generation makes for sharper, more discriminative testing.

“Programming is actually a very general-purpose discipline...and yet it seems the capabilities in a very general way are still being...captured well on a specialist arena like web dev arena.” — Anjane Mittha (Host) [22:09]

6. Toward Personalized Evaluation

[26:03–28:41]

The vision is for every user (and every task) to have a bespoke leaderboard—reflecting the models that work best for them and their needs.

“It should be personalized just for you. You should understand which models are best for you.” — Anastasios N. Angelopoulos [26:25]

7. The LMArena Origin Story & Technical Innovations

[30:06–43:47]

Sparked by Vicuna (one of the first open-source ChatGPT alternatives), technical and logistical challenges in evaluation led to creative adoption of LLMs as judges, user voting, and adopting ranking algorithms (ELO, then Bradley-Terry).
Berkeley’s interdisciplinary environment enables nimble, innovative teamwork—a stark contrast with the slow, siloed approach of large industrial labs.

“The inspiration was how humans in real life rate players or teams... head to head. That’s how we adopted...battle mode.” — Jan Stoica [37:08]
“It was started as a fun project.” — Weyland Chang [37:18]

8. From Research Project to Company – The Strategic Shift

[47:40–55:34]

As usage and demand exploded, extending to hundreds of models and diverse users, it became clear sustained impact required a company, funding, and productization.
New features (like “prompt-to-leaderboard”) leverage massive user data to provide granular, even personal, model recommendations.

“Prompt to leaderboard... you give me your prompt. Can we tell you which models are best for that prompt specifically?” — Anastasios N. Angelopoulos [52:25]

9. Overfitting, Fairness, and the Evolving Benchmarking Debate

[58:43–63:40]

Benchmarks = static, supervised; Arena = reinforcement, user-in-the-loop, ever-fresh.
Overfitting in Arena is essentially moot; doing well means you are serving real user needs, not gaming the system.

“People still think about Arena as a benchmark... But what hasn’t permeated is when you have fresh data, you can’t overfit.” — Anastasios N. Angelopoulos [61:14]

10. Building the Best Testing Platform for an Evolving AI Landscape

[68:43–82:11]

Testing robustness and infrastructure-scale are major technical challenges; collecting high-quality, intrinsically-motivated user votes ensures data fidelity.
As AI “apps” become more integrated—layering in memory, context, and application-specific logic—the platform evolves. LMArena plans SDKs for integration, moving evaluation ever closer to the customer and use-case.

“You need something similar [to CI/CD] for these [AI] models right now...you also want to test your models, your checkpoints on Sherbot Arena for all the reasons we mentioned.” — Jan Stoica [68:43]

11. Red Team Arena and Security, Reliability Concerns

[95:10–99:45]

LMArena’s approach extends to safety: Red Team Arena lets a community of “jailbreakers” actively test and leaderboard models for their security/safety characteristics—mirroring user-driven, real-world red teaming at unprecedented scale.

“In Red Team arena we have a leaderboard. Not just for model, but for jawbreaker — who is best at identifying issues.” — Weyland Chang [97:21]

12. Open Source, Neutrality, and Core Values Going Forward

[91:34–94:26]

Openness with code and data, maintaining academic neutrality, and community engagement are non-negotiable values as the project grows as a company.
Openness builds trust in evaluation results and attracts the best talent and most innovative collaborators.

“If people want to ask the question, ‘Hey, how are models performing?’ ... Just go look at the data. That’s what we did with Llama—we just released the data.” — Anastasios N. Angelopoulos [92:49]

Notable Quotes & Moments

On the end of “hard exams” era:
"Static exams were useful three years ago. The future is about real-time evaluation, real-time systems, real-time testing in the wild." — Anastasios N. Angelopoulos [01:26]
On democratizing expertise:
"Everybody actually has their own opinions and... there’s so many natural experts in the world... Their vote means so much." — Anastasios N. Angelopoulos [07:57]
On subjectivity & bias:
"People vote for longer responses preferentially. Can we learn this bias and actually adjust for it? The answer is yes. That’s why we're making style control default." — Anastasios N. Angelopoulos [13:44]
On why fresh data is disruptive:
"Overfitting means you’re doing well on the test data but only on the train data. There cannot be overfitting [in Arena] because we have continuously fresh data." — Jan Stoica [63:40]
On the vision for company & platform:
"Neutrality, innovation, trust... We want the world to know what the best ways are of evaluating these models and accelerating the ecosystem." — Anastasios N. Angelopoulos [92:49]
On open source & transparency:
"If people want to ask the question, hey, how are models performing on their why are they performing well? Go look at the data. That's what we did with Llama, right? Just go look." — Anastasios N. Angelopoulos [92:49]

Key Timestamps for Major Segments

[01:26] Real-Time Testing vs. Static Benchmarks
[05:18] Working with Labs (Big and Small), Pre-Release Testing
[13:44] Addressing Voting Bias, Style Control
[19:29] Preventing Overfitting, Ensuring Freshness
[22:09] Specialist Arenas (e.g., Coding), Objectivity
[26:25] Personalizing Evaluation, User-Specific Leaderboards
[30:12] Vicuna Origin Story, LLMs as Judges
[37:08] Battle Mode and Rating Algorithms
[52:25] Prompt-to-Leaderboard and the Need for Scale
[61:14] Misconceptions about Overfitting
[68:43] Arena as CI/CD-equivalent for AI
[80:42] Integrating Evaluation into External Apps, SDKs
[95:10] Red Team Arena for Model Security Testing
[92:49] Commitment to Openness, Neutrality, Ongoing Innovation

Roadmap and Looking Forward

Personalization: Deep focus on user-centric, individualized leaderboards and evaluation (aligning incentives and feedback loops).
Open Source & Transparency: Continuation and expansion; all new methodologies and data-sharing practices to maintain trust and foster innovation.
Integrations: Arena SDKs for direct in-app, real-world user evaluation and rapid model improvement.
Scaling Red Teaming: Community-driven adversarial testing to surface and fix emerging AI safety issues.
Evolving as the Landscape Changes: As AI moves from static models to systems and agents—Arena’s methodology, infrastructure, and UI will adapt to maintain robust, real-world relevance.

Final Thoughts

LMArena’s journey is a case study in how large-scale, user-driven, transparent testing is reshaping the definition of “good” in AI—moving from abstract, expert-defined benchmarks to immediate, empirical, and subjective measures shaped by the entire community. Their commitment to openness, neutrality, and enabling both the industry and individual users signals a foundational shift in both the form and substance of AI progress.

AI + a16z Podcast: Beyond Leaderboards – LMArena’s Mission to Make AI Reliable

Date: May 30, 2025
Guests: Anastasios N. Angelopoulos, Weyland Chang, Jan Stoica (LMArena founders)
Host: Anjane Mittha (a16z General Partner)

Episode Overview

Key Discussion Points & Insights

1. The Case for Real-Time, Community-Driven AI Evaluation

[01:26–04:54]

Traditional benchmarks (like MMLU) are necessary but insufficient for today’s fast-evolving, mission-critical AI systems.
LMArena aims to scale to millions of users to tap into diverse, representative feedback across industries, enabling targeted, robust evaluations—for everything from healthcare to shipping to defense.
The majority of real-world questions are subjective, even in science and enterprise. LMArena embraces this reality rather than ignoring it.

“The future is about real-time evaluation... in the wild.” — Anastasios N. Angelopoulos [01:26]

2. Scaling: From Open Source Roots to Broad Industry Impact

[02:26–06:54]

LMArena aspires to support both global communities and private, in-house evaluation by labs or enterprises, offering “private arenas” for specialized use.
The platform assists both big and small labs, providing tools for pre-release and ongoing model testing to inform which models are production-ready.

“In fact, one of the things that we help everybody do is pre-release testing of their models.” — Anastasios N. Angelopoulos [05:18]

3. The Value and Challenge of Subjectivity in Evaluation

[06:54–14:52]

LMArena leverages the collective wisdom of its users, countering the idea that only "experts" should define evaluation standards.
The team acknowledges (and technically addresses) criticism regarding biases (e.g., toward longer, more emoji-laden answers) through “style control”—a method to decompose and adjust for stylistic influences in model evaluation.

“What we’re building is an ever richer evaluation that can tell us all the factors that go into response, how you can optimize people's preferences, keeping style fixed.” — Anastasios N. Angelopoulos [13:44]

4. Fresh Data vs. Static Exam Overfitting

[19:29–21:24]

Unlike static benchmarks prone to “overfeeding” (where models are implicitly trained on test sets), LMArena constantly generates new, fresh prompts and votes, insulating against overfitting.

“Chatbot arena is immune from overfitting by design. Means you're always getting fresh questions.” — Anastasios N. Angelopoulos [21:24]

5. Specialist Arenas: Going Beyond Chat into New Domains

[15:00–24:54]

The expansion into domains like coding (Web Dev Arena) exemplifies how LMArena accommodates more “objective” tasks while maintaining user-driven evaluation.
The inherent difficulty in tasks like real-time code generation makes for sharper, more discriminative testing.

“Programming is actually a very general-purpose discipline...and yet it seems the capabilities in a very general way are still being...captured well on a specialist arena like web dev arena.” — Anjane Mittha (Host) [22:09]

6. Toward Personalized Evaluation

[26:03–28:41]

The vision is for every user (and every task) to have a bespoke leaderboard—reflecting the models that work best for them and their needs.

“It should be personalized just for you. You should understand which models are best for you.” — Anastasios N. Angelopoulos [26:25]

7. The LMArena Origin Story & Technical Innovations

[30:06–43:47]

Sparked by Vicuna (one of the first open-source ChatGPT alternatives), technical and logistical challenges in evaluation led to creative adoption of LLMs as judges, user voting, and adopting ranking algorithms (ELO, then Bradley-Terry).
Berkeley’s interdisciplinary environment enables nimble, innovative teamwork—a stark contrast with the slow, siloed approach of large industrial labs.

“The inspiration was how humans in real life rate players or teams... head to head. That’s how we adopted...battle mode.” — Jan Stoica [37:08]
“It was started as a fun project.” — Weyland Chang [37:18]

8. From Research Project to Company – The Strategic Shift

[47:40–55:34]

As usage and demand exploded, extending to hundreds of models and diverse users, it became clear sustained impact required a company, funding, and productization.
New features (like “prompt-to-leaderboard”) leverage massive user data to provide granular, even personal, model recommendations.

“Prompt to leaderboard... you give me your prompt. Can we tell you which models are best for that prompt specifically?” — Anastasios N. Angelopoulos [52:25]

9. Overfitting, Fairness, and the Evolving Benchmarking Debate

[58:43–63:40]

Benchmarks = static, supervised; Arena = reinforcement, user-in-the-loop, ever-fresh.
Overfitting in Arena is essentially moot; doing well means you are serving real user needs, not gaming the system.

“People still think about Arena as a benchmark... But what hasn’t permeated is when you have fresh data, you can’t overfit.” — Anastasios N. Angelopoulos [61:14]

10. Building the Best Testing Platform for an Evolving AI Landscape

[68:43–82:11]

Testing robustness and infrastructure-scale are major technical challenges; collecting high-quality, intrinsically-motivated user votes ensures data fidelity.
As AI “apps” become more integrated—layering in memory, context, and application-specific logic—the platform evolves. LMArena plans SDKs for integration, moving evaluation ever closer to the customer and use-case.

“You need something similar [to CI/CD] for these [AI] models right now...you also want to test your models, your checkpoints on Sherbot Arena for all the reasons we mentioned.” — Jan Stoica [68:43]

11. Red Team Arena and Security, Reliability Concerns

[95:10–99:45]

LMArena’s approach extends to safety: Red Team Arena lets a community of “jailbreakers” actively test and leaderboard models for their security/safety characteristics—mirroring user-driven, real-world red teaming at unprecedented scale.

“In Red Team arena we have a leaderboard. Not just for model, but for jawbreaker — who is best at identifying issues.” — Weyland Chang [97:21]

12. Open Source, Neutrality, and Core Values Going Forward

[91:34–94:26]

Openness with code and data, maintaining academic neutrality, and community engagement are non-negotiable values as the project grows as a company.
Openness builds trust in evaluation results and attracts the best talent and most innovative collaborators.

“If people want to ask the question, ‘Hey, how are models performing?’ ... Just go look at the data. That’s what we did with Llama—we just released the data.” — Anastasios N. Angelopoulos [92:49]

Notable Quotes & Moments

On the end of “hard exams” era:
"Static exams were useful three years ago. The future is about real-time evaluation, real-time systems, real-time testing in the wild." — Anastasios N. Angelopoulos [01:26]
On democratizing expertise:
"Everybody actually has their own opinions and... there’s so many natural experts in the world... Their vote means so much." — Anastasios N. Angelopoulos [07:57]
On subjectivity & bias:
"People vote for longer responses preferentially. Can we learn this bias and actually adjust for it? The answer is yes. That’s why we're making style control default." — Anastasios N. Angelopoulos [13:44]
On why fresh data is disruptive:
"Overfitting means you’re doing well on the test data but only on the train data. There cannot be overfitting [in Arena] because we have continuously fresh data." — Jan Stoica [63:40]
On the vision for company & platform:
"Neutrality, innovation, trust... We want the world to know what the best ways are of evaluating these models and accelerating the ecosystem." — Anastasios N. Angelopoulos [92:49]
On open source & transparency:
"If people want to ask the question, hey, how are models performing on their why are they performing well? Go look at the data. That's what we did with Llama, right? Just go look." — Anastasios N. Angelopoulos [92:49]

Key Timestamps for Major Segments

[01:26] Real-Time Testing vs. Static Benchmarks
[05:18] Working with Labs (Big and Small), Pre-Release Testing
[13:44] Addressing Voting Bias, Style Control
[19:29] Preventing Overfitting, Ensuring Freshness
[22:09] Specialist Arenas (e.g., Coding), Objectivity
[26:25] Personalizing Evaluation, User-Specific Leaderboards
[30:12] Vicuna Origin Story, LLMs as Judges
[37:08] Battle Mode and Rating Algorithms
[52:25] Prompt-to-Leaderboard and the Need for Scale
[61:14] Misconceptions about Overfitting
[68:43] Arena as CI/CD-equivalent for AI
[80:42] Integrating Evaluation into External Apps, SDKs
[95:10] Red Team Arena for Model Security Testing
[92:49] Commitment to Openness, Neutrality, Ongoing Innovation

Roadmap and Looking Forward

Personalization: Deep focus on user-centric, individualized leaderboards and evaluation (aligning incentives and feedback loops).
Open Source & Transparency: Continuation and expansion; all new methodologies and data-sharing practices to maintain trust and foster innovation.
Integrations: Arena SDKs for direct in-app, real-world user evaluation and rapid model improvement.
Scaling Red Teaming: Community-driven adversarial testing to surface and fix emerging AI safety issues.
Evolving as the Landscape Changes: As AI moves from static models to systems and agents—Arena’s methodology, infrastructure, and UI will adapt to maintain robust, real-world relevance.

Beyond Leaderboards: LMArena’s Mission to Make AI Reliable

Powered by Wave AI

Summary

AI + a16z Podcast: Beyond Leaderboards – LMArena’s Mission to Make AI Reliable

Episode Overview

Key Discussion Points & Insights

1. The Case for Real-Time, Community-Driven AI Evaluation

2. Scaling: From Open Source Roots to Broad Industry Impact

3. The Value and Challenge of Subjectivity in Evaluation

4. Fresh Data vs. Static Exam Overfitting

5. Specialist Arenas: Going Beyond Chat into New Domains

6. Toward Personalized Evaluation

7. The LMArena Origin Story & Technical Innovations

8. From Research Project to Company – The Strategic Shift

9. Overfitting, Fairness, and the Evolving Benchmarking Debate

10. Building the Best Testing Platform for an Evolving AI Landscape

11. Red Team Arena and Security, Reliability Concerns

12. Open Source, Neutrality, and Core Values Going Forward

Notable Quotes & Moments

Key Timestamps for Major Segments

Roadmap and Looking Forward

Final Thoughts

Summary

AI + a16z Podcast: Beyond Leaderboards – LMArena’s Mission to Make AI Reliable

Episode Overview

Key Discussion Points & Insights

1. The Case for Real-Time, Community-Driven AI Evaluation

2. Scaling: From Open Source Roots to Broad Industry Impact

3. The Value and Challenge of Subjectivity in Evaluation

4. Fresh Data vs. Static Exam Overfitting

5. Specialist Arenas: Going Beyond Chat into New Domains

6. Toward Personalized Evaluation

7. The LMArena Origin Story & Technical Innovations

8. From Research Project to Company – The Strategic Shift

9. Overfitting, Fairness, and the Evolving Benchmarking Debate

10. Building the Best Testing Platform for an Evolving AI Landscape

11. Red Team Arena and Security, Reliability Concerns

12. Open Source, Neutrality, and Core Values Going Forward

Notable Quotes & Moments

Key Timestamps for Major Segments

Roadmap and Looking Forward

Final Thoughts