A (1:33)

He does. He's been incredible to us.

B (1:34)

You want to shout out some stuff that you did?

A (1:36)

Yeah, absolutely. So, you know, the way the company started was as an incubation by. On.

B (1:41)

Yeah.

A (1:41)

So what he did is he kind of like, found us at Berkeley and picked us out of the basement and was like, hey, these guys seem like they're onto something. And started working with us really early. Gave it, you know, gave us some grants. He was not, you know, a 16 was not the only wanted to do this. We also had a great grant from Sequoia, but ONS was in particular, quite, quite supportive of us and, you know, gave us some resources in order to continue building out arena before we even were committed to starting a business. And in that capacity, he sort of like, you know, formed an entity for us and this and that. And, you know, he was like, hey, you guys can walk away at any time if you guys don't want to start a business. I Mean, it was really incredible, very aggressive investment move by him. Right. Because of course, any money that he spent, at the end of the day, like, we could walk away and leave him with nothing. But I think he knew wisely that the right thing for arena was to start a company out of it. It was the only way that we could scale and that, you know, Weylan Yan and I would, would ultimately see that and be excited about doing it ourselves, which is which. Which ended up being the case.

B (2:42)

Was there a moment for you where you were, I'm sure you were debating it yourself. You had other opportunities. What was the deciding factor for you?

A (2:48)

It became clear that the only way to scale what we were building was to build a company out of it. That the world really needed something like Arena. Arena being really a place to sort of measure, understand and advance the frontier AI capabilities on real world users, on real world usage based on organic feedback. And that in order to achieve the scale and, you know, distribution necessary and the quality, of course, of the platform necessary to do this effectively, we would need to start a company out of it. You know, we considered other options. Are we going to keep doing this as an academic project? Are we going to do it as a nonprofit, blah, blah, blah. But ultimately, under those constructs, we didn't feel like we'd have the resources necessary to accomplish our mission.

B (3:36)

So you raised 100 million or 80? 100. $100 million. That's a lot of resources.

A (3:42)

Great.

B (3:42)

Yeah. What's it for?

A (3:43)

Well, you know, obviously on behalf of.

B (3:45)

Like, everyone who's like, yeah, everybody. Dude, it's an Arena.

A (3:48)

How are you going to spend your money? Yeah, yeah, it's great. Yeah, I think so. First of all, we don't necessarily need to spend all that money. Right? The, the purpose of money at a company is to give you cards to flip. It's to say, hey, you. You have enough resources necessary so that if your first bet fails, you can make another bet and another bet, of course. So that's not to say that we're going to spend all of it. Of course, you want to spend things responsibly. Having said that, the platform is actually quite expensive to run. We fund all of the inference on the platform. You know, the way, the way that it works is the platform.

B (4:17)

You pay market rates, they don't give you.

A (4:19)

No, no, we get discounts, but they're. But they are standard enterprise discounts, the same that would be given to any other customer.

B (4:27)

And if you close any. I don't know any numbers. I see numbers of Votes. But like what's that in like monthly tokens or. I don't know about tokens.

A (4:37)

You know I'd have to back. Back that out. But we have like, you know, let's say again this is off the cuff but I can safely say we have more than 5 million now. 5, 6 million. We have probably 250 million conversations that happen over the course of the platform. We're on the order of, you know, mid tens of millions of conversations every month that are having on the platform. It's actually quite a large, you know, it's one of the largest consumer platforms for LLMs. Of course nothing is really comparing to.

B (5:03)

Chat GPT but you know, but no still I mean I think the largest skilled ones I would benefit of this.

A (5:08)

Is that like it's actually quite a diverse population. So 25% of the people on our platform, for example, do software for a living. Still at this scale.

B (5:16)

How do you know?

A (5:17)

Because we like do all sorts of. We either survey them or we'll like analyze the prompt distribution that's coming into the platform. Happy. Also to share more, we've done something called Expert arena which is like trying to understand the distribution of experts that are coming to the platform.

B (5:30)

A lot of that can be like unauthenticated or whatever usage it is.

A (5:34)

But about half of our users now are logged in.

B (5:36)

Yeah.

A (5:37)

And so we have some ability to understand them. And we also have surveys that we run on the platform that tell us a bit more about who are the actual users. Of course there's always response bias in surveys so you have to take it with a grain of salt. But nonetheless.

B (5:51)

And if you don't know Anastasia's background, he's the guy to correct for response bias. Yeah. Okay.

A (5:57)

There's a lot of guys like that. And girls.

B (6:00)

And girls. You're not the only player. There are this artificial analysis starting the arena.

A (6:05)

You.

B (6:05)

Yup. AI. Yep. It's like some crypto people that started this. I don't know if you have a conversation with them on like well hey this is our thing or like let's work together on something. I don't know if you've. What the.

A (6:16)

No, you know. So I've talked to the artificial. I've actually talked to both groups.

B (6:20)

Both seem, you know, am I missing any major players? It's just those.

A (6:24)

No, I think those. I don't know. Yeah, I don't think so.

B (6:27)

Okay.

A (6:27)

I think those are like some of course large depending on how you define the term. Artificial analysis obviously has like huge like market mind share around the analysis of different AI systems.

B (6:41)

Yeah, they told me they were just like, we are going to be gardener of AI.

A (6:44)

Yeah, that's kind of their goal. And I think they're going after that consulting market and so on. Artificial analysis, from what I understand is like a team of consultants that are doing this seem like really, really nice guys going after that particular market. It's different from our platform in the sense that their analysis is based on sort of like aggregating public benchmarks and turning those into analytics. Independently rerunning and independently rerunning.

B (7:10)

Which.

A (7:11)

Yeah, yeah, yeah, which matters. Yes. And using those in order to compile sort of reports and so on that educate the field on the performance of all these different models.

B (7:22)

But they also have arenas.

A (7:23)

They have arenas, but the arenas are not based on organic usage. Like the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a level of realism that platform doesn't have. Of course they specialize in a slightly different thing, but I see those platforms kind of diverging in that sense.

B (7:46)

Yeah. And sometimes it's like it's the only way to do this. So like for example, for aa, their video arena, it's pre generated videos. You can't enter in your own video.

A (7:55)

That's correct. But we're doing it organically.

B (7:57)

Yeah, exactly. And so as a voter, it does help in terms of like, I don't have to wait.

A (8:03)

It does. But also why would you go? Do you actually care about like other people's videos?

B (8:09)

Form your own intuition maybe.

A (8:10)

Yeah. Maybe you're like interested in comparing like.

B (8:12)

I'm a shitty prompter, right?

A (8:14)

Are you? I don't believe that.

B (8:15)

I'm not a terrible prompter. I learned by example.

A (8:18)

Don't denigrate yourself. Don't denigrate yourself.

B (8:21)

There are many prompts that's much better than me, let's say that. Right. That's a fact.

A (8:24)

People have all sorts of cool ways of prompting LLMs. Yeah.

B (8:27)

It's educational to say the only way to learn is by like, well, look at other people's prompts and see the results and go like, oh, I didn't know you could do that. And that's hard.

A (8:35)

Totally.

B (8:35)

Yeah, yeah, yeah. Okay, so let's come back to Arena. Oh, one thing I do want to say is the number one use of funds is getting off Gradio.

A (8:42)

Oh yeah. Yes. Well, listen, Gradio, incredible platform. Gradio scaled us to a million mal. Yeah, that's Incredible.

B (8:51)

And of course, you tell the Hugging Face wants that.

A (8:52)

Of course. Yeah. We're really, really grateful to Gradio for taking us so far. Eventually, you know, it became time for us to move off of that and go to a React.

B (9:01)

I'm sure Hugging Face would have loved you to stay on.

A (9:04)

They would have, I'm sure.

B (9:05)

Was there a technical, like, reason you just couldn't get the performance?

A (9:09)

Yeah, it just became hard to develop. And there were all these tools that we wanted in React and, you know, to do all the fancy things that you can do in React became kind of like one example.

B (9:19)

I don't know what's a feature that you really wanted?

A (9:21)

Let's say we wanted to create like our own custom, like loading icons for video with notifications.

B (9:27)

Okay.

A (9:27)

How are we going to do that in React? It's hard.

B (9:30)

Yeah. I mean, we'll make a custom component.

A (9:33)

I'm sure the Hugging Face guys are going to come in and say like, hey, you can do that in Gradio, which maybe you can, but also fewer developers know. How are we going to hire for that? We have to reskill them. People are less familiar with that stack, you know.

B (9:45)

So anyway, full React next year. So all this.

A (9:47)

Yeah, yeah, yeah, all that.

B (9:48)

Okay, cool. Other use of funds that might be interesting. Like, you know, basically.

A (9:53)

No, that's basically it.

B (9:54)

Resources. Okay.

A (9:55)

Yeah, it's. It's. It's on primarily inference that funds the free usage of the platform and. And then also hiring, of course, Headcount.

B (10:04)

Yeah.

A (10:05)

We have an office, you know, that's an sf.

B (10:09)

I'll tackle one of the major things this year, which I'm sure you're tired of thinking about, but I. For people who are not in the loop, this is going to be news to them. The Leaderboard illusion, the whole thing with Cohere. Let's summarize the. What they said and then your response.

A (10:23)

So Leaderboard Illusion is a paper that critiques Alam Marina and the main.

B (10:28)

Pretty, like brutally.

A (10:30)

Well, you know, I would say unscientifically.

B (10:34)

And let's be clear, Cohere wasn't doing.

A (10:35)

That well on the cohere was like 74. It's all good. You know, it's actually not. It's. It's a respectable place that they had on the leaderboard. I don't even think it was really coherent people like the coherent model developers doing this. It was more their research side. But in any case, what is the. What does the Leaderboard Illusion say? It says that Alamarina was what they're the claim is that Alamarina was doing this undisclosed, quote unquote private testing on our platform, that model providers will send us pre release models and we'll expose them and so on and so forth and that this creates so called inequities in the leaderboard. You know, due to that pre release testing, for example, they will, they cited that meta at some point, tested amount of models with us. Of course we can't disclose all of the details of how all that was done, but that is the main claim of the paper. Now our response to that paper, it's online, you can find it on response to Leaderboard Illusion. And our response to that paper is essentially pointing out a series of factual mistakes in the paper that that question the validity of the claims. So you can go look at the first version of the paper on Arxiv yourself and you'll see the claims. I think most scientists would view that.

B (11:42)

As, oh, they've corrected it.

A (11:43)

They've correct of course, because we, I mean, but they didn't correct everything. They just corrected some aspects that were just blatantly unscientific and false. But you know, for example, said that we only sampled like 9% open source models and like you know, 60% like closed source models. And this created a gap between open and closed source. But in reality we're actually really supportive open source models and it was more like 60, 40. And so that was, you know, one of the examples of an error in the claims. Another example is that they were claiming that there was some sort of bias introduced by this prerelease testing and that it was undisclosed. And reality is, you probably know we've been doing this prerelease testing for a long time. Our community loves it. They love basically getting like secret code names. Yeah, the secret code like Nano Banana. Yeah, all that. So Nano Banana, by the way, started on you, started on us. Right. And people loved it. Went like global sensation. Like non, non zero fraction of the global population.

B (12:38)

Using talk to Lena about naming it Banana. Or was that no decision?

A (12:42)

No, it was, it was their decision, I believe. But it was sort of this randomly generated thing and it just went no, no.

B (12:48)

So apparently Naina, who's a pm.

A (12:50)

Yeah, yeah.

B (12:51)

Is named after her because her nickname is Nana. Oh.

A (12:54)

Oh, that's sweet.

B (12:54)

I didn't know that Nana put Banana on it. Yeah.

A (12:57)

Oh, that's sweet. Yeah, I didn't know that. Didn't know the origin story. Yeah. I mean to us it just looked like sort of a random thing but.

B (13:03)

It was like clearly heads and shoulders above huge which like before that there was Reeve Image, remember?

A (13:09)

Yeah, I do, of course, yeah.

B (13:10)

And also of BFL and all those.

A (13:13)

I mean all those models are also great. Yeah. And I think those teams are also improving quite quickly. But Nano Banana was a sensation. I mean, that moment alone changed Google's like roadmap. Yeah. Market share.

B (13:26)

Yeah, seriously.

A (13:27)

I mean, Google stock, billions of dollars are moving because of Nano and now.

B (13:31)

There'S like an OpenAI code red and everything.

A (13:34)

I don't know about that. But yes, the information reported this.

B (13:38)

I would say, like the image generation, I would say has been like this weird part of AI overall because it's not strictly AGI critical. Like it's not reasoning, it's not feeding more context into the model. It is the model generating a visual representation. So it's basically, I always think like, well, Gemini used to get a lot of complaints for generating racist images or whatever.

A (14:06)

Yeah, that was a hilarious moment.

B (14:08)

And ChatGPT also had it in the past. And I'm like, well, can we just get rid of this? Do we have to do image generation? Because let's just focus the positive reputation of AI in general on language models and coding and the other stuff. But I'm wrong. I'm such a huge Nano and Banana pro show.

A (14:27)

Yeah, I totally agree. I was also kind of wrong about this. I didn't see the positive benefits. But actually I think that these multimodal models are going to become some of the most economically valuable aspects of AI, both in consumer and also an enterprise. Because one of the fastest growing segments, market segments in AI adoption is marketing and marketing and design.

B (14:49)

Yeah, ads. And so I'm a content creator.

A (14:52)

Right, yeah, of course. I'm sure you're using it all the time.

B (14:54)

Infinite supplies of diagrams and explainers and totally infographics.

A (14:58)

Yeah, soon we're not going to be even making the papers. Yeah, they're just going to be our paper figures are going to be made by analysis.

B (15:04)

Yes, yes, I do think they actually. One shot. So Deepseek came out with v3.2 recently. I took their explanations, which are very wordy. They like very concise papers. It's 23 pages long, but it's very dense. And so I just took their explanations of the RL environment stuff and I fed it into nanobananapro and it spread an image that I used to understand the paper better. And the fact that I can just casually generate a paper quality diagram that would usually take a PhD student a month in Photoshop or something to do is incredible.

A (15:35)

Yeah, it's incredible. It is amazing.

B (15:37)

I want to ask about your principles running Arena. I think you manage a giant community 5 million miles. What have you decided are the core principles? I guess before becoming a company and now that you're a company, I don't know if there's anything that's changed for you.

A (15:51)

I don't think anything has really changed. We want to provide the North Star of the industry and center the use cases of real users, foreground those so that people know what to target. The goal is to create a benchmark that is constantly fresh, that does not suffer overfitting because of the fact that we constantly have new data points coming in that tracks all the different new models, all the different new use cases of AI and gives the whole world sort of ground truth for how real users are using these models and how good they are on those use cases. We continue to do quite a few open source data releases. We've probably released more data than basically anybody on the real world use cases of AI. Millions and millions of conversations, real world conversations from real users that the community's using to study and improve on.

B (16:46)

Yeah. And then I think like in terms of what you will build versus what will not build. I guess I'm not necessarily caught up on everything that you've launched. I know recently you've done the dev or code arena.

A (16:59)

Yeah, code arena.

B (17:00)

That's the most Code arena.

A (17:01)

Expert arena.

B (17:02)

Yeah, Expo arena. So basically like what is in the critical path for you, let's say for next year and what, what have you decided you'll never do?

A (17:11)

So let me first talk about things that I'll never do. The platform integrity comes first to the platform. The basically the public leaderboard that we show on Alan Marina, I think of as a charity, it's a loss leader for us. We don't really make money on the public leaderboard. You can't pay to get on the public leaderboard. It's not like a Gartner in that sense. It's not like any of these like, you know, play systems. Never going to be like that. Models are going to be listed on the leaderboard whether or not the providers pay and whether or not they're getting a good score. They can't pay to take it off either. And so what that means, that's very important. And so what that means is that the leaderboard has a certain integrity that will never be compromised. Of course.

B (17:53)

But not all preview models will make it onto this.

A (17:55)

No, but that's okay. Those preview models have never been released. Yeah, right. Who cares about putting Unreleased models on the leaderboard. The point is that for every released model, the score that you see on the leaderboard is statistically sound. It reflects the real world capabilities of the model. Why? Because millions of people from around the world have voted for it and that's where that number comes from. All we do to compute that number is millions of people are voting. We take those votes, we turn them into a number that's always going to remain transparent and fair reflection of model performance. Where are we going? Lots of different new categories. I don't know if you recently saw we exposed occupational and expert categories. So now single digit percentage of our user base. We're millions to tens of millions of users, right? So single digit percentages means a lot. Single digit percentage of our user base are in medicine, in legal, in business, you know, finance, accounting, creative marketing, stuff like this. And we're able to show the performance of these models in all these different verticals because we have all these users in our, in our user base and we're working more towards multimodal, you know, video. We're soon to launch on the site at some point, you know, later this year or early next. So lots of things in the pipeline.

B (19:08)

Amazing. Would you expose an API?

A (19:11)

We've thought about it, yeah. I think it's a, it's a possibility, yeah.

B (19:15)

What are the counter arguments? Why not?

A (19:18)

Well, there's obviously a need for an API. The question is more of focus of our company just because we're a startup and so we really should be doing one thing well.

B (19:31)

Arenas.

A (19:31)

Yeah, arenas. So I'm not sure how far we want to sort of splay out and on what timeline we'd want to do that.

B (19:37)

Yeah. Any other sort of like community management tips more broadly? Every AI company really wants to grow their community. You're obviously one of the strongest in the world. What's really, really worked well.

A (19:51)

So first of all, I want to give a shout out to our community manager, Greg, who is doing an awesome job managing our community. Whether that's on Discord or on Elamarina. He's really incredible. So I would say hire Greg, but don't. Don't hire Greg.

B (20:04)

Don't hire Greg.

A (20:05)

Don't hire Greg.

B (20:05)

Find he's ours.

A (20:06)

Find a Greg.

B (20:07)

Find a Greg.

A (20:08)

But in general, you know, the question of how do you get to so many users?

B (20:12)

That is a tough question.

A (20:13)

And keep and retain them. That is a tough question because consumer is one of the hardest markets in the world. There's a lot of websites in the world that people can go to, you know, why should they go to yours? And the reality is, if you want to create a really dominant product, you have to provide people value. And to be frank, I don't think we're all the way there yet. It's not like I have the solution and answer for how to build a great consumer product. If I did, we wouldn't be attending tens of millions of users. We'd be at hundreds or, you know, we'd be at a billion users. We'd be like, is there a world.

B (20:43)

You like are bigger than ChatGPT?

A (20:45)

I don't know. I don't. I don't know that we need to be. Yeah. And I don't know that we ever will be because that's a. That's an extraordinary generational product that they built. Right. It took a lot of time. And then to some extent, it also involved luck. There's a lot of lightning in the bottle moments like nanobano was for us, where our user base just like goes up by a lot of. But when those users come, they can just as easily leave. So the way I think about it is every user is earned. You have to earn them every single day. They can leave at any moment. They're fickle. And so all the time you have to be thinking about, how do I provide this person value learning how are they using my website, what more could I give them? And how do I build in all the retention mechanisms so that they stay? And then they're also bringing their friends.

B (21:29)

Is there one that's working in terms of retention? Like you said, a lot of people are signing in now.

A (21:33)

Yeah, sign in was a big driver of retention.

B (21:35)

No, no, but what did you give them in order to encourage, like, history, persistent history. That's it. That's enough. Yeah, that's.

A (21:40)

That's one thing that has had a big impact.

B (21:43)

Okay. Yeah, cool. What do you want from people? What, what are you looking for help on, like any call section?

A (21:49)

Yeah, I. We are always looking for people to come and join us. If you are one of the best people in the world in your area, whether that's consumer product, whether that is machine learning, whether that is, you know, know B2B, go to market, marketing, all these things. We need you at Arena. We're building like a high performance team of real experts in everything that they do. And, you know, I'm always looking for excellent people to work with.

B (22:11)

Do you need, like, what about partnerships? Right. Like, let's say I make cognition. I want to partner with El Marina or just Arena. What works for you. What existing partnerships do you already have?

A (22:22)

That.

B (22:23)

That's really fruitful.

A (22:25)

Yeah. So, I mean, we of course partner with all of the major model labs.

B (22:29)

Yeah. And that's. This is straightforward. Like, hey, we have a new model here.

A (22:32)

Here you go. Exactly. So I think the. The most straightforward thing would be for someone like cognition, it's like, let's evaluate that agent.

B (22:38)

Yeah.

A (22:39)

But we should be continuing to shape our. Well, Codarin is an agent evaluation.

B (22:43)

Is true.

A (22:43)

Is true.

B (22:44)

And focused on like all these arenas tend to focus on the model rather than the harness. So.

A (22:50)

But that maybe should change. Maybe we should be evolving towards that direction. And I think the code arena is a good example of an arena that. A full featured harness, like a Devon. Yeah. And so in my view, if I'm talking to cognition, I'm saying, hey, let's get Devin on the arena and figure out how to loop together the Devon harness so that we can. I'm sure that there's something that could be really valuable there, especially given Devin. Last week, people talking about Devin dead. Did you see that?

B (23:18)

Yeah.

A (23:18)

People were saying, devin's gone. Devin's not gone.

B (23:21)

Devin's everywhere, but people.

A (23:23)

So can we highlight that for people and show them, hey, Devin is actually the best or one of the best in the world at doing what it does. Al Marina can actually do that. And our. Our place as a central evaluation platform allows. Allows that to happen. Yeah.

B (23:36)

Love it. All right. Thank you for owning the state of Vivals.

A (23:39)

Thanks so much.

B (23:40)

Congrats on a wonderful year.

A (23:41)

Appreciate it. Congrats to you too. Congrats on all the growing, you know, momentum in your podcast and in your career.

B (23:47)

Thank you.

A (23:47)

It's really impressive to see. Sam.