The Engine Powering Open-Source AI

Summary

The a16z Show — How Open-Source AI Became Critical Infrastructure

Date: August 6, 2026
Guests: Simon Mo (Co-founder & CEO, Infrac and lead maintainer of VLLM), Matt Borenstein (a16z General Partner), Elena Berger (Host, Andreessen Horowitz)

---

Episode Overview

This episode explores the transformation of open-source AI from a grassroots movement to critical infrastructure supporting enterprises and developers worldwide. Simon Mo, co-founder of Infrac and maintainer of VLLM—an inference engine running on hundreds of thousands of GPUs—joins Andreessen Horowitz’s Matt Borenstein and host Elena Berger to discuss how open-source inference, model licensing, economic sustainability, and control are shaping the future of AI. Memorable industry anecdotes, technical deep-dives, and sector-wide implications highlight why open-source is now central to how cutting-edge AI is developed, distributed, and deployed.

---

Key Discussion Points & Insights

1. Origins and Evolution of Open-Source AI

[01:41 – 05:27]

• Transition from Hobbyist to Infrastructure:
  • The journey from early open-source demos (pre-ChatGPT) to VLLM’s place as a mission-critical engine began by addressing fundamental challenges: large language models (LLMs) need accelerators (GPUs/TPUs), complex batching, and non-deterministic outputs.
  • "Serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs, and it is a computationally intensive process that will require a lot of engineering" – Simon Mo [02:31]
• Key Milestones:
  • Early models (e.g., BERT, Resnet) could run on commodity hardware, albeit slowly, but newer models demanded powerful, purpose-built infrastructure.
  • “All the frontier AI work was being open source... people could mostly run these models sort of on hardware or computers they already had.” – Matt Borenstein [03:39]

2. Open-Source as Enterprise Backbone

[05:48 – 09:47]

• Open-Source Becomes Essential:
  • The tipping point was around 2023, with widespread reliance on open-source models and infrastructure (“open frontier intelligence”) as backbone for day-to-day productivity—GitHub Copilot, ChatGPT, and similar tools became irreplaceable.
  • Enterprises needed flexibility, control, and the ability to tune AI to their needs.
• VLLM’s Role:
  • VLLM sits between models and hardware, acting as the universal engine powering endpoints for intelligence.
  • "VLLM is an inference engine... it is kind of like databases and operating system and other critical software to power this economy or power the AGI." – Simon Mo [08:25]
  • Supports thousands of model architectures, proprietary and open-weight, often on “day-zero” of release.

3. Behind the Scenes: Model Releases & Ecosystem Collaboration

[09:47 – 12:50]

• Complex Release Process:
  • Releasing a “hero” model involves deep coordination: model labs, hardware vendors (Nvidia, AMD, Google, etc.), inference clouds, format/model hub vendors (Hugging Face), and platform providers.
  • "Every model race...involves the model lab, hardware vendors, involves us, involves Hugging Face..." – Simon Mo [10:54]
• Anecdote:
  • Mistral’s first model: released via torrent, rapidly adapted for VLLM support, showcasing the collaborative, chaotic, and community-driven nature of open AI launches.

4. The Open Weights Pledge and Marketplace Dynamics

[12:50 – 16:33]

• Market Response:
  • Companies (Infra, a16z, Meta, Amazon) signed the Open Weights & AI Leadership letter, defending the necessity of open, non-proprietary models.
  • “The world cannot just be controlled by proprietary API where open way, open development and research of these models are blocked or banned.” – Simon Mo [13:26]
• Customer Drivers – Control vs. Cost:
  • Control is a primary concern (customization, compliance, SLA guarantees), though cost has grown in importance with rising usage and token spend.
  • "Control matters a lot...cost really matters starting from people trying to migrate off their expensive coding plan..." – Simon Mo [14:45]
• Economic Reality:
  • Some open models (e.g., Kimi-3) are nearly as expensive as closed-source, but offer flexibility, deeper performance tuning, full control over data, speed, and compliance.

5. Licensing, Sustainability, and Economics of Open Models

[19:47 – 23:57]

• Evolving Licenses:
  • From permissive (Apache 2.0) to nuanced—usage/monetization thresholds, derivative work clauses—to help labs fund continued model development.
  • "Model labs are trying to understand a way to economically fund their own model development... we're seeing a very healthy ecosystem development." – Simon Mo [19:56]
• Economic Analogy:
  • Compared to pharmaceutical R&D: substantial up-front costs/risk, ongoing need for returns to sustain innovation.
  • “It's about how do you make sure that all this initial capex almost to train the model...is proper sustainable method.” – Simon Mo [22:55]

6. Infrastructure vs. Model Maintenance

[24:25 – 26:58]

• AI ≠ Software:
  • Open-source software is sustained by donated time; open models require massive, expensive compute resources.
  • Maintenance focus: “Once it's out there is a whole community effort trying to optimize this model... it's about how do you turn a use case of one now to a use case of almost infinity.” – Simon Mo [26:09]
• Community Power:
  • Optimization, specialization, reliability improvements are crowd-sourced, mimicking successful open-source software development at scale.
  • “That's cool because that really is like open source software... anybody can contribute and make these better.” – Matt Borenstein [26:58]

7. Scaling, Inference, and Necessity of Open Source

[27:39 – 30:00]

• Inference Challenges:
  • Modern LLMs require massive scale and optimization, both in data centers and for specialized, “edge” use-cases.
  • Open source draws on collective validation, bug-fixing, and performance tuning at scale.
  • "This is where open source inference is the current leading way right now instead of a closed source inference engine." – Simon Mo [28:42]

8. Unique Era of AI Startups and Open-Source Collaboration

[30:00 – 31:38]

• A Wave of Innovation:
  • Teams like VLLM, OpenRouter, Ollama predated ChatGPT boom, motivated by curiosity and open-source ethos.
  • "Open source is where you meet the most mission aligned people together." – Simon Mo [30:57]

9. Case Study: Hugging Face’s Use of Chinese Open-Source Model in Security

[31:38 – 36:22]

• Incident as Proof of Control:
  • HuggingFace used a Chinese open model to control/contain a rogue AI in sandbox; closed models’ guardrails can overreact or block legitimate tasks.
  • “If moderation is never solved... then there's always a place where you have a model where you know and trust that you are publishing to.” – Simon Mo [32:12]
• Analogy to Social Media:
  • AI moderation challenges echo the struggle with platform responsibility and liability in social media.

10. Advice and Reflections: Building Open Source Companies

[36:22 – 37:53]

• Guidance from Industry Leaders:
  • Learning from Databricks’ Ian Stoica focuses on how to create value and fill crucial infrastructure gaps while being open-source first.
  • "His focus really stemmed from how do we build... where this software is being used by so, so, so many people on such a critical project..." – Simon Mo [36:49]

---

Future Outlook & Big Questions

1. Will Open-Source Close the Gap With Frontier Models?

[37:53 – 40:38]

• Little Capability Difference Today:
  • The distinction is now more about distribution and productization strategy, not technical capability.
  • “I don't really see a big gap, not even today because for how these models are coming to being, they're really starting from the first principle.” – Simon Mo [38:26]
• Data & Environment as Differentiators:
  • The “race” will focus on constructing better environments for recursive self-improvement, not just better data.

2. Brilliant Researchers, Global Dynamics & Distillation Myths

[40:38 – 44:54]

• Talent is Global:
  • Brilliant researchers gravitate to impacting work, not open/closed dogmas. The biggest lever is open weights allowing greater impact and collaboration.
  • "They're attracted to interesting problems, not necessarily on the open or closed stance... open weight model does give people a really, really good boost on the impact of such models." – Simon Mo [40:57]
• Distillation is Overstated:
  • Real progress is driven by new environments and learning processes, not copying (distilling) from other models.
  • "I really don't think... distillation is a big cornerstone of what's powering the progress today. In the end, what's powering the progress is still just really smart people..." – Simon Mo [43:23]

3. Open Innovation as a “Racetrack” for Faster Progress

[44:54 – End]

• Open Models Accelerate Global Progress:
  • Open weights foster innovation by allowing others to learn, copy, iterate, and improve upon each other's work.
  • “Open source and open way really helps innovation because it set out this racetrack where everybody can learn from each other and... improve yourself.” – Simon Mo [44:54]
• Investment Outlook:
  • a16z seeks open-source model training “all over the world…because you get that sort of magic of collaboration.”

---

Notable Quotes & Memorable Moments

• On the early days of LLM infrastructure:

  • “All the memories are flooding…Hugging Face had a thousand BERT variants on it.” – Matt Borenstein [04:53]

• On the release chaos of new models:

  • “Even from the beginning of 2023, 2024, if you remember when Mistral dropped their first model, they just drop a torrent link…and then we're working behind the scene…” – Simon Mo [11:31]

• On moderation and control:

  • “For the trusted use case, people will go to openway by default because that is where you know for sure you can control your guardrail for trusted use cases.” – Simon Mo [32:12]

• On distillation and real innovation:

  • “I really don’t think... distillation is a big cornerstone of what's powering the progress today. In the end, what's powering the progress is still just really smart people with very interesting algorithms, data, environment. And they will produce...the models.” – Simon Mo [43:23]

---

Timestamps for Key Segments

• Brief history and challenges of LLM inference: [01:41 – 05:27]
• Enterprise adoption of open-source models: [05:48 – 09:47]
• Anecdotes on model releases and ecosystem coordination: [09:47 – 12:50]
• Open Weights pledge & infrastructure economic realities: [12:50 – 16:33]
• Model licensing evolution & sustainability analogy: [19:47 – 23:57]
• Maintaining models & infrastructure post-release: [24:25 – 26:58]
• Scaling inference & open source necessity: [27:39 – 30:00]
• Curiosity and the accidental revolution of open AI startups: [30:00 – 31:38]
• Incident: Hugging Face using open models for security: [31:38 – 36:22]
• Advice from Ian Stoica on building open source companies: [36:22 – 37:53]
• The future gap between open and closed models, and the road ahead: [37:53 – End]

---

Final Thoughts

This episode frames open-source AI as not just an ideological preference but a pragmatic necessity for the current and future AI landscape. The conversation moves fluidly from vivid technical and historical anecdotes to reflections on control, economics, and global innovation—highlighting how open weights are accelerating the race and making AI more flexible, adaptable, and innovative. The future, the speakers agree, will be defined by infrastructure and community, not just proprietary breakthroughs.

Transcript

[Matt Borenstein]

The fun thought experiment is if GPUs dropped in price by 99% then do we get back to a real open source world?

[Simon Mo]

If moderation is never solved in the future people will go to Openway by default because that is where you know for sure you can control your guardrail for trusted use cases.

[Elena Berger]

And can you talk about where VLLM sits in that stack?

[Simon Mo]

VLM is a inference engine. It is kind of like databases and operating system and other critical software to power AGI, Nvidia, amd, goog, their newest chip will make sure VEOM can run on them and in a lot of cases they use VRM as a benchmark while bridging almost a 10x gap. For proprietary model there is a regular mode and fast mode, but for open wait, every provider can offer potentially even 10 different levels of speed five years from now.

[Elena Berger]

Open source AI models have they closed

[Simon Mo]

the gap with frontier models capability wise? I don't really see a big gap,

[Podcast Host/Announcer]

not even today, because open source AI has become one of the most important forces shaping the industry. But making frontier models available to the world requires far more than releasing model weights. It requires an entirely new infrastructure layer. In this episode, Elena Berger and Matt Borenstein are joined by Simon Mo, co founder and CEO of Infrac, to discuss how open source inference became critical infrastructure for AI, why enterprises are increasingly turning to open weight models, and what the next generation of AI systems will require. They also explore model licensing, inference economics, and the future of OpenAI.

[Elena Berger]

Today we're here with Simon Mo, co founder of Infract and a lead maintainer of Vllm, the open source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Borenstein, an A16Z general partner. Simon Matt, thank you so much for joining us. I think first we should start with open source AI and kind of the more recent history of open source AI. So VLLM actually has its origins kind of. Back in 2022 pre chatgpt and your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems. So can you talk about what made serving an LLM so fundamentally different from the workloads everyone already knew how to run?

[Simon Mo]

Yep, good to be here. So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs, and it is a computationally intensive process that will require a lot of engineering and ensuring that for each request you user can See, the LLMs response quickly and efficiently. So this typically means we need to handle differences in input distribution, how long each request is output distribution, which is non deterministic and batching and scheduling a lot more in the. At the core of the inference engine.

[Elena Berger]

Yeah. And so the team, the project has been around for about four years now, but the company is a little bit more recent. So I want to throw this out to either of you. Matt, I know you known the team and observed the team for a very long time. So at what point did you see this sort of transitioning from being a much beloved open source project to critical infrastructure and then a company?

[Matt Borenstein]

Yeah, I mean I think you have to go back a little bit. Right. Open source was the norm for AI models early on. Right. I mean we literally have this company called OpenAI which, you know, it's become a little bit of a joke. It's not as open as it once was, or not nearly as open as it once was. But early on all, all the frontier AI work was being open source or at least released into open weights, which is a little bit different than true open source. And people could mostly run these models sort of on hardware or computers that they already had. I'm curious Simon, like maybe you remember like what was like the first model that like you actually needed to go out and get special software and like a special set of computers to be

[Simon Mo]

able to run probably BERT. And before that it was like Resnet for computation, like images, computer vision, classification. So Resnet already need to run on Nvidia K80 which is kind of one of the first S class on AWS and other places and but way over. But even at this point resnet you can still kind of run on a commodity, even CPU devices. It's just very slow. But for Berge where running at it is like wow, you have to run it on GPU to make it anything faster and efficient for anything translation or any task. So that was like before 2020 even. Yeah.

[Matt Borenstein]

It's so funny thinking about this. Yeah, I'm like all the memories are flooding hugging face had a thousand BERT variants on it. Yeah, you have to go find like the right BERT variant for your particular task and yeah, and you're right, you had to. Some people I guess had their own GPUs and could run it. But yeah, a lot of people had to go provision cloud. Exactly, stuff like that. That's really funny. And so. Yeah, so so look, I mean BERT was an early language model that newer models are much bigger, much more sophisticated, take up A lot more memory, a lot more compute. And so Vllm really from the early days. Right. Was about running these more powerful models that you couldn't just sort of do it, figure it out on your own.

[Elena Berger]

Yeah, yeah. And I think to get us to this present day, I think it would be good to talk about kind of at what point it really became critical infrastructure for these even larger open source models and when did we even start to see these larger open source models kind of come into the field.

[Simon Mo]

While we really see the criticality of the software stack is always about bringing open frontier intelligence. Open frontier intelligence is a catchphrase for the K3 model release this time. And it's also about VLM's mission, about inferencing infrastructure. So if we look at it, when do they start being critical is when people are relying on it for their application, for their day to day productivity. So maybe 2023, when GitHub, Copilot and ChatGPT are something that people just cannot live out with anymore. At that point open weight model is already backing and become a cornerstone of how people are living every day. And at that point we will need the accelerator running open wait model and open wait software to open source software to make sure it works well.

[Matt Borenstein]

Yeah.

[Simon Mo]

And have all flexibility and control over it.

[Matt Borenstein]

Yeah. It's sort of interesting from a startup standpoint, like you mentioned sort of GPT3 or like early ChatGPT, those closed source solutions were starting to become critical to like a small group of people around that time and open source existed. But it was a little bit of a curiosity or sort of an enthusiast thing. As the frontier has expanded, particularly with closed source models, more and more open source has been dragged in as kind of like critical behind it. Like if that makes sense, like at any given point in time, including now, I think models from OpenAI and Anthropic are kind of more widely used and more critical kind of in general than open source models. But I do think we passed a threshold and I want to say about a year ago where a bunch of smaller companies or like new application companies, as they were trying to figure out how do I really build an AI without just being a wrapper on top of OpenAI, the answer to that question turned out to be open source. I mean this is what Cursor did, this is what sort of Decagon and Harvey are in the process of doing now, and a bunch of other like really, really strong applic level startups sort of made the determination we can't build just on closed source we need to do our own mid training, our own post training, our own sort of inference and deployment tricks and all of that means it must be built on top of open source. The closed source vendors won't give you the access to do this. So my read is like kind of a year ish ago, open source became really central in a way that's not always visible because it's deeply embedded in some of these products. But some of the most innovative products and applications now really depend on this very deeply.

[Elena Berger]

Yeah, yeah. And can you talk about where VLLM sits in that stack? Where we do have these larger enterprise companies that are choosing to use open source models. Like where does VLLM sit in the stack for them?

[Matt Borenstein]

Yeah, I mean you should like just about everybody uses vllm. You should describe it.

[Simon Mo]

Just about everybody uses vllm. VLM is a inference engine. That means its job is to turn available GPUs into a running endpoint for intelligence. So that means it is kind of like databases and operating system and other critical software to power this economy or power the AGI that everybody really uses today to ensure they can have cost effectiveness, efficiency, reliability and also always staying on the frontier. Because for VLM we support more than a thousand model architecture up to today. And a lot of those are proprietary, but also a lot of those are open weight. Right. And a lot of those model architecture, when they're becoming transitioning from a research prototype to world accessible open way model architecture, they are live on VLM immediately. So that's a process we call day zero model release. And additionally VEOM also work closely with all the hardware vendors. So that means across like Nvidia, amd, Google and Amazon, intel and a lot more, their newest chip will make sure VEOM can run on them. And then a lot of cases they use VEOM as a benchmark to make sure it runs well on them. So this kind of fusion of where models run and where it gets to meet the hardware is where the magic happen. And this is where VLM is.

[Matt Borenstein]

And you've told me some of the behind the scenes stories like it's actually not easy these days. Hero model releases, it's like a lot of human drama in addition to like technical work I guess. Are there any stories there that you think are okay to share?

[Simon Mo]

Oh, it's actually a very fun co design process because from model labs point of view, right, These are brilliant researchers who have built this model. Now their biggest question becomes how do we get this out of the world and make sure everybody's Able to use it and run it well. And we have work with model labs that are very just because they just use VRM already in production or in their research process, they will just done everything for you. Because this is a moment when we go to them it's like hi, where's the VOM team? We would like to support your open source model. We would like to offer in a way this kind of open source. But why gloss service to get your model running well on architecture and then you we get the model labs like oh, we'll get it working already because we're running it for the RL process. Here you go. Just review our code and merge our pull request. And then on the other end we really have model labs that just don't know how this can work.

[Matt Borenstein]

Because systems is not like their core.

[Simon Mo]

Yeah, because system is not their core and they have been training or maybe they have their internal inference engine that just don't know how it will adapt to the open web ecosystem. And by the way this is also a very much a multi party kind of involvement process. Every model race typically involves the model lab involves primary or secondary hardware vendors, involves us involves hugging face who are the model format and like model hub vendors. And then depending on the appetite of the model lab involves 10 or 20 different kinds of release partners. These could be inference cloud, these could be public hyperscalers. Whoever is going to run this model and you want them to ensure success, the model are running successfully. Right. So even up to today, if you look at the Case 3 model release, it's a whole partnership and a drive to make sure that the model is once the model is released because it's just a few terabytes of file sitting on the Internet that people are actually going to be using really really well. Even from the beginning of 2023, 2024, if you remember when Mistral dropped their first model, they just drop a torrent link for PTP and then everybody's like struggling and trying to get it up and running and then we're working behind the scene with Mistral team trying to get the inference engine support working in VR and this is one of the most probably early on exciting weekend that we are able to spend on this. And after the weekend when everybody's trying to really analyze what's going on. And Monday, Tuesday, well Mistral and us just announced here, you can run it on VLM successfully here and everybody will able to easily reuse, use a lot of the work and start building on top of it.

[Matt Borenstein]

That was sort of a Fun time where like enthusiasts like me could just like scramble to like download the model and like get it running somewhere. Yeah, yeah. I'm glad the professionals take it over because it never worked like very well. But it was fun. It was like a fun moment in time. Yeah.

[Elena Berger]

So, so to bring things, you know, forward to the present, I think open source models and also, you know, distillation have, have been in the news recently. Infra signed the Nvidia Open Weights and AI Leadership letter that that was signed by also a 16z meta, Amazon, dozens of other companies. Can you just talk about, you know, your decision to sign that and sort of what, what you were really kind of responding to in the market and kind of in the news.

[Simon Mo]

Yeah. So for us what really want to stand behind is openway absolutely matters in the ecosystem. The world cannot just be controlled by proprietary AP where open way, open development and research of these models are blocked or banned. Right. The pledge that Infra Gray signed up for is we want to help and foster this ecosystem where we are typically in little bit downstream of this ecosystem. Right. Inference engine are not part of the pre training process nor the RL process, but where the model actually meets the world. And from what we're seeing, people are just really using their imagination and ability to materialize this imagination of openway model. They're able to leverage these open way models so much effectively.

[Matt Borenstein]

There's almost two pieces to this, right. There's like the cost thing where it's like the closed models are too expensive and then there's sort of the control thing where I want to sort of be in control of my infrastructure and control the model. Right. If I need to extend it or put on my own guardrails or anything. I'm just curious, have you heard from from customers, like are both those things important to them or are they kind of willing to pay as long as they have the control? Or maybe they're different use cases.

[Simon Mo]

I think it fluctuates over time. So control matters a lot over the last few years and then costs just start to matter over the last few months. So cost really matters starting from people trying to migrate off their expensive coding plan and every skyrocketing token maxing spend. But control has always been in the backbone of this is they want to even in a way to control the cost. Right. But also it's about controlling the system performance against what they're paying for. So there are, for example for a voice agent company, they want to control their own model so that they can make sure the model actually respond by their required time. So the customer, when they're on the phone, they can ensure the agent is responding according to a sla. And this sometimes is only you can do with your controlled intelligence because you know the whole hardware you're running and the whole system you're monitoring versus signing up for relying on your critical infrastructure with a proprietary API where they might go down anytime or have violation of the contract at any time.

[Elena Berger]

Simon, you also to go back to the cost point, you actually make the point in an essay you recently wrote about the release of Kimik 3 that actually the economics is besides the point. And it's actually, you know, in, in the case of these, you know, just really, really great open weight models that are on the frontier that were designed by really brilliant researchers like, like these models are in some cases just as expensive as, you know, the closed source models. So in those cases, kind of what is the point of running them and kind of what do we learn architecturally in the, in the course of running them?

[Simon Mo]

Yeah, so first on cost, it's not necessarily they are as expensive as a proprietary model, but rather first the cost discourse has been discussed over and over again with even glm 5.2 a few months back. So openvm model are sometimes definitely a lot cheaper. But for this model there's a big sort of step change where we're bridging almost a 10x gap, but strike somewhere in the middle. Where kimik3 is not as expensive as plaude or gptsoul, but it is a lot more expensive than glm 5.2. Why is that? And I do believe this is the point of we're pricing intelligence with the market correctly and understanding where it is. But then the majority part of the discourse should be focusing on, wow, this model is bringing a Opus 4.8 level model to our own infrastructure that I can use, I can run, I can fine tune, I can build, able to understand exactly how many tokens do I need, understand the exact performance profile. The reason here, for example, is for proprietary model there is regular mode and fast mode and that's only the two switch here. But for open wait, when you are running it, every provider can offer potentially even 10 different levels of speed going from the slowest mode, which can be a lot cheaper, to 400 tokens per second, almost up to 500 in many cases that workloads and this is typically 2x or 3x faster than the fast mode out there today. So this kind of level of control, even in terms of performance and then let Alone control over how customer interacting with the model, control over data retention. Keep in mind Fable doesn't have zero data retention policy and at least a lot of the data need to be staying there and let it control security and compliance and a lot more. Yeah, this is why I'm particularly excited about case three. Not just from the cost perspective, but a lot more on bringing this level intelligence to something people can own in

[Elena Berger]

terms of calibrating things like speed, calibrating other things just sort of on the back end what needs to happen and kind of what are you seeing your users do and who is being really clever about this.

[Simon Mo]

So we do see users are able to get the maximum benefit out of this model when they're in enable fast mode. Like what I'm talking about here of course is VR ON's own fast mode getting up to 400 and 500 tokens per second because it is really a big step change from like especially when developer interacting with the model they can see oh I can really just get my task done faster here. And the model are no longer stuck in syncing, rather it is just executing executing and interacting with the environment. So for premium developer blocking focused task we're seeing is very benefiting. But also case three are just be able to have the ability for people to modify it and fine tune on top of it allow them to make it better for their own workload. And this is definitely happening today as well.

[Matt Borenstein]

Can you just explain what the licensing term is for the most recent open source models compared to the past and like why you think they're doing that?

[Simon Mo]

Oh yeah. So historically the. The openway model are just Apache to our software which is take it, modify it, do whatever you want with it. Here is a gift to the world. And then recently the model lab are trying to understand a way to economically fund their own model development. After all, model training and research and the data are very, very expensive. So we have been starting to see terms even to the llama days when Meta was releasing Llama they do have a term of if you're daily active user or annual recurring revenue exceeds some stress. Hope please enter into a commercial agreement with Meta specifically.

[Matt Borenstein]

Right. I do remember that the numbers were like specifically chosen at that time that you could go find it was like two companies in the world that fit the definition that they had excluded from their license.

[Simon Mo]

Yeah, exactly. But people have taken a hint from that especially now the labs are trying to figure out a way to economically fund it especially when they're open source the model everybody can just take it and run it themselves. Whereas nobody will use your API anymore in many cases while their API currently still taking up shape. Right. And now we're seeing a very healthy ecosystem development starting from even Minimax. Recently when they're releasing the M2.7 model, they have a term specifically focusing on usage. And Kimi initially also has like, if you have derivative works, like this is kind of big news back then with fireworks and cursor about how they build on top of Kimi model.

[Matt Borenstein]

And it's. If I, if I could just expand on that a little. Like I don't think it's greed. At least what I've seen from open source model labs. Right? Yeah. Open source models really what we're talking about are open weights. Right. And it's just not software. Right. Like an AI model is not software at the end of the day. And so open source software used to be supported by people donating their time or big companies kind of authorizing their employees to donate their time. So it's sort of like a bulk in kind donation of people's time. That really doesn't work in AI. Right. Like I can't just like go home at night and like train a frontier open source model with friends for fun. Like we need millions or billions of dollars of computing resources enabled in order to do it. So I think it does support your point that like obviously there need to be economic incentives and there need to be funding mechanisms in place, but frankly I think even more so with Chinese models than with domestically produced models. Right. If there's no source of economic, if there's no source of funding for Moonshot to continue to train models, we know where the funding will come from instead. And it's not something we write, it's government and things that are actually worse for us, I think. So I think you raised an interesting point that this is an important economic structure and I think this means we'll see more of this in the future. Would you agree with that?

[Simon Mo]

Yeah, it's really about sustainability in the end. It's about how do you make sure that all this initial capex almost to train the model fail again and again and train the model again. How do you really pay it back? And how do you make sure that there's enough confidence and funding from everybody involved to go to do the next one. Right. And I recently heard someone, I recently heard someone making an analogy to this to the pharmaceutical industry. It's almost like how do you make sure that the R and D process of new drugs are properly funded and is proper Sustainable method to making sure that people are willing to take big risk, big bad, to go to do research for new drugs and then later, because they know there's a economic incentive in the end, when the new drug released to the market, a portion of those, of course, besides just distribution channels. Right. A portion of those revenue will flow back to continue to fund the next R and D effort. And this is where we're kind of seeing. Similar to the model development now.

[Matt Borenstein]

Yeah, that's a really interesting analogy because it's like once a drug, a molecule is released, you have the strongest possible control, which is nobody else can manufacture it at all. It's like the most closed possible source. Right? Like it's like a secret. But yeah, like in the case of models, especially open source models, you know, once it's out there, anybody can take it, use it, extend it, et cetera. So having some economics attached to it probably does make sense.

[Elena Berger]

Yeah, well actually that raises a question for me too, which Matt, you were alluding to this earlier about how different open source models are from the dynamics of open source software maintenance. When it comes to open source AI, what needs to be maintained? Is it the infrastructure around it? Do the models themselves need maintenance at all? Just kind of. What, what are those dynamics? Because I think even the, the developer behavior around it is, is pretty different.

[Matt Borenstein]

Yeah, I mean Simon sort of said this already, but maybe, maybe I'll just expand a bit which is, you know, you, you see the results of, of a big training run, you know, where training now means it's pre training and then, and then you know, RL kind of mid training or sort of post training on these things or you know, pre training sft, rl.

[Podcast Host/Announcer]

Right.

[Matt Borenstein]

Like it's sort of complicated pipeline. We see only the result of this at the end. And the numbers are big. You're like oh, you know, this was a hundred million dollar training run. But what you often forget is like there may have been five failed training run, you know, large scale failed training runs before you even get to that.

[Simon Mo]

Blood, sweat and tears.

[Matt Borenstein]

Yeah, yeah, yeah, exactly. One of my favorite artifacts, maybe we could even track down the link is one of the early llama models. They published the whole conversation log between the people who are babysitting the training clusters while the models were training. And it's so funny, it's just like oh no, everything's gone wrong. Like yeah, us like panic. Then the next comment is like okay, we solved it, everything's okay. You know, clusters up, losses going down. So there's a lot a Lot, A lot that goes in behind the scenes before these models get released. You know, once they're out there, I guess it's a little bit more in your, in your zone to kind of make sure it's operationalized.

[Simon Mo]

Oh yeah, but this is also a very interesting point. Once it's out there is a whole community effort trying to optimize this model because the model is trained on a given type of hardware, on given type of architecture. But when it's out in the wild, everybody has different cluster topology end use cases and it's about how do you turn a use case of one now to a use case of almost infinity now you have people trying to adapt it to the edge devices and people trying to run it at largest scale ever, adapt it, making sure it runs for voice agent, but also for coding agent, which are entirely different kind of use cases. So this is a whole community effort trying to further optimize, specialize and making sure the running of it is reliable and continue to be able to optimize against it. So that's a whole village later throughout the open source to make sure it's improved.

[Matt Borenstein]

And that's cool because that really is like open source software. I mean, and this is what you do. But you know, like this is like anybody can contribute and make these better. The fun thought experiment is if GPUs dropped in price by 99%, if GPU based compute actually became kind of cheap and widely available, then do we get back to a real open source world where one person sitting in their basement or 100 people working their free time can come up with something new? Try many of these model training paths that are in the queue somewhere at one of the big companies know, kind of, kind of see, you know, really expand and advance the field collectively.

[Elena Berger]

Yeah, yeah. Well, I mean this relates to what you were talking about at the beginning. It's like, you know, at the beginning the amount of compute you needed to be at the quote unquote frontier was negligible. And now it's just like it's, it's enormous. And how do you, how do you get that, you know, back to consumer parody again?

[Matt Borenstein]

I've, I've said this on the podcast before, but I'll keep saying it. Alexnet first, you know, kind of like neural network to run on GPUs that we care about ran on two GPUs. And that's not, that's not like there are no missing decimal points or commas in there. Literally two now that would get you literally nowhere.

[Elena Berger]

Yeah, yeah, so I guess this, this relates to, to another question that we've had, which is, which is inference has gotten harder over the past 18 months because of this combination of scale and diversity and the kinds of models that we have. And of course like agents that are doing increasingly long running tasks. So what makes open source in this world and in this scenario not nice and nice thing to have, but absolutely necessary?

[Simon Mo]

Yeah. So scale comes from a few points. It comes from whether or not you can run this gigantic model on a data center rack. But also it comes from can you optimize it to the extreme point, to the speed of light, so that you are getting the most value out of it. But also you need a whole community and collaboration and partners of effort to validate it and making sure that there is no that last remaining bugs that appear like 0.0001% of the time. So this is where scaling up the deployment and then making sure more people are running it like largest footprint possible will ensure everybody's experience of running this model is better. And this is where kind of, this is why open source inference is the current leading way right now instead of a closed source inference engine. And frankly, right, A lot of the inference cloud and API as a service today do use and leverage open source inference engine under the hood. And the reason to do that is just there's just so much battle tested and learned recipes and things that they can build on top of. And this is where open source is the current leading way to run models.

[Elena Berger]

Something that I find kind of funny and like looking back at not just the history of Vllm and Interact, but also a company like OpenRouter or even Ollama. All of these different teams kind of got started around 2022 and 2023, some of them even before ChatGPT in your case, built around open models. And what kind of special thing do you think was happening at that time? Just like in the world of research and AI where we see these companies now and we look at them and you think of Vllm as mission critical or openrouter as mission critical. And what do you, what do you think was going on at that time where even before we had, you know, a big consumer use case, we had, you know, teams like yours that were, were building out these things and kind of how did you guys know? I guess is is the question.

[Simon Mo]

I think there's two parts to this. Our team always have an open source kind of angle. Where from UC Berkeley, a long tradition of open source software and our system research principles but also just curiosity. Overall everybody was so curious about how are these AI models going to benefit the humanity and how can we use it better. And this is where open source is, where you meet the most mission aligned people together. Openrouter Ollama, as you mentioned, they're all great partners and friends in the ecosystem because we're there to understand how everybody else can leverage AI model better and making sure they have a good use of it.

[Elena Berger]

Speaking of leveraging open, we open source models really well. So I think another thing that was in the news really recently was Hugging face using a Chinese open source model to help contain a cyber attack carried out by a rogue UN sandboxed OpenAI model that was being tested. So like, like what can we take away from something like that? And can you just first, I mean walk us through your impressions and understanding of what happened and then kind of what, what you think the, the takeaways there are?

[Simon Mo]

Yeah, like this kind of goes back to our previous point about control. So for the huggy face incident, they break it down pretty, pretty well on their website and blogs. So really thanks for their transparency there. In the end it's about all the closed proprietary model API. Their guardrails are a little bit arbitrary but also very difficult to enforce. That means they have so much false positive in the guardrails that will have legitimate use cases just being blocked. And this is like a evergreen problem even in the social media days. Like how do you design content filters correctly and be able to do the moderation correctly? So if moderation is never solved, which is going to be very, very hard, then there's always a place where you have a model where you know and trust that you are publishing to and be able to use from. So hugging face they have to use openway model. But in the future also see for the trusted use case, people will go to openway by default because that is where you know for sure that the guardrail is lessened or you can control your guardrail for trusted use cases. Even for us, this is actually applied to us today where a lot of the anthropic models are banning frontier AI research. And then when we're studying GPU kernels, even as a invalid memory access error, we are triggering the red line. And so a lot of our developer within infrastruct and full VL are retreating from using Fable 5 because, because you have a two hour job and you trigger the red line which is false positive and then you have to lose all of your work. And so a lot of our developers are using like Kimi K3 today, even just making sure, because it's similar quality and it has a guardrail that makes sense to us. There it goes. While using it.

[Matt Borenstein]

I think your social media analogy is a really apt one because in both cases what's kind of happened is you've taken distributed human activity and kind of centralized it in one place. Like, like before social media people talked to each other, right. And published articles and like messaged each other on message boards. But then you centralize it all under one kind of like profit seeking enterprise. And all of a sudden the incentives change like a lot. Right. And we, we have a specific carve out for social media which is you are not responsible for what people say on your platform. And that allowed the moderation problem to be tractable, I think. Right. It's like, okay, we're gonna do our best to eliminate obviously illegal things, you know, like things that are just gonna degrade experience on the site, you know, you know, in, in really bad ways. But like, you know, if it's on the edge, you know, it's something that's maybe out of fashion to say, but not illegal.

[Simon Mo]

Yeah.

[Matt Borenstein]

Like we're just, we, we, we just can't police everything. We can't be the police of like all human communications. I think something similar is sort of happening in AI, right, where a lot of work, not just talking, but work work is kind of being consolidated in one place. If I need to write code or create a spreadsheet or get health advice or anything, it's all happening in one or two sites and they don't have that carve out, they don't have that exemption of hey, we're not responsible for what actually happens on the site. And some of them, especially anthropic, is kind of going further than even what would be legally required. And they're sort of taking ethical stances on these things, which may be right or maybe right. That's sort of their decisions. And how do you actually do this? Right. Like the problem's actually bigger than just communication because it's like actual work and action and you don't have this exemption and you have this sort of layer of ethical stuff on top. So I, yeah, I'm just much harder. It's, I'm just totally with you. It's like, and it seems like some of these companies have gone like erred on the side of caution, which is probably a reasonable thing to do. But very, very, very frustrating as, as a user. Right. Like I, you know, I saw online some translation attempts are blocked because they think the particular combination of like language and content is somehow like triggering like politically sensitive topics and you know, things like this.

[Elena Berger]

Yeah, Simon, we're, we're nearing the close of the conversation and I just wanted to take a step back a bit and ask just about Infract and you know, running the company. And I know that Ian Stoica of Databricks is an advisor and a co founder of Infract and I'm just curious what you've learned from him in terms of taking an open source project like Vllm and building a company.

[Matt Borenstein]

Yeah.

[Simon Mo]

Young as a co founder, he has always been thinking about open source and where, how do you support open source better? And then now with experience from databricks and Anyscale and even arena, which is a public platform, and now at Infrac, his focus really stemmed from how do we build such a. In a world where this software is being used by so, so, so many people on such a critical project, how do you make sure they're getting the best quality and where the value come from? Right. If the open source movement and open source inference engine is a starting point, then what are the gaps that we can fill as infrared and then what are the gaps that we should absolutely put in? Young has always been open source first. So for whatever we need to build, we would like to build them in open source and then for whatever value we can deliver, we want to really close the last mile and making sure that customer and partners are getting the best out of it.

[Elena Berger]

Just I guess to close this out five years from now. Do you think open weight, open source AI models, have they closed the gap with frontier models completely? Are frontier models always one step ahead? Kind of. How, how do you see that shaking up five years?

[Simon Mo]

Five years is a lot.

[Elena Berger]

All right, all right. One year, one year, five years.

[Matt Borenstein]

Who knows, we're either, you know, we're all going to be like just floating around in our wall egg.

[Simon Mo]

For me really at this point, there's kind of a point we haven't talked too much about is what really differentiate open way model from closed way model. Right. In the end, there's not much differentiation. It's more about the distribution strategy and go to market strategy and the capability wise. I don't really see a big gap, not even today because for how these models are coming to being, they're really starting from the first principle.

[Matt Borenstein]

Right.

[Simon Mo]

You have a compute cluster, you have training data and you have brilliant researchers that group together and really to build this amazing artifact that is this Pre trained model and then later our old post trend model and that the world can use. But if you look at the ingredients, one of the most important part just the data is about who gets what data and then what are the environment you are building to let the model improve on itself and make better. Right. One of the very useful benchmark that we have on arena for K3 has been front end coding. That means for Moonshot they have built some of the best environment for front end coding.

[Matt Borenstein]

Right.

[Simon Mo]

They have published amazing demo on the ability for this model to code and then see what the render is and then kind of continue looping and this iterative process. Now this is about their environment to improve the model. It's not about just source data, it's not about where they get the data from. Rather it's who can build the best environment and who can make the most through of optimization and algorithmic choices to leverage Aldo's learning from this environment. So the next year is all going to be about that. It's about how open way model labs are differentiating and really getting the model to meet the real world and have this kind of what people are popular today like recursive self improvement almost to really improve the model overall. And so really project out in your ear. There's not going to be any difference.

[Matt Borenstein]

Yeah, and you've used this term brilliant researchers a few times. There are brilliant researchers everywhere in the world. Clearly. Why do you think in the US all the smart researchers are working on closed models and in China all the smart researchers are working on open models.

[Simon Mo]

I mean from my point of view they are attracted to interesting problems, not necessarily on the open or closed stance, but however, open weight model does give people a really, really good boost on the impact of such models. So that is like a plus. And I think all the brilliant researchers are attracted to how to improve the model overall. Right. Actually one interesting point about this, maybe fairly technical for this kimik3 model is they removed a rotary positional embedded. So rope has always been there for a lot of the transformers model. And guess who removed it is the inventor of rope.

[Matt Borenstein]

Who are.

[Simon Mo]

Yeah, like Jenning, he wrote the first paper introducing rope as a concept and then he now also wrote the explanation of why you don't need it as part of the technical report in this case model. So when we read it it's like really come full circle is you have all these brilliant humble researchers that are able to really study how this work and really study the secret of training and pre training and share it across the whole world and recognizing and iterating on their past. Right. So really a miracle, I would say, for this model to come alive.

[Matt Borenstein]

It's so interesting. Like, you know, AI is in this funny zone where empirically it works incredibly well. But then you go ask the theorists and they have like no idea what's going on. Right. And so like you have these kind of iterative things where when you go the, you know, primer on transformers. Yeah. You read about positional embeddings and why or positional encodings and why it's so important because otherwise you can't sort of like understand meaning. And then it turns out once you understand one level deeper, because we've been doing this for a few years and you have all these smart people like, oh, actually you don't need. Exactly, you know, simpler actually is better.

[Simon Mo]

Yeah.

[Matt Borenstein]

We didn't talk about distillation much so far in this conversation, but I think it's very relevant to this. Like. Yeah, I have just one question which is like, I'm not going to ask, like is distillation happening? I think this is kind of speculation on the part of everybody, you know in the world. But like you work a lot with these Chinese labs. Do you think distillation, like is a critical component of what they do? Or like are they kind of just doing good work and you know, distillation, if it's done, is sort of an incidental part of it.

[Simon Mo]

I will lean to the latter part specifically. As I mentioned previously, environment matters so much today. So these are RO environments, right. These cannot be distilled. Like you don't have other people's environment to really distill a copy from. It's about constructing it. Also understanding the learning process. You cannot distill how the model learns within environment. A lot of these are just not doable today. There are things potentially you can do with rewriting the data set, making better pre training data. But again you can do it with any models. Any models that are going to follow instruction are going to be useful in terms of utility there. So I really don't think from currently what we're seeing, this is a big cornerstone of what's powering the progress today. In the end, what's powering the progress is still just really smart people with very interesting algorithms, data, environment. And they will produce, of course, compute, they will produce the models.

[Matt Borenstein]

I think it has really interesting policy implications. I tend to agree with you, by the way, that we have smart people everywhere working on a bunch of smart things. And it's not about distilling data from any one place has really interesting policy implications because it's almost tempting if you're in the White House to say, oh, sure, we'll just turn off distillation, all our problems will be solved. But like, you know, I think it's more the case that they're just, you know, smart people doing interesting things. And so it's, so it's like, how do we, how do we kind of like adapt. Adapt to that? I think it's.

[Simon Mo]

Yeah. And creative innovations. Right. Like one part in my essay we kind of mentioned that open source and open way really helps innovation because it set out this racetrack where everybody can learn from each other and see where each person, like every, each player is in this racetrack and then you are able to improve and, and stand on shoulder of each other kind of to improve yourself. So that is where everybody can move forward faster.

[Matt Borenstein]

Yeah. And one thing we're looking for a lot from an investment standpoint is people doing more open source model training all over the world, not just in China, not just in the United States, but. All right. Because you get that sort of magic of collaboration when everybody's doing it and achieve global harmony and.

[Elena Berger]

All right, well, I think that's a good note. 10 on Matt Simon, thank you so much for joining us. Thanks for tuning in.

[Simon Mo]

Thanks so much. Cold thanks, Sean.

[Podcast Host/Announcer]

Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes go to YouTube, Apple Podcasts and Spotify. Follow us on x16zz and subscribe to our [e-mail] thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investment investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com disclosures.