Eric (2:33)

And Fei. Fei, why did you have to have Martine as the first investor?

Fei-Fei Li (2:36)

Well, first of all, I knew Martin for more than a decade, long time. You know, I joined Stanford in 2009 as a young assistant professor and Martin was finishing his PhD there. So I always know. And of course, Martin's advisor, Nick McCune, was a good friend, and I always know Martin went on to became a very successful entrepreneur and very successful investor. So we see each other, we talk about things. But as I was formulating the idea of World Labs, I was looking for what I would call my unicorn investor. I don't know if that's a word, but that's how I think about this. Who is not only obviously a very established and successful investor, who can be with entrepreneurs on this journey through the ups and downs, who can be very insightful, who can bring the kind of knowledge, advice resource. But I was also particularly looking for an intellectual partner, because what we are doing at World Labs is very deep tech. We are trying to do something no one else has done. We know with a lot of conviction it will change the world, literally. But I need someone who is a computer scientist, who is a student of AI, understand product, market, go to market, go to market, and just can be on the phone or in person with me every moment of the day as an intellectual partner. And here we are. We talk almost every single day.

Martin Casado (4:06)

It is true.

Fei-Fei Li (4:07)

Yeah.

Eric (4:07)

Amazing.

Martin Casado (4:08)

Actually, the origin story of us first connecting is actually pretty interesting. So Fevi has clearly been thinking about this idea for a very long time, like, well before starting, so maybe years even. And she has this very deep intuition of what AI needs in order to basically navigate the world. Right? But we were at one of Mark's fancy lunches and there's a bunch of AI people, and everybody was so excited about LLMs, right? And it was talking about language. And I'd come to this independent conclusion just because I've actually done a lot of image investing, that, like, that wasn't the end of the story. And so the end of this table, all these people talking about it. Vivian leans over to me and she's like, you know what we're missing? I said, what are we missing? She said, we're missing a world model. And I'm like, yes. And it fell into place then because I'd been like, thinking about stuff at a high level. But as she does, she just kind of Perfectly articulated this. So she had a year's worth of thinking about this and talked to people, et cetera. And so in some way we kind of in our own crooked past had arrived at a very similar intuition. Hers was like way more filled out, mine was just this kind of fancy thing. But then after that, we actually had a number of conversations where we both agreed that we were aligned on this kind of idea.

Fei-Fei Li (5:09)

Actually, I don't know if you know this, so of course during that lunch we hit it off on this world model idea. But I was at that point already talking to various people, not just computer scientists, technologists, but also investors, potentially business partners. And to be honest, most people didn't get it. You know, when I say role model, they nod, but I can just tell that was just a polite nod. So I called Martin. I'm like, do you mind coming over to Stanford campus and have coffee with me?

Martin Casado (5:38)

Cup of cafe?

Fei-Fei Li (5:39)

Yeah, go to cafe. And then as soon as Martin came and sat down, I said, Martin, can you define your world model to me? I really wanted to hear if Martin actually meant it. And the way he defined it about an AI model that truly understand the 3D structure, shape and the compositionality of the world was exactly what I was talking about. And I was like, wow, he's the only person so far I've talked to who actually meant it. It's not just nodding.

Eric (6:10)

Wow. Okay, so we're going to get to World Labs and the specifics of this. But first I want to take you back Both to your PhD days, your professor days, and reflect on if you could go back in time and sort of have knowledge of what's happened the preceding 10 years in AI? What do you think would have been the biggest surprises? Or what's the thing that you didn't see coming that would have shocked your younger self? Or did you have a good sense of how this feels play out?

Fei-Fei Li (6:32)

Yeah. It's ironic to say because as Martin said, I was the person who brought data into the AI world. But I still continue to be so surprised. Not surprised intellectually, but surprised emotionally that the data hungry models, the data driven AI, can come this far and genuinely have incredible emergent behaviors of thinking machine. Right, yeah.

Eric (7:01)

Let's get into the specifics. Why start another foundation model company? Why aren't LLMs enough?

Fei-Fei Li (7:07)

My intellectual journey is not about company or papers. It's about finding the North Star problem. So it's not like I woke up and said I have to do a company. I woke up every day, day after day for the past few years thinking that there is so much more than language. The language is an incredibly powerful encoding of thoughts and information, but it's actually not a powerful encoding of what the 3D physical world that all animals and living things live in. And if you look at human intelligence, so much is beyond the realm of language. Language is a lossy way to capture the world. And also one subtlety of language is purely generative. Language doesn't exist in nature. We look around, there's not a syllabus or word, whereas the entire physical, perceptual, visual world is there. And animals entire evolutionary history is built upon so much perceptual and eventually embodied intelligence. Humans, not only we survive, live, work, but we build civilization beyond language upon constructing the world and changing the world. So that's the problem I want to tackle. And in order to tackle that problem, obviously research was important and I spent years doing that as an academic and it's still fun. But I do realize, and especially talking to Martine, that the time has come that concentrated industry grade effort, focus effort in terms of compute data talent is really the answer to bringing this to life. And that's why I wanted to start World Apps.

Martin Casado (8:59)

Amazing, Eric. You can do a very simple thought experiment that kind of highlights the difference between language and space. So if I put you in a room and I blindfolded you and I just described the room and then I asked you to do a task, the chances of you being able to do it are very little. I'm like, oh, 10 foot in front of you is like a cop. It's this very inaccurate way to convey reality because reality is so complex and it's so exact, right. On the other hand, if I took off the blindfold and you could see the actual space, right? And what your brain is doing is actually reconstructing the 3D, right. Then you can actually go and manipulate things and touch things. Right. And so one way to think about is we do a lot of language processing and we use that to communicate and high level ideas, et cetera. But when it comes to navigating the actual world, we really, really rely on the world itself and our ability to reconstruct that.

Eric (9:48)

And how and when did you realize that language weren't enough? Because it seems like it's not super widely known. I don't hear about this all the time.

Martin Casado (9:55)

Well, so if you ask me, like what is this surprising breakthrough? It's that language went first because we've worked so hard on robotics, right. I mean I feel like even look at autonomous vehicles as an industry, we've invested like $100 billion in it. I remember when Sebastian Thrun, like actually won like the DARPA Grand Challenge 2006 and we're like, Hooray. AV is done, right. And then 20 years later, like we're finally there. $100 billion in etc. This is like a 2D problem.

Fei-Fei Li (10:23)

Ye.

Martin Casado (10:24)

So that was the path we were going on is do you actually solve world navigation? And it's hard. Then out of Nowhere comes these LLMs and they are unit economic positive. They solve all of these language problems basically immediately. And so it just took me a moment actually. Fei fei said it beautifully early on when we were talking, which is the part of our brain that actually deals with language is actually pretty recent. And so we're actually pretty inefficient at it. Right. And so the fact that a computer does it better is not super surprising, but the part of the brain that actually does the navigation, you know, the spatial, has been around. It's a million brains, maybe the reptilian brain, about 4 million.

Fei-Fei Li (11:01)

It's even more than that. It's a trilobite break.

Martin Casado (11:04)

Yeah, yeah, right.

Fei-Fei Li (11:04)

Trilobite head break.

Martin Casado (11:05)

Right.

Fei-Fei Li (11:06)

500 million years.

Martin Casado (11:07)

Yeah. So it's almost like we're unrolling evolution. Right. So the language part is actually very, very important for like high level concepts and like the laptop class type work, which is what it's impacting right now. But when it comes to space, and this is everything from robotics, so anything where you're trying to construct something physical, you have to solve this problem. And then we know from AV that it's a very tough problem. And then maybe this is what is worth talking about. Like the generative wave gave us some insight in how you might want to do it. So it really felt like that was the time.

Fei-Fei Li (11:33)

My journey is very different because I've always been vision. Right. So I feel like I didn't need LLM to convince me LWM is important. I do want to say we're not here bashing language. I'm just so excited. In fact, seeing ChatGPT and LLMs and these foundation models having such breakthrough success inspires us to realize the moment is closer for world models. But Martin said it so beautifully. It's that space, the 3D space, the space out there, the space in your mind's eye, the spatial intelligence that enable people to do so many things. That's beyond language is a critical part of intelligence. It goes from ancient animals all the way to humanity's most Innovative findings such as the structure of DNA. Right, that double helix in 3D space. There's no way you can use language alone to reason that out. So that's just one example. Another one of my favorite scientific example is buckyball carbon molecule structure that is so beautifully constructed. That kind of example shows how incredibly profound space and 3D world is.

Eric (12:49)

Let's paint even more of a picture. When World Labs has achieved its vision or language, World models have achieved their vision. What are some applications or use cases that we can present to the audience to help make it concrete?

Fei-Fei Li (12:58)

Yeah, there is a lot, right? For example, creativity is very visual. We have creators. From design to movie to architecture to industry design creativity is not just only for entertainment. It could be for productivity, for machinery, for many things. That alone is a highly visual perceptual spatial area or areas of work. Of course, we mentioned robotics. Robotics to me is any embodied machines. It's not just humanoids or cars, there's so much in between. But all of them have to somehow figure out the 3D space it lives in, have to be trained to understand the 3D space, and have to do things, sometimes even collaboratively with humans. And that needs spatial intelligence. And of course, I think one thing that's very exciting for me is that for the entirety of human civilization, we all collectively as people, lived in one 3D world. And that is the physical Earth 3D world. A few of us went to the moon, but, you know, it's a very small number. But that's one world. But that's what makes the digital virtual world incredible. With this technology, which we should talk about is the combination of generation and reconstruction. Suddenly we can actually create infinite universes. Some are for robots, some are for creativity, some are for socialization, some are for travel, some are for storytelling. It suddenly will enable us to live in a multiverse way. The imagination is boundless.

Martin Casado (14:44)

I think it's very important because these conversations can sound abstract, but they're actually not. But the reason they sound abstract is because it's truly horizontal, just like LLMs are, right? So if you guys say, what are LLMs good at? The same LLM we use for an emotional conversation. We use it to write code, we use to do lists, we use it for self actualization. And so I think we can get actually pretty concrete about what these models do. Right? And so let me just give it a shot. And then Fei. Fei is the expert, of course. So with these models, you can take a view of the world, like a 2D view of the world, and Then you could actually create a 3D full representation, including what you're not seeing, like the back of the table, for example, within the computer. So given just a 2D view, you have the full thing. And then you ask, okay, well, what can you do with that thing, for example? Well, you can manipulate it, you can move it, you can measure it, you can stack it. So anything that you would do, a space you could do, Right. That means you could do architecture, you could do design. But it turns out the ability to fill out the back of the table means that you can fill out stuff that was never there to begin with. Right. So let's say that I just had a 2D picture of this. I could create a 360 of everything. Right. And so now you have fully generative. And so what does that mean? That means that's video games, that's creativity. And so it's a super horizontal piece that takes basically a computer with a single view in the world, or maybe multiple views in the world, and creates a full 3D representation that that computer then can act on. And so you can see that that's a very concrete, pivotal thing from everything from like robotics to video games to art and design.

Eric (16:13)

Yeah, it seems like we haven't fully been appreciating sort of 3D components until now. Is that fair to say?

Fei-Fei Li (16:18)

It is fair to say. In fact, I think it took evolution a long time. 3D is not an easy problem, but I always come back to the fact that I had a conversation with my 6 year old years ago about why trees don't have eyes. And the fundamental thing is trees don't move, they don't need eyes. So the fact that the entire basis of animal life is moving and doing things and interacting gives life to perception and spatial intelligence. And in turn, spatial intelligence is going to reinvent horizontally, as Martin said, so many of the way of work and life that humans are doing.

Eric (17:04)

Yeah, fascinating.

Martin Casado (17:04)

But it is definitely worth asking the question, why can't you just use 2D video for this? Right. Like, 3D is very, very fundamental to.

Eric (17:10)

This Vivi you suggested. Let's get deeper into the technology. What can we share more about how it works or what the breakthrough is, or what's worth commenting on the technology, to Martine's point, does it need to be 3D or why can't you just use 2D?

Fei-Fei Li (17:21)

I think you can do a lot of things using 2D, but the fact is that 2D will get you very far. In fact, today's multimodal LLMs is already Making a big difference in the robotic learning world, helping guiding you to know what's next, the state of the world. But fundamentally, physics happens in 3D, and interaction happens in 3D. Navigating behind the back of the table needs to happen in 3D, composing the world, whether physically, digitally, needs to happen in 3D. So fundamentally, the problem is a 3D problem.

Martin Casado (18:02)

One way to think about it is if it's a human being looking at, say, a 2D video, the human being can reconstruct the 3D in their head, Right? But let's say I've got a robot that has the output of the model. If that's 2D, and then you ask the robot to do, I don't know, distance or to grab something, that information's missing. You've got the XYZ plane, the Z plane just isn't there at all. Right. And so for many things that are spatial, you need to provide that information to the computer so that you can actually navigate in 3D space. And so 2D video is great if it's a human, because we already can turn it into 3D. But like, for any computer program, it'll need to be 3D.

Fei-Fei Li (18:37)

Actually, I want to tell you a personal story. About five years ago, ironically, I lost my stereo vision for a few months because I had a cornea injury. And that means I was literally seen with one eye. And like Martine said, my whole life has been trained with stereo vision. So even if I was seen with one eye, I kind of know what the 3D world looked like. But it was a fascinating period as a computer vision scientist, for me to experiment what the world is. And one thing that truly drove home literally, was I was frightened to drive.

Eric (19:12)

Wow.

Fei-Fei Li (19:12)

First of all, I couldn't get on highway that speed. I could not, you know, but I was just driving in my own neighborhood, and I realized I don't have a good distance measure between my car and the parked car on a local small road. Even though I have perfect understanding of how big is my car, almost how big is the neighbors, the parked cars, I know the roads for years and years, but just driving there, I had to be so slow, like almost 10 miles an hour, so that I don't scratch the cars. And that was exactly why we needed stereo vision.

Martin Casado (19:47)

That's actually a great articulation of why 3D is just actually key if you're doing some processing, right?

Fei-Fei Li (19:52)

Yeah. So I don't recommend it. But if you're stereo, park your car 1 and drive your car 2 with one eye and Feel it. That's your own car.

Eric (20:01)

On the tech side with LLMs, a lot of the research was done at the big companies. What's the state of the research here?

Fei-Fei Li (20:06)

This is definitely a newer area of research compared to LLM. It's not totally fair to say new because in computer vision as a field, we have been doing bits and pieces. For example, one important revolution that has happened in 3D computer vision was Neural Radiant Field, or NERF. And that was done by our co founder, Ben Mildenhall and his colleagues at Berkeley. And that was a way to do 3D reconstruction using deep learning that was really taking the world by storm about four years ago. We've also got a co founder, Christoph Lassner, whose pioneering work was part of the reason Gaussian splat representation started to again become really popular as a way to represent volumetric 3D. And of course, Justin Johnson, who was my former student, also co founder of World Labs, were among the first generation of deep learning computer vision students who did so much foundational work in image generation when before Transformer were out, we were using GANS to do image generation and then style transfer, which was really popularized. Some of the components or ingredients of what we're doing here. So things were happening in academia, things were happening in industry. But I agree with what is exciting now is that at World Lab we just have the conviction that we're going to be all in on this one singular big North Star problem, concentrating on the world's smartest people in computer vision, in diffusion models, in computer graphics, in optimization, in AI, in data. All of them coming to this one team and try to make this work and to productize this.

Martin Casado (22:03)

I will say from an outsider standpoint, and so I'm not an expert in any of these spaces, but it really feels like to solve this problem you need experts both in AI. And that's like the data and the models, like the actual model architecture and graphics, which is like, how do you actually represent these things in memory in a computer and then on the screen? So it takes a very special team to actually crack this problem, which Fei Fei has managed to put together.

Eric (22:28)

Well, that's an inspiring note to wrap on. Fei Fei, thank you so much for joining us.

Fei-Fei Li (22:31)

Thank you. Thank you, Eric.

Podcast Host (22:35)

Thanks for listening to the A16Z podcast. If you enjoyed the episode, let us know by leaving a review@ratethispodcast.com a16z. We've got more great conversations coming your way. See you next time.