← The a16z Show
The a16z Show
Marc Andreessen and Amjad Masad: English As the New Programming Language
01:11:38
Amjad Masad, founder and CEO of Replit, joins a16z’s Marc Andreessen and Erik Torenberg to discuss the new world of AI agents, the future of programming, and how software itself is beginning to build software. They trace the history of computing to the rise of AI agents that can now plan, reason, and code for hours without breaking, and explore how Replit is making it possible for anyone to create complex applications in natural language. Amjad explains how RL unlocked reasoning for modern models, why verification loops changed everything, whether LLMs are hitting diminishing returns — and if “good enough” AI might actually block progress toward true general intelligence.
Loading summary
Transcript259 lines
- [00:00]
Marc Andreessen
We're dealing with magic here that we, I think, probably all would have thought was impossible five years ago or certainly 10 years ago. This is the most amazing technology ever. And it's moving really fast and yet we're still, like, really disappointed. Like, it's not moving fast enough. And like, it's like, maybe right on the verge of stalling out. We should both be, like, hyper excited, but also on the verge of like, slitting our wrists. The gravy train is coming to an end, right?
- [00:18]
Amjad Masad
It is faster, but it's not at computer speed. Right what we expect computer speed to be.
- [00:23]
Marc Andreessen
It's sort of like watching a person work.
- [00:25]
Amjad Masad
It's like watching John Carmack on cocaine.
- [00:29]
Marc Andreessen
Okay? The world's best programmer on a stimulant. Stimulant.
- [00:33]
Amjad Masad
Yeah, that's right.
- [00:35]
Host
Every few decades, programming takes a massive leap forward, and this might be the biggest one yet. In this episode, Marc Andreessen and I are joined by Amjad Massad, CEO and founder of replit, to talk about how AI agents are changing what it means to code. We discussed the end of syntax, the rise of agents that can think and build software for hours, and how reinforcement learning and verification loops are pushing AI towards something that looks a lot like reasoning. And finally, Amjad shares his story, from hacking his university database in Jordan to building one of the most powerful developer tools in the world. Let's get into it.
- [01:12]
Marc Andreessen
So let's start with. Let's assume that I'm a sort of a novice programmer. So maybe I'm a student or maybe I'm just somebody. I took a few coding classes and I've hacked around a little bit or I don't know, I do Excel macros or something like that, but I'm like, not. So as I'm not like a master craftsman at coding and somebody tells me about ReKit, and specifically AI and ReKit. What's my experience when I launch in with what ReKit is today? With AI?
- [01:33]
Amjad Masad
Yeah, I think the experience of someone with no coding experience or some coding experience is largely the same. When you go into replit, the first thing we try to do is get all the nonsense away from setting up development environment and all of that stuff and just have you focus on your idea. So what do you want to build? Do you want to build a product? Do you want to solve a problem? Do you want to do a data visualization? So the prompt box is really open for you. You can put in anything there. So let's say you want to build a startup. You have an idea For a startup, I would start with a paragraph long kind of description of what I want to build. The agents will read that it will.
- [02:09]
Marc Andreessen
Just type it in standard English. In standard English, you just type it in, I want to sell grapes online. So you just like type in I want to sell grapes online.
- [02:15]
Amjad Masad
It literally could be that four words or five words.
- [02:18]
Marc Andreessen
Okay.
- [02:18]
Amjad Masad
Or it could be if you have a programming language you prefer or Stack you prefer, you could do that. But we actually prefer for you not to do that because we're going to pick the best thing for. We're going to classify the best stack for that request. Right. If it's a data app, we'll pick Python, Streamlit, whatever. If it's like a web app, we'll pick JavaScript, JavaScript and Postgres and things like that. So you just type that or you.
- [02:39]
Marc Andreessen
Can decide, you can say, and I want to do it in I know Python. Or I'm learning Python at school and I want to do it in Python.
- [02:44]
Amjad Masad
That's right. The Cool thing about Replit is we've been around for almost 10 years now and we built all this infrastructure. Replit runs any programming language, so if you're comfortable with Python, you can go in and do that for sure.
- [02:54]
Marc Andreessen
Okay. And then just again, I know this is obvious, people have used it, but like, I'm dealing in English. Yes, go ahead.
- [02:59]
Amjad Masad
Yes, you're fully in English. I mean, just a little bit of a sort of background here. Like when I came here and pitched to you like 10 years ago ago or whatever, seven years ago, what we were saying is we were exactly describing this future is that everyone would want to build software. And the thing that's kind of getting in people's ways is all the, what Fred Brooks called the accidental complexity of programming. Right. They're like essential complexity, which is like, how do I bring my startup to market and how do I build the business? And all of that accidental complexity is what package manager do I use? All of that stuff. We've been abstracting away that for so many years and the last thing we had to abstract away its code. I had this realization last year, which is, I think we built an amazing platform, but the business is not performing. And the reason the business is not performing is that code is the bottleneck. Yes, all the other stuff is important to solve, but syntax is still an issue. Syntax is just an unnatural thing for people. So ultimately, English is the programming language.
- [04:02]
Marc Andreessen
By the way, just to. Does it work with other world languages other than English at this point?
- [04:05]
Amjad Masad
Yes, you can Write in Japanese. And we have a lot of users, especially Japanese, that tends to be very.
- [04:10]
Marc Andreessen
So does it support these. Does AI support every language or is it still. Do you still have to do custom work to craft a new language?
- [04:15]
Amjad Masad
No, most mainstream languages that has 100 million plus people who speak it, AI is pretty good at it.
- [04:20]
Marc Andreessen
Okay. Yeah, yeah.
- [04:21]
Amjad Masad
So I did a bit of historical research recently. For some reason, I just want to just understand the moment we're in. And because it's such a special moment, it's important to contextualize it. And I read this quote from Grace Hopper. So Grace Hopper invented the compiler. As you know, at the time, people were programming machine code. And that's what programmers do, that's what specialists do.
- [04:43]
Marc Andreessen
Yes.
- [04:44]
Amjad Masad
And she said specialists will always be the specialist. They have to learn the underlying machinery of computers. But I want to get to a world where people are programming English. That's what she said. That's before karpathy.
- [04:54]
Marc Andreessen
Right.
- [04:55]
Amjad Masad
That's 75 years ago. And that's why I invented the compiler. And in her mind, like C programming is English. But that was just the start of it. You had C and then you go higher level Python, JavaScript. And I think we're at a moment where it's the next step. Instead of typing syntax, you're actually typing thoughts. Right. Which is what we ultimately want.
- [05:15]
Marc Andreessen
And the machine writes the code.
- [05:16]
Amjad Masad
And the machine writes the code. Right, right.
- [05:17]
Marc Andreessen
Yeah, I remember. You're probably not old enough to remember, but I remember when I was a kid there were higher level languages by the 70s like BASIC and so Forth and FORTRAN and C. But you still would run into people who were doing assembly programming. Assembly language, which by the way, you still do, like game companies or whatever, still do assembly to get.
- [05:31]
Amjad Masad
And they were hating on the kids that were doing basic.
- [05:33]
Marc Andreessen
So the assembly people were hating on kids doing basic. But there were also older coders who hated on the assembly programmers for doing assembly. And not. And not. No, no, no. Not doing direct machine code. Right. Not doing direct 011 machine code. So for people who don't know, assembly language is sort of this very low level programming language that sort of compiles to actual machine code. It's incomprehensible gibberish to most programmers, Even.
- [05:51]
Amjad Masad
Most programmers, you're writing an octal, you're.
- [05:53]
Marc Andreessen
Writing like very, very close to the hardware. But even still, it's still a language that compiles to zeros and ones, whereas the actual real programmers actually wrote in zeros and ones. Yeah. And so there's always this tendency for the pros to be looked down the nose.
- [06:03]
Amjad Masad
Yeah.
- [06:04]
Marc Andreessen
And say the new people are being basically sloppy. They don't understand what's happening, they don't really understand the machine. And then of course, what the higher level abstractions do is they.
- [06:12]
Amjad Masad
The absolute irony is I was part of the JavaScript revolution. I was at Facebook before starting Replit and we built the modern JavaScript stack. We built React JS and all the tooling around it and we got a lot of hate from the programmers that you should type vanilla JavaScript directly. And I was like, okay, whatever. And yeah, and now that's mainstream. And now those guys that built their careers on the last wave we invented are hating on this new wave. People never change.
- [06:38]
Marc Andreessen
Okay, got it. Okay, so you're typing English. I want to sell crepes online. I want to do this, I want to have a T shirt, whatever the business is. Okay, what happens? Yeah.
- [06:44]
Amjad Masad
And then a replit agent will show you what IT understood. So it's trying to build a common understanding between you and it. And I think there's a lot of things we can do better there in terms of ui. But for now I'll show you a list of tasks. I'll tell you, I'm going to go set up a database because you need to store your data somewhere. We need to set up Shopify or Stripe because we need to accept payments. And then it shows you this list and gives you two options Initially, do you want to start with a design so that we can iterate back and forth to get locked design down or do you want to build a full thing? Hey, if you want to build a full thing, we'll go for 20, 30, 40 minutes and the agent will tell you here, install the app. I'm going to go set up the database, do the migrations, write the SQL, build the site. I'm going to also test it. So this is a recent innovation we did with Agent three is that after it writes, the software spins up a browser, goes around and tests in the browser and then any issue it kind of iterates, kind of goes and fix the code. So I'll spend 20, 30 minutes building that. I'll send you a notification, I'll tell you the app is ready so you can test it on your phone. You go back to your computer, you'll see, maybe you'll find a bug or an issue. You'll describe it to the agent and say, hey, it's not exactly doing what I expected. Or if it's perfect, you're ready to go and that's it. By the way, there's a lot of examples where people just get their idea in 20, 30 minutes, which is amazing. You just hit publish. You hit publish, couple clicks, you'll be up in the cloud. We'll set up a virtual machine in the cloud. The database is deployed, everything's done, and now you have a production database. So think about the steps needed just two or three years ago in order to get to that stuff. You have to set up your local development environment, you have to sign up for an AWS account, you have to provision the databases, the virtual machines, you have to create the entire deployment pipeline. All of that is done for you. And it just. A kid can do it, a layered person can do it. If you're a programmer and you're curious about what the agent did. The cool thing about replit, because we have this history of being an ide, you can peel the layers, you can open the file tree and you look at the files. You can open Git, you can push to GitHub, you can connect it to your editor. If you want, you can open it in Emacs. So the cool thing about replit, yes, it is a vibe coding platform that abstracts away all the complexities, but all the layers out there for you to look at.
- [08:57]
Marc Andreessen
That was great. But let's go back to. You said, you say, I've got my idea. You plug it in and it says, it gives you this list of things. And then when you describe it, you said, I'm going to do this, I'm.
- [09:03]
Amjad Masad
Going to do that.
- [09:03]
Marc Andreessen
The either in that case was the agent as opposed to the user.
- [09:07]
Amjad Masad
Yes.
- [09:07]
Marc Andreessen
And so the agent lists the set of things that it's going to do and then the agent actually does those things.
- [09:11]
Amjad Masad
Agent does those things. Okay, yeah, that's a very important point. When we did this shift, we hadn't realized internally at replit how much the actual user stopped being the human user and it's actually the. The agent programmer.
- [09:23]
Marc Andreessen
Right.
- [09:24]
Amjad Masad
So one really funny thing happened is we had servers in Asia and the reason we had servers in Asia because we wanted our Indian or Japanese users to have a shorter time to the servers. When we launched the agent, their experience got significantly worse. And we're like, what happened? Like it's supposed to be faster. Well, turns out it's worse. It's because the AIs are sitting in the United States and so the programmer is actually in the United States.
- [09:47]
Marc Andreessen
It's.
- [09:47]
Amjad Masad
You're sending the request to the programmer and the Programmer is interfacing with the machine across the world. And so, yes, suddenly the agent is programmer.
- [09:55]
Marc Andreessen
Okay, so the new terminology agent is a software program that is basically using the rest of the system as if it were a. As if it were a human user, but it's not. It's a bot.
- [10:04]
Amjad Masad
That's right. It has access to tools such as write a file, edit a file, delete a file, search the package index, install a package provision, a database provision, object object storage. It is a programmer that has the tools and interface. It has sort of an interface that, that is very similar to a human programmer.
- [10:23]
Marc Andreessen
And then we'll talk more about how this all works. But a debate inside the AI industry with kind of this idea now of having agents that do things on your behalf and then go out and kind of accomplish missions. There's this kind of debate which is, okay, how obviously it's a big deal even to have an AI agent that can do relatively simple things. To do complex things, of course, is one of the great technical challenges of the last 80 years to do that. And then there's sort of this question of like, can the agent go out and run and operate on its own for five minutes, for 15 minutes, minutes, for an hour, for eight hours? And meaning sort of like, how long does it maintain coherence? How long does it actually stay in full control of its faculties and not kind of spin out? Because at least the early agents or the early AIs, if you set them off to do this, they might be able to run for two or three minutes and then they would start to get confused and go down rabbit holes and kind of spin out. More recently, we've seen that agents can run a lot longer and do more complex tasks. Where are we on the curve of agents being able to run for how long and for what complex tasks before they break?
- [11:26]
Amjad Masad
That's absolutely, I think the main metric we're looking at, even back in 2023, had the idea for software agents four or five years ago. Now the problem, every time we attempt them, the problem of coherence, they'll go on for a minute or two and then they'll just compound in errors in a way that they just can't recover.
- [11:47]
Marc Andreessen
And you can actually see it, right? Because if you watch them operate, they get increasingly confused and then maybe even deranged.
- [11:54]
Amjad Masad
Yeah, they vary deranged and they go into a weird areas and sometimes they start speaking Chinese and doing really weird things. But I would say sometime around last year, we maybe crossed a 3, 4, 5 minute mark. And it fell to us that, okay, we're on a path where long, you know, long horizon reasoning is getting solved. And so we may. We made a bet.
- [12:20]
Marc Andreessen
And I tell my team, so long horizon reasoning, meaning. Reasoning meaning, like dealing in, like facts and logic in a sort of complex way. And the long horizon being over a long period of time. Yes. With many, many steps to a reasoning process.
- [12:34]
Amjad Masad
Yeah, that's right. So if you think about the way large language models work is that they have a context. This context is basically the memory, all the text, all your prompt, and also all the internal talk that the AI is doing as it's reasoning. So when the AI is reasoning, it's actually talking to itself. It's like, oh, now I need to go set up a database. Well, what kind of tool do I have? Oh, there's a tool here that says postgres. Okay, let me try using that. Okay, I used that. I got feedback. Let me look at the feedback and read it. And it'll read the feedback. And so that prompt box or context is where both the user input, the environment input, and the internal thoughts of the machine are all within. It's sort of like a program memory in memory space. And so reasoning over that was the challenge for a long time. That's when AIs just like went off track. And now they're able to kind of think through this entire thing and maintain coherence. And there's now techniques around compression of contexts. So there's still the context length is still a problem. Right. So I would say LLMs today, you know, they're marketed as a million token length, which is like a million words almost. In reality, it's about 200,000. And then they start to struggle. So we do a lot of, you know, we sop, we compress the memory. So if a memory, if a portion of the memory is saying that I'm getting all the logs from the database. You can summarize paragraphs of logs with one statement or the database setup. That's it. Right. And so every once in a while, we'll compress the context so that we make sure we maintain coherence. So there's a lot of innovation happened outside of the foundation models as well in order to enable that long context coherence.
- [14:19]
Marc Andreessen
And what was the key technical breakthrough in the foundation models that made this possible, do you think?
- [14:24]
Amjad Masad
I think it's rl. I think it's reinforcement learning. The way pre training works is pre training is the first step of training a large language model. It reads a piece of text. It covers the last words and tries to guess it. That's how it's trained. That doesn't really imply long context reasoning. It turns out to be very, very effective. It can learn language that way. But the reason we weren't able to move past that limitation is that that modality of just wasn't good enough. And what you want is you want a type of problem solving over long context. So what reinforcement learning, especially from code execution, gave us is the ability for the LLM to roll out what we call trajectories in AI. So trajectory is a step by step reasoning chain in order to reach a solution. So the way as I understand reinforcement learning works is they put the LLM in a programming environment like Replit and say, hey, here's a code base, here's a bug in the code base and we want you to solve it. Now the human trainer already knows what the solution would look like. So we have a pull request that we have on GitHub so we know exactly or we have a unit test that we can run and verify the solution. So what it does is it rolls out a lot of different trajectories that those they sample the model and maybe one of those trajectories will reach and a lot of them will just go off track, but one of them will reach the solution by solving the bug and it reinforces on that. So that gets a reward and the model gets trained that, okay, this is how you solve these type of problems. So that's how we were able to extend these reasoning chains.
- [16:17]
Marc Andreessen
Got it. And how a two part question is how good are the models now at long reasoning? I would say. And how do we know is that established?
- [16:31]
Amjad Masad
There is a nonprofit called Meter that is measuring useful has a benchmark to measure how long a model runs while maintaining coherence and doing useful things, Whether it's programming or other benchmark tasks that they've done. And they put up a paper I think late last year that said every seven months the minutes that a model can run is doubling. So you go from two minutes to, you know, four minutes in seven months. I think they vastly underestimated that.
- [17:08]
Marc Andreessen
Is that right? Vastly. It's doubling more often than seven months.
- [17:11]
Amjad Masad
We so Agent three, we measure that, you know, very closely and we measure that in real tasks from real users. So we're not doing benchmarking, we're actually doing AB tests and we're looking at the data that how users are successful or not. For us, the data, the absolute sign of success is you made an app. And you published it, because when you publish it, you're paying extra money. You're saying, this app is economically useful, I'm going to publish it. So that's as clear cut as possible. And so what we're seeing is in agent one, the agent can run for two minutes and then perhaps struggle. Agent two came out in February, it ran for 20 minutes, agent three, 200 minutes. Some users are pushing it to like 12 hours and things like that. I'm less confident that it is as good when it goes to these stratospheres, but at like 2, 3 hours timeline, it is really, it's insanely good. And the main innovation outside of the models is a verification loop. Actually, I remember reading a research paper from Nvidia. So what Nvidia did is they're trying to write GPU kernels using Deep SEQ. And that was like perhaps seven months ago ago when DeepSEEK came out. And what they found is that if we add a verify in the loop, if we can run the kernel and verify it's working, we're able to run deepseek for like 20 minutes. And it was generating actually optimized kernels. And so it was like, okay. The next thing for us, obviously, as an, as a sort of a agent lab or like app lay, our company, we're not doing the foundation model stuff, but we're doing a lot of research on top of that. And so, okay, we know that agents can run for 10, 20 minutes now or LLMs can stay coherent for longer, but for you to push them to 200, 300 minutes, you need a verifier in the loop. So that's why we spend all our time creating scaffolds to make it so that the agent can spin up a browser and do computer use style testing. So once you put that in the middle, what's happening is it works for 20 minutes. Another agent spins up a browser, tests the work of the previous agent. So it's multi agent system and.
- [19:26]
Marc Andreessen
If.
- [19:26]
Amjad Masad
It founds a bug, it starts a new trajectory and says, okay, good work. Let's summarize what you did the last 20 minutes. Now that plus what the bug that we found, that's a prompt for a new trajectory. So you stack those on each other and you can go endlessly.
- [19:41]
Marc Andreessen
So it's like setting up a marathon or like a relay race. As long as each step is done properly, you could do in sort of an infinite number of steps.
- [19:48]
Amjad Masad
That's right, that's right. You can always go across the previous step into a paragraph and that becomes a prompt. So it's. It's an agent prompting the next agent. Right, right, right.
- [19:55]
Marc Andreessen
That's amazing. So, and then when, when an agent, like when a modern agent, like running on Modern, Modern LLMs that are trained this way, when it. Let's say it runs for 200 minutes. Like, when you watch the agent run, is it like running. Is it like processing through, like, logic and tasks at the same pace that, like a human being is? Or slower or faster?
- [20:12]
Amjad Masad
It's actually, I would say it is faster, but not that much. Significantly faster. It's not at computer speed. Right. What we expect computer speed to be.
- [20:22]
Marc Andreessen
It's like watching a per. Like if you watch the. If you. If it's describing what it's doing, it's sort of like watching a person work.
- [20:26]
Amjad Masad
It's like watching John Carmack on cocaine work the world.
- [20:30]
Marc Andreessen
Okay. The world's the world's best programmer. Yeah. The world's best programmer. On a stimuli. On a stimulant. On a stimulant. Yeah, that's right. Working for you. Yeah.
- [20:40]
Amjad Masad
So it's very fast and you can see the file diffs running through, but every once in a while it'll stop and it'll start thinking. It'll show you the reasoning. It's like, oh, I did this and I did this. Am I on the right track? It kind of really tries to reflect, and then it might review its work and decide the next step, or it might kick into the testing agent or, you know, so you're seeing it do all of that, and every once in a while it calls a tool, for example, it stops and says, well, we ran into an issue. You know, Postgres 15 is not compatible with this database orm package that I have. Okay, this is a problem I haven't seen before. I'm going to go search the web. So it has a web search tool. Go do that. And so it looks like a human programmer. And it's really fascinating to watch. It's one of my favorite things to do is just to watch the tool chain and reasoning chain and the testing chain. Yeah. It is like watching a hyperproductive programmer.
- [21:38]
Marc Andreessen
Right. So we're kind of getting into here kind of the holy grail of AI, which is sort of generalized reasoning by the machine. So you mentioned this a couple times with this idea of verification. So just for folks listening to podcasts who maybe aren't in the details, let me try to describe this and see if I have it right. So just a large language model. The way you would have experienced with like chatgpt out of the gate two years ago or whatever would have been. It's like, it's incredible how fluid it is at language. It's incredible how good it is at like writing Shakespearean sonnets or rap lyrics. It's amazing how good it is a human conversation. But if you start to ask it like problems that involve like rational thinking or problem solving, all of a sudden like you'd. Or the math the whole show. And in the very beginning it was. You could ask. If you asked a very basic math problem so you know, it would, it would not be able to do them.
- [22:25]
Amjad Masad
That's right.
- [22:25]
Marc Andreessen
But then even when it got better at those, if you started to ask it to like, you know, it could maybe add two small numbers together, but it couldn't add two large numbers together. Or if it could add two large numbers, it couldn't multiply them. And it's just like, all right, this is true. And then it had this. There was this famous. The famous. Was the strawberry. The strawberry test. The famous strawberry test, which is how many Rs are in the word strawberry. That's right. And there was this long period where it would just guess wrong and it would say there are only two Rs in the word strawberry. And it turns out there are three. So it was this thing. And so people were. And there was even this term that was being used kind of the slur that was being used at the time was stochastic parrot.
- [22:58]
Amjad Masad
I was thinking clanker.
- [22:59]
Marc Andreessen
Well, clanker is the new slur. Clanker is just the full on racial slur. I guess AI is a species. But the technical critique was so called stochastic parrot. Stochastic means random. So sort of random parrot meaning basically this thing was sort of large language models were like a. They were like a mirage where they were like repeating back to you things that they thought that you wanted to hear but they didn't understand.
- [23:21]
Amjad Masad
And in a way it's true in the pure pre training LLM world.
- [23:24]
Marc Andreessen
Right. For the very basic layer. But then what happened is as you said over the last year or something, there was this layering in of reinforcement learning and then. But the key to it's not new.
- [23:33]
Amjad Masad
Crucially it's like it's alphago. Right.
- [23:36]
Marc Andreessen
So describe that for a second.
- [23:37]
Amjad Masad
Yeah. So we had this breakthrough before in 2015 was the AlphaGo breakthrough. I think 2015, 2016, where it is emerging of sort of, you would know a lot better than me. The old AI debate between the connectionists the people who think neural networks are the true sort of way of doing AI and the symbolic systems, I think, or like the people that think that discrete reasonings, F statements and knowledge bases, whatever, this is the way to go. And so there was a merging of these two worlds where the way AlphaGo worked is it had a neural network, but it had a Monte Carlo tree search algorithm on top of that. So the new neural network would generate a list of potential moves and then you had a more discrete algorithm sort those moves and find the best based on just tree search, based on just trying to verify. Again, this is sort of a verifier in the loop trying to verify which move might yield the best based on more classical way of doing algorithms. And so that's a resurgence of that movement where we have this amazing generative neural network that is the LLM. And now let's layer on more discrete ways of trying to verify whether it's doing the right thing or not. And let's put that in a training loop. And once you do that, the LLM will start gaining new capabilities such as reasoning over math and code and things like that.
- [25:06]
Marc Andreessen
Exactly right. Okay, and then that's great. And then the key thing there though, for RL to work, for LLMs to reason, the key is that it be a problem statement that there is a defined and verifiable answer.
- [25:17]
Amjad Masad
That's right.
- [25:18]
Marc Andreessen
Is that right? And you might think about this as like, give a bunch of examples. Like in medicine, this might be like, you know, a diagnosis that like a panel of human doctors agrees with or, or, or by the way, or a diagnosis that actually, you know, solves the condition. In law, this would be a, you know, an argument that in front of a jury actually results in an acquittal or, or something like that. In math, it's an equation that actually solves properly. In physics, it's a result that actually works in the real world. I don't know. In civil engineering, it's a bridge that doesn't collapse. Right. So, so, so there, there, there's always some, some test.
- [25:51]
Amjad Masad
The first two do not work very well just yet. I would say law and healthcare, they're still a little too squishy, a little too soft. It's unlike math or code. The way they're training on math, they're using this sort of like a program language, provable language called Lean for proofs. Right. So you can run a Lean statement, you can run a computer code, perhaps you can run a physics simulation or civil engineering sort of physics simulation, but you can't run a diagnosis. Okay, so I would say that.
- [26:25]
Marc Andreessen
But you could verify it with human answers or not.
- [26:27]
Amjad Masad
Yeah, so that's a more HF in a way. So it is not the like sort of autonomous RL trained, like fully scalable autonomous, which is why coding is moving faster than any other domain is because we can generate these problems and verify them on the fly.
- [26:46]
Marc Andreessen
But there's two with coding, as anybody who's coded knows there's coding. There's two tests, which is one is does a code compile?
- [26:50]
Amjad Masad
Right.
- [26:51]
Marc Andreessen
And then the other is does it produce the right output? And just because it compiles doesn't mean it produces the right output. And you tell me. But verifying that it's the correct output is harder.
- [26:58]
Amjad Masad
Yeah. So SUI bench is a collection of verified pull request end states. So it is not just about compiling. So a group of scientists. So swedbench is the main benchmark used to test whether AI is good at software engineering tasks. And, and we're almost saturating that. So last year we're at like maybe 5%, early 24 or less and now we're like 82% or something like that. With cloths on at 4.5, that's state of the art. And that's like a really nice hell climb that's happening right now. And basically they went and looked on GitHub. They found the most complex repositories, they found bug statements that are very clear and they found proquests that actually solve those bug statements with unit tests and everything. So there is an existing corpus on GitHub of tasks that the AIs can solve and you can also generate them. Those are not too hard to generate what's called synthetic data. But you're right, it's not infinitely scalable because some human verifiers still need to kind of look at the task. But maybe the foundation models have found a way to have the synthetic training go all the way.
- [28:19]
Marc Andreessen
Right. And then what's happening, I think, I think because what's happening is the foundation model companies are, in some cases they are hiring. They're actually hiring human experts to generate new training data.
- [28:26]
Amjad Masad
Yes.
- [28:27]
Marc Andreessen
So they're actually hiring mathematicians and physicists and coders to basically sit and you know, they're hiring human programmers. Putting them on the cocaine. Yes. And having them probably coffee and having them actually write code and then, and then write code in a way where there's a known result of the code running such that the, this RL loop can be trained properly.
- [28:46]
Amjad Masad
That's right.
- [28:46]
Marc Andreessen
And then the other, the other. And then the other thing these companies are doing is you said they're building, building systems where the software itself generates the training data, generates the tests, generates the validated results. And that's so called synthetic training data.
- [28:59]
Amjad Masad
That's right. But again those work in the very hard domains. It works to some extent in the software domains and I think there's some transfer learning. You can see the reasoning work when it comes to tools like deep research and things like that, but we're not making as fast as progress and the, in the more soft domain, so softer.
- [29:20]
Marc Andreessen
Domains meaning like domains in which it's, it's harder, harder or even impossible to actually verify correctness of result in a sort of a deterministic, factual, grounded, non controversial way.
- [29:31]
Amjad Masad
Like if you have a chronic disease, you could, you could have, you know, you have a POTS or you know, whatever, EDS syndrome or. And they're all, they're all clusters and it's. Because it is the domain of abstraction, it is not as concrete as code, code and math and things like that. So I think there's still long ways to go there.
- [29:52]
Marc Andreessen
Right. So sort of the more concrete the problem, like it's the concreteness of the problem that is the key variable, not the difficulty of the problem. Would that be a way to think about it?
- [29:59]
Amjad Masad
Yeah, I think the concreteness in a sense of can you get a true or false verifiable output?
- [30:05]
Marc Andreessen
Right. But like in any domain, in any domain of human effort in which there's a verifiable answer, we should expect extremely rapid progress. Yes.
- [30:11]
Amjad Masad
Right, okay, absolutely. And I think that's what we're seeing.
- [30:13]
Marc Andreessen
Right. And that for sure includes math. That for sure includes physics, for sure includes chemistry. For sure includes large areas of code. Code.
- [30:19]
Amjad Masad
That's right, right.
- [30:20]
Marc Andreessen
What else does that include, do you think?
- [30:21]
Amjad Masad
Bio, like we're seeing with a protein like genomic.
- [30:24]
Marc Andreessen
Yeah, okay.
- [30:25]
Amjad Masad
Yeah, yeah, things like that. I think some, some areas of robotics.
- [30:30]
Marc Andreessen
Right.
- [30:31]
Amjad Masad
There's a clear outcome.
- [30:33]
Marc Andreessen
Right.
- [30:34]
Amjad Masad
But, but it's not that many. I mean, surprisingly.
- [30:37]
Marc Andreessen
Well, yeah, it depends. Yeah, depends on your point of view. Some people might say that's a lot. So. And then you mentioned that, you mentioned the pace of improvement. So what would you expect from the pace of improvement going forward for this?
- [30:48]
Amjad Masad
I, I think we're, we're ripping on coding. Like, I think, I think it's just, it's going like, I think it's going to be like what we're working on with, with Agent four right now is by, by next year we think you're going to be sitting instead of rep in front of replit and you're shooting off multiple agents at a time. You're like planning a new feature. So I, I want you know, social network on top of my storefront and another one is like hey, refactor the database in your running parallel agents. So you have five, 10 agents kind of working in the background and they're merging the code and taking care of all of that. But you also have a really nice interface on top of that that you're doing design and you're interacting with AI in a more creative way, maybe using visuals and charts and things like that. So there's a multimodal angle of that interaction. So I think, you know, creating software is going to be such an exciting area and, and, and I think that the lay person will be as good as a, what a senior software engineer that works at Google is today. So I think, I think that's happening very soon. But, but you know, I don't see them and be curious about your point of view but like my experience between as, as a sort of a, you know, on the, let's say healthcare side or more, you know, write me an essay side or more creative side haven't seen as much of a rapid improvement as what we're seeing in code. So, so I think I going to go to the moon. Math is probably as well. Some, some you know, scientific domains, bio, things like that, those are, are going to move really fast.
- [32:26]
Marc Andreessen
Yeah. So this is, there's this, there's this weird dynamic. See if you agree with this. And Eric, also curious your point of view on this. Like there's this weird dynamic that we have. We have this in the office here a lot and I also have this with like the leading edge entrepreneurs a lot which is this thing of like, like wow, this is the most amazing technology ever and it's moving really fast and yet we're still like really disappointed and like it's not moving fast enough and like it's like maybe right on the verge of stalling out and like, you know, be hyper excited, but also on the verge of slitting our wrists. The gravy train is coming to an end, right? And I always wonder, it's like on the one hand it's like okay, not all, I don't know, ladders go to the moon. Just because something looks like it works or doesn't mean it's going to be able to, you're going to be able to scale it up and have it work to the fullest extent. So it's important to recognize practical limits and not just extrapolate everything to infinity. On the other hand, we're dealing with magic here that we, I think, probably all would have thought was impossible five years ago or certainly ago.
- [33:19]
Amjad Masad
Ten years.
- [33:19]
Marc Andreessen
Ten years ago. Like, I didn't, you know, look, I, you know, I got my CS degree in the late 80s, early 90s. I never, I didn't think I would live to see any of this, right? Like, this is just amazing that this is actually happening in my lifetime.
- [33:30]
Amjad Masad
But there's a huge bet on AGI, right? Like, whether it's the foundation models. I think now the entire US economy is sort of a bet on AGI. And there are crucial questions to ask, whether are we on track to AGI or not? Because there are some ways that I can tell you it doesn't seem like we're on track to AGI because there doesn't seem to be transfer learning across these domains that are, that are, you know, significance, right? So if we get a lot better at code, we're not immediately getting better at, like, generalized reasoning. We need to go also, you know, get training data and create RL environment for bio or chemistry or physics or math or law. And this has been the sort of point of discussion now in the AI community after the Dwarkesh and Richard Sutton interview, where, you know, Richard Sutton kind of poured this cold water on the Bitter Lesson. So everyone was using this essay that he wrote called the Bitter Lesson. The idea is that there are infinitely scalable ways of doing AI research. And anytime you can pour more compute and more data and get more performance out, you're just, that's the ultimate way of getting to AGI.
- [34:47]
Marc Andreessen
And.
- [34:48]
Amjad Masad
And some people interpreted that interview that perhaps he's doubtful that we're even on a better lesson path here. And perhaps the current training regime is actually very much the opposite, in which we are so dependent on human data and human annotation and all of that stuff. So I think I agree with you. I mean, as a company, we're excited about where things are headed, but there's a question of, like, are we on track to AGI or not? And be curious what you think so.
- [35:22]
Marc Andreessen
And you know, Ilya, I think, you know, Ilya Sensker makes a specific form of this argument which is basically like, we're just literally running out of training data.
- [35:28]
Amjad Masad
It's the fossil fuel argument, right?
- [35:30]
Marc Andreessen
Like, we've slurped all the training data. Fundamentally, we've slurped all the data off the Internet, that is where almost all the data is at this point. There's a little bit more data that's in like, you know, private dark pool somewhere that we're going to go get, but we have it all. And then. Right. We're in this business now trying to generate new data. But generating new data is hard and expensive, you know, compared to just like slurping things off the Internet. So there are these arguments, you know, having said that, you know, you get into definitional questions here really quick which are kind of a rabbit hole. But having said that, like you mentioned transfer learning. So transfer learning is the ability of the machine to.
- [35:56]
Amjad Masad
Right.
- [35:56]
Marc Andreessen
To be an expert in one domain and then, and then generalize that into another domain. My answer to that is like have you met people? And how many people do you know are able to do transfer learning?
- [36:05]
Amjad Masad
Not many.
- [36:06]
Marc Andreessen
Not many. Right.
- [36:07]
Amjad Masad
Well, because they're, because there's quite the opposite actually. The nerdier. They are in a certain domain that kind of, you know, often they have blind spots. We joke about how everyone's just retarded in one area or they make some like massive mistake and don't trust them on this. But another topic, you know.
- [36:21]
Marc Andreessen
Right.
- [36:22]
Amjad Masad
Yeah.
- [36:22]
Marc Andreessen
Well and this is a well known thing among like for example, public intellect. So this happens. There's actually been whole books written about this on so called public intellectual. So you get these people who show up on TV and they're experts and what happens is they're like an expert in economics. Right. And then they show up on TV and they talk about politics and they don't know anything about politics. Right. Or they don't know anything about like medicine or they don't know anything about the law or they don't know anything about computers. You know, this is the Paul Gregman talking about how the Internet is going to be no more significant than the fax machine Facts. Yeah, He's a brilliant economist. He has no idea what a computer Is he a brilliant economist.
- [36:48]
Amjad Masad
Economist.
- [36:49]
Marc Andreessen
Well, at one point. At one point. At one point. Even if, even if he's a brilliant. Well, this is the thing. Like what does that mean? Should a brilliant economist be able to extrapolate the Internet is a good question. But the point being, even if he is a. Or take anybody. Oh by the way, or like Einstein's actually my favorite example, I think you'd agree. Einstein was a brilliant physicist. He was a Stalinist. Yeah. He was a socialist and he was a Stalinist and he was like. Well, he thought Stalin was fantastic.
- [37:19]
Amjad Masad
Jury's out. So.
- [37:21]
Marc Andreessen
Yeah, okay, all right.
- [37:22]
Amjad Masad
True socialism.
- [37:23]
Marc Andreessen
All right, all right, Einstein. You know, I'll take your word for it. But, like, once he got into politics, he was just like, totally loopy or, you know, or even right or wrong. It's just. He just sounded like all of a sudden like an undergraduate lunatic, like somebody in a dorm room. Like, he. There was no transfer learning from physics into politics. Like, he. Right or wrong. He didn't. There was no. There was clearly. There was nothing new in his political analysis. It was the same rote, routine bullshit you get out of, you know.
- [37:47]
Amjad Masad
Yeah. So in a way, the argument you're making is like, we already a human level AI. I mean, perhaps the definition of AGI is something totally different. It's like above a human level. That. Something that totally generalizes across domains. It's not something that we've seen.
- [38:02]
Marc Andreessen
Yeah, like, we've. Ideal. Yeah. I was saying. So we've, we've. And you know, look, we should. We should shoot big. But we've idealized a. We've idealized a goal that may be idealized in a way that, like, number one, it's just. It's like so far beyond what people can do that it's no longer a relevant comparison to people. And usually AGI is defined as, you know, able to do everything better than a person.
- [38:20]
Amjad Masad
Person can.
- [38:20]
Marc Andreessen
And it's like, well, okay, so if doing everything better than a person can, it's like, if a person can't do any transfer learning at all, doing even a little bit, a marginal bit might actually be better, or it might not matter just because no human can do it. And so therefore you just stack up the domains. There's also this well known phenomenon in AI. Typically this works the other way, which is a phenomenon AI engineers always complain about and scientists always complain about, which is the definition of AI is always the next thing that the machine can't do. And so the definition of AI for a long time was like, can it beat humans at chess? And then the minute it could beat humans at chess, that was no longer AI. That was just like, oh, that's just like boring.
- [38:54]
Amjad Masad
That's computer chess. It became AI.
- [38:55]
Marc Andreessen
It's computer chess. It's just like, boring. And now it's an app on your iPhone. And nobody. And nobody and nobody cares. Right? And it's immediately.
- [39:00]
Amjad Masad
The Turing test was the next.
- [39:02]
Marc Andreessen
The Turing Test.
- [39:02]
Amjad Masad
And then we passed it and nobody.
- [39:03]
Marc Andreessen
We blew. This is a really big deal.
- [39:05]
Amjad Masad
There was no celebration.
- [39:06]
Marc Andreessen
There was no parties. That's exactly right. There was no party for 80 years. The Turing Test. I mean, they made a movie about it. Like the whole thing, that was the thing. And like we blew right through it and nobody even registered it, nobody cares. It gets no credit for it. We're just like, ah, it's still, you know, complete piece of shit. That's right. Right. And so there's this thing where. So the AI scientists are used to compl, complaining basically, that they're always being judged against the next thing as opposed to all the things they've already solved. But that's maybe the other side of it, which is they're also putting out for themselves an unreasonable goal. An unreasonable goal. And then doing this sort of self flagellation kind of along the way. And I kind of wonder, Yeah, I wonder kind of which way that cuts.
- [39:42]
Amjad Masad
Yeah, yeah. It's an interesting question. Like, I started thinking about this idea of like, it doesn't matter whether it's truly AGI. And the way I define AGI is that you put an A AI system in any environment and efficiently learns. It doesn't have to have that much prior knowledge in order to kind of learn something, but also can transfer that knowledge across different domains. But we can get to functional AGI and what functional AGI is. Just collect data on every useful economic activity in the world today and train an LLM on top of that or train the same foundation model on top of that. And we'll go, we'll target every sector economy and you can automate a big part of labor that way. So I think, yeah, I think we're on that track for sure.
- [40:32]
Host
You tweeted after GPT5 came out that you were feeling the diminishing returns. What were you expecting and what needs to be done? Do we need another breakthrough to get back to the pace of growth or.
- [40:41]
Amjad Masad
What are your thoughts there? I mean, this whole discussion is sort of about that. And my feeling is that GPT5 got good at verifiable domains. It didn't feel that much better at anything else. The more human angle of it. I felt like it regressed and like you had this sort of Reddit pitchfork sort of movement against, against Sam and OpenAI because they felt like they lost a friend. GPT4O felt a lot more human and closer, whereas GPT5 felt a lot more robotic, you know, very in its head, kind of trying to think through, through everything. And, and so I, I, I would have just expected like when we went from, from GPT 2 to 3, it was clear I was getting a lot more Human. It was a lot closer to our experience. You can feel like it's actually, oh, it gets me. Like there's something about it that understands the world better. Similarly, 3 to 4, 4 to 5 didn't feel like it was a better overall being, as it were.
- [41:47]
Marc Andreessen
But is that, is that, is that, is that a, is the question there? Like, is it emotionality?
- [41:53]
Amjad Masad
Is it partly emotionality, but, but again partly, like I like to ask models, like very controversial things, can it reason through? I don't know how deep we want to go here, but like what happened with World Trade 7?
- [42:11]
Marc Andreessen
Right, right, sure.
- [42:13]
Amjad Masad
It's an interesting question, right? Like I'm not, I'm not putting out a theory, but like it's interesting, like how did it, you know, and can it, can it think through controversial questions in the same way that it can go think through a coding problem and there hasn't been any movement there, like all the reasoning and all of that stuff haven't said. And not just that, you know, that's a cute example, but like Covid, right? Like, you know, the origins of COVID right? You know, go, you know, dig up GPT4 or other models and go to GPT5. You're not going to find that much difference of, okay, let's reason together. Let's try to figure out what was the original of COVID because it's still an unanswered question, you know, and I don't see them making progress on that. I mean, you play a lot with them.
- [42:57]
Marc Andreessen
Do you feel like I use it differently? I don't know, maybe I have different expectations. I, I'm, I, the way I, my main use case actually is sort of, sort of PhD and everything at my beck and call. And so I'm trying to get it to explain things to me more than I'm trying to like, you know, have conversations with it. Maybe, maybe I'm just unusual with that.
- [43:13]
Amjad Masad
But, and that, that gets better.
- [43:14]
Marc Andreessen
Well, so what I, what I, what I found specifically is a combination of like GPT 5 Pro plus deep reasoning or like Grok 4 heavy, like, you know, the highest end like that, you know, they now basically generate 30 to 40 page, you know, essentially books on demand on any topic. And so anytime I get curious about something, you just take it. Maybe it's my version of it, but it's something like, I don't know, here's a good example. When an advanced economy puts a tariff on a raw material or on a finished good, like who pays? You know, is it the consumer? Is it the Importer, Is it the exporter or is it the producer? And this is actually a very complicated, it turns out, very complicated questions. And it's a big, big, big thing that economists study a lot. And it's just like, okay, you know, who pays? And what I found like for that kind of thing is it's outstanding.
- [43:57]
Amjad Masad
Well, but it's outstanding at sort of going out of the web, getting information, synthesizing it.
- [44:02]
Marc Andreessen
Correct. It gives me a synthesized 20, 30, 40 page. It basically tops out of 40 pages of PDF. Yeah, but I can get it, I get up to 40 pages of PDF, but it's a completely coherent and as far as I can tell for everything I've cross checked a completely like, like world class. Like if I hired, you know, for a question like that, if I hired like a great, you know, econ postdoc at Stanford who just like went out and did that work, like it would maybe be that good. Yeah, but, but then of course the significance is, it's like, it's like, you know, at least for, it's, this is true for many domains, you know, kind of PhD and everything.
- [44:33]
Amjad Masad
And so, but, but this is synthesizing knowledge, not trying to create new knowledge.
- [44:36]
Marc Andreessen
Well, but this, this, this gets to the sort of, you know, of course you get into the angels dancing on the head of a pin thing, which is like, what, what, you know, what's the difference how many, how much new knowledge ever actually is there anyway? What do you actually expect from people when you ask them questions? And so what, what I'm looking for is like, yes, explain this to me in like the, the, the most complex, most like complete way that it's possible for somebody to, you know, for a real expert to be able to explain things to me. And that's what I use it for. And again, as far as I can tell from the cross checking, like I'm getting, you know, like almost like basically 100 out of 100. Like I don't even think I've had an issue in months where it's like for sure had a problem in it. Yeah, and it's like, yeah, you can say, yeah, it's synthesizing as opposed to creating new information, but like it's generating a 40 page, it's basically generating a 40 page book.
- [45:18]
Amjad Masad
That's amazing.
- [45:19]
Marc Andreessen
That's like incredibly like fluid. It's, you know, it's, you know, the logical coherence of the entire, like it's, it's a great writing. Like if you evaluated a human author on it, you would say, wow, That's a great author. Are people who write books creating new knowledge? Well, yeah, well, sort of. Not because a lot of what they're doing is building on everything that came before them and synthesizing. But also a book is a creative accomplishment. Right. And so, yeah, one of the things.
- [45:46]
Amjad Masad
I'm interested in, I'm hoping AI could help us solve is just like how confusing the information ecosystem right now. Everything feels like propaganda. It doesn't feel like you're getting read on information from anywhere. So I really want an AI that could help me reason from first principles about what's happening in the world for me to actually get real information. And maybe that's an unreasonable sort of ask of the AI researchers, but I don't think we have made any progress there.
- [46:14]
Marc Andreessen
So maybe I'm over being in my line of work. Maybe I'm over focused on arguing with people as opposed to trying to get. As a trust, trying to get the underlying truths. But. Well, here's the thing I do a lot with this is I just say take a provocative point of view and then steel man, the position. Take your Covid thing. Steel man. So I often, I often pair these Steel man, the position that it was a lab leak and the steel man, the position that it was natural origins. And. And again, like, is this creativity or not? I don't know. But like, what comes back is like 30 pages each of like, wow. Like, that is like the most compelling case in the world I can imagine with like every, you know, everything marshaled against it. Like, the argument structured in like the most possible.
- [46:48]
Amjad Masad
Part of the reason that started happening is because it stopped being taboo to talk about a human origin when it was taboo.
- [46:54]
Marc Andreessen
Yes.
- [46:54]
Amjad Masad
The AIs would talk down to you. It's like, oh, you're a conspiracy theorist. And so there's a period of time so to take something truly controversial and they can't reason about it because of all RLHF and onsets, all the limitations.
- [47:11]
Marc Andreessen
And as you know, I won't pick specific ones here, but there are certain big models that will still lecture you that you're a bad person for asking that question. But some of them are just really, really open now to being able to do these things. And. And then. Yeah, so, okay. Yeah, so. Okay, so, yeah, so there's this. Yeah. So basically, like, ultimately what you're looking for, like the ultimate thing would be if there's something that's like. I don't think anybody's really defined this really well, because it's not. Because again, it's like conventional. All the conventional definitions of AGI are like basically comparing to people.
- [47:43]
Amjad Masad
Yeah.
- [47:44]
Marc Andreessen
And there it's always like, you know, the conventional explanations of AGI always for me strike me a lot like the debate around like whether a self driving car works or not. Which is does a self driving car work, become it's a perfect driver or does it work because it is better than the human driver? And better than the human driver I think is actually quite just like with the chess thing and the go thing. I actually think that that's like a real thing. And then there's the like is it a perfect driver? Which is what obviously the self driving car companies are working for. But then I think you're looking for something beyond the perfect driver. You're looking for the car who knows where to go.
- [48:18]
Amjad Masad
So I'm of two minds, right. So one mind is the sort of practical on entrepreneur.
- [48:23]
Marc Andreessen
Right.
- [48:24]
Amjad Masad
And I just, I have so many toys to play with to build like stop AI progress today and repl will continue to get better for the next five years. Like we have so much we could do just on the app app layer and the infrastructure layer. So you know, I, but, but I think that will, you know, the foundation models will continue to get better as well. And so it's, it's in a very exciting time in our industry. The other mind is more academic because as a kid I've always been interested in the nature of consciousness, nature of intelligence. You was always interested in and reading the literature there and I would point to the RL literature. So Richard Sodden, there's another guy I think co founder of DeepMind, Shane Lagg wrote, wrote a paper trying to define what AGI is. And there I think that the definition of AGI I think is the original, perhaps correct one, which is efficient continual learning. If you truly want to build an artificial general intelligence that you can drop in any domain, you can drop in a that much prior knowledge about cars and within, you know, how long does it take a human to learn how to drive? Within months be able to drive a.
- [49:36]
Marc Andreessen
Car really well, sort of generalized skill acquisition, generalized understanding acquisition, generalized reasoning acquisition.
- [49:44]
Amjad Masad
And I think that's the thing that will like truly change the world. That's the thing that would give us a better understanding of the human mind, of human consciousness. And that's the thing that will propel us to the next level of human civilization, I think. So on a civilization level I think that's a really deep question. But separate from the economy and the industry, which is all exciting but there's an academic aspect of it that I'm really so.
- [50:11]
Marc Andreessen
And what odds. If we're on Cal State today, what odds do we place on that?
- [50:15]
Amjad Masad
I'm kind of bearish on true AGI breakthrough because what we built is so useful and economically valuable. So in a way, good enough.
- [50:28]
Marc Andreessen
Good enough is the enemy.
- [50:30]
Amjad Masad
Yeah, yeah. Do you remember that essay, Worse is better. Worse is better.
- [50:34]
Marc Andreessen
Worse is better. Worse is better. So there's like a local. There's like a trap. There's like a local maximum trap.
- [50:39]
Amjad Masad
We're in a local maximum trap.
- [50:40]
Marc Andreessen
Local maximum trap where it's because. Because it's good enough for so much economically productive work. Yes. It relieves the pressure in the system to create the generalized answer.
- [50:49]
Amjad Masad
Yes. And then you have the weirdos like Rich Sutton and others that are still trying to go that. Down that path and maybe they'll succeed. But there's enormous optimization energy behind the current thing that we're hell climbing on this like, local maximum.
- [51:02]
Marc Andreessen
Right, right, right. And the irony of it is everybody's worried about, like, the, you know, the gazillions of dollars going into building out all this stuff. And so the most ironic thing in the world would be if the gazillions of dollars are going into the local maximum.
- [51:12]
Amjad Masad
That's right.
- [51:12]
Marc Andreessen
As opposed to a counterfactual world in which they're going into solving the general problem.
- [51:16]
Amjad Masad
But it's also potentially irrational. Like maybe the general problem is actually, you know, not within our lifetimes. Who knows? How much further do you think. Do you think we squeeze most of the out of LLMs in general then, or are there any other research directions.
- [51:29]
Host
That you're particularly excited about?
- [51:33]
Amjad Masad
Well, that's the thing. I think the problem is there aren't that many. I think the breakthroughs in RL are incredibly exciting, but we also knew about them now for like over 10 years, where you marry generative systems with tree search and things like that. But there's a lot more to go there. And I think, again, the. The original minds behind reinforcement learning are trying to go down that path and try to kind of bootstrap intelligence from scratch. Carmack is going down that path. As far as I understand Carmack, you guys maybe invested, but they're not trying to go down the LLM path. So there are people that are trying to do that, but I'm not seeing a lot of progress or outcome there. But I watch it kind of from far.
- [52:17]
Marc Andreessen
Although for all we know, there's already a bot on X somewhere. Maybe. You never know. It Might not be a big announcement, it might just be one day there's just like a bot, an X that starts winning all the arguments.
- [52:27]
Amjad Masad
Yeah, could be.
- [52:29]
Marc Andreessen
Or a code, a user Reddit and all of a sudden generating incredible software. Okay, let's spend our remaining minutes, let's talk about you. So yeah, take us. Start from the beginning with your life. And how did you get from being born to being in Silicon Valley?
- [52:45]
Amjad Masad
Okay, in two minutes. Yeah, I'm joking, but yeah, I got introduced to computers very, very early on. And so for what? And so I was born in Amman, Jordan, and for whatever reason, my, my dad, who was just a government engineer at the time, decided that computers were important and he didn't have a lot of money, took out of debt, bought a computer. It was the first computer in, in our neighborhood. First computer of anyone I know. And I just one of my earliest memories, I was 6 years old just watching my dad unpack this machine and sort of open up this huge manual and kind of finger type CD ls, mkd and like I would, you know, be behind his shoulder and just like watching him, you know, type these commands and seeing the sort of machine kind of respond and do exactly what he's asked it to do.
- [53:41]
Marc Andreessen
Popping Tylenol as your.
- [53:45]
Amjad Masad
Exactly.
- [53:47]
Marc Andreessen
Autism activated.
- [53:48]
Amjad Masad
Of course you have to.
- [53:52]
Marc Andreessen
Exactly what kinds of, what kind of.
- [53:55]
Amjad Masad
Computer was, was an IBM as far as I remember?
- [53:59]
Marc Andreessen
IBM PC.
- [54:00]
Amjad Masad
It was IBM PC.
- [54:01]
Marc Andreessen
So what year was this?
- [54:03]
Amjad Masad
1993.
- [54:04]
Marc Andreessen
1993. Okay, so did it have Windows at that point?
- [54:07]
Amjad Masad
No, it didn't have Windows.
- [54:08]
Marc Andreessen
Right before Windows?
- [54:09]
Amjad Masad
Yeah, right before Windows, but I think Windows had been out. But you would.
- [54:12]
Marc Andreessen
It was an add on.
- [54:13]
Amjad Masad
It was an add on. You wouldn't boot it up. So I think we bought the disk for Windows and you had to kind of bootloaded from the disk and it will open Windows and you can click around. It wasn't that interest because there wasn't a lot on it. So a lot of time I just spent in DOS and writing batch files and opening games and messing around with that. But it wasn't until Visual Basic that I started. So like after Windows 95 that I started making real software. And the first idea I had was I used to be a huge gamer, so I used to go to these lang gaming cafes and play Counter Strike. And I would go there and the whole thing is full of computers, but they don't use any software to run their business. It was just like people running around just like writing down your machine number, how much time you spend on it and how much did you pay? And kind of tapping on your shoulders, like, hey, you need to pay a little more for that. And I asked him, like, why don't you, like, just build a piece of software that allows me to log in and have a time or whatever. I was like, yeah, we don't know how to do that. And I was like, okay, I think I know how to do that. So I spent. I was like 12 or something like that. I spent like two years building that and then went out and tried to sell it and was able to sell it and was making so much money. I remember McDonald's opened in Jordan around the time. When I was 13, 14, I took my entire class at McDonald's. It was very expensive. But I was balling all this money and I was showing off. And so that was the first business that I created. And then when it came to. And at the time I started kind of learning about AI, you know, reading sci fi and all of that stuff. And when it came time to go to college, I didn't want to go to computer science because I felt like coding is on its way to get automated.
- [56:03]
Marc Andreessen
Okay.
- [56:03]
Amjad Masad
I remember using these wizards. Do you remember wizards?
- [56:07]
Marc Andreessen
Yes. So wizards, basically, it's like extremely crude early bots that generate code. Generate code, yeah, yeah.
- [56:13]
Amjad Masad
And I remember you could like, you know, type in a few things, like, here's my project, here's what it does, whatever. And then click, click, click. I would just like, scaffold a lot of code. I was like, oh, I think that's the future. Like, coding is such a.
- [56:23]
Marc Andreessen
It's almost solved.
- [56:24]
Amjad Masad
Yeah, it's solved. You know, why, why should I go into coding? I was, okay, if AI can do the code, what should I do? Well, someone needs to build and maintain the computers. And went to the computer engineering and did that for a while, but then rediscovered my love for programming, reading program essays on Lisp and things like that, and started messing around with Scheme and programming languages like that. But then I found it incredibly difficult to just learn different programming languages. I didn't have a laptop at the time. And so every time I go to wanting to learn Python or Java, I would go to the computer lab, download gigabytes of software, try to set it up, type a little bit of code, try to run it. You know, run into missing DLL issue or. I was like, man, this is so primitive. Like, at the time, it was 2008, something like that. You know, we had Google Docs, we had Gmail, you could like open the browser. And partially thanks to you and be able to kind of use software on, on the Internet. And I thought the web is the ultimate software platform. Like everything should go on the web. Okay, who's building an online development environment? And no one. And it felt like I found like a hundred dollar bill on the, you know, on the floor of Grand Central Station. Like, surely someone should be building this. But no, no one was building this. And so I was like, okay, I'll try to build it. And I got something done in like a couple hours, which was a text box. You type in some JavaScript and there's a, there's a button that says eval. You click eval and EV is it shows you in an alert box. So one plus one two. I was like, oh, I have a programming environment. I showed it to my friends, people started using it. I added a few additional things like saving the program. I was like, okay, all right, there's a real idea here. People love it. And then again, it took me two or three years to actually be able to build anything because the browser can only run JavaScript. And it took a breakthrough at the time, Mozilla had a research project called Emscripten that allowed you to compile different programming languages like C C into JavaScript. And for the browser to be able to run something like Python, I needed to compile CPython to JavaScript. So I was the first to do it in the world. So built contributed to that project and built a lot of the scaffolding around it. And my friends and I compiled Python into JavaScript and I was like, okay, we did it for Python, let's do it for Ruby, let's do it for Loa, let's do it. And that's how the emergence of the idea for replit came, is that when you need a repl, you should get it, you should repl it. And so REPL is the most primitive programming environment possible. So I added all these programming languages and again, all this time my friends were using it and excited about it. And I was on GitHub at the time. And just my standard thing is like when I make a piece of software, open source it. And so I was open sourcing all the things I was, you know, years building just like this underlying infrastructure to be able to just run code in the browser. And then it went viral, right? Went viral in hacker news and it coincided with the MOOC era. So massively online courses, Udacity was coming online, Coursera and most famously Codecademy. So Codecademy was the first kind of website that Allowed you to code in the browser interactively and learn how to code. And they built a lot of it on my software that I was open sourcing all the way from Jordan. And so I remember seeing them on Hacker News and they were going super fast, viral. I was like, hey, that's. You know, I recognize this. What are you using? And so I left a Hacker News comments. I was like, oh, you're using my open source package. And so they reached out to me and they're like, hey, we'd like to hire you. I was like, I'm not interested. I want to start a startup. I want to start this thing called Replit. And they're like, well, no, you should come work with us. We can do the same stuff. And I kept saying no. I was like, okay, I'll contract with you. They were paying me $12 an hour. I was really excited about it, back from a month. But they came out to their. To their credit. They came out of Jordan to recruit me and spent a few. A few days there. And then I kept saying no. And in the end, they gave me an offer I can't refuse, and they got me an O1 visa, came to the United States.
- [60:36]
Marc Andreessen
That's when you moved. So when was the first. Because you were born, what year?
- [60:40]
Amjad Masad
1987.
- [60:41]
Marc Andreessen
What was the first year that you could remember where you had the idea that you might not live your life in Jordan and you might actually move.
- [60:47]
Amjad Masad
To the US When I watched Pirates of Silicon Valley.
- [60:50]
Marc Andreessen
Is that right? Okay, got it.
- [60:51]
Amjad Masad
All right. Maybe 98 or 99. I don't know when it came out.
- [60:55]
Marc Andreessen
That might be a good place.
- [60:56]
Amjad Masad
Yeah.
- [60:57]
Host
Is it worth telling the hacker story?
- [60:58]
Marc Andreessen
Because there's a version of the world.
- [61:00]
Host
Where you didn't actually, like, if that.
- [61:01]
Amjad Masad
Changed differently, maybe you wouldn't have gone to America.
- [61:03]
Marc Andreessen
Right?
- [61:03]
Amjad Masad
Right. Yeah. So in school, I was programming the whole time. So I just want to start businesses. I just, like, I'm exploding with ideas all the time. And, like, the reason Replit exists is because I have ideas all the time. I just want to go type it on a computer and like, build them. So I wasn't going to school. It was like, incredibly boring for me. And part of the reason why Repl a mobile app today is because I always wanted to program under the desk, like, just to do things. And so the. At school, they kept failing me for attendance so I would get A's, but I just didn't show up. And so they. They would fail me. And so I felt it was incredibly unfair. And all my Friends were graduating now this year it was like 2011, I've been like for six years in college, it should be like a three or four year. And I was like incredibly depressed. I really wanted to be in school. Silicon Valley. And so I was like, oh, what if I change my grades? There we go in the university database. And so I went into my parents basement and implemented the polyphasic sleep. Are you familiar with that? I am Leonardo da Vinci's polyphasic sleep. I didn't hear from Leonardo da Vinci, I heard it from Seinfeld. Because there's an episode where John Kramer goes on falling basic sleep.
- [62:25]
Marc Andreessen
What, 20 minutes every four hours?
- [62:26]
Amjad Masad
Yes, 20 minutes every four hours. Yeah.
- [62:29]
Marc Andreessen
And yes. And this somehow is going to work?
- [62:30]
Amjad Masad
Well, yeah. And hacking, if you've ever done anything.
- [62:34]
Marc Andreessen
As the meme goes, this has never worked for anybody else, but it might work, but it might work for me. Yes.
- [62:39]
Amjad Masad
And a lot of what hacking is is that you're coming up with ideas for finding certain security holes and writing a script and then running that script. And that script will take 20, 30 minutes to, to run. And so you'll take that, you know, 20, 30 minutes to sleep and go on it. So I spent two weeks just going mad like trying to hack into the university database. And finally I found a way. I found a SQL injection somewhere on the site and I found a way to like be able to edit the records, but I didn't want to risk it. So I went to my neighbor who was going to the same school. I think till this day no one caught him. But I went to him and I said, hey, I have this way to change course grades. Like, would you want to be my guinea pig? And I was honest about it. I was like, I'm not going to do it. Are you open to do it? He's like, yeah, yeah.
- [63:25]
Marc Andreessen
They call this human trials. This is how medicine works.
- [63:31]
Amjad Masad
So we went and we went and changed his grades and he went and pulled his transcript and the update wasn't there and went back to the basement. Well, turned out that I had access to the slave database. I didn't have access to the master database. So find a way through the network. Privileged escalation. It was an Oracle database that had a vulnerability and then found the real database and then I just did it for myself. Changed the grades and went and pulled my transcripts and sure enough it actually changed. Went and bought the gown, went to graduation parties, did all that and we're graduating and then one day I'm at home, it's like maybe 6 or 7pm I get the telephone at home. Rings.
- [64:23]
Marc Andreessen
Ominous ring sound.
- [64:24]
Amjad Masad
Yes, well, hello. And he's like, hey, this is the university registration system. And I knew the guy that run it. He's like, look, we're having this problem. The system's been down all day and it keeps coming back to your record. There's an anomaly in your record where you're both pass. You have a passing grade, but you're also banned from that final exam of subject. I was like, oh, shit. Well, it turns out the database is not normalized. So typically when they ban you from an exam, the grades resets to 35 out of 100. But apparently there's a Boolean flag. And by the way, all the column names in the database are single letters. That was the hardest thing. It's security by obscurity. And turns out there's a flag that I didn't check. So when you go over attendance, when you don't attend and they want to fail you, they ban you from the final exam. So I changed the grades and that, that, that created an issue and brought down the system. So they were calling me and I thought at the time I was like, you know, I could, I could potentially lie and I'll. It'll be a huge issue or I just like, I'll just, I'll just fess up. So I said, hey, listen, look, yeah, I might know something about it. Hey, let me, let me come tomorrow and kind of talk to you about what happened. So I go in and I open the door and it's the deans of all the, all the schools. It's like computer science. Computer. They were working on it for like days because it's like, it's like, it's a very computer heavy, you know, university. And it was like a problem. And they're all kind of really intrigued about what happened. And so I pull up a whiteboard and started explaining what I did. And everyone was engaged. I gave them a lecture.
- [66:05]
Marc Andreessen
Basically your oral exam for your PhD. Yeah, this is great.
- [66:09]
Amjad Masad
They were, they were really excited and I think it was endearing to them. I was like, oh, wow, this is a, that's a very interesting problem. And then I was like, okay, great, thank you. And I was like, hey, wait, wait. We don't know what to do with you. Do we send you to jail? And I was like, hey, we have to escalate to the university president. And he was a great man and I think he gave me a second chance in life. And I went to him and I Explained the situation. I said, I'm really frustrated. I need to graduate. I need to get on with my life. I've been here for six years and I just can't say in school the stuff I already know. I'm a really good programmer. And he gave me a Spider man line at the time. It's like, with the great power comes great responsibility, and you have a great power. And it really affected me. And I think he was right at the moment. And so he said, well, we're going to let you go, but you're going to have to help the system administrators secure the system.
- [67:08]
Marc Andreessen
There we go.
- [67:09]
Amjad Masad
For the summer. I was like, happy to do it. And I show up and all the programmers there hate me, hate my guts. And they would lock me out. I would see them, they would be outside, I would knock on the door and no one would listen. It was like, they don't want to let me in. I tried to help them a little bit. They weren't collaborative. And so I was like, all right, whatever. And so it came time for me to actually graduate. It was the final project. And one of the computer science dean came to me and he said, look, I need to call a favor. I was a big part of the reason we kind of let you go and we didn't kind of prosecute you. So I want you to work with me on the final project, and it's going to be around security and hacking. I was like, no, I'm done with that. I just want to build programming environments and things like that. And he's like, no, you have to do it. I was like, okay. So I thought I'd do something more productive. So I wrote a security scanner that was very proud of that kind of crawls a different side, that tries to SQL injection and all sorts of things. And actually my security scanner found another vulnerability in this system.
- [68:11]
Marc Andreessen
Amazing.
- [68:11]
Amjad Masad
And so I went to the defense and he's like, you need to run this security scanner live and show that there's a vulnerability. And I didn't understand what was going on at the time, but I just, okay. So I give the presentation about how the system works. And I was like, oh, let's run it. And it showed that the security vulnerability. Okay, let's try to get a shell. So the system automatically runs all the security stuff and gets you a shell. And then the other dean that turned out, he was giving the mandate to secure the system. And now I started to realize I'm a pawn in some kind of rivalry here. And his face turned red and it's like, no, it's impossible. We secured the system. You're lying. I was like, you know, you're accusing me of lying. All right, what should we know? Should we know your. Your salary or your password? What do you want me to look up? And I was like, yeah, look up my password. So I look up his password, and it was like gibberish. It was encrypted. And I was like, oh, that's not my password. See, you're lying. I was like, well, there's a decrypt function that the programmers put it there. So I do decrypt and it shows his password. It was something embarrassing.
- [69:19]
Marc Andreessen
I forgot what it was. And so.
- [69:21]
Amjad Masad
So he gets up really angry, shakes my hand, and leaves to change his password. I was able to hack into the university another time. Luckily, I was able to graduate. Gave them the software, they secured the system. But, yeah, later on I would realize that, yeah, he wanted to embarrass the other guy, which was why I was.
- [69:40]
Marc Andreessen
In the middle of politics. Well, I think the moral of the story is if you can successfully hack into your school system and change your grade, you deserve the grade and you deserve to graduate.
- [69:48]
Amjad Masad
I think so.
- [69:50]
Marc Andreessen
And just for any parents out there or children out there, you can cite. Cite me as Mark Andreessen as the moral. You can cite. You can cite on Shout out to me as the moral authority.
- [69:58]
Amjad Masad
Moral authority on this one maybe lesson. I think that is very relevant for the AI age. I think that the traditional sort of more conformist path is paying less and less dividends. And I think kids coming up today should use all the tools available to be able to discover and chart their own paths. Because I feel like just, you know, listening to the traditional advice and doing the same things that people have always done is just not as. It's not working out as much as we'd like. Yeah, that's right. Thanks for the podcast.
- [70:33]
Marc Andreessen
Thank you, man. Fantastic.
- [70:37]
Host
Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to, like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts and Spotify. Follow us on X16Z and subscribe to our substack@a16z.substack.com thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com disclosures.