The Professor of Outputmaxxing — Anjney Midha, AMP

Summary

Latent Space: The AI Engineer Podcast

Episode: The Professor of Outputmaxxing — Anjney Midha, AMP

Date: June 18, 2026

---

Episode Overview

This episode features Anjney Midha, CEO and Founder of AMP, in a deep-dive conversation about AI infrastructure, optimal systems design, scaling challenges, output maximization ("outputmaxxing"), organizational alignment, and the philosophy behind building both data centers and AI companies for sustainable, responsible growth. The discussion explores the intersection of technical, organizational, and human aspects of scaling AI, with a focus on minimizing waste, fostering culture, and pushing the boundaries of what's possible in the AI ecosystem.

---

Key Discussion Points & Insights

1. Data Center Utilization—Efficiency and Common Sense in AI Scaling

00:08–03:13

• Types of Utilization: Node allocation (percent of GPUs used in a data center) vs. MFU (Model-Focused Utilization).

  • At top-tier companies (like Google), 95%+ node utilization is expected—falling below this is considered unacceptable.
  • MFU "best in class today is somewhere between 60 and 70%." (Co-founder, 00:53)

• Leadership and Alignment in Utilization:

  • The challenge is keeping everyone involved in the data center—funding, management, measurement—aligned as companies scale rapidly.
  • “If you have two arcs that are just off by a few degrees, that spreads out right at scale...the wastage just compounds really fast.” (Co-founder, 01:11)

• Iterative Approaches & Common Sense:

  • Moving fast doesn't excuse abandoning established best practices from the semiconductor/data-center industries.
  • "Common sense should always be in fashion." (Andrej Karpathy, 02:17)

• Responsible Scaling:

  • Emphasizing the need to "move fast with responsible infrastructure," reflecting on the evolution from Facebook’s "move fast and break things."

2. Social License, Local Impact, and Net-Positive Data Centers

03:17–05:56

• Marginal Unit Economics Approach: Ideas about allocating part of compute costs directly as cash benefits to local communities hosting centers—increasing public goodwill and reliability of operations.

  • “I’d be happy to pay an additional 50 cents per hour at scale because if that means the public benefit is so clear…” (Co-founder, 03:32)

• Community Backlash Risks:

  • "Up to 20% of all data centers this year...are at risk of community backlash." (Anj Mitta, 04:12)
  • It’s not just about local jobs; communities care about power grid strain, environment, and other externalities.

• Long-term Trustworthy Partners vs. “Neo Clouds”:

  • Preference for established, reliable infrastructure partners over new, flashy, cloud startups.
  • “There are really good, reliable, trusted data center providers in America...I love those folks. They know how to—are they sponsoring happy hours at NeurIPS? No...But they're adults.” (Co-founder, 05:35–05:42)

3. Architecture, Alignment, and the Compute Grid Vision

06:07–13:39

• Full Stack vs. Horizontal Alignment:

  • Explores the difference between monolithic/integrated (vertical) and pooled/resource-sharing (horizontal) systems.
  • AMP’s vision: create a “compute grid” analogous to the electric grid, pooling resources across clouds and silicon vendors.

• Independent System Operator Model:

  • AMP positions itself as an independent system operator, coordinating multiple suppliers and consumers of compute for peak efficiency.
  • “We see ourselves as...an independent system operator. Once it became legible...that pooling is a good idea, we should pool our generators instead of all having a generator running at half capacity in our backyard.” (Co-founder, 08:18)

• Dynamic Prioritization & Credit/Token Systems:

  • Describes mechanisms, like at Google, where job priority and interruptible demand are managed by a bidding/credits system, but notes potential pitfalls of too much decentralization and the risk of missing out on centralized breakthroughs ("Google missing GPT").

4. Venture Structure, Strategic Investment, and Unlocking Research

12:41–15:58

• AMP's Organizational Structure:

  • Mirrors Alphabet: AMP Holdings, infrastructure business, and venture arm (Foundry), investing in labs like Periodic and Anthropic.

• Research Bottlenecks & Hoarding:

  • Concern about DeepMind/Google research not being released—either embargoed, unpublished, or not productionized, hindering broader progress.
  • “There’s negative externalities of research being hoarded and so there’s a market failure.” (Andrej Karpathy, 14:31)

• Gigawatt Scale Aspirations:

  • AMP working to guarantee 1.3 gigawatts base load, aiming for up to 6 gigawatts over four years for their network of labs.

5. AI for End-of-life Care—Personal Journey and Societal Impact

15:58–23:17

• Personal Passion for Medical AI:

  • Anjney traces his journey from grad school in bioinformatics at Stanford, working on end-of-life prediction models with Professor Nigam Shah.
  • Contrasts cultural attitudes toward death in India/Singapore vs. the US, and how that impacts medical systems and spending.

• Challenges in Clinical AI:

  • Regulatory hurdles: shifting liability for diagnosis from doctors to algorithms remains unresolved.
  • “The problem remains, then and now, is regulatory. You...can’t shift the burden of the wrong clinical diagnoses from the physician to the AI system.” (Andrej Karpathy, 22:01)

• Ongoing Commitment:

  • “I haven't been able to get this out of my mind a single day for the last 14 years. This is the hill I would like to die on.” (Andrej Karpathy, 22:40)

• Dual Mission:

  • Public interest in both medically-empowering patients and building net-positive, community-friendly data centers as key AI bottlenecks.

6. Outputmaxxing as a Discipline: Anti-waste in Tech and Society

24:08–27:36

• Defining “Outputmaxxing”:

  • The engineering discipline of maximizing useful outcomes with existing resources, whether GPUs or Medicare dollars.
  • “From an engineering perspective, it's very simple. It's outputmaxxing...I'm a huge believer in optimal outcomes.” (Andrej Karpathy, 24:45, 24:53)

• Cultural/Technical Standardization:

  • Advocates for standardization (as with transformers at Anthropic) or disruptive new capabilities to reduce loss in scaling.
  • "You either have to standardize on protocols...or you can come with a whole new capability that unlocks so much abundance." (Andrej Karpathy, 27:19–27:33)

7. Bootstrapping Open Compute Protocols & Expanding Supplier Diversity

27:36–32:39

• Futures Contracts for Compute:

  • Discussing attempts to create compute futures exchanges (e.g., SF Compute) and the challenges of trust boundaries and interoperability.

• Multi-Silicon, Multi-Cloud Vision:

  • Highlighted by example of new chip companies (e.g., MatX) designing to Nvidia standards to interoperate and “piggyback” on existing infrastructure.
  • “He decided that he didn't want to be a fully integrated chip provider...so then he's dependent on that part of the ecosystem to host his chips.” (Andrej Karpathy, 30:46–31:16)

8. Founder Mindset, Researcher-Founders, and Culture as a (Fragile) Moat

33:49–53:58

• Why Fund Researchers as CEOs:

  • Dispels VC clichés labeling technical founders: “Being a good CEO is hard. Being a great CEO actually requires a level of performance that scientists...have accomplished.” (Andrej Karpathy, 35:29)

• Distilling Company Culture:

  • Culture is not a durable moat by itself; it's fragile and must be repeatedly demonstrated.
  • “Culture is not a set of beliefs, it's a set of actions.” (Referencing a quote seen at a16z, 49:48)

• Anthropic’s Mission Alignment:

  • Mission clarity and hardship in early days contributed to resilience and safety focus:
  • "The number of people who said no...forces you to really understand what is the hill you want to die on at the expense of everything else. What's the P0?" (Co-founder, ~52:36)

9. Case Study—Periodic Labs: Technical and Organizational Hardships

54:02–55:49

• Technical Constraint:

  • Physics is the key hardness—AI is not yet good at analyzing scientific/physics data.
  • “If you said, hey, could you analyze the scientific data coming out of a condensed matter physics lab? I was...in the group at Stanford, it was terrible.” (Co-founder, 54:55–55:00)

• The Reality of Startups:

  • Stories of team members leaving for more lucrative options but returning after a technical breakthrough, reinforcing the importance of company culture and loyalty.

10. Personal Narrative—Backgrounds, Grit, and Money as a Tool

56:10–58:38

• Humble Beginnings:
  • Anecdotes about growing up in modest circumstances in India and Singapore.
  • “What I'm saying is I don't need much to be happy in life...When it stops becoming just a byproduct and more of a measure, it stops having meaning.” (Andrej Karpathy, 58:17–58:38)

---

Notable Quotes & Memorable Moments

• “Common sense should always be in fashion.”
  Andrej Karpathy, 02:17

• "We're trying to do for compute what the power grid did for electricity."
  Andrej Karpathy, 07:00

• “I haven't been able to get this out of my mind a single day for the last 14 years. This is the hill I would like to die on.”
  Andrej Karpathy, 22:40

• "Outputmaxxing...the department of outputmaxxing...I’m a huge believer in optimal outcomes."
  Andrej Karpathy, 24:48–24:53

• “Culture is not a set of beliefs, it's a set of actions...if you stop taking the actions...your culture starts to fray.”
  Andrej Karpathy, referencing a16z wall, 49:48

• “Being a great CEO actually requires a level of performance that scientists who have already published at the top of their field have accomplished.”
  Andrej Karpathy, 35:29

• “Money is a way, I think, sometimes we measure ourselves...when it stops becoming just a byproduct and more of a measure, it stops having meaning.”
  Andrej Karpathy, 58:17–58:38

---

Timestamps for Important Segments

• Data center utilization & scaling pitfalls: 00:08–03:13
• Social impact & net-positive data centers: 03:17–05:56
• AMP and the compute grid analogy: 06:07–13:39
• AI and end-of-life prediction: 15:58–23:17
• Defining and practicing Outputmaxxing: 24:08–27:36
• Standardization and compute futures: 27:36–32:39
• Founder mindset & culture as a moat: 33:49–53:58
• Periodic Labs culture/physics focus: 54:02–55:49
• Personal journey & philosophy: 56:10–58:38

---

Tone, Style, and Takeaways

The conversation is open, thoughtful, and frequently philosophical, blending stories of technical strategy with personal anecdotes and a sense of mission-driven urgency. The speakers emphasize nuance, responsible acceleration, and operating at the cutting edge—not just in technology, but in culture, incentives, and the very definitions of value and waste. The session is rich in both specifics and frameworks, offering listeners a window into both AI’s infrastructural present and its speculative future.

Transcript

[Anj Mitta]

Foreign, we're in periodic labs with Anj Mitta, CEO, Founder of amp. Welcome.

[Andrej Karpathy]

Thanks for having me at Google. If utilization. So there's two types of utilization usually, right, that you're measuring in these clusters.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

One is node allocation and then the other is mfu.

[Andrej Karpathy]

So node utilization is usually like what

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

percentage of cards in the data center are just like used? And that if it's not at like 95%, there's no excuse.

[Andrej Karpathy]

There's no excuse, right?

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Like I think 95% at Google, which is where my co founder Seb came

[Andrej Karpathy]

from, he built the Borg X Borg

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

GQM scheduler at Google and there I think 95% was considered an outage. So 96% node utilization should be standard and most single tank clusters are not running at that. So that's one.

[Andrej Karpathy]

And then MFU utilization should be, I would say, the best in class today,

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

somewhere between 60 and 70%. I think this is a leadership question, right? Fundamentally it's an alignment question, which is

[Andrej Karpathy]

are the people who are funding the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

cluster and then deploying the cluster actually aligned?

[Andrej Karpathy]

And sometimes theoretically they are, but in practice the number of people in the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

chain, the supply chain between the capital and all the way to whoever's managing the cluster and then whoever's measuring what the output is, are just so many degrees of separation away that like the, you know, have you heard that sort of radiant metaphor which is at the beginning of an arc, if you have two arcs that are two lines that are just off by a few degrees, that it spreads out right at scale. And I think what's happening is a lot of cluster implementations and infrastructure, a lot of Frontier Labs and other teams. That's what's happening is they initialize the plan, which is kind of like North Star with a team that wants to do good, but then they're required to scale so fast instead of iteratively that the wastage just compounds really fast at scale. And so I think we know the answer, which is just do iterative bring ups. You know, if you spend time with people who've been in the semiconductor industry or the data center industry for a long time, this is not new. And I don't think AI should be an excuse like sure something what is new? Okay, we have a lot new capabilities, but that doesn't mean just abandon common sense.

[Andrej Karpathy]

Common sense should always be in fashion.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

AI scaling doesn't change the. In fact, if anything, AI scaling should be putting a premium on the value of common sense and infrastructure because the margin of error now is so much lower and the cost of wastage are so much higher. And the cost of wastage, by the way, is not just economic. Obviously I'm an investor or I'm an investor by background over the last few years. Now we're running an AI infrastructure business called Amp. And I think that it's okay to say this time is different on the capabilities front. Like we are genuinely getting capabilities of a kind we haven't had before. That doesn't give you an excuse to say this time is different for everything, especially infrastructure.

[Andrej Karpathy]

So look, I love the hacker mindset

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

and the hustler mindset and that's great for the startup mindset, but do you remember this moment where Zuck went from saying move fast, break things to like move faster.

[Anj Mitta]

Stable infrastructure.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Move fast with stable infrastructure.

[Andrej Karpathy]

I think now we need to move

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

fast with like responsible infrastructure.

[Anj Mitta]

Yeah.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

They're going to say like, where is the impact? You know, there was a really. In our class yesterday, Scott Nolan, who was the founder of General Matter, came by at Stanford to speak about energy bottlenecks and he had a phenomenal idea.

[Andrej Karpathy]

He said if you look at the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

marginal unit economics of computer per hour, let's call it like $4 an hour. If you're having to bring up a new data center in a new community, why not just say we're going to charge 450 an hour and that marginal impact or that marginal increase, we just literally take that and give it to the local community as cash. I can tell you as a customer of that compute, I would love that. I'd be happy to pay an additional 50 cents per hour at scale because if that means the public benefit is so clear to the communities that the data centers are coming up in, I'm going to feel like that COMPUTE is much more reliable. Up to 20% of all data centers this year in the U.S. my understandings

[Anj Mitta]

are at risk of community backlash.

[Andrej Karpathy]

Correct.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Of not getting the community support they need to get brought up.

[Anj Mitta]

Wow, that's a huge number.

[Andrej Karpathy]

Now I think we should dig into

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

what that number is. I think it's a little bit of overstated. These things can get over reported, but

[Anj Mitta]

they don't just care about jobs, they care about all the other stuff around it. Right? Like they care about power grid, they

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

care about environment, power grid permitting and so on.

[Andrej Karpathy]

And imagine, I think if you said

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

there's a new AI deal, if we're bringing up a data center in your community, we're actually going to reduce the cost of your electricity bill. Okay, now we're talking, right? The community is Going okay, now this is a deal. I feel like a partner in this right now. That's not happening.

[Anj Mitta]

Mm.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

There will be audits, there will be investigations. And when the, when the regulators come, I don't know when it's going to be. The folks who are moving fast and breaking things in the name of AI progress better be prepared. That's certainly not how we're procuring compute or we're trying as much as we can to work with partners who have long term track records, many of whom by the way, are not like AI providers. I think this whole idea of Neo Clouds being somehow this new category is a lot of marketing speak. There are really good, reliable, trusted data center providers in America who've been around 20 plus years. I love those folks. They know how to. Sure have they. Are they sponsoring happy hours at Neurips? No. Are they legibly bitter lesson pill?

[Andrej Karpathy]

No. Are they hanging out in, you know,

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

in like situationally aware parties? No.

[Andrej Karpathy]

But they're adults.

[Anj Mitta]

Yeah.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

I trust them.

[Anj Mitta]

They can run land, they can run

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

powershell, they have credit histories. We sit down, we have a conversations. Many of them live in Silicon Valley.

[Andrej Karpathy]

They've, you know, they've had to deal with the boom and bust cycles of

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

the Internet and I love those folks.

[Andrej Karpathy]

You know, they, they are stable infrastructure partners and thinkers and I think there's

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

a lot of short term thinking going on in the compute layer and it's going to catch up to us. It, it's not going to be good.

[Anj Mitta]

You talk about aligning incentives and you know, I would think that aligning incentives means you have the full stack in one company, which is Xai and OpenAI. Right? So you as a standalone infrastructure layer, why are you somehow more aligned to your portfolio companies than people who just own the whole thing?

[Andrej Karpathy]

In systems design, right, there's two regimes of architecture, right? You have integration and then you have

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

pooling and utilization, right?

[Andrej Karpathy]

Or rather the way to increase utilization

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

often is you can do systems integration where you collapse a lot of process

[Andrej Karpathy]

into one node, or you can pull

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

out a process from a node and share that amongst that resource amongst several different nodes.

[Andrej Karpathy]

And so we see the AMP grid,

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

which is the system we're building here, which is basically a compute grid.

[Andrej Karpathy]

We're trying to do for compute what

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

the electric grid, what the power grid did for electricity.

[Andrej Karpathy]

This is a pooling and utilization layer across clouds. And so we're actually the opposite of

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

a full stack integration approach. It's much more horizontal and it's multi cloud, it's multi silicon. The Goal is to try to make flops flow like megawatts. And that is very hard to do today for many reasons. There's stranded pools of compute all over the place and there's no fungibility. And so right now we do it at the level of scheduling and we often do it at the economic layer. But as we start to announce what we're working on, it's extraordinary, like how many folks are coming out of the woodworks and saying, hey, I'm actually working on a way to make compute fungible at this part of the stack and that part. And as a grid, we'd like all of these folks to participate on the grid. You know, people often ask me, Andre, you're a new cloud. And I go, no, actually Neo clouds are suppliers. Or sometimes they'll ask are you a venture capital firm? I go, no, actually they are, they are demand like sort of off takers of the grid. We see ourselves as what's called an independent system operator. So if you study the history of the electric grid, once it became legible to a lot of factories and industrial sort of participants that hey, actually it turns out pooling is a good idea. We should pool our generators instead of all having a generator running at half

[Andrej Karpathy]

capacity in our backyard, there was a

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

need for an independent entity who could coordinate all these parties.

[Andrej Karpathy]

Transmission line, power generation facilities, transmission lines, factories. And that neutral coordination mechanism is very critical in order.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

If you study like the history of grids, the most enduring ones were those that never owned their own assets.

[Andrej Karpathy]

They were ones that had already, often started with long term anchors who were

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

uncorrelated sources of demand. A steel factory, a shoe mill or whatever in a particular town who weren't competitive.

[Andrej Karpathy]

Where the steel factory wanted to spike

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

up at night, the shoemake wanted to spike up during the day. So then you pool and you share.

[Andrej Karpathy]

Right. So each of you is guaranteed some baseload, but then you kind of schedule

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

your spikes to drive a peak utilization across the town.

[Andrej Karpathy]

The gold standard, so to speak, historically

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

has been these utility companies like PGM Interconnect in the northeast of America where

[Andrej Karpathy]

they over many, many years became this,

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

what's called an ISO, an independent system operator of the grid.

[Andrej Karpathy]

So that's how we see ourselves. Yeah, economically that's what we are for. From a technical perspective, we started at the scheduling layer because Seb and Mihaly

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

who run engineering here, built that scheduling. They did that at Google.

[Anj Mitta]

And you have infrashops from Discord as well. I don't know if Discord is the Primary identity, but whatever, I'm just.

[Andrej Karpathy]

No, Discord was choosing a well known name. Well, so I was running the developer platform there, the internal infrastructure I was not responsible for that was actually a

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

guy by the name of Mark Smith who was extraordinary.

[Andrej Karpathy]

And yes, Discord did pool. So Discord is actually a counterexample. I guess I had the chance to learn a lot about full stack info there because same thing. Yeah, it's the other architecture which is Discord built its own WebRTC voice and video infra. So like Discord did not use. Yeah, did not for communication. Discord did not use third party infra. It was all built in house. And then the way you maximize utilization was you pull demand from the world's 200 million-plus monthly active gamers.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Right.

[Andrej Karpathy]

And so that's how those stacks were constructed. Again, in systems design, the two concepts that keep coming up over and over again are abstraction and composition.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Right.

[Anj Mitta]

Bundling and unbundling.

[Andrej Karpathy]

And unbundling abstraction, composition, like verticalization and horizontalization. So in that sense, AMP is an independent system operator of the grid. We pulled supply from a number of

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

partners we trust at about 1.3 gigawatt scale over four years.

[Andrej Karpathy]

And then we pull demand from some

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

of the world's best research labs and so on. We're sitting at one periodic labs who need extraordinary long term demand.

[Andrej Karpathy]

And the idea is that each of them is guaranteed base load on the grid, but they can spike up and down flexibly for compute with much shorter timelines as needed. That was roughly the design of the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

program I came up with at a 16z called oxygen.

[Andrej Karpathy]

The same. That was the same design of the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

GQM Borg x Borg GQM implementation at Google that Mihai and Seb had built,

[Andrej Karpathy]

which is that how do you allow teams inside of Google on the internal infrastructure to be guaranteed capacity for their base workloads, but when they need to

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

spike up on research, how could they ensure that that was sufficiently there?

[Andrej Karpathy]

And of course the big innovation that was discovered, not discovered, but kind of implemented in the space, this infraspace maybe

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

three, four years ago at Google was

[Andrej Karpathy]

the idea of interruptible demand, right? Where you just queue up a bunch

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

of jobs and through this like sort of credit system there can be a bidding mechanism, priorities.

[Andrej Karpathy]

It's a dynamic prioritization basically. And jobs can get interrupted based on

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

somebody else who's saying, you know What?

[Andrej Karpathy]

I have 10 tokens, 10 credits I

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

want to spend on this job.

[Andrej Karpathy]

Another business like Team lead, research leads, genie 3 or whatever is only worth 5 credits. And nanobanana 2 is worth 10 credits

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

and so the nanobanano gel gets priority. That's a made up example.

[Anj Mitta]

It's very real. Brain Marketplace was real and we've covered this on the POD with David Luan who was there. And the criticism is that, well actually sometimes you need central commands to go all in on the thing and actually sometimes capitalism via credits doesn't work. It's not criticism of amp. I'm just saying this is a thing that has been tried internally within Google and it led to Google missing GPT.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Like we structured ourselves essentially very similarly to Google. We are structured as a holdings company. So Alphabet holdings is Alphabet holdings and then they've got these subsidiaries called Google and other bets, other bets and so on. We've got AMP holdings and we've got our infrastructure business and then we've got a capital business called Foundry that incubates New Frontier AI Labs invests in them as venture capital, like Periodic. We put a few hundred million dollars into Anthropic from our fund earlier this year.

[Andrej Karpathy]

So wherever we feel like teams are

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

making progress, especially researchers and so on, who've pushed the frontier inside of existing labs like DeepMind, I find there comes

[Andrej Karpathy]

a point where they feel misaligned with

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

the dictatorship of Alphabet holdings. And at that point sometimes the dictatorship doesn't want them anymore and they're like thank you, you've done your job here. You've kind of helped us through the 0 to 1 phase and for whatever reason we're going to deprioritize your amazing like omni model or whatever it is and instead we're going to prioritize coding and I think that's a tragedy.

[Andrej Karpathy]

But I get it, like they're, you

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

know, Sergey and team are running their own business there, but that doesn't mean we should. The rest of us should sit around waiting for that progress to get unlocked for the rest of the world and humanity. I mean if, if you think about how much extraordinary research has happened inside of DeepMind over the last 10 years. I mean Demis and Sergey and those guys did such a great job.

[Andrej Karpathy]

But at the end of the day, so much of that has never seen

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

the light of day.

[Anj Mitta]

Or they're like papers only, but they never actually shifted to production.

[Andrej Karpathy]

I mean, what's worse is the paper is actually not even being published anymore

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

because there's a six month embargo inside of DeepMind. Right. We've heard about this where a paper comes out and then I think it has a six month embargo window where

[Andrej Karpathy]

if anybody on the business team says this could be interesting, it's embargoed for life.

[Anj Mitta]

Exactly. So the stuff that gets published is the stuff that's not good enough. There's an adverse selection problem.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Basically. Yeah.

[Anj Mitta]

At this point it's a common complaint at Neuros, by the way. That's like, well, why would I look at the papers that are the trash of GDM again?

[Andrej Karpathy]

I think it's a tragedy. I mean, I get it, they're running their business, but the rest of the space, I think there's negative externalities of research being hoarded and so there's a market failure and somebody needs to unlock that research and we can't do it on our own. We only have 1.3 gigawatts of compute. That's nothing. That's about $40 billion of cloud spend.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

We're going to need a lot of.

[Anj Mitta]

That's a new number. I haven't come across that gigawatt number. That's huge.

[Andrej Karpathy]

Yeah. And to be clear, we haven't secured all of it. That's how much demand we have started to secure. I think publicly we haven't actually confirmed how much we have for this year.

[Anj Mitta]

Where do you want to get to?

[Andrej Karpathy]

I think the steady state would be that we have a base load pool of 1.3 gigawatts at all times of baseload capacity for spike capacity. Right now my estimate is we need

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

roughly 6 gigawatts over the next four

[Andrej Karpathy]

years for all our teams to feel like they were able to keep moving the frontier.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Whatever they're working on, whether it's like superconductor discovery over here.

[Andrej Karpathy]

There's a new investment we're working on right now which is in the end

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

of life prediction space in, in healthcare. It's extraordinary how much you can, you can give people.

[Andrej Karpathy]

You know. This was actually my graduate school work.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

I went to grad school for bioinformatics at Stanford Med.

[Anj Mitta]

Yes. And I know we econ, MCS bio.

[Andrej Karpathy]

So my. I was this really weird cat where like I was never satisfied with my major options. So at one point I was an econ major, then I was a CS major, then I was a MCS major called mathematical computational science and they decided they were going to end that major. So I took all that coursework and I applied it to grad school.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

My graduate degree in bioinformatics, which was the master's program. And then I thought I was going to do a PhD. I never ended up doing. I dropped out and went to work at Kleiner.

[Andrej Karpathy]

But I was lucky enough to apprentice with this professor at Stanford Med. His name is Nigam Shah and he

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

was working on end of life prediction.

[Andrej Karpathy]

And Stanford is one of the only research facilities in America that has a longitudinal patient data set that's larger at scale. I think it's at least 12 million patient lives. The only larger data set is the VA, the Veterans affairs of America. And to do research, like do any deep learning and so on that data set, it was called the stride data set at that time you had to be a Stanford Med School affiliate, which

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

is why I went and enrolled in the bioinformatics department.

[Andrej Karpathy]

Wow. And look, deep learning was early. Nigam Shah had the visibility, like the vision to see that like you could do end of life prediction to help palliative care in, you know, in America, the like over 30% of all Medicare, Medicaid spend, at least at that time was spent on end of life care. And what's, you know, we grew up in Asia, so we all.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Yeah.

[Andrej Karpathy]

At least I won't speak for you, but I have a very different relationship with death than I find folks who

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

grew up in America do.

[Andrej Karpathy]

In America, spiritually and culturally, especially in Western societies where Christianity, the Christian tradition sort of frames death as this terminal point, there's often a judgment day and so on. The way we view death is with a finality. In Indian culture and Hindu culture, death is one Buddhist as well. You're a Buddhist? Yeah. So it's one, it's one step in a journey of many lives. Right. And so I grew up in this city called Chennai in the south of India. And when people die, you dance on the street. You know, there's like a procession where your body is carried to be cremated and your family like celebrates and there's drums and so on. It's this huge thing. And it's because the idea is that you're going to be reincarnated. You know, you've been liberated from the responsibilities of this life and now you're onto your next. It's a new advent. It's like going off to a new college or whatever. Right. And so it was so alien to me when I got here as an undergrad that the medical system works backwards from that assumption that we have to view death as this terminal thing and delay it, postpone it. It's a bad thing. And so at the time, clinical decision support in, in the United States was this very primitive field. Even to this day, physicians in the United States often will tell you when you have a terminal disease. This is your. We've diagnosed you, which is great. Our ability to diagnose you is extraordinary. You have somewhere between six months to

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

six years to live.

[Andrej Karpathy]

What do you do with that information? The error bars are so high that then in times of uncertainty, we default to culture. And when the culture is, this is a bad thing, I've got to prolong my life, then you start doing things like. And just to sort of, from a systems perspective, what's going on there is physicians often feel like they need to

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

provide such high error bars because there's

[Andrej Karpathy]

always some uncertainty in end of life diagnosis. And if you provide the wrong diagnosis or recommendation to your patient, you can be sued for medical malpractice, and then your license can be taken away. It can be catastrophic for your career. In contrast, if in countries where that's not the case, what you often observe is that patients, like physicians, are quite prescriptive with their recommendation. They say, hey, this is your condition. The literature says that you probably have this much time on earth left. My expert opinion is that you are an outlier or whatever. And they try to be more prescriptive. And that empowers a patient because a patient can say, I trust my doctor. They said, on average, I have six months to live. But if I do these things, I may have a shot because of my particular predispositions or my genetic history or whatever. And that empowers you to go about your life in actually a more scientific way then leaning on religion, culture, spirituality and so on. In contrast here, because of that medical malpractice sort of thing looming over your head, a physician never gives you a clear recommendation. So instead you say, okay, Doc, well, let's try it all. And then you start a whole regime of drugs and therapies. And then you often spend weeks and weeks in the hospital, and that deteriorates your quality of life. And when that deteriorates your quality of life, instead of spending your last few days doing the things you love with your family, you're spending on a hospital bed. And that ends up being 30% of Medicare and Medicaid. So it's worse for the patients. The doctors feel terrible. The American taxpayer is paying a huge amount of money. And so this is why Nigam Shah, who was this professor at Stanford, said, anj, if there's. I kind of sat down with him. I was this young guy, you know, I was 21, and I was like, I want to work on a big problem. And he's like, the big problem is

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

end of life care.

[Andrej Karpathy]

And so we Tried to do deep learning to say. So we started trying to run deep learning on these TRI patient data sets to say, could you have an AI system make a recommendation that is orders of magnitude more precise about how much time you have left once you've been diagnosed with a terminal condition than a human? And then if we can get that precision to be high enough, then you can empower the patient. And it turns out the tech works once you get the data set like RL works. Honestly, even regression models work. You don't need to get that fancy. At the time, we were just doing very simple neural nets. Today, what we can do with RL is extraordinary. The problem remains, then and now is regulatory. And because you actually can't shift the burden of the wrong clinical diagnoses from the physician to the AI system. And so at that time, I got quite disillusioned 10 years ago, 12 years ago, because I felt I just didn't have the resources to influence regulation. Today, I'm very lucky. I'm in a different place. I'm a lot older. And so I've been spending a lot of time on my next incubation, which is how can we unlock the patient empowerment by training AI models to do end of life prediction with much more precision.

[Anj Mitta]

You're still focused on this the whole time?

[Andrej Karpathy]

I haven't been able to get this out of my mind a single day for the last 14 years. This is the hill I would like to die on. There's two. I would say, you know what? I actually prefer not to die. Yeah. But I think two bipartisan issues. I think two issues that should be bipartisan in America are how do we empower patients to make the right clinical decisions at the end of their life such that we're reducing the taxpayer burden with science. It's just good old science. And AI can help here. And the second is net positive data centers. Because I think that's the biggest critical bottleneck on training good enough AI models

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

to help people at the end of their life.

[Andrej Karpathy]

There's two sides of the same scaling bottleneck curve, but those two. We formed AMP as a public benefit corporation. My wife and I, who you've met, you know, you've met Viv.

[Anj Mitta]

Yeah.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Her passion is education.

[Andrej Karpathy]

Her family is a long line of

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

educators and so on, and physicists.

[Andrej Karpathy]

And so this class is my attempt to stop being the black sheep of the family and be an educator. But if I'm not educating, the thing I would be doing is working on

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

these two problems, whether on the political spectrum or as A researcher back at in some lab.

[Andrej Karpathy]

And my hope is if anyone's listening

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

to this podcast, if they're passionate about

[Andrej Karpathy]

either of those two topics, I'd love

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

to hear from them. We can share the contact in the show notes. But we're looking for people to join both of those missions on the political side as well as on the medical side, on the research side.

[Anj Mitta]

You said this is a discipline that you want to form. It's called variously called Frontier System. It's variously called one person, Frontier Lab. What is the ideal name or shape of this? What is the mission of the class, of the discipline that you're I guess exploring? Right. The class is called Frontier Systems. But for me, maybe one phrase is you're just anti waste. Right. Which is waste in GPUs, waste in human and Medicare. But is there a broader theme that maybe you can encapsulate more succinctly?

[Andrej Karpathy]

Yeah, yeah. From an engineering perspective, it's very simple. It's output.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Maxing

[Andrej Karpathy]

the department of output matches the

[Anj Mitta]

most of what we have.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Exactly.

[Andrej Karpathy]

I'm a huge believer in optimal outcomes. You know, I. I think both in America and other countries we are losing our appreciation for nuance. And this is the thing, AI is the same case, right? Oh, the bitter lesson holds. Okay, fine. But that doesn't mean you just like throw 500 GB, 300, 500,000 GB, 3 hundreds at your like, you know, suboptimal model scaling and you waste a bunch of computer compute. It also doesn't mean that the Most optimalists have 50 different architectures where there isn't enough standardization. One of the reasons Anthropic has had extraordinary velocity is because they picked the transform architecture and said, this is simple,

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

let's double down on it.

[Andrej Karpathy]

And now, luckily there's enough investment going

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

to the space that we can afford other architectures.

[Andrej Karpathy]

But at the time, investment was just

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

too fragmented into other architectures, so that arguably unlocked scaling.

[Andrej Karpathy]

So I think there's a philosophy, I think we all owe it to ourselves to do output maxing with a new capability called AI on a global level. I think if I was starting a new department at Stanford, depending on how

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

fuzzy or technical I wanted to be, I'd probably call it the department of Alignment.

[Andrej Karpathy]

It's an overloaded term, but alignment really is a hard problem. And I think when you unlock it, full stack alignment is super hard in any organization, in any system, like in a venture capital firm, if you can have full stack alignment between your limited

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

partners and the founders who are creating

[Andrej Karpathy]

the value and ultimately the public that owns the IPO stock, that is a gift that keeps giving. And when you study the history of these systems, when they start off, they usually start out small scale, where the feedback loop is actually so tight that there's alignment. And then the more you try to scale, the more division of labor happens, the more specialization happens. And at each step you add abstractions. And wherever there's an API interface, there's like loss, there's communication loss. And so I think a really cool thing would be for us to figure out, is there a way for us to have our cake and eat it too?

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

As an engineering discipline, is there a

[Andrej Karpathy]

way to actually scale up and scale out without losing any alignment, without lossy transmission mean standards. So standards is one way. The other way is you just have net new capabilities. So, like what we're trying to do here is discover new superconductors. A room temperature superconductor would be a lossless transmission mechanism for energy. I mean, we would have flying cars right within a few years of having a room temperature superconductor. So I think those are the two. You either have to standardize on protocols or API specs that allow lossless communication, or you can come with a whole new capability that unlocks so much abundance.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

The standardization doesn't matter because you just

[Andrej Karpathy]

unlock net new capacity. So this is what I spend my

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

days thinking about these days.

[Anj Mitta]

I mean, no, I think every infra person who wants scale and wants to output max does eventually end up thinking about this. We don't have time to go into it, but we have done an episode with SF Compute that is trying to standardize the futures contract for compute.

[Andrej Karpathy]

Right.

[Anj Mitta]

I don't know how that's going, by the way, but at some point this

[Andrej Karpathy]

will be, oh, I think Evan is awesome. And SF Compute is the kind of effort that I hope we can accelerate. Because what often happens is these exchanges are very hard to get. It's hard to bootstrap them right, because they often require there's many inefficiencies between parties. There's trust boundary inefficiencies, and infrastructure because you don't trust one part of the stack, doesn't trust another part of the stack to give them visibility. There's capital markets, inefficiencies, there's operational efficiencies. So if you can inject like a single shock to the system of a ton of compute demand or supply, then you can accelerate these new flywheels. And so my hope is one day or soon, if SF Compute needs extra like has Excess capacity, they just hook it up to the grid and they

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

get flooded with demand from us.

[Andrej Karpathy]

And on the other side if they have a ton of demand but they don't have supply, they again hook up to the grid and it's a two

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

way protocol where they can just hook

[Andrej Karpathy]

up to our capacity.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

And I don't think we're too far from that today.

[Andrej Karpathy]

Our working implementation of it is mostly through a group of labs, universities and a few sort of trusted parties who all feel like they're in alignment to borrow an over sort of used word. But our hope is to just have

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

it be an open protocol that anyone can hook up to on hook up

[Anj Mitta]

for demand or hook up for supply and primarily demand. It sounds like like you would want to offer demand.

[Andrej Karpathy]

Both. Yeah. Unfortunately what's happened in the last six weeks is you know, we thought we'd have a bunch of excess capacity by the end of this year. It's all gone.

[Anj Mitta]

It's exploding.

[Andrej Karpathy]

Yeah, it's all gone. And so I have. My text messages are full of friends. I mean we know many of these people, these are founders who've raised billions

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

of dollars in San Francisco going on.

[Andrej Karpathy]

Any chance you have like 50 nodes in the next few weeks, what is

[Anj Mitta]

the scope for non Nvidia. Right. You have Lisa Su coming and Reiner Pope as well. And so there is a lot of demand for more performance, alternative architectures and all that. At the same time this hurts your standardization?

[Andrej Karpathy]

I don't think so. So actually Reiner is a great example.

[Anj Mitta]

Right.

[Andrej Karpathy]

Reiner is the CEO and founder of matx. I actually had him by for the office hours in the class earlier today and there was an insight he brought up that I hadn't considered before which is when they decided to pick the standard for their data center they picked the Nvidia reference architecture. So the MATX chips just plug in to any site that has an Nvidia bring up planned and you know, it's

[Anj Mitta]

just software then it's not the hardware.

[Andrej Karpathy]

Well from an input and I O perspective it's the same footprint as an Nvidia rack. Makes sense where they have done innovated a bunch from what I can tell is on systems co design which is where a lot of the gains are to be had. And so he picked, he was like anj, there's just so much work to do when you're building a new chip company. Can't fight every front, just can't fight on every front. So my question to him was well you're working on this new chip their tape out is next year. Who are you going to partner with to host the chips? He said, whoever will host them. That's not my focus. And I said, but how did you. You decided. Back to our earlier systems design question. He decided that he didn't want to be a fully integrated chip provider. The bottleneck they're focused on is the logic die and he feels they can crank out a ton of performance gains through co design there. But then that means you delegate to our question earlier. He's like the data center provider is a different part of the stack and so then he's dependent on that part of the ecosystem to host his chips to get the performance gains to the customer. So now you have another abstraction and you might have loss. So I asked him how do you prevent loss? And back to your point, he said, I just picked the Nvidia standard because I didn't want to like, I wanted to piggyback off of an existing protocol. And what's great about Nvidia is that reference architecture is known, it's open, they've published it. So Jensen's actually enabled someone like Reiner to build a chip company like Maddox. And I don't see them as competitive. The compute demand is so high. Like I don't, I think Nvidia's not able to meet the demands of production. So we just need more chips. And I think it's very smart what Maddox has done, which is to say we're just going to, we're not going to innovate on the data center design because actually, thank you Jensen.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

You've done all the hard work.

[Andrej Karpathy]

Where we can innovate is somewhere else and I think that's very healthy.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

I think that's how we'll unblock new bottlenecks.

[Andrej Karpathy]

And my view is these chip teams like matx who have arrived at the insight that co design is the way the primary bottleneck for them is trust boundary. To do co design well, you need visibility into the next model generation as soon as possible because it takes two years to tape out. So if by the time I bring my chip to market, your model architecture's changed. I'm host now. When he was inside Google, he was sitting next to the Gemini team.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

He was on Palm or whatever.

[Anj Mitta]

His co founder was one of the Palm guys, I think.

[Andrej Karpathy]

Yes, yes, exact. So when you're inside the trust boundary of Google, then your system's co design loop is super tight when you leave. As a founder, one of the biggest risks you take is now you're outside the trust Boundary. And so what I love doing is helping chip teams who can help us unlock more capacity for the independent ecosystem, access to trust. Because when I, if I've been like involved with a lab from day one and I was lucky enough to work

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

with Anthropic, and then I'm on the board of Mistral and Apple, Black Forest

[Andrej Karpathy]

Labs get started, I think at this point I'm on six or seven different teams.

[Anj Mitta]

Only six? I feel like my mental number was going to be 13.

[Andrej Karpathy]

But yeah, dudes, no, I go deep with one at a time.

[Anj Mitta]

You were founding CEO of Arena? No, that wasn't, that was administrative CEO.

[Andrej Karpathy]

It was an administrative five month gig where Waylon and Anastasia were graduating from their PhDs and they didn't need a product team. So I helped recruit the head of engineering, product and design. But Anastasia has always been the CEO of that company. I played a pinch hitting job. I'm an intern. I was CEO intern for five months.

[Anj Mitta]

I interviewed him and he's very, very well spoken. I mean, I think he's a debate, former debate champion, but also very quantitative and mathematical, which is such a unicorn.

[Andrej Karpathy]

So you know what's amazing about him? If you look at his output, he's an output maxer. Like by the time he was graduating from his PhD, which he only graduated last year, he had published more work with a citation count than like people twice his age. But at the same time, he'd already started a project called Ala Marina that was being used by millions of people as a side project. And time and time again, what I've realized is venture capitalists suck at seeing human beings as like dynamic agents where,

[Anj Mitta]

where they want to put you in a box.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

They want to put you in a box.

[Anj Mitta]

This is your thing.

[Andrej Karpathy]

So the first time I got introduced to Anastasios, somebody had told me like, oh, he's amazing, but you know, he's a researcher. Yeah. I was like, what? What do you mean he's a researcher?

[Anj Mitta]

Not a CEO.

[Andrej Karpathy]

Not a founder, not a CEO. Exactly. I was like, are, are you crazy? Have you met Dario? Dario's a scientist. He's gone from zero to like what will soon be a trillion dollar company in four years. Being a CEO, nominally speaking, is not that hard. Being a good CEO is hard. Being a great CEO actually requires a level of performance that scientists who have already published the top of their field have accomplished. It is super hard to be a competitive scientist to publish in academia over the last 20, 30 years to make it a top of your discipline at a place like Berkeley. You were a star athlete. Like you, you are an athlete of the mind, and you perform at the highest levels. And to get there, whether you're, you know, Anastasius or Whelan at Berkeley, or you are Robin, who with Black Forest and created stable diffusion, or if you're like Guillaume at Meta, who created Llama, before you start Mistral, like the amount of human leadership you have to demonstrate to get the resources, get the trust of the organization, publish it, put it up. I mean, I would just fund researchers all day, right.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Who have contributed already to the field.

[Andrej Karpathy]

If they've put soda out there, they're star athletes already. If they haven't done soda, look, they can still be good CEOs, but then I find the failure mode is that they just don't want to be CEOs. They primarily want to publish, and that's okay too. You know, one of the things we do with the amp grid is we donate excess compute we have to nonprofits like university labs. We carved out like a couple thousand H1 hundreds. But I do think there's extraordinary research

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

being done on university campuses.

[Andrej Karpathy]

You know, my father in law is a physicist, he's a professor. Extraordinary work in physics, and we need that. But if you want to be a CEO, what you need to be willing to do is be super confrontational outside of science, like within the scientific community, some of the best researchers are very confrontational about their convictions. Right.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

This architecture is right.

[Andrej Karpathy]

To be a great CEO, you basically have to be willing to be confrontational

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

up and down the stack to your

[Andrej Karpathy]

own team, to your own team, hiring, recruiting customers. Well, I would say, yeah, pretty much to everybody.

[Anj Mitta]

I feel a little bit of that in my own work, but like, yeah, I can't imagine the stakes that Dario has had to go through.

[Andrej Karpathy]

It's, I don't think mistakes are that different from how you're feeling it. Right. Stakes are personal scaling vectors.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Right.

[Andrej Karpathy]

Like the stakes that seem so low to you, like having this podcast where you can talk to somebody and just have a conversation. I mean, you're an extraordinary communicator.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Right.

[Andrej Karpathy]

Like, already in this conversation, you've pulled more out of me than most people, you know, And I've been on 12 podcasts in the last two weeks.

[Anj Mitta]

I think we've just seen each other enough that there's some base trust. And I know that you, you know that I've done my homework and like, I, I know that trust is a big deal for you, so.

[Andrej Karpathy]

Right. Yes, I, I, I think trust is about consistency. And you and I have seen each other in the community for years.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Right.

[Andrej Karpathy]

Like, I remember the first time we met was at New UPS and New Orleans. I don't know if you remember that.

[Anj Mitta]

Oh, my God.

[Andrej Karpathy]

Reiko had set up this. You know, Reiko's amazing. And he set up this luncheon.

[Anj Mitta]

I was like, who's this Discord guy? I'm like, okay, no, you weren't. No, you made some investment.

[Andrej Karpathy]

You were much less polite. You were like, who's this vc?

[Anj Mitta]

No. Was I? Oh, my God, I'm so sorry.

[Andrej Karpathy]

It was visible on your face.

[Anj Mitta]

I'm so sorry. But you weren't. The introduction was bad. I was. I didn't know who you were.

[Andrej Karpathy]

See, this is the thing about context.

[Anj Mitta]

Right?

[Andrej Karpathy]

Like, but then I think I heard your accent.

[Anj Mitta]

Yeah.

[Andrej Karpathy]

And I was like, are you, Are you Singaporean? And you're like, yeah. And I said, I went to high school JC in Singapore. And then the ice broke. Right? But this is the. You know, there are, in the scientific community, sometimes the stakes are very high for people who haven't had the emotional. We know what is called EQ coaching and mentorship. Right. Which is like, to have scientific impact, you often need to be an extraordinary emotional, like, emotionally in tune person with the folks you're trying to influence. And so what comes so naturally to you is actually a super high stakes thing to other people. And so I wouldn't assume that Dario is more stressed out than you. These things are like, you'll be surprised how similar and small sometimes the problems are to you that some of the world's biggest leaders are facing. And that's what I've learned from this class. The guest speakers are Sam Satya Jensen.

[Anj Mitta]

AI Coachella.

[Andrej Karpathy]

Yeah, it's AI Coachella. Right? So we gotta get all the headliners. And I'm very lucky that some of these people have either mentored me over the years or I've done business with them. And when you take the performative stuff out and any assumptions you may have about these people that you read in the press or on Twitter, we're all just humans trying to get along. And what's so special about this moment is AI is forcing. Scaling the bitter lesson is forcing a lot of people to revise their assumptions for how the world works and go back to first principles or go and educate themselves. So the kind of people, you know, I won't name who this person is, but I was at an event last week in Texas and ran to somebody who said, anj, I came across the class. What do you think about Real time action prediction models. And I was, you know, I don't know how happy it made me feel when they asked me that question. I know they've done the work, they've challenged it. They didn't ask me, what do you think of world models? They said, what do you think of real time action prediction models? World models, don't get me wrong, are cool and everything, but you and I both know that that is a layer of abstraction that is sometimes not usefully precise enough. Yeah, right.

[Anj Mitta]

There's like four different kinds of world models.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Exactly.

[Anj Mitta]

We've done the part with general intuition, by the way, which is very focused on, oh, cool.

[Andrej Karpathy]

Yes, I love bim. PIM is great. And this is what I love about people who've done that level of work. They realize they're not in competition with people who the rest of the world thinks they're in competition with.

[Anj Mitta]

Yeah. Because they're not in the category they're in the specific thing they're trying to do.

[Andrej Karpathy]

They're focused on their mission and they have a systems understanding of the bottleneck they're trying to solve. And when somebody else says, I'm working on real time action prediction models too, PIM goes, oh, I love that person.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

I can learn from them.

[Andrej Karpathy]

But the minute they're like, oh, that person's a world model person, it's like, ah, like which type of world model person. Mostly they're just trying to figure out if it's a waste of their time because we don't have enough time. So, you know, pim, for example, is super. Loves this other company I work with

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

we've talked about called Black Forest Labs,

[Andrej Karpathy]

you know, and he's mentioned me multiple times that he's. So he thinks what Flux is doing is really cool, you know, Andy Blotman came by and spoke in the class. And what I find over and over again is for people who do the work, who can be usefully precise enough about what is actually going on in the world of frontier research. The sense of camaraderie is still well and alive, but it gets lost sometimes when you have to abstract the technical complexities in business terms. And then the VCs are like, how are you different from that world model coming? Say, where do I even start to explain this stuff? And then the misalignment.

[Anj Mitta]

Yeah, I think people listening get a sense of what it is like to operate at a real level like yourself, rather than at the journalist level, where you have to sort of put everyone in a rough category and create a narrative of competition. And who's winning today? Who's behind?

[Andrej Karpathy]

Yeah, yeah. This idea of winning is so weird to me.

[Anj Mitta]

You do want to win. You want competitiveness.

[Andrej Karpathy]

No, I think you want to lead.

[Anj Mitta]

You want soda?

[Andrej Karpathy]

No, I think you want to lead. Yes. So you want to push the frontier. You want to push the state of the art. You want to do something that hasn't been done before. You want to capture value, but you don't want to capture so much value that, like, people think you're unaligned with your mission or trying to do what's best for the world. You want to capture enough value that you can keep innovating. Right. And I think that people want to lead. They don't really. This idea of winning and losing, like, you know, again, I love Jensen. He's a leader. The mindset that he talked about on Dwarkesh's podcast, right. He was like, I didn't wake up with a loser mindset. I think that was awesome. Right. Because he's an engineer. Dwarkesh has done the work. So there's at least. Even though to me it was very obvious they're talking about the same thing. They just passed each other, basically. Jensen has this five layer cake abstraction of how the industry works. And Dwarkesh, I think from that podcast had more of like a pre training, mid training, post training, systems loop concept.

[Anj Mitta]

It's just a factor of who he talks to. Right. It's very clear.

[Andrej Karpathy]

It's the abstraction, the mental models, the. It's the whole dude. So much of the problem in the world is reasoning by analogy and then the assumptions that are held invisibly.

[Anj Mitta]

Yeah. I've said, like, this is actually the best time in human history for first principles thinkers. Because everything you think will happen is actually now coming true.

[Andrej Karpathy]

Correct. And the venture capital community is, like, notorious for this. Where people look in times of uncertainty, they cling to axioms that ended up being true from the previous era. And they. They kind of like proclaim them with confidence as if they're truths, but they're not. And it's very important to see the distinction between a heuristic and an axiom.

[Anj Mitta]

Yeah.

[Andrej Karpathy]

An axiom can be proven, like, from internal consistency. Internal consistency. A heuristic is a way you kind of a shortcut. And my God, the number of people I have had to put up with over the last few years who proclaim, like, use heuristics as axioms to judge people, to judge which companies are going to succeed. I mean, the number of people were like, oh, yeah, yeah, yeah. Anthropic, they're just training models right now. But this one continued, they go to B2, B SaaS. Yeah, like which? Which over the fullness of time, if you squint at it, maybe. But the way you arrive there is so important that you can dismiss people. Here's what happened. Right. What happened is Anthropic basically achieved takeoff in October of last year. That training run, whatever 3.7, I forget

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

the numbers now, but whatever that checkpoint

[Anj Mitta]

was, we saw it accomplish it.

[Andrej Karpathy]

Yeah, right. You probably to those of us in the community, especially once post training was done and it was released in December.

[Anj Mitta]

Yeah. Can I sneak a sneaky question in there? I don't know if you have a perspective. Maybe you don't. I just. The number one question is how did anthropic crack coding? Right. Because Claude one, Claude two. Okay, it was part of it, but it wasn't a big deal. And the leading hypothesis, it's a lucky dice roll that was then compounded. Right. It was mildly better, but then they saw it and they were like, okay, let's really invest.

[Andrej Karpathy]

I had this very annoying teacher. I went to this boarding school called Rishi Valley in India, which is like this bird preserve, like 350 acres of, of bird preserve in, in rural India. And there was no technology for seven years. There was this teacher. I won't name them, but they would have this. I hated it every time he said this to me. He was like, luck favors the prepared mind. Which is like a common saying, but the way he delivered it, like always grade me. Cuz he was always trying. Like I was always one of those kids who got like a good grade without trying very hard. Because like high school, middle school is not that hard if you're generally like paying attention and so on. And there was this one time where. But then I would get an 80% grade and he would keep pushing me to say like the reason you didn't get the 95 percent is because you're not that lucky. And I would say, what do you mean? Because I would think that I deserve that grade. And I would sometimes argue with him and he'd say you didn't have a prepared mind if you want to get lucky again. There was basically one time where I got like 95 or 96 on this subject. And now that I felt entitled, I was like, okay, I'm going to keep doing this.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

And I didn't.

[Andrej Karpathy]

And then he was like, luck favors a prepared man. You got lucky last time, but you got to stay prepared. And I Didn't understand what he meant. Now, as I'm older, I'm like, okay, these adults actually knew a thing or two. Anthropic has been the most prepared company for four years. And so then when the right, like, context data comes in, the right developers start sending in the right context diffs. Sure, you could say you got lucky, but if you ask me, they're pretty damn prepared with paranoia for, like, four years. And you have to remember it was so hard for them to get going early on that they had to do so much more with so much less that you just have to be prepared to be so efficient.

[Anj Mitta]

Yes, there's numbers on their burn compared to OpenAI. I've written about it. But they are so much more efficient. Not even close.

[Andrej Karpathy]

Yeah, but it's so clear, right? Like, how to output max for the world. They have been prepared. And you could call that luck, but luck favors the prepared mind.

[Anj Mitta]

This is one of those things that I was going over some of your old lectures, and you were like, data people think it's a moat, and actually it's culture, and actually it's team. There's different levels of moats, and this is the ultimate one that determines everything else, which you can then compound.

[Andrej Karpathy]

You're saying culture is the ultimate mode. Yeah, but the thing about culture is it's very fragile. So moats, I don't think. There's very few moats I found that are actually moats. It's a nice concept, but in reality, you have to replenish your culture. Ben Horowitz was the speaker in CS153 on Tuesday, and I asked him this question about the culture bottleneck in teams, because, you know, there are several AI teams.

[Anj Mitta]

His book, like, Hard Thing About Hard Things.

[Andrej Karpathy]

Hard Thing About Hard Things. But more concretely, there are so many AI labs today that have all the cash they need, they have all the compute they need, and they're still not able to ship anything. Soda. And then you start seeing people leave and so on. And my diagnosis is it's the culture. And so I asked him, Ben, you know, he's been one of the most aggressive investors in AI labs. He goes back to this thing, which resonates in my mind a lot. When I used to work at A16Z, I would book a conference room, and right outside the conference room, which is closest to the toilet, because it was the fastest way for me to go use the bathroom between Zoom meetings.

[Anj Mitta]

Oh, my God. Output maxing by toilet optimization.

[Andrej Karpathy]

It was not healthy in hindsight, but maybe this is tmi. But anyway, outside that conference on the wall was this quote that was printed that said culture is not a set of beliefs, it's a set of actions. And it's by Bushido is a Japanese philosopher. And if you stop taking the actions that demonstrate the mission alignment to what you've said to your team and to the world matters to you, then your culture starts to fray. So it's not actually a moat, I would say it's a very, very brittle, fragile thing that requires daily tending to like a garden. But if you figure out the system to keep that garden tended, which I think ultimately comes down to knowing yourself because you most naturally, if you're authentic and so on, you'll naturally make trade offs that seem effortless to you, but that reinforce your culture and then that becomes this very hard thing for other people to catch up to. And that anthropic. From day one there was this missionary zeal and belief that hey, these capabilities will scale. These systems are stochastic, not deterministic. There will be error bars and until we crack interpretability, there's risk. And at some point people will stop using Claude just for coding. They'll use it in some mission critical context where it'll throw off a bug and then people are going to come blame them. And they want to be on the

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

right side of history where they said

[Andrej Karpathy]

yes, this is a powerful technology, we think it's going to change the world. And we want to be very measured and scientific about the fact that hey guys, these are stats, models, statistical models. That's how statistics works. Ultimately when you're training neural nets, it is just a statistical system. I think that that belief that safety is important and that it might seem toy like in the early days and sometimes you could say ansh, they totally over exaggerated the risk like two years

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

ago when they said let's not launch Claude 1 or whatever.

[Andrej Karpathy]

Well, okay, maybe in hindsight, but hindsight is 2020 and at the time they didn't know how that model would be used. And to them it felt existential. If somebody came and said you weren't responsible, this wrote a bug. The liability associated with that is massive. So how do you prevent against that? Well, day in, day out, you say safety, safety, safety, safety. And when you start deviating from that, you have the team hold you accountable, you have the world hold you accountable. And I think that becomes a moat over time. At some point that moat will get challenged and so on, and then it become fragile. I hope it endures because that's the beauty of Having founders run the show because they can make really hard trade offs to do mission alignment. The hardest part is in the earliest days when you don't have a group of people who are going through difficulty, stress, crisis together, then your culture doesn't get defined sharply enough. And that's what I'm worried about right now is there's so much money going to these labs. There's no hardship. There's no 21 knows. There's no 21 knows. And that in hindsight was a feature, not a bug for anthropic. The number of people who said no, the number of people who said, sorry, we're all investors in OpenAI. That is competitive difference. It forces you to really understand what is the hill you want to die on at the expense of everything else. What's the P0? And there P0 from day one was coding the reason mechanism system. There was. If we crack coding, then we will crack AGI. Our mission is AGI. We want to get there safely. If we focus on coding, it's such a generally powerful capability that it can accelerate all kinds of work on a computer. And if we can accelerate all kinds

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

of work on a computer, we can get to AGI.

[Andrej Karpathy]

You know, as a result, they've had to say no to so much other stuff here. Superconductivity is the mission, coding is not the mission. So we use Claude, we'll use Claude. We don't care about that. The mission defines everything. And I think teams who can raise too much money too fast, too early, who don't have to define what the P0 is because that's the only thing when you have scarce resources, you got to. Yeah, you got to invest in those

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

cultures end up being the most fragile and brittle and they almost don't even make it to take off.

[Anj Mitta]

So let's apply this to periodic since we're here.

[Andrej Karpathy]

Sure.

[Anj Mitta]

What is the constraint or the hardship that they were forcing themselves to go through?

[Andrej Karpathy]

Dude, here. Are you crazy?

[Anj Mitta]

No.

[Andrej Karpathy]

Well, yeah. Okay. So on a technical level, it's physics. It's literally reality.

[Anj Mitta]

I mean is there another one that's like the company building?

[Andrej Karpathy]

Yeah, when I mean Liam was a co creator of ChatGPT and Doge was skip level from Demis at DeepMind had created genomes, one of the most important tools to come out of DeepMind. Now at the time I was a visiting scientist at the Stanford physics department and we had started benchmarking frontier models on physics and science capabilities and they

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

were not very good.

[Andrej Karpathy]

They're good at doing things like Summarization of papers. But if you said, hey, could you analyze the scientific data coming out of a condensed matter physics lab?

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

I was in the condensed matter physics group at Stanford. It was terrible.

[Andrej Karpathy]

So it was not popular. 12 months ago periodical. And I won't go into details, but there were people who said as recently as a few months ago, who said they wanted to join the company and they, for whatever reason, took a job elsewhere. They kind of reneged on their commitments. They took a job elsewhere that offered more money. Then we had a technical breakthrough, created a soda system. And like, okay, I decided to cover it.

[Anj Mitta]

We'll be doing a separate part on periodic.

[Andrej Karpathy]

And then they wanted to come back. And I said, no, no way. If you come here, you had your shot. You had your shot.

[Anj Mitta]

Because it's actually about culture, of course, first principles. Yeah.

[Andrej Karpathy]

You know, and look, I believe in second chances and so on, but time will need to heal some of those wounds were.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

They will leave deep, deep for them, will leave deep scars.

[Andrej Karpathy]

But because I started my company at 24, 25, I went through the whole cycle of betrayal and drama. And so you realize, you know, Silicon Valley is both a very missionary place, it's also a very mercenary place. Sometimes people lose their minds

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

when big

[Andrej Karpathy]

money gets involved, which is in the grand scheme of things, quite small money. Like, you know, I guess you're taking

[Anj Mitta]

life changing to me, maybe less to you. But, you know, like, a lot of people have not been taught how to deal with money. And yeah, we didn't come up from like, that privilege of a background.

[Andrej Karpathy]

I'm a street dog, man. Yeah, look, I grew up in Rishi Valley. We didn't have like, this was enforced brutalism. Jiddu Krishna with you started, the school was like, you will sleep on a hard slab of stone. Like, my mattress was this thin, you know, I mean, you grew up in Singapore. When I got to Singapore, I used to sleep. I was part of the scholarship program. But which. Which was amazing. I'm very grateful to the Singaporean government, but I was at St. Andrews JC and our dorm, which was by Boon Kang mrt, which is not a prestigious neighborhood, it was a transition dorm because you're building this beautiful residential campus on site at SAJC in Potong Pasir. But we were the last, I think the second last batch to be in the transition site, which was some old. I think it was like an immigrant labor.

[Anj Mitta]

Yes. That's where we keep the people who work on the factories and stuff. Right.

[Andrej Karpathy]

So I lived in A. My 11th and 12th grade I slept in a bedroom the size of this. Like, literally from from there to here. Yeah, right. They were like bunk beds. And so one bunk bed here, one bunk bed there. One on top, one up, one more here. And then here was where our, like, we kept our toiletries and clothes and stuff. And when one guy would climb onto his bed there, this one would shake.

[Anj Mitta]

Oh, my God.

[Andrej Karpathy]

And one of my roommates who was from. And it was amazing. I loved every minute of it. You know, my roommates were a guy who was a top ranked dota player from prc, from China, didn't speak a lingua English.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Loved him.

[Andrej Karpathy]

Amazing guy.

[Anj Mitta]

I mean, all the Singapore scholars are fantastic. And honestly, we should treat you guys better because of what you go on to do, but cool to know.

[Andrej Karpathy]

What I'm saying is I don't need much to be happy in life. You know, when you've lived through that, money is a way, I think sometimes we measure ourselves. But you know, when it's. When it stops becoming, you know, war. Goodhart's law. When it stops becoming just a byproduct and more of a measure, it stops having meaning.

[Anj Mitta]

You use it to do more meaningful things. Resources to pursue your missions. I've kept you longer than I am supposed to, but we should continue this part two. I really enjoy this. Yeah. I mean, you're so inspirational. And yeah, there's more I want to dig into about how you've set everything up, every single one of your investments, how amp is going. But we're running out of time for that. But thank you so much for joining us.

[Andrej Karpathy]

It was great to see you, man. Let's get chicken rice sometime.

[Anj Mitta]

Yes, actually, tomorrow. I'll send you details.

[Andrej Karpathy]

Okay.

[Anj Mitta]

I'm hosting a birthday party.

[Andrej Karpathy]

I don't get an invite.

[Anj Mitta]

It has to be a Singaporean birthday party. Yes, you're getting an invite right now.

[Co-founder / Technical Lead (possibly Mihaly or Seb)]

Okay, perfect. All right.

[Anj Mitta]

Thank you.

[Andrej Karpathy]

All right. Thanks, man.