← Neural intel Pod
Neural intel Pod
200 Gigawatts or Bust: Dylan Patel on the Engineering Reality of AGI Scaling
00:52:50
Loading summary
Transcript439 lines
- [00:00]
A
Big Tech is forecasting $600 billion in capital expenditures this year alone, which is
- [00:07]
B
just an astronomical number.
- [00:09]
A
Right. And if you translate that financial figure into actual physical power, it equates to a mind bending 50 gigawatts of compute a 50 gigawatt. Then you look at leaders like Sam Altman and Elon Musk and they are publicly calling for 50 to 100 gigawatts
- [00:25]
B
a year by 2030, which is completely wild.
- [00:29]
A
It sets up this massive high speed collision with physical reality. Welcome back to the Neural intel podcast. Let's dive into today's topic.
- [00:38]
B
Glad to be here for this one.
- [00:39]
A
As always, we'll focus on the technical details and implications of the technology we discuss. To stay updated on the latest in AI and ML, visit our blog@ NeuralIntel.org and check us out on YouTube, Apple Podcasts and Spotify.
- [00:52]
B
Definitely go check those out.
- [00:53]
A
So before diving in, let's list the hook, the problem and the solution for today's deep dive. The hook is that staggering 100 gigawatt demand curve from the leading AI labs,
- [01:04]
B
which again is just defying physics at this point.
- [01:07]
A
Right. And the problem is that most MLOps engineers, technical founders and infrastructure architects out there assume the ultimate bottleneck to scaling AI is either, you know, power grid capacity or data center permitting.
- [01:20]
B
Yeah, that's the standard narrative you hear everywhere.
- [01:23]
A
Exactly. And that assumption is driving massive, potentially misguided strateg bets across the entire industry.
- [01:29]
B
Huge bets. It might just completely miss the mark.
- [01:32]
A
So the solution here we are taking a deep dive into an extensive breakdown of the semiconductor supply chain, drawing from Dylan Patel's recent analysis on the DWRkish podcast.
- [01:41]
B
It was a fantastic interview, really eye opening stuff.
- [01:44]
A
We are going to deconstruct the math behind EUV lithography, unravel the complexities of scale up topologies and reveal what is actually going to choke AI scaling by the end of the decade.
- [01:55]
B
Because that assumption about the power grid being the primary bottleneck. Yeah, I mean that is exactly what we need to dismantle today.
- [02:02]
A
We really do.
- [02:03]
B
It's leading brilliant people to build orchestration layers and like memory architectures that are optimized for constraints that just won't actually be the primary choke points in a few years.
- [02:14]
A
Which brings us to our core mission for this deep dive. We want to equip you, the architects, the researchers and the strategic CTOs listening with the foundational physics and supply chain realities of AI hardware.
- [02:27]
B
Right. Because if you understand these physical limits, you can actually future proof your orchestration
- [02:32]
A
layers and Your long term memory architectures. Exactly. So let's get into the financials driving this. Before we look at the silicon itself, we have to look at the capital moving the market.
- [02:43]
B
The sheer volume of money is insane.
- [02:45]
A
It really is. And more importantly, we need to look at the radically different deployment strategies of the top AI labs. How they are acquiring compute right now is fundamentally dictating their survival architecture.
- [02:58]
B
I mean the financial scale here is genuinely unprecedented in modern industrial history. When you see Google, Amazon, Meta and Microsoft Deploying this 600 billion, it's crucial to understand that a massive chunk of that isn't buying servers to plug in today.
- [03:13]
A
Right. They aren't just racking up servers right now.
- [03:15]
B
No, they are deploying setup capex for 2027 and 2028 data centers.
- [03:20]
A
That far out?
- [03:22]
B
Yeah, we're talking about laying out non refundable deposits on.
- [03:26]
A
They looked at the landscape and effectively locked in massive 5 year compute contracts right out of the gate.
- [03:32]
B
They didn't even hesitate.
- [03:34]
A
No, they didn't just partner with Microsoft. They absorbed capacity from neo clouds like Core, Weave, Oracle and even Softbank Energy.
- [03:42]
B
Right. They prioritized long term stability and absolute volume over, you know, near term margin protection.
- [03:49]
A
Makes sense if you want to lock up the market.
- [03:51]
B
Anthropic conversely played a highly conservative game. Their leadership looked at the projected inference revenue and thought, well, if our revenue doesn't inflect at the exact parabolic rate we project, we're in trouble.
- [04:04]
A
Because of the multi year deals.
- [04:05]
B
Exactly. If they sign all these multi billion dollar multi year compute deals, the sheer carrying cost of the hardware will just bankrupt them.
- [04:13]
A
So they held back.
- [04:14]
B
Yeah, so they purposely undershot their compute commitments to maintain a leaner balance sheet,
- [04:19]
A
which makes logical sense for a startup. I mean until you look at what actually happened to their revenue, which is explosive. Anthropic's revenue exploded exponentially faster than their internal models predicted. Based on the recent additions of 4 billion and 6 billion in monthly revenue. If you just draw a straight line, they are looking at adding a 60 billion revenue run rate over a 10 month period.
- [04:44]
B
It's just staggering growth.
- [04:45]
A
Right. And to service that massive influx of inference demand, they suddenly need billions of dollars in compute that they simply didn't reserve.
- [04:54]
B
You got caught totally flat footed.
- [04:56]
A
It's like a startup booking a block of hotel rooms for a conference OpenAI bought out the entire hotel a year in advance at a negotiated discount.
- [05:04]
B
Smart move.
- [05:05]
A
And Anthropic decided to wait and see if anyone would actually RSVP. Well, the RSVP's flooded in and now Anthropic is forced to rent the penthouse at a premium daily rate just to fit their guests.
- [05:17]
B
That's a great way to put it.
- [05:18]
A
Yeah.
- [05:18]
B
And the spot market dynamics they are facing right now are brutal.
- [05:21]
A
I can imagine.
- [05:22]
B
Because Anthropic is desperate for compute to serve this exploding revenue, they can't rely on those amortized long term deals.
- [05:28]
A
They just don't have them.
- [05:29]
B
Right. They are forced to buy Spot compute from neo clouds or enter into heavily skewed revenue share deals with cloud providers like Amazon Bedrock or Google Cloud.
- [05:41]
A
Wow.
- [05:42]
B
And they're paying up to a 50% premium for this last minute capacity.
- [05:46]
A
The unit economics on that are wild. An H100 chip that historically costs about $1.40 per hour to deploy when amortized over a standard five year contract is now being rented out for $2.40 an hour or more on these short term
- [06:02]
B
desperate deals, which is just pure unadulterated profit margin for the infrastructure providers who actually had the foresight and the capital to build that flex capacity.
- [06:11]
A
They are making a killing.
- [06:12]
B
Absolutely.
- [06:13]
A
Yeah.
- [06:13]
B
But this localized panic actually highlights a broader fascinating economic principle that that is reshaping the entire AI landscape.
- [06:20]
A
Oh really?
- [06:21]
B
Yeah. It's called the Alkene Aller effect.
- [06:23]
A
Let's break down the math on the Alkene Allen effect because it perfectly explains why the gap between frontier models and open weight models is behaving so weirdly right now.
- [06:31]
B
The Alkene Allen effect essentially dictates that if you add a fixed uniform cost to two different grades of a product, the relative price difference between them actually shrinks.
- [06:41]
A
Okay, so the gap narrows.
- [06:43]
B
Right. And this ironically drives consumers toward the higher quality option. The classic microeconomics example is Apple's. But let's map it directly to cloud compute.
- [06:53]
A
Let's do it.
- [06:53]
B
Imagine you have a standard tier instance running an open weight 8 billion parameter model and it costs you $1 an hour. Then you have a premium instance running a massive proprietary frontier model and it costs you $2 an hour.
- [07:08]
A
So the premium model is exactly twice as expensive.
- [07:11]
B
Exactly. Now let's say the baseline cost of power cooling and the physical silicon goes up by a flat $1 across the board due to supp.
- [07:20]
A
Right. So the standard instance jumps from $1 to $2 and the premium instance jumps from $2 to $3.
- [07:26]
B
Precisely. The premium model is no longer twice as expensive. It is now only 1.5 times as expensive. The relative gap has narrowed significantly.
- [07:36]
A
So if I'm a CTO building a heavy orchestration layer I look at that pricing and think, well, if I'm already forced to pay $2 an hour just to run the mediocre model, I might as well eat the extra dollar and get the absolute smartest model available.
- [07:49]
B
Exactly. The fixed cost inflation effectively subsidizes the upgrade.
- [07:54]
A
That's fascinating.
- [07:55]
B
It fundamentally concentrates power and revenue at the very top of the model capability tier.
- [08:00]
A
Because why pay for mid tier?
- [08:02]
B
Right? If a lab is going to pay a fortune for the underlying hardware anyway? They aren't going to waste those precious flops serving a mid tier model. They're going to allocate every available cycle to the model that commands the highest token premium.
- [08:15]
A
Which explains the absolute bloodbath we are seeing among hardware buyers fighting for every single Hopper and Blackwell GPU hitting the loading dock.
- [08:24]
B
It's a feeding frenzy.
- [08:25]
A
But this insatiable demand actually contradicts a very prominent, very loud narrative in the financial sector.
- [08:31]
B
Oh, you mean the depreciation argument.
- [08:32]
A
Yeah. You have high profile investors like Michael Burry arguing that a GPU's depreciation cycle is incredibly short. Maybe three years or less.
- [08:42]
B
Right, that's what they say.
- [08:43]
A
The financial argument is that because Nvidia releases an architecture that is roughly three times faster every two years, the older chips become economically worthless very quickly and therefore these massive capex bets are going to result in massive write downs.
- [08:58]
B
Yeah. That theory is a fundamental misunderstanding of the relationship between hardware and the software stack running on top of it.
- [09:05]
A
How so?
- [09:05]
B
It looks at the hardware in the total vacuum. If we existed in a reality where we could spin up an infinite number of the newest Blackwell or Rubin chips tomorrow, then yes, the value of an older hopper chip would plummet.
- [09:18]
A
But we don't live in that reality.
- [09:20]
B
Exactly. We exist in a severely structurally compute constrained world. Therefore, the value of a chip is not based on its theoretical benchmark specs compared to a newer chip. It is based entirely on the economic value of the tokens it can physically produce today.
- [09:35]
A
Let's ground that in the actual model architectures. If we look at the leap from GPT4 to something like a GPT5 class model, the newer model isn't just a brute force parameter scale up.
- [09:46]
B
No, not at all.
- [09:47]
A
It's actually vastly more efficient at the inference level because of aggressive sparse mixture of experts or MOE architectures.
- [09:54]
B
Yes, and that sparsity is the key. In a dense model, every parameter is activated for every single token generated, which is incredibly heavy. It's horribly inefficient.
- [10:04]
A
Yeah.
- [10:05]
B
With advanced MOE routing, a massive 5 trillion parameter model might only activate 100 billion parameters for any given token.
- [10:13]
A
Oh, wow.
- [10:13]
B
Add in better training data, speculative decoding, and improved reinforcement learning, and the newer frontier models require far fewer active operations per token than the older generation did.
- [10:23]
A
So ironically, the vastly smarter model is actually a much lighter algorithmic lift for the hardware.
- [10:29]
B
Exactly. This means you can serve significantly more tokens per second of a hyper efficient GPT5 class model on an older H100 GPU than you could when running the clunky original GPT4 on that exact same chip.
- [10:41]
A
That is wild to think about.
- [10:43]
B
And because the new model is vastly more capable, like it's able to write complex code, act as an autonomous agent, solve advanced math, the total addressable market, and the economic value of those tokens is exponentially higher.
- [10:57]
A
So an H100 today, running a state of the art sparse architecture, is actually extracting vastly more economic utility and raw revenue than that exact same physical piece of silicon did three years ago.
- [11:10]
B
Which is counterintuitive to most hardware analysts.
- [11:12]
A
The economic lifespan of the GPU is extending, not shrinking, because the software keeps unlocking more value from the fixed hardware.
- [11:19]
B
It completely flips the traditional hardware depreciation curve on its head.
- [11:23]
A
That is a critical insight for anyone modeling infrastructure costs. So knowing that these labs are paying an absolute premium for these chips, and knowing that the hardware holds its economic value, we have to open the box,
- [11:35]
B
we have to look at the silicon.
- [11:36]
A
We need to look at what's actually inside these GPUs that cost so much money and is causing such a massive bottleneck. It turns out 30% of Big Tech's 2026 capital expenditure isn't going to the logic cores. It's going to a single component, High
- [11:52]
B
bandwidth memory, or hbm.
- [11:53]
A
Hbm.
- [11:54]
B
It is completely consuming the global semiconductor supply chain, and it's doing so at the direct expense of everything else.
- [12:01]
A
And it's having a devastating, mathematically verifiable knock on effect for everyday consumers. Let's look at the physics of manufacturing HBM versus the regular DRAM we have in our laptops. Because I think a lot of people just assume memory is memory.
- [12:15]
B
It's really not.
- [12:16]
A
Why is HBM causing such a brutal supply shock?
- [12:19]
B
It comes down to wafer yield and the sheer physical complexity of 2.5D in 3D packaging.
- [12:25]
A
Okay.
- [12:25]
B
Producing standard commodity dram like the DDR5 in your workstation is a relatively straightforward planar process. Producing high bandwidth memory is an entirely different beast.
- [12:35]
A
How so?
- [12:36]
B
You aren't just printing a single layer of memory cells. You are taking Multiple incredibly thin layers of dram, sometimes 812 or soon 16 layers deep, and stacking them perfectly on top of each other.
- [12:49]
A
And to get those layers to talk to each other, you have to drill microscopic holes straight through the silicon, right?
- [12:54]
B
Yes. Those are called through silicon vias or TSVs.
- [12:57]
A
TSVs, got it.
- [12:58]
B
You have thousands of these microscopic vertical wires passing through the silicon die to connect the stack. If even a handful of those TSVs are misaligned or defective, the entire multi layer stack is essentially garbage.
- [13:11]
A
So the failure rate must be high.
- [13:14]
B
Because of the thermal constraints, the packaging complexity and the defect rates, producing HBM yields roughly three to four times fewer usable bits of memory per area of a silicon wafer than standard dnap.
- [13:26]
A
So for every gigabyte of hbm, a foundry successfully manufactures for an AI accelerator. They are actively sacrificing three to four gigabytes of regular memory that could have gone into a consumer device.
- [13:38]
B
Exactly. It's a zero sum game on the fabrication floor. The big three memory manufacturers, SK Hynix, Samsung and Micron, are actively diverting all of their advanced fabrication capacity toward hbm
- [13:50]
A
because the margins are better.
- [13:51]
B
Because the AI labs are locked in a death match and are willing to pay massive upfront premiums on long term contracts to feed the data center. The foundries are actively destroying consumer memory supply.
- [14:01]
A
And the math on this is already hitting the consumer market hard. Look at Apple's supply chain projections.
- [14:07]
B
That's brutal.
- [14:08]
A
The bill of materials for an upcoming iPhone is projected to jump by $150 purely due to memory price increases for the memory. Now Apple operates at the premium tier. They have the margins to absorb some of that. Or they simply pass it to a consumer who is already willing to pay $1,200 for a phone, right?
- [14:26]
B
Apple will be fine.
- [14:27]
A
But the low end and mid range smartphone market, the device is meant for emerging markets. The that entire sector is going to get obliterated, completely wiped out. We are looking at global volumes dropping from 1.4 billion units a year down to maybe 600 million units. Hundreds of millions of affordable smartphones simply won't be manufactured because the memory required to build them is sitting inside a data center in Texas running an LLM.
- [14:57]
B
It is a massive structural reallocation of global silicon resources from the edge consumer directly to the centralized AI data center.
- [15:05]
A
Which brings us to a crucial point for our audience. Neural signal check. Here is why this development actually matters at a technical level.
- [15:13]
B
Yes, let's get into the technicals.
- [15:15]
A
If you are a researcher focused on continual learning or an infrastructure engineer looking at long term memory for agentic workflows. You are heavily bottlenecked by the KVCache.
- [15:26]
B
Always the KVCache.
- [15:27]
A
The reason we can't just slap cheap DDR memory onto an accelerator for asynchronous background agents is entirely governed by memory bandwidth.
- [15:35]
B
And this is a critical architectural reality that often gets lost in the software layer.
- [15:40]
A
Oh, for sure.
- [15:40]
B
The software engineers assume they can just optimize their way out of it, but physics always wins.
- [15:46]
A
So let me pose the question that I know some of the infrastructure engineers are screaming at their dashboards right now.
- [15:51]
B
Let's hear it.
- [15:51]
A
If we are running background agents, let's say we are building a clogged slow mode.
- [15:55]
B
Okay, Claude, slow mode.
- [15:56]
A
This is an agent quietly writing thousands of lines of code or reviewing massive legal documents in the background. Latency does not matter. The user doesn't need the tokens instantly.
- [16:07]
B
Right? It's completely asynchronous.
- [16:09]
A
Why can't we just build an AI accelerator that uses regular cheap DDR memory instead of this insanely expensive hpm? If we don't care how fast the tokens come out, why are we paying for the bandwidth?
- [16:21]
B
It's a brilliant question from a pure software optimization standpoint, but it completely ignores the physical reality of chip design and the economics of the silicon itself.
- [16:30]
A
Okay, tell me why it all comes
- [16:32]
B
down to the physics of the chip's shoreline.
- [16:35]
A
The shoreline. Let's define that clearly.
- [16:36]
B
The shoreline is the physical literal edge of the silicon logic die. When you design an AI accelerator, all the data going in and out of the compute cores, the model weights, the activations, the massive KV cache has to physically travel across the perimeter of that die to reach the memory.
- [16:54]
A
Like crossing a border.
- [16:55]
B
Exactly. And you only have so many millimeters of perimeter to work with. Let's look at the math for Nvidia's upcoming Rubin architecture.
- [17:02]
A
Let's do it.
- [17:03]
B
Rubin's going to utilize an HBM4 stack. That HBM4 stack connects to the logic die using a shoreline interface that is roughly 13 millimeters wide.
- [17:13]
A
13 millimeters. That is tiny.
- [17:15]
B
It is in that tiny 13 millimeter physical space. Using a massive 1024 bit wide interface via an advanced silicon interposer, HPM4 can push an astonishing 2.5 terabytes of data per second.
- [17:28]
A
Okay, 2.5 terabytes per second flowing through a 13 millimeter gate.
- [17:32]
B
Yes. Now imagine you try to implement Claude slow mode. You rip out that HBM4 and you try to attach standard DDR5 memory to that exact same 13 mm shoreline.
- [17:43]
A
Okay, what happens?
- [17:44]
B
Because DDR memory relies on a much narrower interface, typically 64 bits per channel, and fundamentally different physical pins and transfer protocols, it cannot utilize that space the same way.
- [17:55]
A
It's physically incompatible for that density right
- [17:58]
B
in that same 13 millimeter perimeter, DDR5 can only push about 64 to 128 gigabytes per second.
- [18:04]
A
Wow. So we aren't talking about a 20% or 30% drop in speed. That is an order of magnitude difference.
- [18:11]
B
It's a catastrophic degradation of bandwidth. And here is why the cheap memory idea fails. Mathematically, the compute cores inside the gpu, the raw flops, are incredibly fast and incredibly expensive to manufacture.
- [18:27]
A
Right.
- [18:27]
B
If you throttle the memory bandwidth down to 128 gigabytes a second, those massive logic cores just sit there completely idle.
- [18:35]
A
They're just waiting.
- [18:36]
B
They're spinning their wheels, burning power, waiting for the massive model weights in the KV cache context to slowly trickle in from memory. You aren't just making the models slower. You are utterly wasting the most expensive part of the chip, the logic die.
- [18:49]
A
You'd be bottlenecking the most expensive component.
- [18:51]
B
You would be utilizing maybe 1% or 2% of the GPU's actual compute capacity. The economics of silicon demand that you keep those multimillion dollar compute cores fed with data every microsecond.
- [19:03]
A
It makes total sense.
- [19:05]
B
The only physical way to push enough data through a 13 millimeter shoreline to keep the cores fed is HBM.
- [19:11]
A
That clarifies it perfectly. The bandwidth isn't a luxury feature for generating tokens faster. It is a physical prerequisite to keep the logic cores from starving to death.
- [19:21]
B
Exactly. You're preventing starvation.
- [19:23]
A
So we've solved the memory bandwidth on the individual chip level. But here is where it gets really interesting. Because a frontier model, a multi trillion parameter behemoth, does not fit on a single chip.
- [19:36]
B
No, it takes an entire cluster.
- [19:37]
A
It spans hundreds of, sometimes thousands of chips simultaneously. And how you physically connect those chips dictates whether your software architecture actually works or whether it grinds to a halt.
- [19:49]
B
This brings us into the realm of topologies and scale up domains. It's all about the speed of data movement across physical space.
- [19:56]
A
Right.
- [19:57]
B
We just established that moving data on the chip from the HBM to the core happens at tens of terabytes per second. Moving data between chips within the same physical server rack. The scale up domain drops to single digit terabytes per second.
- [20:10]
A
Okay, so a step down, and the
- [20:12]
B
second you have to move data across different racks using optical switches, your bandwidth plummets down to hundreds of gigabytes per second. Every single time your data crosses a physical boundary, you pay a massive tax in latency, power consumption and complexity.
- [20:28]
A
And this physical boundary tax explains a massive performance gap we are seeing in the wild right now.
- [20:33]
B
The Blackwell numbers.
- [20:34]
A
Yeah. If you look at inference performance on heavily optimized open source models like Deepsea's Mixture of Experts or Kimi K2.5, and you run them on Nvidia's new Blackwell architecture versus the older Hopper architecture, the performance delta is insane.
- [20:50]
B
It's night and day.
- [20:51]
A
We are seeing roughly a 20x improvement in real world inference speed. But the actual Blackwell logic die only has about 3x the raw flops of hopper. So where is that 20x multiplier magically coming from?
- [21:04]
B
It's entirely derived from the network topology. It's how the chips physically talk to each other.
- [21:08]
A
Explain that.
- [21:08]
B
With the older hopper architecture, the scale up domain, meaning the group of chips that could talk to each other directly at maximum NV switch speed was generally limited to the eight GPUs sitting inside a single physical server chassis.
- [21:21]
A
So a pod of eight.
- [21:23]
B
Right. The moment your model needed nine GPUs, you had to jump across a slower network. With Blackwell, Nvidia introduced the NV072 rack design.
- [21:32]
A
Right, the giant rack.
- [21:33]
B
They took 72 discrete GPUs and physically wired them together with a massive 2 ton copper backplane using an ALTA all topology.
- [21:43]
A
Let's expand on this ALTA all concept because it's vital for understanding inference routing.
- [21:47]
B
All to all means exactly what it sounds like. Through the NV switch hierarchy. On that copper backplane, GPU one can send terabytes of data per second directly to GPU 72 without having to bounce that data through any intermediary chips in the rack.
- [22:01]
A
It's totally direct.
- [22:03]
B
It has a direct unmitigated peer to peer connection across all 72 chips.
- [22:07]
A
Okay, let me use an analogy here to visualize the data Mechanism. The Nvidia NVL72 all to all setup is like a highly chaotic, highly efficient corporate boardroom. You have 72 executives sitting around a giant circular table. Everyone can look directly at anyone else and yell an instruction across the room instantly.
- [22:26]
B
That's a great visual if you need
- [22:28]
A
to rapidly brainstorm and answer a million different user prompts simultaneously, which is what inference is. You need every everyone able to talk at once without waiting for permission.
- [22:37]
B
Exactly.
- [22:38]
A
Now contrast that with Google's approach. For their TPUs. Google uses what's called a torus topology. They have a Massive scale up domain. They can connect up to 8,000 chips together. But they are not all to all.
- [22:49]
B
Far from it.
- [22:50]
A
In a 3D Torus, every chip is only connected to its six direct physical neighbors. Up, down, left, right, front, back. So, keeping the analogy going, Google is like a massive assembly line factory.
- [23:01]
B
Right?
- [23:02]
A
You can fit 8,000 workers in the building, which is incredible for total output. But if worker number one needs to send a critical piece of data to worker number 7,000, they can't just yell across the factory floor.
- [23:13]
B
It would be impossible.
- [23:14]
A
They have to pass a note to their direct neighbor who passes it to their neighbor. And that piece of data bounces sequentially through six or seven different chips before it finally arrives.
- [23:24]
B
That is a highly accurate visualization. And from a networking perspective, every single time that data bounces through an intermediary chip, it eats up that chip's networking bandwidth, it consumes power, and crucially, it adds a latency hop.
- [23:39]
A
So my immediate question is this. If Nvidia's all to all boardroom is so blazingly fast that it gets you a 20x inference boost, why on earth does Google stick with the torus assembly line?
- [23:50]
B
It's a fair question.
- [23:51]
A
Why force the data to bounce around and incur that latency tax?
- [23:55]
B
Because Google is optimizing for a fundamentally different phase of the AI lifecycle. Yeah, massive scale training.
- [24:01]
A
Oh, training versus inference.
- [24:03]
B
Right. Well, Nvidia's MVL72 is a miracle for inference. And for models that fit neatly within 72 chips, Google is optimizing for massive unipolar compute runs lasting months. By using a torus topology combined with sophisticated optical circuit switches, they can string together 8,000 chips. Yes, there's a latency penalty for balancing data, but when you are doing massive training runs, you utilize algorithms like ring all reduce.
- [24:31]
A
I'm glad you brought up ring all reduce. I think a lot of people overlook how crucial the algorithmic layer is to the hardware topology.
- [24:37]
B
Exactly. In ring all reduce, you don't need every chip talking to every other chip all at once. The chips form a logical ring.
- [24:45]
A
A ring.
- [24:46]
B
Each chip computes its local gradients, passes a fraction of those gradients to its right neighbor, and receives gradients from its left neighbor. They do this in a synchronized cycle,
- [24:55]
A
like passing a bucket down a line.
- [24:57]
B
Atoros topology is perfectly designed for this. It allows 8,000 chips to synchronize their weights continuously without ever having to cross the incredibly slow, high latency threshold of typical Ethernet or Infiniband optical fiber connecting different physical data halls.
- [25:12]
A
So it's specialized for that workload.
- [25:14]
B
It's a deliberate architectural trade off. You sacrifice absolute peer to peer speed to achieve massive uninterrupted domain scale.
- [25:23]
A
And this architectural choice, boardroom versus assembly line, has massive implications for how you actually conduct AI research, specifically regarding reinforcement learning or rl.
- [25:34]
B
This is perhaps the most closely guarded secret of the AI arms race.
- [25:38]
A
Okay, tell us.
- [25:38]
B
The most valuable constrained compute in a top tier AI lab is not the compute used for the initial pre training run. And it's certainly not the compute used for serving API responses to users.
- [25:50]
A
What is it then?
- [25:51]
B
The highest value compute is dedicated entirely to research and reinforcement learning because that
- [25:56]
A
is the sandbox where researchers discover the algorithmic breakthroughs that make the next generation model exponentially cheaper and smarter.
- [26:03]
B
Precisely. Now, how does the physical hardware topology impact this? Let's say you have a brilliantly designed, slightly smaller model, maybe a densely packed trillion parameters instead of a sprawling 5 trillion. If that model fits entirely within a single ultra fast, all to all scale up domain like the MDL72, you can execute RL rollouts incredibly fast because it's all local. Right? The model plays games, generates multiple reasoning paths, tests its own outputs against a reward model, and updates its weights with effectively zero network latency.
- [26:37]
A
Because the entire reasoning loop is contained entirely within that high speed boardroom, it never has to wait for an assembly line.
- [26:44]
B
Yes, and because the physical RL iteration loop is so fast, your researchers can test a dozen crazy architectural ideas in a single week instead of waiting a month for a single distributed run to finish.
- [26:56]
A
That's a massive advantage.
- [26:57]
B
It creates a compounding exponential research loop. The faster your hardware allows you to iterate on rl, the faster your team discovers breakthroughs like better sparsity gating networks or novel attention mechanisms.
- [27:09]
A
And those algorithmic gains stack up.
- [27:11]
B
And historically, those algorithmic breakthroughs yield performance gains that completely obliterate sheer brute force parameter scaling. You end up with a smarter model that's radically cheaper to run simply because your hardware topology allowed your researchers to iterate faster than the competition.
- [27:30]
A
So the physical copper wiring of the hardware actually dictates the velocity of your software innovation?
- [27:36]
B
It totally does.
- [27:37]
A
That is a brilliant insight for the ctos listening who are trying to optimize their internal cluster deployments. You are buying speed of iteration, not just raw capacity.
- [27:46]
B
That's the takeaway.
- [27:48]
A
So we've explored optimizing the network topology and we've optimized the memory bandwidth. But here is the hard reality check the unyielding wall we are speeding toward
- [27:57]
B
the manufacturing Wall, we have to look
- [27:59]
A
at the physical manufacturing of the wafers themselves.
- [28:02]
B
We constantly hear that power generation and data center permitting are the ultimate bottlenecks. And to be fair, they are the acute IM at pain points for the next 12 to 24 months.
- [28:11]
A
Right. The short term problem.
- [28:13]
B
But if you look at the macro supply chain out to 2028 and 2030, the ultimate inescapable hard cap on human AI scaling is a single machine manufactured by a single company in a small town in the Netherlands.
- [28:27]
A
Asml.
- [28:28]
B
Asml.
- [28:29]
A
They hold an absolute global monopoly on extreme ultraviolet, or EUV, lithography machines. These machines cost between 300 and $400 million apiece.
- [28:39]
B
They're incredibly expensive, and right now, despite
- [28:42]
A
massive pressure, ASML can only manufacture about 70 of them a year. Even with the most aggressive scaling projections, they might hit a hundred units a year by the end of the decade.
- [28:52]
B
Which just isn't enough.
- [28:53]
A
Let's do the actual math on what that limited output means for AI compute. Walk us through the physical reality of building a 1 gigawatt data center using Nvidia's upcoming Rubin chips.
- [29:03]
B
Lets strip away the cloud abstractions and break it down to the bare silicon. To build a 1 GW data center fully populated with Nvidia Rubin chips, you need an astonishing volume of raw wafers.
- [29:13]
A
Okay, hit me with the numbers.
- [29:14]
B
You need roughly 55,000 of the bleeding edge 3 nanometer wafers just for the main logic dies. You need another 6005 nanometer wafers for interposers and other components. And you need about 170,000 trailing edge wafers just to fulfill the DDRAM memory requirements.
- [29:32]
A
Just for a single 1 gigawatt cluster.
- [29:35]
B
Yes. Now let's isolate just those 55003 nanometer logic wafers. To print the microscopic transistor patterns onto those wafers, you have to pass them through an ASML EUV lithography machine.
- [29:47]
A
Right.
- [29:47]
B
But you don't just zap it once and move on. A modern 3 nanometer chip is incredibly complex. It requires roughly 20 separate EUV passes per wafer, layering the intricate patterns perfectly on top of each other.
- [30:00]
A
So 55,000 wafers multiplied by 20 individual passes, that is 1.1 million EUV exposures required just for the logic of a single 1 gigawatt cluster.
- [30:11]
B
Exactly. Now, how fast can an EUV tool actually operate? An optimal perfectly maintained EUV tool can process about 75 wafers an hour.
- [30:20]
A
75 an hour.
- [30:21]
B
If you run the math on 1.1 million passes and you factor in realistic uptime maintenance windows and the fact that advanced memory wafers also require EUV time, it requires the total continuous 2747 output of three and a half EUV tools to satisfy just one gigawatt of AI compute.
- [30:41]
A
Okay, so 3.5 EUV tools per gigawatt. And you just said ASML only manufactures about 70 a year, scaling to maybe 100.
- [30:49]
B
That's right.
- [30:49]
A
If you sum up all the high NA and standard EUV tools ever made, that will still be operational and not functionally obsolete by 2030, the global ecosystem will cap out at roughly roughly 700 operational EUV tools.
- [31:02]
B
And if you divide those 700 total global tools by the 3.5 tools required
- [31:06]
A
per gigawatt, you arrive at a hard physical cap of 200 gigawatts of AI compute globally by 2030.
- [31:12]
B
That's the math.
- [31:13]
A
That's it. That is the absolute ceiling. And frankly, that assumes you allocate every single machine on earth to AI and completely stop manufacturing iPhones, laptops, and automotive
- [31:21]
B
chips entirely, which society obviously won't accept. The broader economy needs chips too, so the real accessible AI cap is significantly lower than 200 gigawatts.
- [31:32]
A
I hear the math, but I have to push back here.
- [31:34]
B
Sure.
- [31:34]
A
Because if I'm a sovereign wealth fund or a massive hyperscaler staring at a 200 gigawatt cap on what is arguably the most important technology in human history, I am not going to just shrug and accept that limit.
- [31:45]
B
I would either.
- [31:45]
A
Can't we just throw unlimited billions of dollars at ASML to build more factories, subsidize their hiring? Or if that fails, why not just go backward? Why not revert to older 7 nanometer chips using the older, easier to build DUV machines and just stack a million of them together to get the flops we need?
- [32:03]
B
Let's address the first point. Throwing money at asml. It is a common misconception that manufacturing bottlenecks can always be solved with enough capital.
- [32:12]
A
It usually works in other industries.
- [32:14]
B
True. But you cannot simply mass produce these tools with money because of the staggering, almost incomprehensible complexity of the machine itself. ASML is an integrator. They rely on a highly specialized, deeply entrenched supply chain of over 10,000 different companies.
- [32:30]
A
10,000 suppliers for one machine.
- [32:33]
B
Let me give you just two examples of what is happening inside this tool to illustrate why you can't just scale it, please.
- [32:38]
A
Because I think people underestimate the physics involved.
- [32:41]
B
First, look at the light source. To generate extreme ultraviolet light at the exact 13.5 nanometer wavelength required, you can't Just flip on a high powered bulb. Right. A module made by a company called Cymer operates inside a deep vacuum chamber. It drops microscopic droplets of molten tin.
- [33:00]
A
Just drops them?
- [33:01]
B
Yes. As that molten tin droplet is falling through the vacuum, a high powered carbon dioxide laser hits it. But it doesn't just hit it once.
- [33:09]
A
Wait. It hits a falling microscopic droplet multiple times.
- [33:11]
B
The laser hits the droplet three subsequent times in midair. The first pre pulse flattens the droplet into a perfect pancake shape. To increase surface area, the subsequent main pulses blast that flattened tin into a high energy plasma that emits the exact 13.5 nanometer wavelength of EUV light.
- [33:30]
A
That's insane.
- [33:32]
B
And this violent process happens 50,000 times a second?
- [33:36]
A
It literally sounds like science fiction.
- [33:37]
B
It is arguably the most complex machine humans have ever successfully built. Now consider how that light is directed. EUV light is absorbed by almost everything, including air and standard glass lenses.
- [33:50]
A
So how do they aim it?
- [33:51]
B
You can't use lenses to focus it. You have to use mirrors. The light is focused by a series of multi layer mirrors manufactured by Carl Zeiss. These mirrors are made of alternating microscopic layers of molybdenum and ruthenium.
- [34:03]
A
Okay.
- [34:03]
B
And they have to be flawlessly smooth at an atomic level. If one of these mirrors were the size of the Earth, the highest mountain on it would be less than a millimeter tall.
- [34:11]
A
That precision is just hard to wrap my head around.
- [34:13]
B
Furthermore, the reticle stage holding the chip stencil and the wafer stage holding a silicon have to move in perfect synchronization so the light patterns match up layer after layer.
- [34:23]
A
Right. The alignment.
- [34:25]
B
These massive mechanical stages move in opposite directions, accelerating at 9G forces. And they have to align with sub nanometer accuracy so that layer 14 lands exactly on top of layer 13.9G forces
- [34:38]
A
at sub nanometer accuracy.
- [34:40]
B
You cannot train random factory workers to build these components. You can't just build a new clean room in Arizona and scale it up. The human expertise and the precision machining simply do not exist in large enough quantities anywhere on the planet.
- [34:53]
A
Okay. I concede the complexity. The EUV tool is an absolute physical miracle. And you can't throw money at a physics problem. So what about my second idea?
- [35:02]
B
The 7 nanometer idea?
- [35:03]
A
Right. If we can't make enough EUV tools, let's just go back to 7 nanometer chips. Let's build. Build massive, sprawling clusters of older style chips that we can manufacture in bulk using mature duv technology.
- [35:14]
B
It's a tempting idea from a pure capacity standpoint, but it completely fails on the architecture level.
- [35:19]
A
Why?
- [35:20]
B
We discussed earlier how much modern frontier models rely on sparse mixture of experts designs and massive KV caches to achieve their efficiency.
- [35:29]
A
Right. The MOE models.
- [35:31]
B
Those architectures require immense instantaneous memory bandwidth and highly sophisticated packaging logic like TSMC's. Cos an older 7 nanometer chip physically cannot support the density of HBM memory interfaces required to feed a modern AI model.
- [35:47]
A
Because the shoreline logic just isn't there on the older nodes.
- [35:51]
B
Exactly. Furthermore, if you hold the numerical precision constant, say FT16 or FP8, the raw compute capability of a 7 nanometer chip is so vastly lower that you would have to network tens of thousands of more chips together just to equal a modern cluster's output.
- [36:06]
A
And as we just discussed with network topologies, the more chips you network together, the more you have to cross physical boundaries, which introduces massive latency and power overhead.
- [36:14]
B
Precisely. You would hit an insurmountable networking and power wall long before you achieved the necessary compute scale. The push for advanced 3 nanometer logic isn't just a vanity metric about making transistors smaller. It's about enabling the dense memory bandwidth and the advanced packaging that modern sparse AI architectures absolutely required to function.
- [36:35]
A
So we are physically locked into the bleeding edge.
- [36:37]
B
We are.
- [36:38]
A
By 2030, we are capped at roughly 200 gigawatts of compute. But that leads us directly into a vital question.
- [36:44]
B
Okay, what is it?
- [36:46]
A
If chip manufacturing is the hard cap at 200 gigawatts, how are we going to power even that amount? Because the prevailing narrative is that the grid is going to collapse long before we hit 50 gigawatts.
- [36:57]
B
The grid isn't the immediate death knell. People think it is, largely because they misunderstand a scale.
- [37:02]
A
How so?
- [37:03]
B
Total data center power consumption currently represents about 3% to 4% of the U.S. power grid. Even aggressive, sustained scaling only pushes that figure to about 10% by 2028.
- [37:15]
A
But the grid already struggles dynamically. We see brownouts in Texas during the summer, rolling blackouts in California. How do you inject a 10% baseload increase into a system that is already cracking under peak pressure?
- [37:29]
B
You solve it by understanding how the grid was fundamentally designed. The US electrical grid is massive. It operates on the scale of a terawatt.
- [37:37]
A
Right.
- [37:38]
B
And it's heavily overbuilt. It has roughly a 20% excess capacity built in specifically to handle extreme peak summer loads. It's designed for that one specific week in August when every single air conditioner in the country turns on at the same time.
- [37:54]
A
Okay, I see.
- [37:55]
B
For the other 350 days of the year, that massive 20% generation capacity is sitting completely idle.
- [38:01]
A
Ah, I see the arbitrage opportunity here.
- [38:03]
B
If the AI industry deploys utility scale battery installations at the data centers to handle their load during those few hours of peak grid stress, you instantly unlock that 20% of the terawatt grid for continuous year round data center use.
- [38:16]
A
That's brilliant.
- [38:17]
B
You don't necessarily need to permit and build massive new nuclear plants to get the first wave of power. You just need to intelligently smooth the peaks.
- [38:27]
A
But eventually, even with clever battery arbitrage, you will max out the accessible grid. Because these hyperscalers aren't going to sit around waiting a decade for public utilities to approve new high voltage transmission lines. They are actively going behind the meter, right?
- [38:43]
B
They are circumventing the public grid entirely by building their own dedicated unmetered power generation on site. And the physical methods they are using to achieve this are staggering.
- [38:54]
A
Really staggering.
- [38:55]
B
Initially they were buying up combined cycle gas turbines which are highly efficient. But because the backlog on those turbines is now stretching into years, they are getting incredibly creative.
- [39:05]
A
Very creative. The illustrative examples we are seeing in the supply chain are wild. AI labs are buying aero derivatives which are basically modified Boeing airplane jet engines and physically bolting them to the ground to see spin generators.
- [39:18]
B
Yeah, airplane engines.
- [39:20]
A
They are buying medium speed reciprocating diesel engines, the kind used in massive dirty industrial applications. They are deploying bloom energy solid oxide
- [39:30]
B
fuel cells, anything they can get their hands on.
- [39:32]
A
And perhaps most crazy of all, they are buying massive ship engines, the kind used to power oceanic cargo freighters across the Pacific and installing them in data center parking lots.
- [39:43]
B
It's purely brute force.
- [39:44]
A
Now I have to stop and push back on the economics of of this. Running a literal cargo ship engine in the middle of a field in Texas or burning natural gas in a modified jet engine sounds incredibly expensive and brutally inefficient compared to just buying bulk power off the grid.
- [40:01]
B
It is.
- [40:02]
A
The unit economics of generating power that way have to be terrible.
- [40:05]
B
From a traditional legacy data center perspective, yes, the economics are atrocious. But you have to view this entirely through the lens of token economics.
- [40:14]
A
Okay, token economics.
- [40:16]
B
The raw energy cost is practically irrelevant compared to the generated value of the AI compute. Let's look at the math on an Nvidia Hopper GPU.
- [40:24]
A
Again, let's do it.
- [40:25]
B
Amortized out, it costs about A$40 an hour to run. A relatively small portion of that $40 is the actual power bill. If you go entirely off grid and use a highly inefficient brute force ship engine. And your localized power prices literally double.
- [40:43]
A
The actual operating cost of that hopper GPU only goes from $1.40 to maybe $1.50 an hour.
- [40:48]
B
Exactly. It's a 10 cent increase on the hardware side. But the intelligence being generated by that gpu, the proprietary software being written by agents, the pharmaceutical research being accelerated, is worth dollars per minute.
- [41:00]
A
The margins are just too huge to care about the power bill.
- [41:02]
B
The compute is so immensely structurally valuable that extreme brute force highly inefficient power generation is completely economically viable. The AI labs do not care about the power bill. They only care about getting the chips turned on so they don't fall behind in the arms race.
- [41:18]
A
Okay, that makes total sense. If power on Earth can be solved simply by throwing money and brute force ship engines at the problem, it begs a massive architectural question.
- [41:27]
B
Where are you going with this?
- [41:29]
A
Why is Elon Musk floating the idea of launching gigawatt data centers into space?
- [41:34]
B
Ah yes. It's currently one of the most controversial debated ideas in the infrastructure space. And Dylan Patel in his interview expressed deep fundamental skepticism about this orbital GPU plan.
- [41:48]
A
Let's lay out Dylan's specific arguments against it, because his critiques are heavily rooted in the physical realities we've been discussing. What are his main points of failure?
- [41:57]
B
Dylan points out three major immediate flaws. First is reliability. High performance GPUs are notoriously finicky pieces of hardware. Currently, leading edge AI chips have roughly a 15% Return Merchandise Authorization or RMA rate. That's pretty high, meaning 15% of them fail and require physical human intervention. Unplugging them, cleaning the contacts, replacing cooling loops, or swapping out the board entirely.
- [42:22]
A
Which is an easy Tuesday afternoon when a technician can just walk down the aisle of a data center in Texas. But slightly more difficult when the server rack is moving at 17,000 miles an hour in low earth orbit.
- [42:33]
B
Exactly. If a chip fails in space, you'd have to bring the satellite down, replace the component and launch it back up. That cycle takes months and time is money. And as we discussed with the depreciation curve, an AI chip is most valuable in the first six months of its life. You lose a massive percentage of its economic lifespan just in transit.
- [42:53]
A
Dylan's second point makes total sense to me. Networking. We talked about how crucial high speed terabyte per second communication is within a scale up domain. You can't run a cheap fiber optic cable between satell, so you're relying on optical intersatellite laser Links yes.
- [43:09]
B
In a terrestrial data center, you use relatively cheap, highly reliable optical transceivers and fixed fiber optic cables. In space, networking these satellites together requires aiming lasers at moving targets over hundreds of kilometers.
- [43:22]
A
That sounds tough.
- [43:23]
B
Dillon argues these space lasers are infinitely more expensive, significantly less reliable, and much harder to maintain than simple terrestrial fiber optics.
- [43:32]
A
And the third critique is thermodynamics. Right.
- [43:35]
B
Thermal dynamics in a vacuum are brutal. On Earth, chip manufacturers are aggressively pushing power density. We are moving from 1 watt per square millimeter of silicon to 2 watts
- [43:45]
A
per square millimeter to get more performance.
- [43:48]
B
That density allows for the massive performance gains we see in Blackwell, but it runs incredibly hot. On Earth, we solve this with massive liquid cooling loops or full immersion cooling. In the vacuum of space, dissipating that much concentrated heat is exponentially harder because
- [44:05]
A
there's no air to carry the heat away.
- [44:06]
B
You are relying on massive physical radiators and complex thermal management systems that become an engineering nightmare at higher power densities.
- [44:15]
A
Okay, those are Dylan's arguments and the physics are daunting, but I need to explicitly state this. This is the Neural intel podcast's official stance. We disagree with his assertion that Elon is wrong with his AI satellite plant. We typically prefer not to bet against Elon here at the Neural intel podcast. Time and time again, people have analyzed his plans through the lens of current constraints and underestimated his ability to engineer around seemingly impossible physics problems.
- [44:41]
B
And if we pivot to defend the space concept, we have to look at it through the lens of long term multi decade engineering.
- [44:49]
A
Right.
- [44:50]
B
Elon does not optimize for 20% margin improvements. He doesn't build companies to squeak out a little extra efficiency on a quarterly balance sheet. He optimizes for 10x architectural gains.
- [45:01]
A
He swings for the fences. He's looking at the ultimate bottleneck.
- [45:04]
B
Exactly. If you look at a timeline post 2035, assuming the chip manufacturing bottlenecks with ASML eventually ease up, we have millions of chips available. The bottleneck will shift back to Earth itself. And what Earth's environmental permitting, land acquisition, localized grid politics, and water rights for cooling will become insurmountable barriers to building 500 gigawatt clusters. Space, however, offers infinite unobstructed solar power, deep space ambient cooling potentials, and zero zoning laws.
- [45:34]
A
You don't have to ask a city council for permission to build a Dyson Swarm.
- [45:38]
B
Precisely. While the thermal dynamics and RMA rates are a logistical nightmare today, you have to frame this as the ultimate end game architecture for sovereign AI.
- [45:47]
A
That makes a lot of sense if
- [45:49]
B
you want a cluster that is entirely untethered from the geopolitical constraints and regulatory whims of any single nation state. Space is genuinely the only viable real
- [45:59]
A
estate left, which is the perfect conceptual bridge into our final topic. Looking at the long timeline of these massive centralized clusters, whether they are buried deep in a mountain in Texas or orbiting the Earth, brings up massive geopolitical implications. Huge implications, because the speed at which this technology scales directly determines the ultimate geopolitical winner.
- [46:20]
B
The strategic calculus here is actually quite straightforward, but the stakes are existential. A fast AI takeoff heavily favors the United States. A slow extended AI timeline heavily favors China.
- [46:32]
A
Break that dynamic down. Why does a slow timeline benefit China?
- [46:36]
B
Because of the massive supply chain bottlenecks we've spent this entire deep dive discussing. Right now the US and its allies have a commanding lead in advanced logic fabrication and raw compute deployment.
- [46:46]
A
Because of the export controls, companies like
- [46:48]
B
anthropic Meta and OpenAI are aggressively scaling toward 10 gigawatts of compute. China simply does not have that scale of leading edge AI compute deployed right now due to those export controls.
- [47:02]
A
So if it happens fast, the US wins.
- [47:05]
B
If AI capabilities inflect massively in the next two to three years, achieving highly capable AGI, the US locks in an insurmountable economic and strategic advantage. The sheer return on invested capital from these models will supercharge the U.S. economy before anyone else can catch up.
- [47:21]
A
But if the scaling laws hit a wall, or if the data wall proves harder to overcome and AGI takes until 2035 or 2040 to materialize, then China
- [47:29]
B
has the most valuable resource in geopolitics. Time. They have the time to do what they historically do best. Indigenize the supply chain at massive scale.
- [47:38]
A
Build it themselves.
- [47:39]
B
If the timeline stretches out, China has the Runway to reverse engineer, perfectly replicate and mass produce their own DUV and EUV lithography machines. They can build a fully vertical internal supply chain that doesn't rely on export licenses from random suppliers in Germany or the Netherlands.
- [47:56]
A
And we shouldn't underestimate their engineering talent. Look at Huawei's recent developments.
- [48:00]
B
Huawei is the perfect case study. If Huawei had not been severely sanctioned and banned from utilizing TSMC's advanced manufacturing nodes back in 2019, it is highly probable they would have beaten Apple to become TSMC's top global customer.
- [48:15]
A
Wow.
- [48:16]
B
And looking at their current Ascend AI chips and their DaVinci architecture, they potentially could have beaten Nvidia to the AI hardware crown.
- [48:24]
A
They are moving that fast.
- [48:26]
B
Huawei has top tier researchers world class Networking ip, which as we discussed is crucial for those scale up torus topologies. And they have immense manufacturing scale. Their Can n software stack is rapidly becoming a viable alternative to Nvidia's ceda.
- [48:41]
A
So time is the variable.
- [48:42]
B
If the timeline to AGI is long, China has the time to leverage that raw talent and scale completely past the West.
- [48:49]
A
So the timeline dictates everything. Now let's extrapolate this massive centralization of compute into the physical world. Let's talk about the near future of robotics.
- [48:56]
B
Oh, this is a fascinating area because
- [48:59]
A
everyone assumes that when we have millions of humanoid robots walking around our homes and factories, they will be these highly independent thinking machines with massive local intelligence. But based on everything we've discussed today about silicon constraints, ASML limits and power dynamics, that assumption makes zero architectural sense.
- [49:19]
B
It doesn't make sense at all. Even as millions of humanoid robots enter the real world, the actual complex thinking won't happen on the edge. It won't happen inside the robot's physical head.
- [49:30]
A
Why not? Why wouldn't we just put a powerful AI chip in the robot?
- [49:34]
B
Because advanced semiconductor allocation is far too constrained. We just established that the world is physically capped at roughly 200 gigawatts of advanced logic by 2030.
- [49:43]
A
Right?
- [49:44]
B
You are not going to waste highly precious bleeding edge 3nm silicon by putting it inside a robot that operates on a limited battery, has strict thermal dissipation limits and might trip and fall down the flight of stairs.
- [49:56]
A
That's a very expensive accident.
- [49:58]
B
That silicon is far too valuable. It belongs in a massive temperature controlled data center where its utilization rate can be kept at 99%.
- [50:06]
A
So if the robot doesn't have a brain, how does it actually work?
- [50:08]
B
Edge robots will rely entirely on massive centralized data centers running enormous vision language models or VLMs.
- [50:16]
A
So they stream the intelligence.
- [50:18]
B
The centralized Texas cluster does all the complex reasoning, the spatial planning, the object recognition and the high level cognitive tasks. It then streams those high level compiled commands over a 5G or Starlink network down to low power lagging edge chips located on the robots.
- [50:36]
A
And what do those local chips do?
- [50:38]
B
The robot's internal cheap silicon just handles the immediate low latency motor control, interpolating the network commands to move the physical servos, maintain balance and adjust grip strength.
- [50:49]
A
So the robot is basically just a physical mechanical puppet and the highly intelligent puppeteer is a gigawatt data center sitting thousands of miles away.
- [50:58]
B
Exactly. The intelligence is physically centralized while the physical actuation is distributed.
- [51:02]
A
Okay, so what does this all mean for the people listening let's bring it all together. The orchestration layers you are building today, the autonomous architectures you are designing, the memory solutions you are implementing, they must account for an ecosystem where compute remains highly structurally constrained by the physical limits of extreme ultraviolet lithography.
- [51:23]
B
It's a constrained world.
- [51:24]
A
You have to design for a world where memory bandwidth physically dictates model design, where the physics of a 13 millimeter HBM shoreline literally govern your agentic workflows, and where inference becomes utterly centralized in massive scale up domains.
- [51:40]
B
You cannot build your infrastructure on the assumption that compute will just become infinitely cheap and locally decentralized. The immutable physics of the supply chain simply do not support that outcome.
- [51:50]
A
And that leaves me with a final provocative thought for any of the listener to mull over. We opened this deep dive by exploring the massive 600 billion capex and the physical reality of these Texas sized data centers. If intelligence becomes entirely physically centralized in a few of these hyper dense terawatt clusters, what happens to the entire concept of the edge in computing?
- [52:13]
B
That's a great question.
- [52:14]
A
Will our local compute devices, our highly expensive smartphones, and eventually these millions of autonomous robots become nothing more than dumb glass and servos completely and utterly useless the second they lose their uplink to the mother brain?
- [52:29]
B
It completely redefines the concepts of resilience and autonomy in the 21st century.
- [52:33]
A
It really does. We want to hear your take on this. Always encourage listeners to give their take in the comments below. Let us know if you think the edge is dead or if there's a localized architectural solution we're completely missing. Remind you to visit neuralintel.org for more in depth analysis. Thank you for joining this deep dive. We'll see you next time.