Inside Nvidia’s Age of Inference

Summary

WSJ Tech News Briefing – "Inside Nvidia’s Age of Inference"

Date: March 17, 2026
Host: Katie Dayton
Guests: Kate Clark (WSJ Reporter), Robbie Whelan (WSJ Reporter)

---

Episode Overview

This episode explores two interlinked phenomena rapidly transforming the tech world: the explosion of AI personal assistants and agents among Silicon Valley workers, and the business and technical shifts at Nvidia as the industry transitions from AI “training” to “inference.” The episode draws on reporting from the ground in San Francisco and at Nvidia’s annual AI conference in San Jose, diving into workplace culture, technical evolution, and industry competition.

---

The Rise of AI Agents in Silicon Valley

Scene Setting

• San Francisco hot spot Dolores Park is now filled with tech workers managing not their own code, but armies of software “agents”—AI-powered bots handling both technical and everyday tasks. (01:53)

Key Insights and Discussion Points

• Widespread Adoption of AI Agents

  • “The AI agents, the AI personal assistants or work assistants have really taken over Silicon Valley and beyond.” — Kate Clark (01:53)
  • AI bots are used for everything from coding and creating spreadsheets to planning vacations, reflecting a huge culture shift.

• Paradox of Productivity and Overwork

  • Despite bots’ promise of efficiency, people spend more time glued to screens, “babysitting” their bots and striving for maximal productivity.
  • “There is this anxiety... they know they can be so much more productive. They really want to make sure they are being as productive as possible.” — Kate Clark (02:29)
  • New tools paradoxically deepen work attachment.

• Underlying Technology

  • Breakneck improvements in large language models (LLMs) from OpenAI, Anthropic, and others are driving this shift.
  • “Every few weeks or every few months, they release model updates that have made... tools built on top just work so much better.” — Kate Clark (03:12)
  • A vibrant ecosystem of startups builds on top of these models. Constant innovation and increased investment fuel a fast-changing environment.

• Pitfalls of Bot Management

  • So far, mishaps are mostly minor—bots make mistakes, delete inboxes, or fumble tasks, creating more work. “People get irritated because... they're just not doing the things that they're asking them to do. And they feel like they're basically babysitting the AI bots.” — Kate Clark (04:02)

• Changing Nature of Software Engineering

  • Long-honed coding expertise risks obsolescence as natural language replaces code for many tasks.
  • Some engineers feel nostalgia or sadness: “People being really upset at the idea that the skill they spent their life learning is no longer relevant.” — Kate Clark (04:54)
  • Others argue foundational understanding is necessary to manage AI. The consensus: massive disruption and job loss are in the pipeline, moving at breakneck speed.

---

Nvidia and the "Age of Inference"

Nvidia GTC 2026
The company’s annual AI conference in San Jose puts “inference” front and center as CEO Jensen Huang announces a major move: licensing chip technology from Groq for $20 billion to beef up Nvidia’s inference hardware (06:40–08:02).

Key Insights and Discussion Points

• Nvidia’s Strategic Shift

  • Nvidia dominates AI “training,” but “doesn’t have quite as much of a foothold” in “AI inference”—the process of running and using trained AI models in real-world applications.
  • The Groq deal (“essentially an acqui-hire”—Nvidia licensed the tech and hired top leadership) positions Nvidia for the next phase in AI infrastructure. — Robbie Whelan (07:18)

• Inference vs. Training: Hardware Needs (08:02–09:58)

  • Inference: Requires vast memory and high bandwidth to quickly access trained model data.
  • Training: Requires raw computing power for “billions and billions of math problems.”
  • “Inference... requires a lot more memory because you have these models that are working to remember everything that they've been trained.”
  • Fast, low-cost inference is vital as AI becomes mass-market. Companies must “drive down costs” while maintaining “reliable response times.”

• Can Nvidia Compete? (09:58–11:08)

  • Nvidia is unlikely to be left behind: “They have essentially locked up their supply [of crucial memory chips] for the next two or three years.”
  • New challenge: thinner profit margins when moving from luxury products (training chips) to more mainstream, cost-driven inference products.
  • “It's very hard to sustain high profit margins.”

• Competitive Landscape (11:13)

  • Cerebra: $10B licensing deal with OpenAI for custom inferencing chips.
  • AR Labs: Focuses on interconnectivity chips using “advanced fiber optics and lasers” for faster inference—highlighting the importance of every part of AI hardware, not just main processors.
  • AMD: Scaling up in inference, gaining traction despite being smaller than Nvidia.

---

Notable Quotes & Moments

• "There is this anxiety... they really want to make sure they are being as productive as possible." — Kate Clark (02:29)
• “They feel like they're basically babysitting the AI bots.” — Kate Clark (04:02)
• “We're very quickly moving into an era where... you and me can just use natural English language to command these agents to create websites, to create products.” — Kate Clark (04:54)
• “Inference requires a lot more memory... Training requires a lot more raw computing power.” — Robbie Whelan (08:06)
• “There's a real worry... that Nvidia’s margins are not going to remain that high for very long.” — Robbie Whelan (10:09)

---

Key Timestamps

• 00:34–05:42 — Rise of AI agents, reporting from San Francisco (Kate Clark)
• 06:40–07:18 — Nvidia GTC conference & strategic moves (Katie Dayton, Robbie Whelan)
• 08:02–09:58 — Technical differences: inference vs. training hardware (Robbie Whelan)
• 10:09–11:13 — Nvidia’s competitive outlook, supply chain, margins (Robbie Whelan)
• 11:13–12:13 — Competition and new players in AI hardware (Robbie Whelan)

---

Takeaway

The episode spotlights both a cultural and technological inflection point: software engineering is being redefined by LLM-powered agents, and industry giants like Nvidia are retooling fast to power the "age of inference" as AI shifts from advanced R&D to everyday use. The pace of change leaves questions about productivity, job security, and who will win—or lose—this next chapter.

---

Transcript

[Xfinity Tech Support]

Xfinity Tech Support I can't get Jurassic park back online without but no worries. Our WI Fi Boosted Mobile keeps you connected in all kinds of places.

[Xfinity Mobile Advertiser]

Cut your mobile bill in half with two lines of unlimited. Learn more@xfinity mobile.com Xfinity Imagine that. Watch Jurassic park with Xfinity restrictions apply. Xfinity Internet Required Savings Comparison based on promotional rate for two lines of Xfinity Mobile Unlimited versus two lines of T Mobile. Experience more Verizon Unlimited plus and AT&T starter as of February 10, 202026 taxes fees extra.

[Katie Dayton]

Welcome to Tech News briefing. It's Tuesday, March 17th. I'm Katie Dayton for the Wall Street Journal. Silicon Valley has a bot obsession. Software engineers used to burning the midnight oil, churning out lines and lines of code, are now using AI models to do the dirty work for them, to varying degrees of success. We're learning how the sudden shift in definition of tech work is changing the coding culture of the Then we're heading south to Nvidia's annual AI conference in San Jose, where the word inference is on everybody's lips. Our reporter on the ground will be with us to explain how we got here. But first, software engineers prone to working solo are finding themselves as managers of ambitious but sometimes unruly workers. And those workers? They're not human. Tech heads are increasingly employing fleets of AI assistants to do their work, so much so that asking what are your bots up to? Has become one of the hottest questions in Silicon Valley. WSJ reporter Kate Clark has been following the trend. So, hey, set the scene for us. It's a beautiful day in San Francisco. The crowds are descending on Dolores park and people are sitting in the sun with their laptops open. What exactly is going on here?

[Kate Clark]

That's exactly right. The AI agents, the AI personal assistants or work assistants have really taken over Silicon Valley and beyond. It is particularly evident that people are obsessed with getting these AI agents set up, whether it's creating a spreadsheet for them and completing some basic work tasks or coding for them, or it's planning a vacation and handling all of the logistical tasks that take up a lot of our time.

[Katie Dayton]

What were some of the stranger anecdotes about this bot management culture that you unearthed reporting this story?

[Kate Clark]

It's really just how obsessed with it people are and how, you know, you would think this would make people more efficient, but it seems to me that people are actually a lot more attached to their devices now than they ever were, and they are staying up into the late hours just kind of making sure that they're AI agents are working. And so there is this anxiety that people feel right now because they know they can be so much more productive. They really want to make sure they are being as productive as possible. So I think this interesting dynamic is happening, which is the people are working more despite having these tools that should be making them work less.

[Katie Dayton]

This behavior appears to have swept the tech world incredibly quickly. What technology or LLM models have come on the scene to make all of this possible?

[Kate Clark]

Basically, the main models from OpenAI Anthropic, have just been getting better and better. Every few weeks or every few months, they release model updates that have made these other tools built on top just work so much better. Even just in the last couple of weeks, OpenAI has made adjustments, improvements to their model that have made Codex, their AI coding tool, even better. So it's just kind of like the natural evolution of this LLM technology. And there are many, many startups that are building tools on top of the LLMs that are powering these agents, and they're just catching on. They're just getting better by the day. There's more money being invested in them. It's just constant change.

[Katie Dayton]

Now, I've watched enough Black Mirror to know that delegating all your grunt work to bots can have terrible repercussions. Did you hear of any instances where this form of software engineering has gone incredibly wrong?

[Kate Clark]

For the most part, it's just they make mistakes and they mess everything up and you have to restart. You hear things about people's entire inboxes being deleted because of the stage we're at, which is really just people learning to use them, learning to see their capabilities. We really haven't gotten to the part where there's horror stories after horror stories for now. I think it's just people get irritated because they've got 1, 2, 3, 4 agents working on their behalf, and they're just not doing the things that they're asking them to do. And they feel like they're basically babysitting the AI bots.

[Katie Dayton]

Right, like human management in a way.

[Kate Clark]

Exactly like human management, which is also a funny part of all of this.

[Katie Dayton]

Software engineering is a skill that a lot of these people probably spent a lot of time honing. And in the Bay Area in particular, your ability to engineer was almost a status signifier. Is there any sense to all of this that this skill has just been rendered obsolete? And if so, is there any sense of nostalgia or sadness wrapped up in that?

[Kate Clark]

Definitely. I came across such a broad range from people being really upset at the idea that the skill they spent their life learning is no longer relevant. I also came across a lot of people that feel like it is still incredibly important to understand the building blocks in order to successfully manage the AI agents. But we're very quickly moving into an era where like you and me can just use natural English language to command these agents to create websites, to create products. And that as we get deeper into that chapter, there will be a lot more concerns about job loss for engineers. There already are signs of that. But you know, in a few more months, just because of how fast this is moving, there'll probably be a much, much bigger story to tell about the true impact on software engineers.

[Katie Dayton]

That was Wall Street Journal reporter Kate Clark. Are you deploying AI assistants in your work? If you're a listener on Spotify, be sure to let us know in the comments. Coming up Demand for inference computing is ramping up. We break down what this means for Nvidia's business model. That's after the break.

[Xfinity Tech Support]

Xfinity Tech Support I can't get Jurassic park back online without but no worries. Our Wi Fi Boosted mobile keeps you connected, connected in all kinds of places.

[Xfinity Mobile Advertiser]

Cut your mobile bill in half with two lines of unlimited. Learn more@xfinity mobile.com Xfinity Imagine that. Watch Jurassic park with Xfinity restrictions apply. Xfinity Internet Required savings comparison based on promotional rate for two lines of Xfinity Mobile Unlimited versus two lines of T Mobile Experience more Verizon Unlimited plus and AT&T Starter as of February 10, 2026. Taxes fees extra

[Katie Dayton]

this week sees the return of Nvidia gtc, the chipmakers mammoth AI conference held in San Jose. All eyes yesterday were on Jensen Huang, the leather jacketed CEO who addressed the rise of inference computing and what that means for the company going forward.

[Xfinity Tech Support]

Finally, AI is able to do productive

[Robbie Whelan]

work and therefore the inflection point of inference has arrived.

[Katie Dayton]

We got the lay of the land from WSJ reporter Robbie Whelan, who spoke to us from a car on the highway while heading to the conference. Robbie, what are we expecting to hear from Nvidia?

[Robbie Whelan]

Nvidia is getting ready to roll out a brand new chip using technology from a company called Groq, which Nvidia recently essentially acquired. Didn't fully acquire the company, but they paid $20 billion to license Groq's technology, which is chip design technology, and to hire Grok's top leadership. So basically it was what's called an aqua hire. And the reason why they did this was because Nvidia at the moment dominates the world of what's known as AI training. But they don't have quite as much of a foothold in what's known as AI inference. And AI inference means how to run models and make them respond to your user queries faster.

[Katie Dayton]

What are the different requirements for inference vs. AI training chips?

[Robbie Whelan]

Inference requires a lot more memory. So every computer chip that is in your iPhone or our laptops, or even in data centers, it's actually a combination of multiple chips. When we talk about a chip, we talk about a processor, which in Nvidia's case is called a GPU. And then there's other types of processors called CPUs, and then there's a whole bunch of chips that are known as memory chips. And those are the ones that allow the computer brain to kind of go back into its memory and access all of its training, access all the databases that have been pumped into it to teach it how to be an AI model. And so inference requires a lot more memory because you have these models that are working to remember everything that they've been trained. And training requires a lot more raw computing power. So if you think of training an AI model as just doing billions and billions of math problems over and over again, whereas inference, which means allowing a model to go out into the wild and respond to user queries, they have to go back to all of the information that's been pumped into the model via training. They have to access it. And that means a lot more memory chips are required, a lot more high bandwidth memory. Inference also requires a lot of computing power. It's not like it doesn't require any good processing at all. It does require fast processors in order to act quickly. But the name of the game right now is driving down costs to consumers and also driving speed to consumers. The phase we're in right now, we've done a lot of growth, a lot of training of models. Now we're trying to monetize those models. And to do that, you have to have really fast, reliable response times. And so that means the quality answer has to be good and has to come quickly. And so that's why inference requires such a different computing profile than training.

[Katie Dayton]

And given Nvidia's focus on AI training chips so far, and this new inference that's coming in, does that put Nvidia on the back foot in this space, will it have a hard time competing here?

[Robbie Whelan]

The short answer is no, I don't think they're going to be too far behind when it comes to sort of developing products quickly that serve the infinite computing needs and more importantly, perhaps locking up the supply chain that is needed for it to produce inference. So, for example, right now there's a big crunch in supply of memory chips. There's a certain number of companies, three in particular, that make memory chips that are usable in AI systems. And Nvidia is one of the biggest buyers of those chips and they have essentially locked up their supply for the next two or three years. But the age of inference, there's a real worry from a lot of people who I talk to in the AI world that Nvidia's margins are not going to remain that high for very long. And the reason for that is because when you go from producing a luxury product that's very expensive and you do it better than anyone else to producing a product that's meant to save users money, which is what's happening in the transition from training to inference, it's very hard to sustain high profit margins.

[Katie Dayton]

And you mentioned there are a few competitors in this space with them. Can you talk us through a few of those?

[Robbie Whelan]

There's a company called Cerebra. It's a startup that has raised a ton of money and recently did a $10 billion licensing deal with OpenAI to help OpenAI build its own custom chips for inferencing. There's a number of other startups that are sort of in stealth mode that I know about but can't talk about yet. There's a company that we talked about called AR Labs that we wrote stories about recently in the Wall Street Journal that makes a sort of a product that's a different type of chip. It's called an interconnectivity chip that uses advanced fiber optics and lasers in order to make inference computing faster. It's not just in the processors, in other words, it's in all the hardware that goes around the main processors that drive AI computing. The entire ecosystem is seeing much more competition. And then Advanced Micro Devices, amd, they are going all in on inference as well and trying to strike deals with model developers. They're a lot smaller than Nvidia, but they are growing very fast and rolling out products very quickly.

[Katie Dayton]

That was wa USJ tech reporter Robbie Whelan. And that's it for Tech News Briefing. If you're a listener on Spotify, be sure to leave us a comment. Today's show was produced by Julie Chang with supervising producer Katie Ferguson. I'm Katie Dayton for the Wall Street Journal. We'll be back later this morning. With TMB Tech Minute. Thanks for listening.

[Xfinity Tech Support]

Xfinity Tech Support I can't get Jurassic park back online without but no worries. Our WI Fi Boosted Mobile keep connected in all kinds of places.

[Xfinity Mobile Advertiser]

Cut your mobile bill in half with two lines of unlimited. Learn more@xfinity mobile.com Xfinity Imagine that. Watch Jurassic park with Xfinity restrictions apply. Xfinity Internet Required Savings comparison based on promotional rate for two lines of Xfinity Mobile Unlimited versus two lines of T Mobile Experience more Verizon Unlimited plus and AT&T Starter as of February 10, 2026 taxes fees extra.