WSJ Tech News Briefing – "Inside Nvidia’s Age of Inference"

Date: March 17, 2026
Host: Katie Dayton
Guests: Kate Clark (WSJ Reporter), Robbie Whelan (WSJ Reporter)

Episode Overview

This episode explores two interlinked phenomena rapidly transforming the tech world: the explosion of AI personal assistants and agents among Silicon Valley workers, and the business and technical shifts at Nvidia as the industry transitions from AI “training” to “inference.” The episode draws on reporting from the ground in San Francisco and at Nvidia’s annual AI conference in San Jose, diving into workplace culture, technical evolution, and industry competition.

The Rise of AI Agents in Silicon Valley

Scene Setting

• San Francisco hot spot Dolores Park is now filled with tech workers managing not their own code, but armies of software “agents”—AI-powered bots handling both technical and everyday tasks. (01:53)

Key Insights and Discussion Points

• Widespread Adoption of AI Agents

  • “The AI agents, the AI personal assistants or work assistants have really taken over Silicon Valley and beyond.” — Kate Clark (01:53)
  • AI bots are used for everything from coding and creating spreadsheets to planning vacations, reflecting a huge culture shift.

• Paradox of Productivity and Overwork

  • Despite bots’ promise of efficiency, people spend more time glued to screens, “babysitting” their bots and striving for maximal productivity.
  • “There is this anxiety... they know they can be so much more productive. They really want to make sure they are being as productive as possible.” — Kate Clark (02:29)
  • New tools paradoxically deepen work attachment.

• Underlying Technology

  • Breakneck improvements in large language models (LLMs) from OpenAI, Anthropic, and others are driving this shift.
  • “Every few weeks or every few months, they release model updates that have made... tools built on top just work so much better.” — Kate Clark (03:12)
  • A vibrant ecosystem of startups builds on top of these models. Constant innovation and increased investment fuel a fast-changing environment.

• Pitfalls of Bot Management

  • So far, mishaps are mostly minor—bots make mistakes, delete inboxes, or fumble tasks, creating more work. “People get irritated because... they're just not doing the things that they're asking them to do. And they feel like they're basically babysitting the AI bots.” — Kate Clark (04:02)

• Changing Nature of Software Engineering

  • Long-honed coding expertise risks obsolescence as natural language replaces code for many tasks.
  • Some engineers feel nostalgia or sadness: “People being really upset at the idea that the skill they spent their life learning is no longer relevant.” — Kate Clark (04:54)
  • Others argue foundational understanding is necessary to manage AI. The consensus: massive disruption and job loss are in the pipeline, moving at breakneck speed.

Nvidia and the "Age of Inference"

Nvidia GTC 2026
The company’s annual AI conference in San Jose puts “inference” front and center as CEO Jensen Huang announces a major move: licensing chip technology from Groq for $20 billion to beef up Nvidia’s inference hardware (06:40–08:02).

Key Insights and Discussion Points

• Nvidia’s Strategic Shift

  • Nvidia dominates AI “training,” but “doesn’t have quite as much of a foothold” in “AI inference”—the process of running and using trained AI models in real-world applications.
  • The Groq deal (“essentially an acqui-hire”—Nvidia licensed the tech and hired top leadership) positions Nvidia for the next phase in AI infrastructure. — Robbie Whelan (07:18)

• Inference vs. Training: Hardware Needs (08:02–09:58)

  • Inference: Requires vast memory and high bandwidth to quickly access trained model data.
  • Training: Requires raw computing power for “billions and billions of math problems.”
  • “Inference... requires a lot more memory because you have these models that are working to remember everything that they've been trained.”
  • Fast, low-cost inference is vital as AI becomes mass-market. Companies must “drive down costs” while maintaining “reliable response times.”

• Can Nvidia Compete? (09:58–11:08)

  • Nvidia is unlikely to be left behind: “They have essentially locked up their supply [of crucial memory chips] for the next two or three years.”
  • New challenge: thinner profit margins when moving from luxury products (training chips) to more mainstream, cost-driven inference products.
  • “It's very hard to sustain high profit margins.”

• Competitive Landscape (11:13)

  • Cerebra: $10B licensing deal with OpenAI for custom inferencing chips.
  • AR Labs: Focuses on interconnectivity chips using “advanced fiber optics and lasers” for faster inference—highlighting the importance of every part of AI hardware, not just main processors.
  • AMD: Scaling up in inference, gaining traction despite being smaller than Nvidia.

Notable Quotes & Moments

• "There is this anxiety... they really want to make sure they are being as productive as possible." — Kate Clark (02:29)
• “They feel like they're basically babysitting the AI bots.” — Kate Clark (04:02)
• “We're very quickly moving into an era where... you and me can just use natural English language to command these agents to create websites, to create products.” — Kate Clark (04:54)
• “Inference requires a lot more memory... Training requires a lot more raw computing power.” — Robbie Whelan (08:06)
• “There's a real worry... that Nvidia’s margins are not going to remain that high for very long.” — Robbie Whelan (10:09)

Key Timestamps

• 00:34–05:42 — Rise of AI agents, reporting from San Francisco (Kate Clark)
• 06:40–07:18 — Nvidia GTC conference & strategic moves (Katie Dayton, Robbie Whelan)
• 08:02–09:58 — Technical differences: inference vs. training hardware (Robbie Whelan)
• 10:09–11:13 — Nvidia’s competitive outlook, supply chain, margins (Robbie Whelan)
• 11:13–12:13 — Competition and new players in AI hardware (Robbie Whelan)

Takeaway

The episode spotlights both a cultural and technological inflection point: software engineering is being redefined by LLM-powered agents, and industry giants like Nvidia are retooling fast to power the "age of inference" as AI shifts from advanced R&D to everyday use. The pace of change leaves questions about productivity, job security, and who will win—or lose—this next chapter.