Generative Now | AI Builders on Creating the Future

Episode: Bill Dally: NVIDIA’s Evolution and Revolution of AI and Computing (Encore)
Host: Michael Mignano, Lightspeed Venture Partners
Guest: Bill Dally, Chief Scientist and Senior VP for Research, NVIDIA
Air Date: January 16, 2025

Episode Overview

This encore episode features a conversation between Michael Mignano and Bill Dally—NVIDIA’s Chief Scientist and Senior VP for Research—exploring the company’s pivotal role in the ongoing AI revolution. The discussion covers Bill Dally’s personal journey from academia to industry leadership, NVIDIA’s groundbreaking advances in hardware and AI research, the rapid evolution of generative AI, and how NVIDIA harnesses its own technology for chip design and innovation. The episode also offers insights into future trends, advice for AI entrepreneurs, and reflections on how AI is transforming both education and work.

Key Discussion Points & Insights

Bill Dally's Background and Path to NVIDIA (02:00–06:00)

• Early Fascination with Neural Networks:
  Bill’s interest began 40 years ago at Caltech, experimenting with multilayer perceptrons and Hopfield nets when compute power was limited.

  “We built little multilayer perceptrons and convnets...but it also impressed me that it was a toy…the COMPUTE wasn't there at the time.” (02:08, Bill Dally)

• Pioneering Parallel Computing:
  As a professor at MIT, Bill saw parallelism as the way to scale performance, even as Moore’s Law made people complacent sticking with serial processes.

• Stream Processing to Cuda:
  At Stanford, developed stream processing and partnered with NVIDIA to create the NV50 (marketed as G80), making parallel processing accessible via CUDA.

• Early Signs of AI Potential:
  Reflected on Stanford’s Grand Challenge win for autonomous vehicles, noting the power of data-driven feature extraction and the era’s compute limits.

• Joining NVIDIA and Enabling Modern AI:
  Breakfast with Andrew Ng (Google Brain) in 2010 was a turning point; Bill realized GPUs could unlock neural networks’ promise, leading to the development of cuDNN.

Speed and Scope of the AI Revolution (06:00–08:16)

• The AI Explosion:
  Bill always believed AI would revolutionize every aspect of life (“how we play, how we work, how we educate, how we get medical care”), but the pace post-ChatGPT surprised him:

  “I thought the change was going to be more gradual and not quite as frenetic as it's turned out to be…when ChatGPT came out, it was like somebody turned the rate knob way up…” (06:31, Bill Dally)

• On Data as a Limitation:
  While web data might soon plateau, synthetic and private data repositories provide ample room for growth (07:37–08:16).

A Day in the Life of NVIDIA's Chief Scientist (08:16–10:49)

• Dual Role:

  • Chief Scientist: Focused on technological vision and pushing for new innovations company-wide.
  • Head of Research Org: Manages a 400-strong team (“giant playground”) clearing obstacles for researchers.

• Culture and Leadership:
  Bill's mantra: “My job is to get obstacles out of their way. I try to enable them by finding out what's blocking them and remove the blockages so they can do amazing things.” (09:22)

Current Frontiers in NVIDIA Research (10:49–12:55)

• Generative AI and Multimodal Models:
  Major focus on advancing generative models—language, vision, video, and especially multimodal approaches.

• Improving Model Understanding and Efficiency:
  The Finland research team’s notable work on diffusion models exemplifies deep dives into the fundamentals of new tech.

• On Data Enabled by Multimodal:
  Incorporating videos, images, and audio exponentially increases the training data pool and brings AI closer to human-like experiential learning.

  "Now our models can do that as well." (12:47, Bill Dally)

Understanding and Scaling Large Models (12:55–14:41)

• Pilot and Ablation Studies:
  NVIDIA performs small-scale studies to understand model mechanics before investing in massive training runs, blending “math behind what’s going on” with empirical validation.

• Production vs. Research:
  Scaling from research to production requires confidence—a failed big run means “a difficult conversation with Jensen at the end of the day.” (14:23, Bill Dally, referencing Jensen Huang, NVIDIA CEO)

Advanced Research Areas: Autonomous Vehicles and Hardware (14:41–17:31)

• Foundation Models for Autonomous Vehicles:

  • Use generative AI to script complex, realistic scenarios for simulation and train perception/planning stacks.
  • Bill notes, “There’s a lot of very exciting things coming together at that nexus of generative AI and autonomous vehicles.” (15:23)

• Staying Ahead in Hardware:
  Working on new number representations, sparsity handling, and efficiency in GPUs—they plan several chip generations ahead.

• Creativity over Moore’s Law:
  Architectural and design innovation matters more now as hardware improvements alone offer diminishing returns.

Realities and Advancements in Autonomous Vehicles (17:31–20:33)

• The ‘Long Tail’ Challenge:
  Handling rare driving scenarios remains tough:

  “A decade ago saying that we were almost there, and a decade later we're not…ultimately it's a tough game of chasing down the rare cases and making sure you handle them well.” (18:00, Bill Dally)

• Modeling Human Behavior:
  Autonomous systems are learning to characterize and predict individual drivers’ behaviors (“this one is aggressive, that one is about to fall asleep”).

AI-Enabled Chip Design at NVIDIA (20:51–25:56)

• Large Language Models for Hardware Design:
  Domain-specific pretrained LLMs help new designers learn faster, summarize bugs, and write code for tool configuration.

• Graph Neural Networks & Reinforcement Learning:

  • Predicting circuit parasitics in seconds (previously days), improving iteration speed.
  • RL-driven adder design beats decades-old human-authored methods.

• Standard Cell Library Automation:
  Designing standard cells now takes an RL model overnight (vs. 90 person-months), producing higher-quality designs.

  “...in an overnight run on one GPU…it achieves a higher quality so that the average cell is smaller than the ones designed by the humans…” (25:34, Bill Dally)

Building and Leading a World-Class Research Team (26:23–28:16)

• Growth and Recruiting:
  From 15 graphics-focused researchers to ~400 across fields, with deliberate hiring of top talent to create a high-bar, low-turnover culture.

• Industry Impact:
  NVIDIA’s breakthroughs spread industry-wide, making recruitment attractive compared to companies where new work stays siloed.

Navigating Platform Shifts & Entrepreneurial Advice (28:48–30:59)

• Core Competencies & Agility:
  NVIDIA’s strengths: parallel processing and domain-specific acceleration. The team continually anticipates new applications and adapts the platform accordingly.

• Advice to Builders:
  “...have a core expertise and get ahead of the applications…” (30:58, Bill Dally)

Preparing for the Future of AI Models (31:18–35:52)

• Staying Nimble:
  New neural network architectures constantly emerge—some (like transformers) replace established methods rapidly post-breakthrough.

• Academic Outreach:
  Bill spends time with universities to scout new ideas, keeping NVIDIA close to early-stage innovation.

• Adoption Trends:
  Notably, “in the AI world, people throw stuff away overnight and tomorrow they have a new model.” (34:45, Bill Dally)

• State Space Models:
  Bill is intrigued by these as a possible successor to transformers if their promise holds.

Education, Young Talent, and the Changing Field (35:52–40:06)

• Advice to Recent Graduates:

  • Prioritize learning and growth: pick workplaces with smart people, cutting-edge problems, and healthy culture.

    “New graduates…what they have is a license to learn…it's important for their first job is to pick the job where they're going to learn a lot and to learn the right set of things.” (36:21, Bill Dally)

  • NVIDIA’s environment “checks all three boxes.”

• Learning from New Generations:
  Young talent bring fresh perspectives; engaging with students helps veterans see beyond established paradigms.

• The Future of Computer Science Education:

  • Core CS fundamentals should remain, but “classical” coding could give way to building applications via LLM APIs.
  • “The generation of students that's coming out…will be thinking about how to plug together AI through a bunch of APIs…classical ways of programming” will diminish. (39:17, Bill Dally)

Notable Quotes

• On the Speed of AI’s Rise:

  “I was convinced…AI…was going to revolutionize all of human endeavor…But then when ChatGPT came out, it was like somebody turned the rate knob way up…” (06:28–07:16, Bill Dally)

• On Multimodal Models and Data:

  “A lot of our experiences is visual…now our models can do that as well.” (12:52, Bill Dally)

• On Building a Research Team:

  “We had to really set the bar high and hold it there…People come and they stay because they get to do what they want to do. They have the resources to do fun, fun experiments…impact…the whole world.” (26:44, Bill Dally)

• On the AI vs. Supercomputing World:

  “In the AI world, people throw stuff away overnight and tomorrow they have a new model. They don't care about the. It's fun, it moves really quickly.” (34:45, Bill Dally)

• On Shifting Role of Programming:

  “Right now they build applications by, you know, getting an API to an LLM and…piping their data into that…I think it'll be the more classical ways of programming [that go away].” (39:17, Bill Dally)

Timestamps for Key Segments

• 02:00–06:00 — Bill Dally’s journey: academia to AI pioneer at NVIDIA
• 06:28–08:16 — Reflections on the frenetic acceleration of AI, explosion of generative models
• 08:25–10:49 — Bill’s dual role and daily leadership at NVIDIA
• 10:49–12:55 — Hot topics: generative AI, foundation models, and data expansion
• 14:50–17:31 — Foundation models in autonomous vehicles, supply-side hardware innovation
• 18:00–20:33 — Challenges and nuances of autonomous driving progress
• 20:51–25:56 — Using AI for chip design: LLMs, GNNs, RL in hardware engineering
• 26:23–28:16 — Recruiting and building a world-class deep tech team
• 28:48–30:59 — Advice for experts and startups: stay agile and anticipatory
• 31:18–35:52 — The AI research horizon: new models, rapid shifts, and adoption
• 35:52–40:06 — Guidance for new grads; how AI is changing the CS curriculum

Memorable Moments

• Turning Andrew Ng’s Cat Detector Code into cuDNN:
  A pivotal moment, initiating NVIDIA’s direct role in accelerating deep learning (03:21–06:00).

• Applying RL to Hardware Design:
  Using reinforcement learning to outperform “textbook” adder designs in GPU circuits (23:39–25:56).

• Education and the ‘API Era’:
  Bill’s vision of a future where plugging together AI services supplants traditional programming skills (38:46–40:06).

Conclusion

This episode offers a sweeping yet grounded exploration of AI’s present and future from the perspective of Bill Dally—a foundational thinker driving NVIDIA’s relentless innovation. Whether discussing hardware breakthroughs, the unpredictable trajectory of AI models, or the changing landscape of education and work, Dally’s blend of technical insight, experience, and candid advice provides listeners with a vivid sense of how the AI revolution is being built from the inside—and where it might go next.