Podcast Summary: The Twenty Minute VC (20VC) – NVIDIA vs Groq: The Future of Training vs Inference
Episode Title: NVIDIA vs Groq: The Future of Training vs Inference | Meta, Google, and Microsoft's Data Center Investments: Who Wins | Data, Compute, Models: The Core Bottlenecks in AI & Where Value Will Distribute with Jonathan Ross, Founder @ Groq
Release Date: February 17, 2025
Host: Harry Stebbings
Guest: Jonathan Ro, Founder and CEO of Groq
Introduction
In this episode of The Twenty Minute VC (20VC), host Harry Stebbings engages in an in-depth conversation with Jonathan Ro, the founder and CEO of Groq—the creator of the world's first Language Processing Unit (LPU). Jonathan brings a wealth of experience from his time at Google, where he contributed to the development of the Tensor Processing Unit (TPU). The discussion delves into the competitive landscape between NVIDIA and Groq, the future of AI training versus inference, data center investments by tech giants, and the core bottlenecks in AI development.
Scaling Laws in AI
Jonathan Ro begins by addressing the scaling laws in AI, referencing a pivotal paper published by OpenAI. He explains that while increasing the number of parameters in a model generally enhances its ability to absorb information, there are diminishing returns due to the quality of data used in training.
“When you are growing faster than exponential, there is no amount of profit that you can make that matters. What matters is getting a toehold in the market and becoming relevant.”
— Jonathan Ro (00:00)
Ro emphasizes the importance of synthetic data over real data, arguing that as models become smarter, they can generate higher quality synthetic data to train themselves more effectively. This iterative improvement helps models transcend the asymptotic limitations traditionally associated with scaling laws.
Competition with NVIDIA
A significant portion of the conversation focuses on Grok's strategic positioning in the AI hardware market, particularly in relation to NVIDIA. Jonathan outlines Groq's unique approach to handling AI inference, contrasting it with NVIDIA's dominance in training.
“We're one of the best things that ever happened to Nvidia because they can make every single GPU that they were going to make and they can sell it for training. ... We are growing faster than exponential.”
— Jonathan Ro (00:00)
Grok aims to own the inference market by providing LPUs that are more cost-effective and energy-efficient compared to NVIDIA's GPUs. While NVIDIA continues to excel in training, Groq focuses on high-volume, low-margin inference operations, allowing both companies to benefit without directly cannibalizing each other's markets.
Data Center and Compute Bottlenecks
Jonathan discusses the bottlenecks in AI, particularly focusing on compute, data, and algorithms. He points out that while compute has traditionally been seen as a less restrictive bottleneck due to its fungibility, the rapid scaling of chip deployment is beginning to strain data center power capacities.
“... the more inference you have, the more training you need and vice versa.”
— Jonathan Ro (22:55)
Grok's partnership with entities like Aramco highlights their strategy to overcome chip supply constraints by leveraging long-term funding and infrastructure support. However, Ro warns of an impending power bottleneck within the next three to four years, as the demand for compute power outpaces available data center capacities.
Business Model Innovations
Grok's innovative business model involves partner-funded deployments, allowing them to scale aggressively without being limited by traditional capital constraints. This model not only accelerates their market penetration but also aligns Groq's incentives with their partners.
“... we have a very positive contribution margin right now. ... we're making money running these open source models.”
— Jonathan Ro (37:47)
By structuring deals where partners fund the capital expenditure (CapEx) for deploying Groq's chips, the company can focus on scaling its infrastructure to meet the burgeoning demand for AI inference without diluting its financial stability.
AI Predictions and Future Outlook
Jonathan shares his vision for the future of AI, predicting significant advancements once current challenges like hallucinations in models are addressed. He outlines several stages of AI evolution, including:
- Solving Hallucinations: Eliminating inaccuracies in AI responses to enable reliable applications in critical fields like medicine and law.
- Breaking Down Subgoals for Agency: Enhancing models to handle complex, multi-step tasks efficiently.
- Invent Stage: Moving beyond predictable outputs to foster creativity and invention.
- Proxy Stage: Allowing AI to make decisions on behalf of humans, streamlining processes and enhancing productivity.
“We’re growing faster than exponential. ... there is no amount of profit that you can make that matters.”
— Jonathan Ro (31:03)
Ro emphasizes that Groq's focus is on maximizing compute deployment to preserve human agency in the age of AI, asserting that rapid growth and relevance in the market are paramount over immediate profitability.
Geopolitics in AI: China and Europe
The discussion shifts to the geopolitical landscape of AI, with Jonathan analyzing China's advancements and Europe's challenges. He acknowledges China's willingness to utilize synthetic data and streamline deployments but highlights potential drawbacks related to censorship and privacy, which could stifle innovation.
“You're saying ... if you are running a Chinese tech company, your fear is that you become Jack Ma.”
— Jonathan Ro (56:32)
Regarding Europe, Ro critiques its regulatory environment, suggesting that excessive regulation and risk aversion are hindering its ability to capitalize on AI advancements. He advocates for fostering an entrepreneurial ecosystem similar to Silicon Valley to spur innovation and competitiveness.
Conclusion
In this enlightening episode, Jonathan Ro of Groq provides a comprehensive analysis of the current and future state of AI infrastructure. By strategically positioning Groq in the inference market and addressing critical bottlenecks in compute and data center capacities, Groq aims to play a pivotal role in shaping the AI landscape. Jonathan's insights into scaling laws, competitive dynamics with NVIDIA, and the geopolitical implications of AI advancements offer valuable perspectives for founders, investors, and technologists navigating the rapidly evolving world of artificial intelligence.
Notable Quotes:
- Jonathan Ro (00:00): “We're one of the best things that ever happened to Nvidia because they can make every single GPU that they were going to make and they can sell it for training. ... We are growing faster than exponential.”
- Jonathan Ro (31:03): “We’re growing faster than exponential. ... there is no amount of profit that you can make that matters.”
- Jonathan Ro (22:55): “... the more inference you have, the more training you need and vice versa.”
- Jonathan Ro (37:47): “... we have a very positive contribution margin right now. ... we're making money running these open source models.”
These quotes encapsulate Groq's strategic vision, emphasizing the importance of scaling and market positioning over short-term profitability.