Catalyst with Shayle Kann

Episode Summary: “The mechanics of data center flexibility”

Release Date: August 28, 2025
Host: Shayle Kann
Guest: Varun Sivaram (Founder & CEO, Emerald AI)

Brief Overview

This episode of Catalyst explores the evolving concept of data center flexibility and its implications for the electricity grid amid massive growth in AI (artificial intelligence) infrastructure. Host Shayle Kann sits down with Varun Sivaram, CEO of Emerald AI, to discuss how data centers—historically seen as rigid, inflexible electricity consumers—can adapt their operations to become dynamic, flexible assets for grid reliability and decarbonization. The conversation dives into the technical, market, and regulatory nuances as data centers’ energy demand skyrockets and pressures power systems.

Key Discussion Points & Insights

1. Data Centers and Their Growing Energy Challenge

• Conventional grid view: Data centers are perceived as flat, unchanging loads, operating 24/7 at peak demand. However, actual operations differ and are often more flexible than assumed.
• Growth & density: AI data center power demand is doubling annually; compute demand is quadrupling (10:54). Rack power density has surged from 5 kW a few years ago to 132 kW today, heading toward 1 MW per rack.
• Quote:

  “AI’s power density is increasing by orders of magnitude, which I don’t think any other electricity application has seen in this short span of time...These massive data centers occupy a tiny footprint and look like small cities.”
  — Varun Sivaram [10:54]

2. How AI Workloads Translate to Electricity Loads

• Training vs. Inference:
  • Model training spikes sharply; unpredictable and energy-intensive.
  • Inference workloads are smoother but still variable.
  • Data center repurposing: Workload types can change over a center’s lifespan, making historical data a poor predictor of future patterns. [13:10]
• Grid complexity:
  • Grid operators must plan for worst-case scenarios (e.g., 10 years at full demand), which leads to overbuilding and challenges interconnection.
  • Quote:

    “You want to predict or plan for a worst-case scenario where the data center...shows up at the absolute worst time of the year...If so, can’t connect it today, have to upgrade the system before we do that.”
    — Varun Sivaram [07:00]

3. The Flexibility Opportunity: "Clever Stuff" Explained

• Physical vs. Digital Flexibility:
  • Physical: Onsite generation, batteries (often limited by regulation).
  • Digital: Orchestrating workloads—slowing, pausing, or shifting computational jobs in response to grid needs. [15:43]
• Temporal vs. Spatial Flexibility:
  • Temporal: Pausing, delaying, or slowing workloads.
  • Spatial: Moving workloads between locations/data centers (often for cost or carbon benefits).
  • Quote:

    “You might slow down a job...change how many chips...are instantaneously being used...or go all the way down to the underlying silicon and...change the clock frequency of the chip to change the rate at which computations happen.”
    — Varun Sivaram [20:00]

4. Service Level Agreements (SLAs) & Types of Workloads

• Historical barrier:
  • Rigid SLAs promising near-perfect uptime made flexibility unattractive.
• New model:
  • Customers increasingly willing to tolerate minor, well-defined interruptions—e.g., allow power capping 100–200 hours/year.
  • Spot vs. Guaranteed Instances: Multiple tiers of compute availability give workaround options.
  • Quote:

    "This is one of those cases where...We've got 50 to 100 gigawatts of latent AI demand...it's just not going to get built unless you have this capability of flexibility."
    — Varun Sivaram [23:30]

5. The Scale and Limits of Data Center Flexibility

• Demonstrated results:
  • Oracle/Nvidia/EPRI/Emerald AI pilot in Phoenix: 25% demand reduction for 3 hours with representative AI workloads (some cases up to 40% reduction) while maintaining user performance [28:19].
  • About 10% of workloads may be completely non-flexible.

    "It was surprising...that just 10% of the workloads...were non-preemptible. That gives us a lot of flexibility to work with."
    — Varun Sivaram [28:19]

• Limits:
  • Not all power (e.g., HVAC) is deferrable.
  • Amount of shift is dictated by the mix of workloads and agreed SLAs.

6. Market and Regulatory Acceptance

• Barriers:
  • Grid operators need rigorous, real-world demonstrations and robust verification before recognizing flexible data centers as reliable grid resources.
• Path forward:
  • Emerald AI’s “Emerald Simulator” (digital twin) forecasts the effects of orchestration, helping to build confidence for grid operators.
  • Ongoing and planned demos with utilities and research institutes are essential to mainstream adoption.
  • Quote:

    "That data, that ground truth reliability information is what's needed for grid operators and utilities to believe that this is actually a thing...They've got to see it to believe it."
    — Varun Sivaram [33:22]

Notable Quotes & Memorable Moments

• [06:34] On grid planning:
  “You want to predict or plan for a worst-case scenario...When a transmission line goes down somewhere and it's a record hot day...Will my 400 megawatt data center request its full 400 megawatts and overload a circuit and if so, can't connect it today, have to upgrade the system before we do that.”
• [12:49] On rapid AI growth:
  "The power demand from data centers has more than doubled every year the last several years...compute demand is more than quadrupling every year."
• [14:48] On workload volatility:
  "A data center will not do a single thing for its lifetime...A single data center may be used for one model, and then it's separated out into multiple different types of workloads."
• [20:00] On mechanisms for flexibility:
  “You might slow down a job. You might change the resource allocation...You might also go all the way down to the underlying silicon and...change the clock frequency of the chip..."
• [23:30] On market need:
  “We've got 50 to 100 gigawatts of latent AI demand in the pipeline...it's just not going to get built unless you have this capability of flexibility.”
• [28:19] On real-world results:
  “It was surprising...that just 10% of the workloads on a representative databricks cluster were non-preemptible. In other words, they absolutely could not be paused or delayed in any way.”
• [33:22] On convincing grid operators:
  "They've got to see it to believe it...that ground truth reliability information is what's needed for grid operators and utilities to believe that this is actually a thing..."
• [34:30] On the stakes for states:
  “…That chairman said, 'I've got the governor knocking on my door every month and saying what have you done for me to bring data centers to my state because I want to economically compete with all the other states?’ Regulators, utilities, system operators are all balancing this trade off..."

Important Timestamps

• 03:49: Data centers as “AI factories”; core concepts of electricity-to-token conversion
• 06:34: How grid planning for data centers is performed—risk aversion, long-term peak scenarios
• 10:54: What makes AI data center loads unique: fast growth and extreme density
• 13:10: Distinction between AI training and inference; workload profiles
• 15:43: Introduction to demand flexibility—physical vs. digital
• 20:00: Practical methods for temporal workload flexibility
• 23:30: Market drivers: why flex is now essential, not optional
• 28:19: Case study—demonstrating data center demand reduction in Phoenix
• 30:43: The critical role of SLAs and customer willingness
• 33:22: The path to regulatory acceptance and trust

Conclusion

This episode lays out a vision for AI data centers as not just an unprecedented demand on the grid, but potentially its most valuable and responsive asset. Data center operators, AI firms, and grid managers will need to move beyond outdated assumptions, develop new SLAs and partnership models, and invest in trust-building pilots. According to Varun Sivaram, flexibility is not only technologically feasible but increasingly economically and operationally essential to the future of both AI and clean electricity in the US.