Software Engineering Daily: Turing Award Special – A Conversation with Jack Dongarra

Released on March 18, 2025

Introduction

In this special Turing Award episode of Software Engineering Daily, host Shawn Falconer engages in an in-depth conversation with Jack Dongarra, a celebrated computer scientist renowned for his groundbreaking work in numerical algorithms and high-performance computing (HPC). Dongarra, a recipient of the 2021 Turing Award, shares insights from his illustrious career, discusses the evolving landscape of supercomputing, and explores future directions in the field.

Defining High Performance Computing

[01:12] Shawn Falconer: “What defines high performance computing and is that a moving target as mainstream computers that we use every day become more powerful over time?”

[01:32] Jack Dongarra: “High performance computing, or supercomputers, are usually specified as the fastest computers at any time. ... Supercomputers are fast in terms of floating point operations... They are characterized by being quite expensive as well. For example, the fastest computer today is located at Lawrence Livermore National Laboratory, costing about $600 million.”

Dongarra explains that the definition of supercomputers is dynamic, with each new generation surpassing the previous in speed and cost. The rapid advancement in mainstream computing power continually redefines what constitutes a supercomputer.

Jack Dongarra’s Career Path

Shawn Falconer delves into Dongarra’s journey into HPC.

[02:51] Jack Dongarra: “I initially wanted to be a high school science teacher... An internship at Argonne National Laboratory transformed my outlook, leading me to pursue computer science instead.”

Dongarra recounts transitioning from education to research, earning his master’s degree at Illinois Institute of Technology, obtaining a PhD from the University of New Mexico, and working at prestigious institutions like Los Alamos National Laboratory. His consistent focus on HPC has spanned decades, primarily at Argonne National Lab and later at the University of Tennessee and Oak Ridge National Laboratory.

Motivation in a Rapidly Evolving Field

[07:09] Shawn Falconer: “Is the moving target nature of supercomputing something that’s helped keep you motivated to focus on this field throughout your career?”

[07:37] Jack Dongarra: “It’s exciting to see new architectures and try to understand how they can effectively be used to solve problems. ... Each architectural change requires us to rethink algorithms, software, and numerical libraries.”

The constant evolution in supercomputing architectures, from scalar to vector computers, and then to parallel and multicore processors, keeps the field vibrant and challenging. Dongarra emphasizes the necessity of adapting software to leverage new hardware advancements continually.

Data Movement: The Bottleneck in Supercomputing

[15:57] Shawn Falconer: “What are some of the approaches to reduce the amount of communication that's happening at the hardware level?”

[16:15] Jack Dongarra: “Data movement is the biggest bottleneck... We need ways to overcome the memory bottleneck, such as embedding processors in memory itself and organizing computations around directed acyclic graphs to maximize parallelism.”

Dongarra highlights that while floating-point operations have become highly efficient, the primary limitation in supercomputing is the movement of data. Overprovisioning of floating-point units without corresponding improvements in data transfer rates leads to inefficiencies, with applications typically achieving only about 10% of a supercomputer's peak performance.

Benchmarking: Beyond Linpack

[22:54] Shawn Falconer: “Is the solve for that, that essentially there should be more than one measurement or KPI that is used to benchmark supercomputers?”

[23:18] Jack Dongarra: “The best benchmark is the application you intend to run. Linpack was developed when floating-point operations were very expensive, but it no longer reflects modern applications. We developed the HPCG benchmark to better represent current scientific computations.”

Dongarra discusses the limitations of the Linpack benchmark, which primarily measures dense matrix operations, and introduces the High Performance Conjugate Gradients (HPCG) benchmark. HPCG focuses on iterative methods used in solving sparse systems of linear equations, providing a more accurate assessment of a supercomputer's real-world performance. He advocates for a diverse set of benchmarks to capture the multifaceted nature of HPC applications.

Impact on the Top 500 List

[24:49] Jack Dongarra: “I want to augment the Top 500 with other benchmarks like HPCG. The Top 500 gives us a handle on peak performance, but additional benchmarks provide a more realistic view of application performance.”

While the Top 500 list remains a valuable tool for tracking the fastest supercomputers, Dongarra suggests complementing it with additional benchmarks to better assess performance across different types of applications. This approach ensures a more comprehensive evaluation of supercomputing capabilities.

AI and Mixed Precision in HPC

[26:15] Shawn Falconer: “There's a shift to AI-driven workloads with 16-bit versus 64-bit floating point arithmetic. How does that change how we measure HPC performance?”

[26:48] Jack Dongarra: “AI is driving the adoption of lower-precision computations. We’re moving from 64-bit to 32-bit, and now to 16-bit and even 8-bit floating point operations. Mixed precision leverages lower precision for speed while using higher precision to maintain accuracy.”

Dongarra explains that AI workloads benefit from lower-precision arithmetic, enabling faster computations and reduced memory traffic. This shift necessitates new algorithm designs that can effectively utilize mixed precision to balance speed and accuracy, fundamentally altering how HPC performance is measured and optimized.

[31:08] Shawn Falconer: Interjects with an advertisement which is skipped in the summary.

Exascale Computing: The Next Frontier

[38:57] Jack Dongarra: “Exascale computers perform 10^18 floating point operations per second. For example, a machine at Livermore National Lab has a peak performance of 2.7 exaflops with 11,000 nodes, each containing multiple CPUs and GPUs. These systems consume enormous power, about 34 megawatts.”

Dongarra provides a detailed overview of exascale supercomputers, emphasizing their immense computational power and energy consumption. He underscores the critical role of GPUs in achieving peak performance and the importance of application parallelism to fully utilize these massive systems.

Future Directions: Beyond Traditional HPC

[42:39] Shawn Falconer: “What are your thoughts on the potential of quantum computing? Is it overhyped?”

[42:57] Jack Dongarra: “Quantum computers won’t replace conventional computers but will augment them. They hold great potential, but currently, only a few algorithms can effectively utilize quantum computing. It’s an exciting research area, though somewhat overhyped.”

Dongarra acknowledges the promise of quantum computing as a complementary technology to traditional HPC. While optimistic about its future applications, he cautions against expecting it to supplant existing computing paradigms in the near term. He also mentions other emerging technologies like neuromorphic and optical computing, which could further diversify HPC architectures.

Key Contributions and Legacy

Reflecting on his career, Dongarra highlights three major contributions:

1. Numerical Libraries for Linear Algebra:

   • Developing software that adapts to evolving hardware architectures, ensuring efficiency and performance in scientific computations.

2. Message Passing Interface (MPI):

   • Establishing a community-driven standard for message passing in parallel computing, fostering interoperability and collaboration across research groups.

3. Performance Benchmarks:

   • Creating and promoting benchmarks like Linpack and HPCG to evaluate and guide the development of supercomputers, enhancing their alignment with real-world applications.

[46:06] Jack Dongarra: “I’ve contributed to numerical libraries, the MPI standard, and performance evaluation through benchmarks like Linpack and HPCG.”

These contributions have significantly shaped the landscape of HPC, providing the tools and standards that underpin modern supercomputing efforts.

Conclusion

[49:14] Shawn Falconer: “Jack, thanks so much for being here. It’s been a real honor.”

[49:23] Jack Dongarra: “Great. Very good, Shawn. Thanks for the opportunity.”

Jack Dongarra’s insights offer a comprehensive view of the challenges and advancements in high-performance computing. His work continues to drive the field forward, bridging the gap between evolving hardware and the demanding needs of scientific research.

Notable Quotes

• On Supercomputing Evolution:

  • Jack Dongarra ([01:32]): “Supercomputers are fast in terms of floating point operations... They are characterized by being quite expensive as well.”

• On Data Movement Bottleneck:

  • Jack Dongarra ([16:15]): “Data movement is the biggest bottleneck... We have machines today which are really over provisioned for floating point operations.”

• On Mixed Precision:

  • Jack Dongarra ([26:48]): “We’re moving from 64-bit to 32-bit, and now to 16-bit and even 8-bit floating point operations.”

• On Quantum Computing:

  • Jack Dongarra ([42:57]): “Quantum computers won’t replace conventional computers but will augment them.”

• On His Contributions:

  • Jack Dongarra ([46:06]): “I’ve contributed to numerical libraries, the MPI standard, and performance evaluation through benchmarks like Linpack and HPCG.”

This conversation provides a valuable exploration of high-performance computing’s past, present, and future, guided by one of its most influential figures. Jack Dongarra’s expertise offers listeners a deep understanding of the intricate balance between hardware advancements and software optimization, underscoring the continuous innovation required to push the boundaries of computational science.