Yesterday, I wrote about the state of deep learning theory circa 2016,[1] as well as the bombshell 2016 paper that arguably signaled its demise, Zhang et al.'s Understanding deep learning requires rethinking generalization.

As a brief summary, I argued that the rise of deep learning posed an existential challenge to the dominant theoretical paradigm of statistical learning theory, because neural networks have a lot of complexity. The response from the field was to attempt to quantify other ways in which the hypothesis class of neural networks in practice was simple, using alternative metrics of complexity. Zhang et al. 2016 showed that the standard neural network architectures trained with standard training methods could memorize large quantities of random labelled data, which showed that no such argument could explain the generalization properties of neural networks.

Today we’re going to look at the aftermath: how did the field of deep learning theory react to this paper? What were the attempts to get around this result using data-dependent generalization bounds? And why did Nagarajan and Kolter's humbly titled Uniform convergence may be unable to explain generalization in deep learning serve as the proverbial final nail in the coffin of this line of work? [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
April 26th, 2026

Source:
https://www.lesswrong.com/posts/zcGmdQHX66NhC69v6/the-other-paper-that-killed-deep-learning-theory

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Mathematical theorem and formula defining spectral complexity for neural networks.

Four log-log plots showing distance, spectral norms, test error, and bounds versus training set size.

Mathematical theorem and setup describing linear classifier weights and dimensional vectors with training error bounds.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.