If you follow AI you might have heard the phrase, “scale is all you need.” The idea that to continue to improve the performance of AI systems, all you need is bigger models and more data. But as AI has continued its rapid advancement the tide is starting to shift on that paradigm. Many of the new AI language and image models released in 2024 have been a fraction of the size of the models we saw in early 2023. But even these smaller models are data hungry.

That’s where today’s guest comes in. In a widely circulated paper from April of this year, Vishaal Udandarao and his coauthors showed that when it comes to AI image models, while more data is better, it takes an exponential increase in data volume to achieve a linear improvement in model performance. With concerns that AI models have already exhausted much of the easily scrapable data from the web Vishaal’s paper has added fuel to the conversation around how AI progress can continue.

Vishaal is a second-year PhD student at the Max Plank Institute at The University of Tuebingen. He’s also affiliated with the European Laboratory for Learning and Intelligent Systems. Vishaal and I talk in detail about his paper’s results and about what solutions might be available to help continue the progress of AI model development by leveraging existing data more efficiently.

Full show notes available here.

Transitioning from scale to efficiency in AI model training