By Walt Hickey

Welcome to the Numlock Sunday edition.

This week, I spoke to Glenn McDonald, author of the new book You Have Not Yet Heard Your Favorite Song: How Streaming Changes Music.

I’ve followed Glenn’s work for years now, and this book is the result of decades of work in the field, and comes from a perspective not only of technology’s bleeding edge but also a sincere, personal love of music.

We spoke about the mechanics of tracking genre data, how streaming has impacted listening trends, and how the model’s economics are holding up.

The book can be found everywhere books are sold.

This interview has been condensed and edited.

Glenn McDonald. Thank you so much for coming on. You are the author of You Have Not Yet Heard Your Favorite Song, which is a really compelling title all about the streaming revolution, but more importantly about this really fascinating moment in music data that for much of the past decade, you have been at the front seat for, or even in the driver's seat for. Your work goes back to a really interesting company called The Echo Nest.

For new listeners and folks who maybe are unfamiliar with your story, can you just tell me a little bit of the history of this field and your place in it? The Echo Nest was a really fascinating company and I think more people ought to know about it.

I had been doing software design for a long time and had worked on a bunch of different things that all had to do with making sense of data for people. None of them had been specific to music data, but I would always use my work tools on my record database or my other various music-related projects. Those were the things that I was really interested in.

At some point I ended up tabulating the Village Voice critics music poll every year. This was what big data was for music in the era before streaming: It was like 800 music critics typing 10 album votes into blanks, with typos and everything. The companies I worked for kept getting acquired and my projects would get shut down or something, so every few years I’d need a new job.

When this happened in about 2011, I just knew through contacts that there was this company in Somerville a couple of media lab people had started called The Echo Nest, which was trying to do something with music data because there suddenly was a lot more music data. The Echo Nest was trying to do recommendations and categorization stuff for streaming services. This was pre-Spotify launching in the U.S., so 7Digital and Rdio at the time were some of the existing players. And I had done enough music data things to convince them that I was a worthwhile person to add to this effort.

I remember my first task at The Echo Nest. I showed up for my first day and they were like, “Oh, Glenn, you're here. Good. We're doing these radio stations for Spotify, this company we're trying to entice into using our services, and we're putting cartoon noises on the Franz Liszt classical station. Can you please figure out why we're doing that and make it stop?”

So that was the beginning of the journey. We did not succeed initially at getting Spotify as a customer, because Spotify recognized, correctly, that to do a really good job we had to have listening data, and there was no way they were going to give us listening data when we were also powering their competitors, even though their competitors were small. I remember we tried really hard to convince them. We were like, “We'll keep your data on a server on the totally other side of the closet where we have our servers.” That obviously didn't fly. After a couple of years of doing a lot of other things along the way, it was a race to see whether Spotify would develop their own recommendations and not need us or whether they would just get enough money to buy us first.

The money happened faster. We got acquired in 2014 and basically officially became the personalization team at Spotify. A bunch of the things we did had to do with understanding music and understanding taste so that you could do personalization, but it wasn't all directly involved with personalization.

That's about when you got on my radar, because I was at the time doing pop culture stuff at 538. I think music was always a bit of an enigma to me, just because there was so much of it, and obviously all the challenges that you were facing at an industrial scale, I was facing on a journalistic scale.

When I saw what you were doing — I don't think people really appreciate enough the moment when The Echo Nest got bought by Spotify. Very soon after, that's when you started seeing the level of personalization on the platform skyrocket. Do you want to speak a little bit about that?

It was a combination of things, because some of that stuff was stuff that we brought, The Echo Nest, but that acquisition was also Spotify's moment where they bought into the idea that personalization was going to be a big part of this Spotify experience. Discover Weekly, for example, which came along shortly after that, was not an Echo Nest thing. That was done by people who had already been at Spotify, and some of them were annoyed that that feature was described as if it somehow came from the Echo Nest work.

But basically everything that came about was because Spotify decided, “All right, personalization is going to be a thing.” At the same time, they acquired another company called Tunigo that was a playlist-making editorial company. And that was the beginning of, “All right, we're really going to have an editorial effort, too.” That was the beginning of both of those areas at once in Spotify's existence.

A lot of the interesting stuff in your book comes out of the complications of using algorithms as opposed to taste, and just the serendipity of some of it. I want to play something, because I think it shows the moment that completely shattered something about how I thought about music and how I thought about what the tech was that y’all were building.

I have a lot of Spotify playlists, just as anyone else does, and I was on a kick and I added a few songs in a row. The following week, all 10 recommended songs had this kick to start it. It’s the “Be My Baby” kick. There’s no way you could ask a DJ or even an expert, “Hey, can you find me 10 songs that all have this kick that I’m apparently into right now?”

But lo and behold, I would go down this recommended songs list and it would all be that. And that showed me that, man, there's a level of depth here that not only could we never accomplish before, but that is going to change the way we really consume a lot of this stuff. I always found it really fascinating how you were really on the front of that for so much of the time at Spotify.

One of the most interesting things to realize for me in this journey was that finding those patterns often comes about not the way you think.

How did that happen?

You imagine that the computer knows there are those drumbeats, it's found that you like them, and knows these songs contain them and lines them up. In fact, in this feature, that's not happening at all. It's just patterns of playlist making.

That recommended feature at the bottom, it uses the playlist title when you don't have anything, but then as soon as you've got stuff in your playlist, it's really just doing a complicated search of songs and playlists from other people that overlap with what you put. Here's what else they did. I found over and over that it was more effective to basically mine listening for the implicit signal that people have created by listening in nonrandom ways than it was to try to find the thing you're actually looking for.

If you try to find bands from Estonia, you get screwed up by metadata mistakes and missing data all the time. But if you can find a few bands that you know are from Estonia and use them to find an audience and use that audience to find what's different about those people's listening, then you find all the rest of the bands from Estonia without having to rely on metadata. Even the system doesn't know what it's doing. People have encoded that knowledge implicitly by listening.

So I did find someone who’d been on a kick of listening to all the “Be My Baby” hooks in a row. It's fascinating stuff.

I want to take it to the book now, because that speaks to a chapter specifically all about how you talk about genres and how genres don't really exist; they're just words that people use to talk about things. You describe them as “distributed communities of interest.” Do you want to speak a little bit about what genres are?

We got into this genre thing at The Echo Nest because we promised somebody that we had genre radio. It was the era of Pandora. Algorithmic radio was mostly track and artists seated. That was how people mostly thought about it.

We had some customer — I've long since forgotten who they were — who was like, that's too complicated. I just want like 16 buttons. It should just say rock and you hit it and it plays some rock music. And we were like, “Oh yeah, totally. We totally have that.” And then we went back to the office and we were like, we don't actually have that. But we better make it really quick.

What we did have was...