← Harvard Data Science Review Podcast
Harvard Data Science Review Podcast
Recreations in Randomness: From Glicko Rating to World Cup
00:39:57
Loading summary
Transcript33 lines
- [00:02]
A
Welcome to the Harvard Data Science Review podcast. I'm your host, Liberty Vitter Capito, and I'm joined by my co host and editor in chief, Shali Meng. Today we're diving into one of the fastest evolving fields in applied data science, sports analytics. Over the last decade, we've moved from simple box scores and batting averages to real time player tracking, computer vision, deep learning models that can predict injury risk before a player even feels pain, and AI driven tactical systems that coaches consult mid match. The numbers have never been richer and the questions have never been harder. We're recording this in the middle of a remarkable sporting summer. The 2026 FIFA World cup is underway right here on home soil. The first 4018 tournament in history, generating a torrent of data that analysts are racing to make sense of. And with the US Open and the Open Championship just around the corner, tennis is about to give us its own annual test of who can perform when the pressure is highest and the margins are thinnest. To make sense of all of it, I have two extraordinary guests today. Mark Glickman is a Senior Lecturer in statistics at Harvard University and the creator of the Glico rating system, a probabilistic framework for measuring competitor strength that has been adopted everywhere from international chess to online gaming. He's one of the leading minds in statistical modeling of competitive outcomes. Stephanie Kovalcic is a sports statistician whose focus on tennis analytics has set the standard for how we understand player performance, aging and pressure at the elite level. She's published extensively built tools the tennis world actually uses and brought genuine scientific, rigorous to a sport that desperately needed it. Mark, Stephanie, welcome. Let's get into it, shall we? Take it away.
- [02:12]
B
Thank you Mark and Steph for joining this podcast. And Mark, I want to first thank you for creating and editing this HDSR columns on recreations, randomness, if I remember correctly, the name actually you created, which is a really wonderful name. It's a column on the data science for the leisure activities and sports is part of it, but there are a lot more than that. And Steph, I want to welcome you as the new co editor working with Mark. But I also want to thank you for your wonderful article back in 2021 for the column on why the tennis is still not ready to play the Moneyball. Because really about the availability of the data, which is something that I've been emphasizing and we all emphasize, no matter how the AI technologies go forward, the most important thing is to have data have good quality data. Without it, this technology cannot do much to speak of technology. The first question I have with both of you is that in the world of sports analytics, what are the one most transformative methodologies or technologies and are we using them, using it wisely? Mark, maybe you go first.
- [03:31]
C
Sure. I guess off the top of my head, there are two that really spring to mind. So one of them is just the notion of using detailed tracking data to be able to characterize very detailed kinds of movements on the pitch or chord or field. That gives just a lot more richness to the kinds of ways of describing how players and teams carry out their, their tactics at a fairly precise level. And then the second is having these very specific models that basically estimate the probability of a team winning at any particular point in the game based on situational factors. So there are a lot of different procedures and applications to particular sports where you can look at, for example, say in basketball that, you know, the game has 15 minutes left and one team is up by a certain margin and they're playing in the home field having just this very specific situational aspect of the game and being able to then have a calculation that says, here's the probability that the game is going to be a win. And the reason that's important is that you can use that information, for example, to start evaluating players by who's actually on the court and when they get substituted in. So you can basically say relative to a player being on the court and then play for a while and then come off the court. You can see how the win probability has changed over that period. And that gives you the basis for being able to evaluate, like, how good players are, how valuable they are to their teams.
- [05:15]
B
Steph?
- [05:16]
D
Yeah, I would really echo Mark's points. I think one of the things that really fascinates me is how the kinds of methodology and questions that researchers ask is really driven by what the state of the art is in data. You'll see that in any field. But within sports analytics, it's quite striking when the typical arc in sports analytics has been beginning with maybe box scores and then being able to do more play by play data and then tracking data. That's sort of the era that we're in, although we're starting to see some movement to even newer types of data beyond that. But one of the things that came out really immediately once more of that data became open was methods like expected possession value. That was one of the, the kind of big ideas that originally came out of the XY hoops group that would have never been possible without having access to that positional data. But what's also interesting is where you still see the fundamentals that preceded it within it. Because although EPV requires spatially driven models, it does kind of sit with on what is fundamentally a Markov model. So it's still something where you always see new methods building on ideas that came before. And so there's just a lot of interesting examples of that. But in general, we are really driven by the data that's at hand.
- [06:56]
B
Okay, so I guess Mark and Steph, what you're saying is these AI technologies basically are doing two things. One is allow us, oh, it doesn't even have to be AI. The technology allows us to collect more data, more precise data, more situational data. And the other is the modeling aspect. But I guess, am I correct that these modeling we're talking about, the better models are inevitably involving more deep learnings like language models, things of that nature, instead of traditional pure kind of a statistical modeling? Am I correct on that?
- [07:31]
D
I think it's actually been the opposite. I think traditionally where you've seen more, let's say neural network based models has been more in the data collection. So in the absence of systems like optical systems or camera based systems like Hawkeye, for example, which is one way to get positional data, other organizations have tried computer vision. And so that would have been a place where more AI would have been applied, but purely for extracting, capturing data. But it was still the case that as far as extracting insights from those data, that's traditionally been more within statistical modeling. It would depend a bit about the nature of the sport. Like for example, the original EPV work was applied to basketball. And then a few years later there were papers that started to appear applying to soccer or European football, let's say. And there's you started to see more neural network approaches actually applied to the analysis as well, partly because of some of the challenges within that sport. Very low scoring, dynamic play, these kinds of things. So somewhat of a mix. But I would say prior to the kind of Gen AI era that we're in right now, I think you would see a lot of traditional statistical models for the actual analysis versus data capture.
- [09:00]
C
Yeah, I completely agree with Steph. I haven't really seen much in the way of LLMs playing much of a role in constructing prediction or inference models in sports. And the types of instances that I've seen of using, you know, particularly deep neural nets has been to basically like combine different optical camera information so that it becomes one usable data set. That's really been the nature. It's More coming from the hardware side and then assembling data that can then be analyzed. And to be fair, the tools that are being used to analyze this kind of data, you know, there's some machine learning tools that are being used, but I don't think they, at least at this point, seem to be pushing the boundaries of using deep neural nets. It just hasn't really gone in that direction. And probably part of it is just the types of applications haven't just lent themselves so solidly to being able to use them just yet. Oh, that's actually quite interesting.
- [10:02]
B
So you're saying both of you are saying essentially these tools are more used for creating better database or combining.
- [10:10]
C
Yeah, data processing, really data processing itself
- [10:13]
B
and not necessarily on the analysis side. I mean, probably. Is that also the reason that maybe because in the sports area you do get lots of replications, lots of human data. So it's sort of, you have a better understanding the modeling aspect and you don't have to be blindly just seeking patterns. This is what most of these large equipment want to do. Well, when humans don't really quite understand what's going on, is that also a fair description of worth while you're really doing a lot more kind of good modeling instead of just relying on, say, hey, let's just find out whatever the pattern has.
- [10:55]
D
I think it will depend a bit on the model developer. I think if you're working for teams, interpretability becomes really important. So even if there might be a predictive edge for, let's say, an XG boost method or deep learning architecture, there's a great disadvantage in being able to sort of debug, tune those models properly and then also be able to interpret what they're doing. So I think that will definitely be a factor. So if you're a sports researcher for a betting company, then your goal may just be pure predictive performance and interpretation is less, less important. And so there you're probably exploring every method that you can get your hands on. So it'll vary with the setting. And then obviously in academia there's more of a concern about advancing the methods themselves and maybe less focus on what you learn from applications, or there's just much more around originality and novelty. So it will vary some. And then, I think, you know, with the popularity of the LLM models now, I imagine there will be more of an interest in applying some of those architectures, like the transformer architecture, to support tracking data. You can already find examples of that. But what's interesting there is the goal is really about Generative outcomes making plausible responses to questions. Whereas in sport, often the questions, usually they center around predicting an outcome of interest or being able to assess a skill or the quality of a decision or action. So how you would sort of fit these generative architectures towards those goals, I think would be kind of the big questions that sports researchers will have to tackle. But I imagine we will see more of that.
- [12:55]
C
Steph is actually reminding me that part of what may be going on is somewhat an economics issue, which is to say that most researchers are situated in universities. And the access to the kind of data that can be analyzed realistically for interesting sports research is actually quite limited because most of these teams in organizations have proprietary data that they're really not willing to make so available that they want, you know, researchers to publish work that just can be shared widely. So there are pockets of people that do get access to these data, but the amount that they can actually publish is really quite limited. So I think part of what's going on is I think academic researchers are stifled a little bit because they just don't have access to the data. While, you know, industry folks, you know, may have plenty of access to the data, but then they need to produce results on a quick timescale and they may not be able to dig in quite so deeply and apply these deep neural net models or do investigation using LLMs on a large scale to be able to get actionable answers to questions. So that plays a role, I think as well.
- [14:18]
B
Yeah, that's a really great insight. That reminds me the article Stephanie, you wrote. So let me put in another question before I turn to liberty. Steph, you wrote that time about five years ago about why the tennis is not ready then to play the Moneyball, because these different leagues, they have different commercial interests, I guess they don't share the data that much with each other. And that was major issues. Have things changed or it's too soon to change or they're still pretty much the same? How does all the revolutions on all the AI technology, everything, does that make people want to share more or share less? So what's the situation there now?
- [15:04]
D
Yeah, it is a difficulty that has been, I mean, certainly like a regret of mine, I guess, for tennis analytics in general, that there is this perception within sport industry that making the state of the art data, which at the time that I started working for Tennis Australia, which was my first role in sports, the state of the art was tracking data. And there's the perception that if that were to be made open, that organizations would lose their ability to. To benefit from the commercial potential that that data has. But I actually think it's the opposite. If you do see, like, examples of other sports that have been more willing to share, particularly with the academic community, that they have seen much more advance in the kind of questions and methods that are getting applied to that data. So I had mentioned before about the XY Hoops group. That's a perfect example of that, because that was only possible because Kirk Goldsberry, who's now sort of famous for his Court vision work, he was given access at that time. It was Sport Vu, one of the main systems that was capturing tracking data within the NBA. Reams and reams of positional data that none of the staff within the league really knew how to even begin to deal with. And so they approached Kirk because he had already presented work at the MIT Sloan Sports Analytics Conference with his court vision methodology. And then it just so happened that Luke had recently joined Harvard's faculty and came with expertise on spatial statistics. And so that was how their collaboration started. And over the next few years, they essentially, you know, became a factory of sports analytics ideas based off of what kind of exploration and questions that they were able to investigate with that data. And then more recently, I think you see the same thing happening with Mike Lopez and the Big Data bowl, where through Mike's role as the analytics director for the National Football League, he set out very early on to create an event that would allow primarily students to have access to tracking data in the NFL. And it's just become increasingly popular over time and not only been a way for the league and for teams to recruit top statistical talent, but also just the knowledge sharing that happens has definitely benefited from the kinds of methods and tools that the league is ultimately able to benefit from. So unfortunately, not all sports has gotten the message. And so there are some that are still very, very closed off and protective of the data. I mean, it is true that in tennis there are, let's say, both economic and political factors that are also at issue because there is not like a single tennis league. There's not like a major league tennis that just owns everything and could just decide to open it up. It's much more fractured. So that's another complication. But at least I do hope, yeah, that more decision makers within sport could kind of recognize those trends of the past and hopefully make similar opportunities for using that data.
- [18:49]
A
I want to sort of bring this back. You know, Mark, your rating system has become sort of the gold standard way far beyond its chess origins, sort of embedded in tennis rankings, video game matchmaking, and probably 100 other things that I don't know about. And with AI now capable of sort of ingesting decades of this granular match data, can a principled probabilistic model still compete with the raw pattern recognition power of a neural network? And are we going to be approaching a threshold or interpretability simply loses accuracy. Do classical probabilistic rating systems still hold their own against neural networks or are they going to be approaching obsolescence?
- [19:48]
C
I haven't seen like any, any like serious machine learning AI attempts at rating competitors in organizations using the kind of data that is typically used in rating systems. The data that's usually used by rating systems is just simply wins and losses and just who the players or teams are that are competing. So the data itself is actually very simple. And at least my experience has been that in such a simple setting it's actually hard to like very seriously improve on methodology that is reasonably classical as opposed to machine learning because just to, you know, variation in the inputs is so constrained. And I think what also makes an extra piece of it even more constrained is that in most of these kinds of settings for analyzing game outcomes, there's an assumption that the team or player abilities are changing over time. So if there's any role for more high powered kinds of analytic techniques, it's really going to be in the tracking of abilities as they change over time. But fundamentally all of that's coming from the game outcomes. And the game outcomes is such a simple structure that my sense is that they'll probably, at least for a pretty long while, still be a role for the kinds of rating systems that are currently in use now. That being said, I do want to mention that there are alternative strategies to measuring ability in games and sports that is taking advantage of, of data that's not just simply wins and losses. So for example, in chess, rather than, you know, basically if you and I are playing a whole bunch of games and you know, I win four games, you win six games, you know, we can basically estimate the probability that you're going to defeat me based on your having a 60% win rate. But in chess, what you could do is using these very strong chess playing programs, you can actually estimate on a move by move basis how strong your move is. At least, you know, since these programs are so strong, they'll come up with reasonably strong evaluations or accurate evaluations of individual moves. And so combining the set of performances, move based performances into one overall game performance is something that is achievable and that I think is a different strategy that might lend itself a little bit better to using the kind of tools that Liberty you were talking about.
- [22:30]
D
You know, Mark, one of the things that I think is really interesting when sort of comparing the gen AI architectures to sports rating systems is that these models, they saw like a big shift once they started to incorporate a more reinforcement learning where they had more manual labeled data that could act like a feedback loop into the model. I think technically it's the human in the loop reinforcement learning. What's interesting about that is that a lot of these systems like yours already have that built in. Essentially there's the self correcting and dynamic nature to the models. And I think really what's a key driver to why they perform so well. But it's kind of funny that it's almost like the JNAI labs maybe could have learned from that earlier on. But I think yeah, there are certainly areas though where you might see some benefit. Like I think one of the problems that's still at least I find challenging with those models is what you do for competitors you've never seen before. And that could be something where maybe taking more about the context that you see them first perform or anything that you might know about their background could maybe set a more reasonable, let's say prior rating in those cases. So maybe that could be a kind of application for these current network based models.
- [24:04]
B
Well, speaking of predicting win and lose these days, that each of us or many of us are doing like four or five times a day now because the World Cup. I want to ask a question about the World Cup. I understand probably neither of you have actual work on it. So it's not a question specifically about the World cup, but rather about your intuition as a statistician. Because I've been watching quite a bit. I love watching soccer. I don't have time, so I'm usually multitasking. And this year this is just a reminder. There are lots of dramas. People think about one team stronger than the others and it turned out to be lots of surprises, which is great for sports. For all the entertainment, that's terrific. The one thing has changed, as you all know, is this year that we have 48 teams instead of 32 teams. Now this is purely a statistical question. In terms of predictions. When we have these more teams on one hand you can argue, well, there will be more data, but somebody else can say, oh, then they have more noise. What makes things better or worse in
- [25:12]
C
terms of predictions, you know, the structure of the World cup is Interesting. You know, it's set up as essentially these set of round robins where, you know, teams are basically advancing from this first round into a knockout tournament. I guess the issue is that if the goal here is to be able to determine and in some ways the best team, you know, are you making sure that you're starting off with the, you know, the best set of teams to begin with, especially after this, this playing round. And there's no guarantees that you're, you're going to be able to come up with the best team. And like, you know, knockout formats are, are known to actually generally be reasonable formats. But like, when you're only playing one game and there's so much, so much variability in game outcomes, probably it leans much more on, on the fan satisfaction or engagement aspect of it than the statistical aspect. Probably the best team that's going to win using this kind of format is probably pretty low. But I, I think, I think it was smart. I think it's smart to actually have a play in round. So at least you've gotten rid of just the, you know, the chaff from the, the wheat and, and you know, it's going to be a reasonable set of teams. Steph, do you have any inside here?
- [26:30]
D
It is a tournament design that does already lend itself to a lot of unpredictability. If you were designing purely to pick the best team in terms of true overall talent, that you would probably try to have longer series where teams are maybe playing each other multiple times. But obviously there are constraints that don't make that possible. So I think that that already incorporates a large component of randomness. And then I think the other factor is that a lot of these teams, you know, throughout the year will be playing with an internationally mixed group. They're not playing within their national team that frequently compared to maybe their typical season schedule. A lot of the teams may be working with the coaching staff. That's entirely new. So there are all kinds of factors that make it a really challenging problem to know what's going to happen on any given game. But I guess in terms of your original question around increasing the number of teams, I mean, I suppose if that means that you're adding to the range of talent, that maybe those initial group stages become a bit more predictable if it means that the overall level of talent becomes a key driver to that outcome. But beyond that, I would think it's, yeah, very, very hard to say.
- [28:02]
A
I think I come back to this AI question of how we're seeing sort of these AI assisted coaching tools that It'si mean it's pretty cool, you know, how they can be deployed really in real time. You know, shot selection algorithms, defensive alignment recommendations in match, tactical adjustments. Is there a risk though, that every franchise optimizing from the same models creates some kind of strategic monoculture, you know, and who in the room, the coach of the algorithm, should have the final word? So I guess really it's like, where's the line between data informed decision making and over automation? And what happens when every team sort of converges on the same model?
- [28:58]
C
Well, so there are two issues that come to my mind. One is I think we're still in a place where even in the most sophisticated kinds of algorithms that end up getting used and presented to coaching staff, if the procedures are somewhat at odds with what the coaching staff thinks based on their own intuition, they are probably going to be resistant to being incorporated. I'm pretty sure we're not at the stage where a lot of these algorithms are being trusted. And part of it is just the way that these tools are put together for use. Maybe making poor assumptions about player skills or, you know, anything that might be omitted in developing some kind of procedure that, you know, ends up getting applied. I mean, it's basically just as simple as, as like if you construct the model and you're, you know, even a simple classical model and you don't include, you know, important predictors, you know, the value of that model may be quite reduced. And so something like that might be going on except at a, you know, a more complex scale. The other thing I wanted to mention is, so even if you were able to find an algorithm that you were trusting that is going to be very useful for, for determining tactics or defensive schemes or, you know, some aspect of play, you still have your own set of players with their own set of skills. So even if two teams are playing against each other purely based on these algorithmic determinations, that I don't think that's going to really take away from the sport because, you know, you have different sets of players, different skills, different random factors that are going to happen in the actual implementation of the play that, you know, still you're going to have, I think, an enjoyable experience and I think it's not going to necessarily entirely determine the outcome.
- [30:57]
D
Yeah, the question I think really goes to the decision making process in sport. I mean, when you're thinking about coaching decisions or recruitment decisions, and I've generally been more on the R and D side of things, but my sense and what I have gathered from my colleagues who have been more on that side of the sport that it's never been the case that coaching staff have set all of their expertise and years of playing the game aside because a model said something different. Usually it's always been an additional piece of information that they would combine with that experience for the game. And I think that's the thing that even as models advance and are able to take all of the Internet when determining their parameters, it's still the case that they're not going to have any one coach or staff member even in an organization that they won't have that expertise. So I wouldn't expect to see that they would sort of default to the suggestion of a model. But being able to access more information and maybe address more questions than they have in the past and be able to take that into account when reaching decision. I think there will be more examples of that. But like Mark was saying, the context is really critical. So I think because of the black box nature of them, I think you might actually see the opposite. Where individuals within a sport organization may be more skeptical if they can't really interrogate like what information is actually being considered. How much context, that nature of those models actually would I think be a detriment to more adoption.
- [33:00]
B
Now, we have been really talking mostly or almost exclusively on the sports analytics, but I know your columns is really a lot broader than just on the sports side. So I would invite, Mark, you first talk a little bit about give our audience a general sense what has been featured in this wonderful column called Recreations in Randomness. And Stephanie and I will invite you as a new co editors. What's your visions, what you bring in to this column, what you want to see. And I broadly wanted just to have this opportunity to tell the audience about this column. Welcome more submissions. But Mark, let me turn that to you.
- [33:45]
C
Yeah, this is an exciting column in the sense that we want to be publishing more papers that have to do with applications of data science and AI to just regular life, which typically involves things having to do with recreation, hobbies we've had in the past recommendations for wine and how AI and just data science more generally is now involved with that. But we've even gone into sort of interesting areas like AI generated poetry, how good AI is in creating humor, how do you generate art or evaluate art. I'll admit one of my favorite articles which was from early on is on cooking, recipe generation and the tools that are available for the upcoming cook. So we're looking for people to contribute to our column. These are relatively short pieces if you're particularly involved in a hobby or a recreational area where you have some knowledge and interest in how AI and data science generally is being applied in that area, we would love for you to contact us and write an article and show that data science is not really just restricted to typical academic pursuits and politics and industry, but can engage pretty much anybody.
- [35:13]
B
Thank you, Mark, for your wonderful stewardship of this column. Steph, I want to turn to you and again, welcome and thank you for your willingness to serve as a co editor.
- [35:24]
D
No, it's my privilege. Mark established a really valuable column that gives space to show the ways in which data science can enter our lives in maybe surprising ways. But I'm sure for those of us that have had some quantitative training that do see an interest in data come up in all aspects of our lives, and that's exactly the kind of thing we'd like to highlight in the column. And so we really invite potential contributors. If you have ideas or experiences in ways in which you see data being used in interesting ways within the recreational sector, we'd really love to hear those stories and yeah, look forward to featuring them in the column.
- [36:16]
A
We always close with the question that sort of separates the analysts from the visionaries and one magic wand, one data set. One question that has just been out of reach. What is it? If you could have access to one type of data that currently doesn't exist or isn't publicly shared, what would it be and what would it finally like you answer.
- [36:45]
C
I can answer the question in sort of a way where like I that I've been struggling to find a data set and finally have it. Does that count? Wow, this goes back to this question on being able to evaluate player ability based on their chess moves. And there's a very particular type of information that I was hoping to achieve in getting such a data set from chess online play. And I've been actually asking for it for like the last five years and I just can't seem to figure out a good way to get that data set. But long story short, I've been working with a collaborator who did manage to get access to such a data set. And so now we have it in hand and so now I'm pretty excited. But that was just from like one particular system. So like I would love to to see if such kind of data is easy to generate on other systems, not just on this one that we have access to.
- [37:45]
D
I think for me, I mean my personal I guess answer would be as a mom of a four year old, I would love to understand his perception of the world and kind of download his mental model like Neo Matrix style. But more seriously, I do think actually the sort of mental component of sport is a really interesting one that we don't kind of have any real data, even things like how a player reacts to, you know, very fast events. That kind of perceptual aspect to athleticism I think is a missing component even in today's sports analysis. So it sort of relates to that as well. I think that would be a pretty fascinating data set if we could have it.
- [38:34]
B
Thank you to both of you again. And I want to say that the one data set I want is how readers react to the hdsr its podcast. You know, what constitutes as readings, listening. You know, there's all these issues about a human's reactions, right? And this is the kind of things that we can use to make a decision. I mean, it's very iron ironic that as a data science journal and platforms like we don't really have those data ourselves. So with that, I want to say that thank to both of you for what you do, both to the sports analytics itself, but for your great contributions to hdsr and I'm really looking forward to working with both of you. Thank you very much.
- [39:25]
A
Thank you for listening to this month's episode of the Harvard Data Science Review podcast. To stay updated with all things HDSR, you can visit our website at HDSR, MITPress, MIT.edu, or follow us on Twitter and instagram @thehdsr. A special thanks to our executive producer Rebecca McLeod and producers Tina, Toby Mack and Aaron Keeswetter. If you liked this episode, please leave us a review on Spotify, Apple or wherever you get your podcasts. This has been the Harvard Data Science Review. Everything Data Science and Data Science for everyone.