← Pablo Torre Finds Out
Pablo Torre Finds Out
How Artificial Intelligence Is Already Changing Sports, with Daryl Morey and Sendhil Mullainathan
00:47:33
Loading summary
Transcript194 lines
- [00:00]
A
Welcome to Pablo Torre Finds out. I am Pablo Torre. And today we're gonna find out what this sound is.
- [00:07]
B
And anger is all that drives Sixers fans. As far as I can tell, right.
- [00:12]
A
After this ad, you're listening to Giraffe Kings.
- [00:32]
C
So I need to jump in at the top of this episode to give you a warning. A couple weeks ago, I was asked if I wanted to record a live episode of this show at the Sloan Sports Analytics Conference, which is a conference, if you did not know, that started at a lecture hall at MIT when I first attended it 16 years ago now. But quickly it moved into a convention center in Boston on account of the thousands and thousands of people who wanted to actually go. And many of those people annually include the most powerful people in sports. At this point, we're talking league commissioners, billionaire owners, Barack Obama. One year, pretty much every data driven team executive in American sports and beyond shows up.
- [01:15]
A
But the guy presiding over all of.
- [01:16]
C
This, the conference's co founder, is Darrell Morey. Darrell is now the president of basketball operations for the Philadelphia 76ers, and he is also, it turns out, a listener of this show. And so I do feel journalistically obligated to point out that the Sixers are now about 20 games under.500 having a.
- [01:36]
A
Truly horrible season, as we'll discuss. But the thing I really wanted to.
- [01:41]
C
Find out about at Sloan, even more pressingly, was a question I did not have a good answer to. I realized, despite how much everybody is always talking about large language models, LLMs like ChatGPT, and also when the robots are finally going to achieve artificial general intelligence, AGI and, you know, think like human beings actually do. All of which is to say what.
- [02:07]
A
I wanted to find out about today.
- [02:09]
C
Is what AI is actually doing to sports in particular. Like, what's the signal here, as in the relevant information worth knowing and what's just noise that we should discard. And so the warning I need to give you is the same warning that I gave our live audience in Boston as I was introducing Daryl and my other honored guest today, a man named Sendhil Muleynathan, who is the perfect person to join us and demystify this subject.
- [02:39]
A
This is gonna be the nerdiest episode of my show. I'm glad to have all of you here listening and watching. This will be on YouTube, I presume, and you will then get to see how nervous Daryl Morey is.
- [02:53]
B
I'm nervous, yes.
- [02:56]
A
You've been complaining about this all weekend.
- [02:58]
B
It's just Sendhill.
- [03:00]
A
I know.
- [03:01]
B
I hope I'm not Insulting you by the genius grant is very intimidating.
- [03:06]
A
It is legitimately fun to be here alongside an actual certified genius and also an executive who gets to be sarcastically referred to AS1.
- [03:19]
B
Is that the only way you can be labeled genius is the MacArthur?
- [03:22]
A
I think so.
- [03:23]
D
We've copyrighted it, so.
- [03:25]
B
Oh, nice. He even said we.
- [03:27]
A
I know, but he's on the MacArthur foundation board. I mean, Sendhil's bio. We could just read his bio for a while. But just to catch you up on why. Daryl and I are both incredibly excited to be sitting next to Sendhil, MIT professor, splitting his time between electrical engineering and computer science and economics, focusing on the intersection, I guess, broadly speaking, of algorithms and people. Co author of arguably the most highly cited economics paper in the 21st century on differences in differences estimation, which is a thing I definitely understand.
- [04:00]
B
You can subtract, right?
- [04:02]
D
Yeah.
- [04:03]
A
And also landmark papers on algorithmic bias and poverty in the criminal justice system. And now training his brain, his human brain, on artificial intelligence. And also. Darrell's here, so, Darrel.
- [04:15]
B
He's killing me.
- [04:16]
A
But the resume for Daryl is.
- [04:19]
B
Hasn't won yet.
- [04:20]
A
Well, historically. Right. Over 60% win rate as a GM. This is, though, barring, I don't know, some act of God or the devil. Your first losing season that you're grappling with.
- [04:35]
B
It is my first losing season, and I think 17 years. Yeah.
- [04:39]
A
So feel vulnerable being here on stage.
- [04:42]
B
I do. I feel very vulnerable. Yes. Yes. Plus, you're a good interlocutor.
- [04:46]
A
That's right.
- [04:47]
B
Whatever they say.
- [04:48]
A
I do want to interlock you to. With you. If that's how I betray my own diploma. How long have you been using large language models? I want to start with this first, and I want to also establish just the very basic premise of how long you've been using it. And also, how are you using AI in your job as president of basketball operations currently? How does that look?
- [05:12]
B
So I've used them a little longer than the OpenAI moment, which was what, November two years ago? Three years ago now with the GPT, I believe, three, you could get some early access. I wasn't really special there, except I knew about it. I knew people who knew about it, and it seemed very cool. Even I didn't realize how cool until it went public. I think that was like sort of a moment that even surprised the AI researchers that I think that they were like, yeah, this is really cool. But then once the public got ahold of it, it was very cool. And then in terms of my job, we use. We've Used a version of AI for a long time. We mentioned this on an earlier panel. AI is just forecasting. My job is making good decisions. It's all about forecasting. We've used machine learning models for forecasting for a long time. And primarily the big areas that would be most relevant would be things like forecasting draft picks. There are a whole bunch of different forecasting techniques. Traditional regression, linear regression, tree models, graphical models. There's a whole set that have been tried and they're all a flavor of forecasting of which even goes into large language models later and then for like pick and roll identification in computer vision. Turns out even humans can't agree on what a pick and roll is or isn't the most basic one everyone can agree on, which is one guy hits another guy which keeps him from guarding his guy and then something happens. But there's all these like phantom pick and rolls and you act like you're going to set it and you don't. Even humans can't ident. Computers do a pretty good job, but not a perfect job at that as well. And then on the LLM side, the traditional stuff is helpful coding, you know, speeding up writing projects, things like that. Those are ones we use and there's some shooting stuff as well. So.
- [07:20]
A
But just to be sort of, I don't know, top line here. When you're making a decision and insert your favorite Sixers decision here, are you asking an LLM for their input?
- [07:35]
B
So two answers there. We absolutely use models as a vote in any decision and how much of that decision depends on how much the assumptions behind those models don't change and their past success rate. So if you have a highly successful model for picking draft picks and we think that the game, you know, Evan hasn't changed the rules yet and so the game isn't changed.
- [08:03]
A
Evan Wash who's in charge.
- [08:05]
B
Thank you.
- [08:06]
A
Of defeating you.
- [08:07]
B
Yes. Then the models get a lot of weight in those decisions. It turns out LLMs do fairly well at prediction. They still are not beating human like superforecasters they're called like in human Judgment Project talks about it, but they do add signal over just scouts and things like that. So we'll treat them almost like one scout. Now over time, if it's shown that they have a better track record than scouts, we might weight the LLMs or use multiple LLMs as one or two votes out of the process. So it turns out I'm building on a lot of things that Senthil knows and you know that, you know, multimodal approaches can put you more on the decision frontier, the optimal decision frontier. And when I say that carefully, you get very similar accuracy with lower risk by using multimodal approaches. And so that's why we do that.
- [09:05]
A
I would just like NBA Central to aggregate what Daryl just said, which is that he asks ChatGPT for help on trades.
- [09:13]
B
Please, not aggregate.
- [09:15]
A
I'm just trying to pay for my kids tuition. Darrel Sendhil look, part of the larger conversation here, I was in San Francisco for All Star weekend. The word cloud over All Star weekend was artificial intelligence. Was AGI artificial general intelligence, specifically. Your view, though, on the ambition of AGI thinking like a human, it sounds like what to you, given your research?
- [09:44]
D
It sounds unambitious. Yeah, I think that AGI on the surface feels like an ambitious goal, but it's a shockingly unambitious goal. Let me articulate why. We already have people who can think like people.
- [09:59]
B
People who can think like what people?
- [10:01]
D
Okay, we have that and we have ways to make more of those people. And they're fun ways, actually.
- [10:09]
A
That's the next panel.
- [10:10]
D
Yeah, that's the next panel. But what I think is missing, and you've done a lot of this, what I think we need is algorithms that can do things people could never do. And to me, that's the fundamental expansion that we're going to get. We need algorithms that can see things that we can't, that can notice things that we never could. And the reason I think the AGI part is unambitious is also too hard in the sense that we've had algorithms, vectors along which algorithms can see things that we absolutely can't. And those are actually the easiest paths to improve performance, to exactly do what we do with very high fidelity. That's actually really hard and not clear. There's that much gain. And so let me see if I can give a good example. My favorite example is the wizard of Oz. By far my favorite example. How many of you have seen the wizard of Oz? Raise your hand. Okay, how many of you have read the book? All right, how many? You read the book? How many of you read all 30 of the books?
- [11:11]
B
No, I read one other and it was weird.
- [11:14]
A
I've just found out there are 30 books exactly. The wizard of Oz cinematic universe.
- [11:19]
D
And in fact, it gets even weirder. There are 30 books. 14 of them were written by Frank Baum. 15 of them was written by this woman, Ruth Plumlee Johnson. And the 15th book is the source of a big literary kerfuffle. Who wrote it? Frank Baum died while that was being written. And so amongst literary critics, they just spent a long time close reading that book and they couldn't get, they couldn't agree. And so in 2002, there's a article in 2002, think of how early this is in it, an article in this magazine that says, I've done nlp. And you read the analysis and it is so clear, like with 100% accuracy, that the algorithm can see that until chapter 12, it's all Baum. After chapter 12, it's all Ruth Plumbley Johnson. How can it be that an algorithm can notice with so much clarity something that humans have utterly not noticed in our own native domain? And we see that again and again, which is algorithms are fantastic at picking up signal on the kind of things we didn't even know to notice for. Because they notice aggregates, they don't have prejudgments of where to look for signal. So to me, I think the ambitious AI goal is actually to find algorithms that can help us know. Like, I loved what you were saying about how you, in scouting, you have another scout. I think, Pablo, you used this phrase, I would love it if there was another scout, not of the players, but as you refer to it, self scouting, something that scouted me. We do that.
- [12:47]
B
We have a self scout. Yeah. Just to flag any. Like if another team was analyzing us.
- [12:52]
A
What would they explain that? Yeah.
- [12:56]
B
Can I go back and then come back to that? Because you mentioned something that we say all the time, which is that on a single game, humans do better than any algorithm. But similar to your L. Frank Baum example, algorithms are very good at seeing these 2 to 5% patterns that humans miss because we see big things, we see big patterns, we see. And we, we find big patterns when there aren't any. And there's all these kinds of things. So we have automated tools that say, when I'm about to play this team, here are the kinds of things, you know, that player is very efficient. We should limit this opportunity here. So we flip that on its head and self scout and say, when they play us, what are they likely saying? We don't know for sure. That's a little different.
- [13:46]
D
What I would love is a model of Daryl that you look at. No, no. But that you look at. And that says to you, hey, just to you. Hey. I think this is the kind of situation where you often do this, like something that helps you get candidate biases of yourself. And in a way, what I like about that is it doesn't need to be perfect at that if it was 20% right at that, boy, what value add? Because it'll notice the type of things we might not notice about you. So I think once you think of algorithms as where do you deploy them to find things we cannot look for, I just think it's going to open up a huge avenue of.
- [14:23]
B
Can I ask a question? Because you give an optimistic take of where we should take AI or even AGI, which left a role for humans, left humans at the center. And I know that as someone who plays a lot of chess, that was also the belief in chess that there would be this role for humans over time. And in fact, there was a time where a human machine combo could be a centaur. A centaur, yes.
- [14:48]
A
One of the great terms of art for anything, a human plus a computer being the most dominant combination of any. Just humans, Just computers. They were the best.
- [14:58]
B
It is called centaur chess, for example, as you say. But it was in the early 2000s where that went away. And actually any human involvement would mess up, would beat a machine with no human involvement, would be at any human involvement. And so I worry that we may face that as well.
- [15:17]
D
So I love the chess example because it so clarifies many things. There's this term, though a bit technical, I think is very helpful, called closed world versus open world problems. Closed world problems are ones where everything is specified. Let's say chess satisfies that. But I'm going to come back on how even chess may not satisfy that in a second. But let's say chess satisfies that all the rules are known, all the objectives are known, everything is there. We will never beat an algorithm in a closed world problem. That the idea that we're going to beat it conditional on rules, conditional on things, is crazy. You can just compute your way to stuff. And there's an analog to closed world problems where closed world problems, where we have enough data, again, we're not going to beat algorithms. But most phenomena in the world are not really closed world problems, they're open world problems. What that means is what are the things that you can do? The moves in chess were prescribed, the moves in life are not prescribed. And some of the most genius insights of someone realizing, hey, I can actually make this move. This is a feasible thing to do in this situation. Similarly, the objectives are not specified. And it may seem like the objectives are specified, but if you think of some of the great innovations that people have had, it has been realizing, oh, that's not the problem we're trying to solve, this is the problem that we're trying to solve. Like, I am not in the burger business. I am in the customer satisfaction business. I'm not in this business. I'm in this business. And so specifying objectives, specifying moves, specifying the rules of the game. Once you've done that, the algorithm will clearly win.
- [16:51]
B
So you're basically saying if we're playing Centaur chess, the computer doesn't know we can still flip the board.
- [16:57]
D
Right. And another example of chess is. Is chess about winning? For example, suppose you're playing with a friend. In one sense, the goal is winning, but in one sense, the goal is to really have, like, a fun game. Can you build an algorithm that'll have a fun game with you? Now, suddenly, that gets interesting, and there are people who do chess algorithms have been trying to figure out, how do we build a fun game algorithm? And so that suddenly that's no longer in the realm of just computing. You've now got to actually get into it, and people get involved.
- [17:26]
A
But that brings us right back to sports.
- [17:28]
D
Yeah.
- [17:29]
A
So to me, the problem, the tension of our time that Darrell has had a hand through this conference actually, in shaping is this tension between entertainment and efficiency. And I always think about when I mentioned the league office trying to keep up with people who are trying to innovate, there is this push and pull of how do we stay ahead of the people who are the most clever when solving these wicked problems, these open questions. And so when it comes to how we're figuring this all out, Right. Currently we're all set up so that Daryl's over there, the players are trying to catch up and figure out, okay, how do we fit into this? And then the league office is somewhere else. Right. Is there a way Sendo, to actually acknowledge that a truly optimal goal here is for everybody to be aligned on what the best outcome is, which involves entertainment.
- [18:23]
D
Yeah, I, I, and I'd be curious what you say about this. I, I, I do think part as a fan, I'm like a big super fan. I'm a super fan of.
- [18:31]
A
You had an account on real GM on the Sixers message board just to out you immediately.
- [18:38]
D
Yeah, yeah. More than an account. I was.
- [18:40]
B
So what's the account I.D. just.
- [18:42]
D
Well, no, it gets worse, Daryl. No, no, it gets much worse. This is when I was trying to get tenure at mit. I was so obsessed with real GM that I then became a moderator for the Sixers message board.
- [18:52]
A
Just like genuine sicko behavior.
- [18:54]
D
Yes. Yes.
- [18:55]
B
Was this when Sam was there?
- [18:56]
D
No, this is 2000. This is when I was trying to. Yeah. I should not have been doing this. It was really quite.
- [19:03]
B
You seem to have done fine.
- [19:04]
A
Yeah.
- [19:05]
B
Yeah. I hope. Yeah. Yeah.
- [19:06]
D
I won't give away my user id, but I do think as a fan, one of the things we can just.
- [19:13]
B
Put NLP on it and figure it out.
- [19:14]
D
Exactly.
- [19:15]
A
I think you should be able to. We're going to find out, Darrel. Don't worry.
- [19:17]
B
Natural language processing.
- [19:20]
D
Yeah. This is a Frank Baum situation all over again. What I do think, and I'm really curious your guys opinion on this. I think as a fan, what I've noticed is that wins matter, but it's shocking to me how much of my pleasure comes from a bunch of other things. The underdog story, the person who just shouldn't be scoring this much but scores this. Like I was happiest as a Sixers fan during the process years, which is a weird thing to say. I would watch every game and I don't know what that reveals about sport.
- [19:53]
A
But it reveals that this season for the Sixers is going to be a really enjoyable one for you.
- [19:57]
D
No, no, no offense, Darrell, but it has not been. And I'll tell you why it has not been. And I will say me neither. We're in the same boat. It's not been because the storylines have not been captivating. And if I think of sports as entertainment, like I would love to know, even I would love to turn an algorithm on what makes a captivating storyline. There are just some storylines that just keep you hooked. You want to go back for the next game, you want to know what's going on. And so is that part of the objective function or is that an illusion? Like, look, you guys are thinking to yourself, storylines matter. We've looked at the data. There's only one thing that matters, it's wins. I don't know. I'd love to know how to win.
- [20:33]
B
I don't focus on wins. I focus on championship. And I would say the one. The Philly fans surprised me in this manner. So I guess it's two seasons ago now. We were up 32 on the Celtics. Objectively in Vegas, we were at the highest odds to win the championship that year because they felt that the team out of the east would win. I guess it wasn't like above 50. I've scraped 50. When we were up on the warriors in 18 and we were up 3 2, obviously we then lost a close game at home, which was very frustrating. And then we had a very bad loss in game seven and you know, it's a little bit similar to the LeBron Michael Jordan argument. Objectively, people should feel better about that. Then the next season we had a very entertaining Knick series, like, almost famously entertaining. I think it was voted best series that year. And for me, it was just stressful. But for everyone else, I'm glad you enjoyed. Was very stressful. And we lost in six games, you know, to a Knicks team that was not as good as the Celtics team we lost to the year before. And I was very down after the loss. And I was like, man, people are going to be so angry. And they were angry. Obviously, everyone wants us to win, but they were less angry than the year before. And anger is all that drives Sixers fans. As far as I can tell, they were less angry.
- [22:07]
D
Anger and irrational.
- [22:08]
B
Yes, yes. The aggregators are going to have a field day with this. Pablo, I know you're excited.
- [22:13]
A
My kid's going to go to prayer, private kindergarten.
- [22:17]
B
But they were less angry, was my perception. I don't know for sure. It's not like we surveyed everyone. How angry are you? And that really threw me. Cause to me, I'm just trying to win the title. And so the year before, for our. The way we measure it was objectively better, even though still a zero on the Boolean scale I'm judged on. So it should all be the same. And that surprised me.
- [22:40]
D
Can I put on my behavioral scientist hat for a second, please? I'm just gonna speculate. Please. I'm just gonna purely speculate.
- [22:44]
B
Well, you're a Sixers fan, so you can be direct. You can't even.
- [22:47]
D
No. Which is that I even think negative emotions are fine. I think the thing that is dangerous is disinterest. And if you think about sports, it's easy to dismiss sports as entertainment. But what else in our culture can have people of very different backgrounds, different generations, talk to each other and feel they've meaningfully touched, communicated, you know, had a meaningful meeting of the minds, you know, parents and children, grandparents. It's quite a unique place as a cultural artifact. And the main thing that it does, if I think of its core contribution to society, is it provides conversation. And I think the worst thing is anger is good. You got something to talk about. Like during that, after that Celtics thing, you meet a Sixers fan. Like, it was easy. You knew what you would do. You would complain. And complaining with another person is its own form of, like, connection. It's those periods when you don't really have anything to talk. Like, it's disinterest. This feeling that the story is old, it's like, kind of stale.
- [23:52]
B
That we're always interesting.
- [23:54]
D
Yeah, right, right.
- [23:55]
A
The curse actually is that.
- [23:58]
D
And it is always interesting. And so I wonder to what degree one should just be maximizing engagement. I know that we've had bad experiences with that.
- [24:05]
B
This is Pablo's department.
- [24:07]
A
That's right, Sendhil. If you want to trash any other geniuses, feel free, but I actually want to bring it back around. So, like, the larger picture here that we're sort of saying without explicitly saying it, is that the nerds won't. Okay, right. Like the whole debate about, like, when are the Lakers going to show up at Sloan? Which is a headline for a long time. The last holdout of this conference from the basketball ops department used to be the Lakers a dozen years ago. Then everybody, everybody shows up and Jeannie.
- [24:48]
B
Buss game actually is the first Laker.
- [24:50]
A
Yes, yes, But I say that because when it comes to the distribution, the wide, broad access to LLMs, and I want to get to beyond LLMs in a second here, but when it comes to that, every team, even the most caveman old school team having access to ostensibly an army of robot Darrells isn't the person who should be worried. The most is the guy who founded the conference, who had as his competitive advantage this natural curiosity and skill and career premised on doing things that you couldn't just automate. Should Darrell be worried? I guess. Sendhil, is that the subtext here of everybody having access to these devices?
- [25:34]
D
The short answer is definitely not. And it's worth talking through. Which experts should be worried and which experts shouldn't, please? Some experts absolutely should be worried. There are experts where the nature of their expertise is the reliable, repeated application of some tacit knowledge. So there are, you know, a very good doctor is a reliable, repeated application of tacit knowledge about diagnostics. Now, it's not that those algorithms today can do that. I think we're far. But it is also totally plausible that we will get to the point where.
- [26:10]
B
You mean like reading an X ray?
- [26:11]
D
Reading an X ray. Yeah. It's like, it's easy to say that's going to happen next year, but that's not going to happen. But it is definitely like it is. Do not tell your kids to go into radiology. You'll be happy. Since we both had moms who told us to go into medicine, we will. The nerds will win over the doctors.
- [26:29]
A
Sindhil and I bonded over the trauma of not becoming a doctor in his office yesterday. It's real.
- [26:36]
D
So that Repeated application of tasks and knowledge. On the other hand, I think the genius of someone like Darrell and experts like that are people who can take an ill formed problem and actually formulate it in a way that it becomes tangible and actionable. So it's easy to say oh the three point but it was recognizing in this ocean of stuff, let's focus on this part of the problem is actually very, very, very hard. The reason algorithms are complementary to that it goes back to the chess thing. Once you formulated the problem, we can now use algorithms to bulldoze through it which then raises the enormous returns to problem formulation. And I think people who can bridge the inchoate under specified who knows what's even going on here and bridge it back into the I've now formulated into a data set or a specific type of activity, that skill is going to go way up. And so I think the data science skills that are going to be worth less are the people who once it's formulated do stuff with it. But that's not been your genius. And does that make sense here?
- [27:45]
A
Yeah. And even just like Darrel, I had the opportunity. I was at the Sixers Celtics game with you and I was very curious. For those who remember Larry Kuhn, Larry Coon, a retired computer scientist who had written the seminal text for understanding the NBA's salary cap and collective bargaining agreement, the MBA salary cap FAQ. Larry was such a scarce resource, that body of knowledge used to be so scarce that everybody would consult the this like blogspot page.
- [28:15]
B
It was Larry. And just a very few people at the legal office and a few teams at the time. Yeah, yeah.
- [28:20]
A
But now ostensibly, why can't an LLM analyze the CBA and take the jobs of the many humans who are currently tasked in the way that lawyers are tasked with analyzing documents.
- [28:34]
B
In fact it can. And so he had me give it what I thought was a medium hard problem and it answered it flawlessly. This is on 4 5, the latest from OpenAI. Then we gave it a medium plus hard problem. It also answered flawlessly. And then I was with Monty McNair from the Kings. He formulated one that he thought was hard and it did fail the hard one. So I think we, yes, we have the same problem though law firms have, which is even though it gets the answer 98.9% right, you can't hand it to a client who's paying you that much money unless it's well, nothing's 100, but you probably need a higher sigma or at least to say there was a human who looked at it because you're like, oh, you just took the algorithm. You're like, yeah, that doesn't work. That will work, though, eventually. Or where people will be like, that's a high enough sigma. I'm going to take the answer from the algorithm.
- [29:29]
D
And then I'd add one thing to that. It's the problem, as LLMs as they are today, is not just their error rate, but the inscrutability of their errors.
- [29:39]
A
Explain that.
- [29:40]
B
Yeah, explain that.
- [29:40]
D
Yeah, we have almost no way of knowing the form factor and where the errors are going to come from. That is, you tried some hard ones and got it to fail. I bet if we had an hour we could find some easy ones that it fails on. And it's unclear to you why it failed on this easy one, but got this easy one.
- [29:57]
B
So this has been my test since the LLMs came out, and only four or five is the first one to have solved this. I would give it. I've. Since the OpenAI first one, I've given an artificial Monty hall problem to the LLMs. And it's perfect because they're trained. There's been so many articles written on it. It's trained on a billion articles. And the answers always switch in these articles. Well, some tiny exceptions. So I've been giving it a money hal problem where. And just so folks know that three doors, you pick a door, I pick door one. The host then opens another door and says, okay, you can either keep the door or do you want to switch? And it turns out if you switch, you get two out of three probability. If not. Most of our audience probably knows this, but the key to the whole Monty hall problem is Monty hall knows which door to open. So I formulated, I have to change it. I put it in sacks, and then I have a person just walk by and say, there's nothing in a sack, but without any knowledge of the others and say, do you want to switch? In that formulation, it's no advantage to switch. And four, five was the first one that got it right. All the other ones were like, switch.
- [31:03]
D
Yeah, yeah, that's fantastic.
- [31:05]
B
And so that's been my little test for how advanced they're getting. But those are the kinds of problems you're talking about where, like, change a little assumption and it changes a major part of the answer. They will miss it because they've been trained heavily on the correct answer based on what they knew.
- [31:21]
D
Yeah, it's tempting to think that they're reasoning their way to an answer when there's something else they're doing. We have this paper we're finishing called Potemkin Understanding, or where it's very easy to generate these little Potemkins. Where it seems to get this right, then why is it getting this other thing utterly wrong? And you can start algorithmically generating. And you see it's kind of like Swiss cheese, their knowledge, this thing.
- [31:44]
B
And this is the New York driving.
- [31:46]
D
Oh, this is another paper we're just finishing now. So it's like an example of this is like you can just give it a bunch of examples and say, what's an ABAB rhyme scheme? Great. And then you write a little thing, take out a word and say, can you fill this in to rhyme it?
- [32:00]
B
And they can or cannot.
- [32:02]
D
Sometimes it gets it right, sometimes it doesn't. And when it doesn't get it right, you take that word and you say to another instance, does this word rhyme with this word? And it says, no. So it's like there's incoherence underneath the object.
- [32:15]
B
Well, that's where the inference time comes in.
- [32:17]
D
That's where you.
- [32:18]
B
They just repetitively ask themselves questions before they answer the first time. Right.
- [32:22]
D
So you're hoping that those type of tricks kind of get this stuff out of the space.
- [32:25]
B
But it is a klude. The more I go into the inference time stuff, it's basically a kludge on maybe a fundamental problem, but I'm not quite sure. I don't know.
- [32:34]
D
It's definitely a fundamental problem. It really is. I think what's happening in this space is that when you play with these LLMs, it's just so amazing. They feel like they've understood. It's hard not to get pulled in and feel like, oh, my God, this is a thinking creature.
- [32:47]
A
What you're saying is that the fact that it's a language model talking to us has fooled us into thinking it's. That there is a depth that is not actually present.
- [32:55]
D
That's exactly right.
- [32:56]
B
But that's probably true of most humans too.
- [33:02]
A
Unfortunately, that is.
- [33:04]
B
This is like in interviews. This is my favorite interview tactic is they. I don't care what they bring up. I sadly might know a little bit about everything, and I'll just, like, start asking. And if they can't get past, like, two layers and you're like, okay, they don't really understand that.
- [33:18]
A
But I do want to get to just the job. Right. So. Okay, so there are versions of what I imagine that CBA problem is where it's like, could you have an LLM call a play, and then you consult in the Huddle. Okay. This is what our robot assistant coach is proposing. You have a person check it. I can understand they're at the table in every decision, but just the job, Darrell, of what it means to be a president of basketball operations, such that Sendhill has faith in your ability to persist past the broad access of everybody having these robot assistant GMs, coaches and everything else. How much of your job is dealing with incoming, like, bad ideas?
- [33:58]
B
Frankly, yeah, we often. When I say we, I mean like me and some other GMs across sports. We feel often our job is sort of bad idea, whack a mole. Because ideas, like, form in these little narratives and they can be good or bad ideas. But part of our job is to smother the bad ideas in the crib before they be. I will say, I won't say the assistant, but when I had Cal Lowry way back early in his career, what made him a great player was his tenacity and everything. But it also made his personality somewhat challenging just to assistant coaches. And basically, we weren't winning. We were winning about half our games. And then Kyle went out and we won six in a row and a narrative started forming. We might be better without Kyle Lowry, which was so absurd in the data that it wouldn't even be something anyone who looks at data would consider. But I could see how, you know us, you know, we find patterns anywhere. Could see that. And I had to, like, work hard to like, to like, be like, no, like this guy. Like, we had an easier schedule. We had all the different things that would go into that. And so that's. But that happens like multiple times a week, at least, if not more, where you're just trying to keep things on, like, let's get the big things right and not let these narratives form that might harm us.
- [35:48]
A
Sendhil. I want to move beyond LLMs, because I know so much of your research is not merely these chatbots that are, again, the talk of every conference you'll go to at this point, but it's also supervised learning and how to train these models once you now account for a more realistic version of what being a. An executive in sports would be like. Can you explain supervised learning and how your fandom of sports has led you to see sports through that lens.
- [36:15]
D
Yeah, so I think supervised learning differs. I'll do one thing might be helpful to people. Supervised learning is I collect a bunch of data of various kinds and I ask the algorithm what in this data predicts this particular outcome. And as long as you can specify an outcome and some variables to predict with it. These models can sort of really start finding signal that we never even imagined. And so I think the playbook that emerges from that is to be very creative about finding things you'd like to predict. And so across disciplines, the best work has been getting clever about that. That's why I love the Frank Baum story, which is the person who did that was super clever. He's like, I'll just get some data and predict whether this was written by Baum or not. And I've got some data that tells me ones that were. Ones that were written by Ruth Plumlee. And so I think for me what makes sports so interesting is that there is so many unknowns that you could imagine collecting data on. And I'm sure you guys do a lot of this already, but even something like injury risk, what is a good predictor of whether someone's going to get a certain kind of injury? And if I just play that forward without knowing what you all do in that space, if I play that forward, it's kind of a nice toy example to start thinking about stuff. Because first you have to be creative to say we're not just about wins and what happens. Like injuries are big, so why can't we turn the gun on that? Then you have to be creative about deciding what input variables are you going to get to do that. And that's where I think people haven't fully appreciated the biggest change that these algorithms have rendered. So that if I said that these algorithms can find a lot of signal in data, then a logical implication of that is we ought to invest in new data collection and there's huge returns to new data collection. So what I would do is say, oh, what would I want to predict injury risk? I don't know. What are some low cost devices I could put on players? What are some low cost devices I could put? Not even during the game, but while people are taking free throws. Maybe if I had video of people taking free throws, I would start noticing small changes in form that indicate things that are wrong at a bio skeletal levels. So you kind of the creativity becomes not in finding patterns in data, but in deciding what data to give the algorithm either as an objective or an input. And that's just a very different kind of open world creativity. And I think that path is, I think going to take off in the next 10 years. We're starting to see it in medicine. Well, you tell me. I think we'll see it in sports. We probably already have for sure.
- [38:54]
B
Yeah. And I, I don't know if you read it, it sounds like you didn't. I wrote a paper years ago on you need better data, not better analysts. And I actually met a lot of people mad at the conference. But yeah, the data is going to be the unique thing, not the analysts. Yeah.
- [39:12]
A
How much parity is there now in terms of the data teams have?
- [39:18]
B
I think it's getting pretty high. I mean, we're following baseball almost in a five year delay and it's pretty homogenous in terms of how the. I mean, I think the quality varies, obviously, but we're in a tighter band, Much, much, much tighter band now, which is not good. Yeah, not good.
- [39:40]
A
The question though of, you know, if someone were to approach Sendhill and say, we vetted your real GM posts and yet still. And yet still, we think you should be the chief intelligence officer for our NBA team, do you have a sense of what you would want to do given your field of study now?
- [40:01]
D
Yeah, I think. And I'd love to hear what you think about this. I think what I would focus on is turn the camera. Since so much of it has already been focused on the game, I would turn the camera elsewhere, for example, on the decision makers. I would say, what are decisions that we think are being made all the time? And how do I start creating what you call the self? Like, how do I start creating algorithms that can help scout me and tell me where my biases, where are the things where I screw up? So that would be an example where I would start. No.
- [40:31]
A
Could you explain you had this landmark study of judges and bail? I just would like everybody to appreciate what it's like when you turn the camera on a decision maker.
- [40:40]
D
Yeah. This study was basically pretrial. Judges have to decide who to release and they're making a prediction of risk. Will the person come back for trial? Will they commit a crime? So the first part of the study was we had an algorithm predict the defendant risk. And it turns out that does very well and could add a lot of value to the judge. When I say a lot of value. Sometimes in sports it's hard to know what the magnitudes are, but in this case, you could close Rikers in July and have the same crime rate. So you could cut prison population, jail population, by 40%. There's a lot of welfare left on the table. But then we did two things that were surprising even to me. So we didn't predict what the defendant would do. We turned the camera and predicted what the judge would do. And the first thing we found was that a Predictor of the judge beats that judge. So that if an algorithm just does whatever the judge would typically do that is better than what the judge actually does. And this is sort of a crisp way to see noise. And so many of our decisions, we found this in domain. Again, domain. Something that could just tell you you seem to be doing something different than you usually do. Like in many domains, medicine. So the predicted you does better than you is quite a scary thing. I find that very disturbing at some level, and it's true across all experts.
- [42:10]
B
That must mean you're just cutting off some outliers and tails. I think that's regressing to the mean more.
- [42:16]
D
Regressing to your own mean.
- [42:17]
B
Yes.
- [42:17]
D
Yeah, it's sort of. Because what happens is we are use variables we really ought not to. It's exactly to your point. It's the Kyle Lowry point all over again. I'm keying in on things in the moment that like, hey, that's not what you normally do. Why are you doing that?
- [42:32]
B
Use variables that you probably want. You have some other reason to be driving an outcome in this case, but.
- [42:39]
A
Can you mention which variables were driving the outcome in this case?
- [42:42]
D
So then we expanded the prediction to not just be the things we thought mattered, but include a bunch of stuff. And you identify. About half of the variation in what judges do that they ought not to be doing comes from one variable in the data set, which is the mugshot. So you can build a predictor of who the judge will release just by using the mugshot. And it does extremely well.
- [43:05]
A
More than any other characteristic, more than.
- [43:07]
D
Whether they've committed a crime, what they're being charged for, how many times they failed to appear. Just the mugshot. And in the mugshot, it's not just the variables you would imagine. It reminded me of your Marc Casal thing, actually. It was variables that even judges and public defenders didn't realize mattered. So, for example, one of the biggest things that has about half the size of having committed a violent crime is if the person has a full or fat face. Now, what's interesting about that one is.
- [43:34]
B
When I asked the good or bad. That's how it is.
- [43:36]
D
Exactly. No, but isn't that weird? We don't know.
- [43:38]
B
We don't know.
- [43:39]
D
No, no, we know, but our intuition doesn't tell us. Even though in the data it's very.
- [43:43]
B
I just. I'm looking at our mug shots.
- [43:44]
D
Yeah, I'm looking at that, too. It's not a good sign, is it?
- [43:46]
A
Yeah, yeah, yeah.
- [43:47]
B
Word for you, Daryl.
- [43:48]
A
Guilty. Sendhil is what we're asking.
- [43:50]
D
So the good news is I think Daryl will not be going to jail.
- [43:55]
B
Oh, yeah. Nice.
- [43:56]
D
You have a slightly fuller groove, so.
- [43:57]
B
I can get away with more.
- [43:57]
D
Yeah, yeah, yeah. But you're not well groomed. That's the other thing you need to. Yeah, yeah. Halfway there.
- [44:03]
B
Well groomed.
- [44:04]
D
Well groomed is good. Slightly rounder face is very good.
- [44:09]
B
That's sad, though. It should not be effective.
- [44:12]
D
Absolutely. Not to say.
- [44:14]
A
Frankly, it's mind blowing. It's mind blowing that they were not aware of this. We would not intuit that based on our predisposition for bias.
- [44:20]
B
Or you might predict the opposite.
- [44:21]
D
You might have predicted the opposite. And so I think these ideas that when the camera turn on us and help us understand what we do and how. What we're doing, that would be probably the first angle I would take because a decision aid like that could help me understand myself.
- [44:35]
B
So eliminate mug shots. Might be your.
- [44:37]
D
Well, they see the person. Oh, right. And so in some sense. But yes, if we could do everything behind a curtain.
- [44:43]
B
Do like orchestras.
- [44:44]
D
Yes. Orchestras behind a curtain would be fantastic. But to me, it's also just. I think these algorithms can start ferreting things out about us that we did not know about ourselves. And that's, again, a very optimistic thing for me. I think it's part of the path that we've been growing on of becoming better stewards of ourselves.
- [45:12]
A
At the end of every episode of Pablo Torre Finds Out a show about finding stuff out, we say what we found out today. Daryl, would you like to begin?
- [45:21]
B
Can we start with Sendhil?
- [45:23]
A
Sendhil, would you like to begin?
- [45:26]
D
I. Oh, my God. I'm in the conversation so much. I'm gonna need you to say something, Daryl. We're just playing hot potato here.
- [45:33]
B
Yeah, I'm always picking up something. I'm trying to figure.
- [45:36]
D
So I did find out that Daryl is. Is very attuned to whether fans are interested, excited, not excited. I think a lot of fans would have foolishly thought, Daryl doesn't care about that, but he is shockingly attuned to that thing, which ex post makes sense, but I think fans would have been miscalibrated on that.
- [45:54]
B
I think everyone who says they don't read it, don't read the comments, don't whatever. It's all not true. We're all humans.
- [46:00]
A
LeBron James reads the comments, as does Derek Moran. We've learned that this week. Absolutely.
- [46:07]
B
I think probably one of the things I took away is the judge, the decision maker. And yeah, look for times maybe even create a decision making unit. We wouldn't be able to make a Darrel decision algorithm because it's federated across a head coach and multiple. We could create a decision making model and then whenever it spits out a different answer, you should start asking questions. So that would be my takeaway.
- [46:37]
A
Yeah, I found out.
- [46:38]
B
What did you find out?
- [46:39]
A
Yeah, I found out that there's a non zero chance that Sendhil has trashed you online. And that at the same time, he also thinks we'll probably be talking about this in 15 years. So Sendhil, Darrell, you guys, for listening. Thank you for doing this. This is great.
- [46:57]
D
Thank you, Pablo.
- [46:58]
B
Okay, thanks. That was fun. Appreciate you coming over. Y.
- [47:09]
A
This has been Pablo Torre Finds Out a Meadowlark Media production and I'll talk to you next time.