Dr. Katie Milkman (3:15)

This is Dean Oliver. He's a data scientist at espn and he helped create the Sports Network's analytics department in 2006. The Celtics needed to rebuild. And to do that meant a big shakeup of their roster, which required ruthless scouting. These recruiting decisions traditionally relied heavily on scouts experience, on their intuition.

Dean Oliver (3:38)

An old school scout is someone who goes and watches players play and uses their eye. They're watching the game. They have responsibility of contributing to making a judgment on whether a team should acquire that player, Whether through the draft or whether through trades or free agency.

Dr. Katie Milkman (3:56)

Scouts were respected because they'd been around the game forever. Some of them believed they could size up a player just by looking at him.

Dean Oliver (4:04)

I heard this from scouts, actually. There are guys I can see for five minutes and I never have to see him again because I know right then and that's obviously not the way a lot of data goes. There are players that have great games and there are players who have bad games that go against their typical pattern.

Dr. Katie Milkman (4:22)

But the Celtics were starting to question whether a scout seeing was the same as a scout knowing. Going into the 0708 season, the Boston Celtics knew they needed a new approach, even if they weren't entirely sure what that would look like.

Dean Oliver (4:36)

Yet they hired Mike Zarin, who came out of law school to work on legal aspects in the NBA. But Mike came in with a knowledge of analytics and trusting numbers. By showing this interest in using numbers, some of what came out of Moneyball. He was hired about the same time that Moneyball came out.

Dr. Katie Milkman (4:57)

You might be familiar with Moneyball. It's the famous book by Michael Lewis about the struggling Oakland A's baseball team and how they used Advanced analytics in 2002 to to hire new talent and turn their team around.

Dean Oliver (5:10)

Well, his on base percentage is all we're looking at now.

Dr. Katie Milkman (5:13)

The Moneyball story served as inspiration for other baseball teams and eventually other sports leagues to adopt a more analytical approach to scouting. With Mike Zarin's influence, analytics became a key part of the decisions that would be used to rebuild the 0708 Celtics roster.

Dean Oliver (5:32)

He is pretty close lipped about the details of it, but I know generally from our conversations on how he thinks about players and what they've had to do to build say a draft model. And they're trying to predict how good a player is going to be in year four or year three or throughout their career. They have metrics for that.

Dr. Katie Milkman (5:54)

The predictive player model that Zarin used offered the Celtics insight that wasn't visible to the public. And it helped them make key trade decisions to build a stronger team. As the season approached, Boston started making moves, big ones.

Dean Oliver (6:10)

The two new players that came to the team were Kevin Garnett and Ray Allen. These are players who had established careers. They were good for sure. They hadn't played necessarily quite as well in the last season.

Dr. Katie Milkman (6:24)

These players weren't unknowns, they were stars, but aging ones.

Dean Oliver (6:29)

There was some sense that they may decline, but the Celtics were able to identify that they had a long way to go before they would decline to a poor level or even an average level. Kevin Garnett, who was not a very big point scorer but was a great defender by all the metrics. He was a genius on the defensive side of the ball. And if you watch highlights around basketball, you will see that 90 plus percent of the highlights are about offense. They don't show what Kevin Garnett does when he's off the ball. And even the old school scouts, old school scouts are supposed to look for that and they're supposed to see it. But it is not as prominent as offense.

Dr. Katie Milkman (7:14)

Mike Zarin and the Celtics general manager Danny Ainge brought together players whose reputations weren't flashy, but the data said they were undervalued. Paired with longtime Celtic Paul Pierce, Allen and Garnett became a force to be reckoned with. Known as the Big Three.

Dean Oliver (7:30)

Celtic buzz is back in a big way. With the additions, of course, of Kevin Garnett and Ray Allen, along with all star Paul Pierce, that Celtics season felt like a season of inevitability. They were so good on a regular basis, it just felt like, okay, you're at the top. It's going to be a mountain for everybody else to climb up to touch them.

Dr. Katie Milkman (7:56)

The Celtics went 66 and 16 in the regular season. That's 66 games won and 16 games lost over the 82 game season. They shut down teams that usually run up the score. They looked composed even when things got tight. By spring, the Celtics had one of the best records in NBA history and they cruised into the Eastern Conference final playoffs. But of course, regular season success doesn't guarantee anything. And the playoffs exposed some weaknesses.

Dean Oliver (8:30)

They struggled a bit in their playoffs. It became clear that, okay, teams may have figured some of their juice out. They all of a sudden had to play teams that were generally better. There was no easy ones for the Celtics anymore. When you get to the playoffs, when you're playing the the top half of the league, okay, every night is work.

Dr. Katie Milkman (8:50)

In the Eastern conference semifinals against LeBron James and the Cleveland Cavaliers, the Celtics pushed it to a Game seven and won by just five points.

Dean Oliver (9:01)

And the Cavaliers fall just short in an absolute game seven thriller. And the Celtics win that'll do it. The Celtics are going back.

Dr. Katie Milkman (9:11)

Then, for just the second time since 1988, Boston reached the Eastern Conference finals. They faced the Detroit Pistons, a team that had dominated the east for much of the decade. After a back and forth series, the Celtics prevailed 4 2, becoming Eastern Conference champions and NBA finalists. But the work was just beginning. They'd have to defeat the LA Lakers in the playoff finals. And no one was calling this a sure thing anymore.

Dean Oliver (9:46)

The Lakers hate the Celtics. The Celtics hate the Lakers. Those two teams, they faced each other enough over time. And the Celtics, with their most titles on record, they were getting challenged by the Lakers. Lakers were getting close, so the rivalry was very real.

Dr. Katie Milkman (10:07)

The Lakers were the Western Conference champions and the Celtics biggest rivals, meeting them in the finals for the 11th time in NBA history. They were led by superstar coach Phil Jackson and the incredible Kobe Bryant. But unlike the Celtics, the Lakers hadn't relied on analytics in crafting their roster.

Dean Oliver (10:27)

What the Lakers frankly had as an advantage is that they were the Lakers. They were Showtime players, wanted to play for them. If you are working for a team in a smaller market and you think you're going to get a player that the Lakers want, you're not going to get them. Basically all the rest of the teams realized that they needed the analytics weapon to deal with someone who stood at the top with their branding like the Lakers.

Dr. Katie Milkman (10:56)

The series was exhilarating to watch. It started at home in Boston with over 18,000 fans in TD Bank North Garden, decked out in green and white and cheering through the rafters.

Dean Oliver (11:10)

NBA Finals games. They are incredibly loud. If you are sitting up near the top of a dome and such like that, you can't hear. Your ears will be ringing at the end. Especially in a place like Boston, people waving the signs, wearing the green when they were facing the Lakers. It was loud.

Dr. Katie Milkman (11:30)

It was the first time the Celtics had been this close to the championship since 1986.

Dean Oliver (11:36)

Their best players, their big three, definitely played well in those games and they ended up winning by around 10. They went all the way across country to face the Lakers and in game three, the Lakers kind of turned it around. They were able to slow down some of the big three. So at that point, what they say is that the series only becomes real when a road team wins. And so the Celtics were facing game four in la and that was when they were able to kind of take the close win. They were able to win by six points in game four. So they had at that point a 31 lead. So. So finally you had a Road team win. The Lakers were then desperate for game five.

Dr. Katie Milkman (12:23)

Game after game, the series swung back and forth. Momentum shifted. Hope wavered. By the time the Celtics returned home for game six, they led three games to two. They were desperate to close out the series.

Dean Oliver (12:38)

The Lakers. Actually, the Lakers did come out a little bit flat. By halftime, the Celtics had a 23 point lead. And so there was a sense that, okay, we've done a lot of the work that we need to do. We just need to maintain so that lead then kind of grows more. In the second half. They didn't take their foot off the pedal.

Dr. Katie Milkman (13:03)

In the final minutes, the energy in the garden was erupting. Fans envisioning the NBA championship banner hanging from the rafters. The Celtics were winning 131 to 91. With just 30 seconds left courtside, Paul Pierce doused coach Doc Rivers with orange Gatorade. Splashing news cameras.

Dean Oliver (13:27)

They end up winning by 39 points.

Dr. Katie Milkman (13:31)

Green and white confetti filled the stadium. Players cried and gave passionate interviews. Garnett shouted, anything is possible. And bowed down on center court to thank Lucky the leprechaun. The Celtics logo. Finally, after 22 years, the Celtics had won the NBA championships. But what few people in the arena could see was that many of the most important choices leading to this remarkable victory were made long before the opening tip. The 0708 Boston Celtics were lauded with awards. Kevin Garnett was named NBA Defensive Player of the Year. Paul Pierce was named NBA Finals MVP. General Manager Danny Ainge was named NBA Executive of the Year. And all of Boston's 41 regular home games sold out. For Mike Zarin, who led the charge on analytics, the win meant everything.

Dean Oliver (14:32)

I know what it meant for Zarin. Zarin, he got the big ring. He was so proud. Being a fan for all his childhood and then being able to add a ring onto a finger. Zarin doesn't brag about a lot, but he definitely was able to show off how happy he was. That's the thing, right? When you're a fan, you feel like you earned it in a sense, that you supported your team. You were loud and things that like, like that. When you're Mike Zarin, you're proud because you actually made a decision that shaped that franchise, and he, he earned it.

Dr. Katie Milkman (15:11)

For Dean, the Celtics win was a watershed moment in demonstrating how useful analytics could be in building a championship NBA team.

Dean Oliver (15:20)

I don't think it's recognized as much as it should be for, for how it kind of changed the league. There were enough circumstances. Oh, yeah. Those players were already good, but it was a big analytical decision behind the scenes. Certainly if they hadn't changed their path, if they hadn't decided to go after Ray Allen and Kevin Garnett, they would have been in a rebuilding period for quite a while. They might have competed, they might have been playoff teams, but it's extremely unlikely. Without having made another deal at some point to get better players, it would have been practically impossible to win a championship.

Dr. Katie Milkman (16:00)

Dean Oliver is a pioneering sports statistician at ESPN and the author of the books Basketball on Paper and Basketball Beyond Paper. You can find more details and links to his writing in the show notes and@schwab.com podcast. The Boston Celtics 2008 NBA win is legendary. Of course, many factors contributed to the team's success in addition to the choice to rely on algorithms. Factors like the individual talent of the players, the chemistry of the Big Three, a relentless focus on defense, and coach Doc Rivers building a culture of trust and teamwork. But without insights from Zarin's player prediction models, it's unlikely the team would have succeeded in the same way today. Algorithms are ubiquitous in the NBA. Every team has an analytics department, and they all strive to project how good players will be in this season and in the future. Algorithms have shaped the game in other ways, too. They've guided teams towards shooting more three pointers, which were previously thought of as high risk and low reward compared to two pointers. Defensive switching has also become more common, where players don't guard the same person all the time. Of course, coaches and scouts still second guess algorithms, especially when a pick that looked perfect goes wrong. We explored people's tendency to lose greater faith in algorithms than in humans for equivalent missteps. In an earlier episode of Choiceology that was back in February of 2019 with my Wharton colleague Cade Massey. The episode looked at algorithm aversion, which describes our tendency after seeing algorithms and humans make the same mistakes, to trust flawed human judgment over superior machine logic. The question of when and how much we'll embrace algorithmic guidance, and when we'll turn away from it, is the focus of my next guest's research. Jennifer Log is an assistant professor of management at Georgetown University's McDonough School of Business, and she joins me to talk about her research on algorithm appreciation. Hi Jen, thank you so much for joining me today.

Jennifer Log (18:16)

Thanks so much for having me. I'm excited to be here.

Dr. Katie Milkman (18:19)

Well, I'm excited to talk about your work on algorithm appreciation. Could you start by just defining what that term is that you and your Collaborators coined to describe a consistent pattern you are noticing in human behavior.

Jennifer Log (18:31)

So we find that people incorporate advice more from an algorithm than from people. What we found was that the people who received advice from an algorithm incorporated more of that advice into their final prediction than the people who thought that advice came from a person. And that's the result that we call algorithm appreciation.

Dr. Katie Milkman (18:55)

I really love this work and I was hoping you could give us some examples of the kinds of algorithms that offer people advice. Just so that we're all on the same page about what we're talking about here when we say an algorithm or a human could offer someone advice.

Jennifer Log (19:08)

I always kind of think of Netflix. Netflix is known for its algorithm generating recommendations. But I could also text my friend if it's a rainy day and ask her, hey, what TV shows have you been binging?

Dr. Katie Milkman (19:21)

And in this era where LLMs are everywhere, would you consider an LLM like a ChatGPT or Claude giving advice to be an algorithm category? That could be compared with, you know, calling my doctor or phoning my neighbor?

Jennifer Log (19:38)

Definitely. Algorithms from 10 years ago versus Genai today. It can do so many more things.

Dr. Katie Milkman (19:47)

I'd love it if you could describe one or two of your favorite studies showing that algorithm appreciation is an important phenomenon.

Jennifer Log (19:56)

So in one of our studies, our participants made predictions about the popularity of songs that came out. For example, what rank will Perfect by Ed Sheeran place on the Billboard magazine Hot 100 this week week. So how popular will it be? Then they gave their numeric answer. On the next page, they received advice that half of our participants read that the advice came from an algorithm, half of our participants read it came from a person. And they made their final prediction so they could decide how much they wanted to listen to or disregard that advice. And then we measured how much people updated from their time 1 prediction to their time 2 prediction. So we found that when people were making these predictions about songs, they listened more to advice when they thought it came from an algorithm than from a person.

Dr. Katie Milkman (20:55)

So the same advice labeled differently as coming from an algorithm versus a person basically produces different decisions. Do you find that experts and novices are similar when they are choosing how to incorporate advice from an algorithm or a human advisor? Because I think you maybe have another study on that, right?

Jennifer Log (21:12)

Yeah. So we also asked people to make business predictions. But in this experiment, we compared how non experts, people who didn't necessarily make forecasts or predictions for a living, compared to a sample of national security professionals, people who on a regular basis for their jobs make predictions. And what we found was that our non expert sample, seeing the same exact questions, responded quite differently than our national security professionals. We found that our non experts were quite happy to listen to algorithmic advice relative to advice from people. However, when the experts, these national security professionals were deciding how much they wanted to listen to the advice that they received in the study, it actually didn't matter if they were the group of people that read that the advice came from the algorithm or advice came from other people. What the national security professionals did was discount advice from everyone. So I expected that maybe the experts knew something about these predictions that I could never expect. And maybe them discounting advice was a good thing for their accuracy. But we found that when we scored all of these predictions on accuracy, our sample of non experts actually made more accurate predictions than our expert sample. And that's because they were willing to listen to the algorithmic advice while the experts discounted all advice.

Dr. Katie Milkman (22:50)

Yeah, the implications of that are a little scary, right? As two experts speaking to one another in a moment when algorithms are taking over sounds like maybe don't listen to us, but listen to the algorithms and you'll make better judgments.

Jennifer Log (23:03)

It's a pretty humbling result.

Dr. Katie Milkman (23:04)

Yeah. Yeah, that's really interesting. Could you tell us a little bit about why it is that people listen to advice from algorithms more than people when the advice is the same? It's not totally intuitive to me. Why getting an estimate that is labeled this came from an algorithm would lead me to say, oh, wow. Okay. Well, the probability forecast I just made of whether or not this geopolitical event will happen, I should really update now, now that I've got an algorithm telling me something more than if I'm told. Here's a piece of advice from a group of wise humans or from one wise human.

Jennifer Log (23:44)

So my answer might not be very satisfying.

Dr. Katie Milkman (23:47)

That's okay.

Jennifer Log (23:48)

One of the things that I found was when we gave our participants 7 simple questions related to just simple mathematical questions, that was our way of capturing their comfort with numbers. So researchers have called it their numeracy. But the more comfortable people were with numerics, the more they relied on algorithmic advice. Now, why my answer might not be totally satisfying is because I tested a lot of other things and they did not actually drive the effect. So one of the things that I tested that didn't explain the effect was age. My prediction was that people who were older might have less experience with algorithms, say Netflix algorithms, compared to younger people. So age I used as a proxy for familiarity with algorithms. And it turns out in Our data age did not correlate with people's responsiveness to algorithmic advice, their willingness to update to algorithmic advice.

Dr. Katie Milkman (24:57)

Is there any sense in which giving people a deeper understanding of how an algorithm works can affect the degree of algorithm appreciation? If we sort of unpack the black box, does that exacerbate algorithm appreciation, or actually, does it reduce it?

Jennifer Log (25:15)

Yeah. So some of the first studies that I ran, I was kind of obsessed with the idea that you just brought up. And I tried to present the algorithm, which ultimately, for a lot of our lives, the algorithms and Gen AI we deal with, they are black box. But I wanted to try to unpack that and understand if people knew different pieces of information about the algorithm, would knowing more encourage them to trust it more than they already seem to do? Or might knowing more actually make them more hesitant? Because now that they understand it more, they might kind of consider how things could go wrong. So I compared if people were shown calculations of what was kind of behind the algorithm, the gears of the clock, if you will, and if it was presented in a simpler way, people didn't respond differently than if the algorithm was presented in a more complex way. And I would like to kind of mention Dan Goldstein's work. So Dan has a paper presenting people with different information about an algorithm, also simple and complex, and he similarly found no difference. People were quite happy to listen more to the algorithm than person, and it didn't matter if people thought that algorithm was complex or not. And I kind of think that's a nice bridge from thinking about algorithms to Gen AI, too. So hopefully the research that we've done on algorithms to some extent also can speak to how people think about Gen AI.

Dr. Katie Milkman (26:53)

What are some situations where you think people would be wisest to privilege algorithmic advice?

Jennifer Log (27:00)

Kind of feels like a tightrope walk. So where and when is this useful? One of my favorite results is actually from the medical domain. And they found this is before Genai. They found that when a biopsy was shown to an algorithm versus a pathologist, the algorithm was better at predicting the severity of breast cancer than pathologists. And the reason why I love this result so much is that they unpacked why it was better. The algorithm that they used was actually identifying new cues that pathologists just didn't even have in their textbook. So there were cues that were otherwise overlooked. So regardless of how you feel about algorithms, even in the abstract, if we can learn from algorithms in Genai, cues that we've otherwise overlooked that are predictive of forecasts that we care about, well, then we can update textbooks. Even if you say, I want a person to have ultimate control, are there

Dr. Katie Milkman (28:09)

situations though where you would say you want people to privilege human judgment over algorithmic judgment that are sort of an edge case to what we've been talking about where algorithmic advice is so powerful?

Jennifer Log (28:23)

So I kind of keep thinking about what Daniel Kahneman wrote before his book on noise came out, that algorithms are a great antidote or solution to human judgment, which can be inconsistent because by definition algorithms are consistent. I think the boundary conditions are, which Kahneman notes, where the data is lower quality. So if the data you're using is based on historic data made by people that might be biased, it might be missing data which could lead to bias.

Dr. Katie Milkman (29:02)

So you could end up with a really low quality algorithm and trust too much in it.

Jennifer Log (29:07)

Right.

Dr. Katie Milkman (29:07)

That's really helpful. What advice do you have for our listeners about how they can make better decisions now that they're familiar with your research on this topic?

Jennifer Log (29:16)

The way I think about how people can improve their lives with generative AI is really to go back to classic research on the wisdom of the crowd. We know that getting multiple predictions and judgments from groups of people is really beneficial for improving our accuracy. And we can think of Gen AI as another source to add to that collective wisdom.

Dr. Katie Milkman (29:45)

Jen, what do you do differently in your own life as a result of having studied algorithm appreciation?

Jennifer Log (29:50)

I think a lot more about data. And if I'm getting advice that seems to be coming from an algorithm, I think more questions come up for me. Is this data complete? You might think you have data that's useful to make predictions for the future, but if you are missing data on so many other demographics, or it could be about people's preferences, you are going to be making the same mistakes over and over again. The algorithm is going to magnify whatever patterns in the input data. So I think think a lot more about where does this data come from. Is this data representative of the things that my organization and I care about? And that is kind of the driving factor behind why I created a class called the Psychology of Big Data, like thinking carefully about measures and what we're feeding into algorithms and Gen AI?

Dr. Katie Milkman (30:59)

That's so helpful. Thank you so much for taking the time to talk to me today, Jen. I really appreciated it.

Jennifer Log (31:04)

Thank you. This was so much fun.

Dr. Katie Milkman (31:08)

Jennifer Log is an assistant professor of management at Georgetown University's McDonough School of Business. You can find her paper Algorithm Appreciation. People prefer algorithmic to human judgment in our show notes or just visit schwab.com choiceology. To hear more about the intersection of behavioral science and sports, check out the Financial Decoder episode Does yous Inner Scorekeeper Skew youw Judgment? You can find it@schwab.com financialdecoder or just search for Financial Decoder in your podcast app. In the past few years, sophisticated algorithms that can offer us guidance on nearly every type of decision have gone from a futuristic concept to a normal part of modern life. Genlog and her collaborators have shown that unless we've just watched an algorithm make a bad decision, we tend to exhibit algorithm appreciation, trusting and incorporating the very same advice more when we're told it came from an algorithm than when we're told it came from another person. This unquestionably reflects our appreciation that algorithms can process fear far more data faster and more reliably than we can. And research by John Bogard and Suzanne Hsu has shown that the more common it's become for people to rely on algorithms in a given setting, be it on the basketball court or in the operating room, the more we see algorithm appreciation arise in that setting. Our trust in algorithms is exciting because it suggests a safer, healthier future is around the corner. Self driving cars will soon be able to prevent tens of thousands of annual traffic deaths if we trust them to take over. And algorithm powered scans will catch cancer, heart failure, and other treatable diseases earlier and more often than doctors ever could. But it's also important that we not blindly hand over our futures to algorithms without recognizing that they aren't all created equal. While many deserve our trust, biases like sexism and racism can be baked into algorithms if they're trained on flawed data. And like humans, recent research shows that large language models exhibit many biases. For instance, Stephen Lehrer and colleagues have shown that they exhibit cognitive dissonance, persistently adopting views they've been randomly assigned to articulate after the fact to maintain a semblance of internal consistency. What other biases have LLMs learned from us or evolved independently? Research on this is still in its infancy, but there is some evidence that prompting LLMs to make rational decisions reduces biases. Another risk of placing too much trust in algorithms is deskilling. Recent medical research found that doctors with many years of experience who performed colonoscopies for just three months with algorithmic assistance were less effective when they returned to performing unassisted operations. If we outsource learning and thinking to algorithms, that may be okay in some arenas but catastrophic in others. So we should lean into our algorithm appreciation, but with care and calibration, and with a recognition that deskilling is a risk, that model quality is limited by training, data, and algorithmic biases, and that blind trust in any powerful tool can lead to missteps. There are limits to the quality of any advice in a world without crystal balls. You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or Feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book how to Change, or sign up for my monthly newsletter, Milkman delivers on Substack. Next time I'll speak with Richard Thaler, Nobel laureate and emeritus professor at the University of Chicago's Booth School of Business. We'll explore a bias that helps explain why most of us polish off our plates at restaurants, even when we're too full to enjoy what we're eating, and why we feel compelled to finish boring books. I'm Dr. Katie Milkman. Talk to you soon. For important disclosures, see the show notes or visit schwab.com choiceology.