The Algorithm Advantage

Summary

Choiceology with Katy Milkman

Episode: "The Algorithm Advantage"

Release Date: March 23, 2026
Host: Dr. Katy Milkman

---

Episode Overview

In "The Algorithm Advantage," host and behavioral scientist Dr. Katy Milkman explores why and when people trust algorithmic advice over human expertise. The episode journeys through the transformation of the Boston Celtics with analytics, examines pioneering research on algorithm appreciation, and considers when humans should—and should not—rely on algorithms. Insights span high-stakes sports, medicine, everyday decision-making, and the profound implications of generative AI.

---

Key Discussion Points & Insights

Human vs. Algorithmic Advice

• Opening Scenario: Katy sets the stage with a relatable dilemma: who is more trusted for dinnertime cooking advice—an algorithm like ChatGPT or a foodie friend?

  • Mixed opinions surface: some trust the speed and concrete guidance of ChatGPT, others the authenticity and tailored touch of a human friend.
  • Quote (Dr. Katy Milkman, 01:06): “I find that recipe suggestions are usually pretty good. I guess the algorithm, because it's faster and you know, at that point, I just want dinner on the table.”

• Main Theme: This everyday conundrum introduces the episode’s focus—how we weigh automated versus human advice in various domains.

---

Analytics and the 2008 Boston Celtics: A Sports Revolution

The Celtics’ Dilemma and the Rise of Analytics

• Celtics’ Low Point (02:19–03:15): The team, long iconic, hit rock bottom in 2006, plagued by injuries and poor results.
• Traditional Scouting (03:15–04:22):
  • Dean Oliver (ESPN data scientist) explains scouts’ reliance on “just watching” and gut feel.
  • Quote (Dean Oliver, 04:04): “There are guys I can see for five minutes and I never have to see him again because I know right then... obviously not the way a lot of data goes.”

The Turn to Analytics

• Moneyball Influence (04:57–05:10): The Celtics, inspired by baseball’s Moneyball movement, bring in Mike Zarin—an analytics-driven decision-maker.
• Building a Predictive Model (05:32–05:54):
  • Zarin’s draft models predict player performance years into the future, revealing undervalued talent invisible to scouts.

Key Trades and “The Big Three”

• Acquiring Garnett & Allen (06:10–07:14): Data identify veteran stars Garnett and Allen as undervalued assets with hidden strengths (notably defense).
  • Quote (Dean Oliver, 06:29): “Kevin Garnett ... was a genius on the defensive side of the ball. ... Even the old school scouts are supposed to see it. But it is not as prominent as offense.”

Triumph and Impact

• Season Success (07:30–13:31):

  • Celtics post one of NBA’s best-ever records; playoff story builds tension and drama.
  • Finals vs. Lakers: Analytics undergird team construction while LA relies on brand and star power.

• Championship Outcome (13:03–13:31): Celtics win Game 6 in a euphoric blowout; green and white confetti rains down as data-driven strategy delivers.

Legacy and Lessons

• Dean Oliver Perspective (15:20): “I don't think it's recognized as much as it should be for, for how it kind of changed the league. ... Certainly if they hadn't changed their path ... it would have been practically impossible to win a championship.”
• Modern NBA (16:00):
  • Every team now has an analytics department. Models drive roster construction, game tactics (like more three-pointers), and player evaluation.
  • Yet, skepticism persists—when algorithms make mistakes, trust can quickly erode.

---

Behavior Science Spotlight: Algorithm Appreciation

What Is "Algorithm Appreciation"? (18:19–18:55)

• Definition (Jennifer Log): “People incorporate advice more from an algorithm than from people. ... we call [that] algorithm appreciation.”

Examples & Experiments

• Netflix & Streaming Recommendations (19:08): Algorithms like Netflix are familiar sources of advice; LLMs like ChatGPT represent a new frontier.

• Classic Study (19:56–20:55):

  • Subjects predict song popularity, receive identical advice labeled as coming from either an "algorithm" or a "person".
  • Finding: People update their beliefs more when told the advice comes from an algorithm.
  • Quote (Dr. Katie Milkman, 20:55): “So the same advice labeled differently as coming from an algorithm versus a person basically produces different decisions.”

• Expert vs. Novice Dynamics (21:12–22:50):

  • National security “experts” discount all advice (human or algorithmic), while non-experts trust algorithms more—and as a result, outperform experts on forecasting accuracy.
  • Quote (Dr. Katie Milkman, 22:50): “The implications of that are a little scary, right? As two experts speaking to one another... maybe don't listen to us, but listen to the algorithms and you'll make better judgments.”

Drivers and Boundaries

• Numeracy Matters (23:48):
  • Participants comfortable with numbers trust algorithmic advice more.
• No Age Effect: Familiarity with technology (proxied by age) doesn't explain algorithm trust.
• Transparency (“Unpacking the Black Box”, 25:15–26:53):
  • Showing people how an algorithm works (“simple” vs. “complex”) doesn’t meaningfully change their level of trust.
  • Support from Dan Goldstein's research: Complexity level doesn’t sway willingness to accept algorithmic advice.

---

When to Prefer Algorithmic Advice—and When to Beware

When Algorithms Excel

• Medicine Example (27:00–28:09): Algorithms outperformed pathologists in diagnosing cancer, spotting cues missed by humans.
• Broader Implication: Algorithms add value by processing vast data and highlighting overlooked patterns—sometimes even “updating textbooks.”

When Humans Should Be Cautious

• Limits and Risks (28:23–29:07):
  • Algorithms are only as good as their training data—biased or incomplete input propagates errors and prejudices.
  • Quote (Katy Milkman, 29:02): “So you could end up with a really low quality algorithm and trust too much in it.”

Practical Takeaways

• Synthesis with the “Wisdom of the Crowd” (29:16–29:45):
  • Incorporate group and algorithmic input for best accuracy—GenAI is a valuable voice in a collective.
• Personal Practice (Jennifer Log, 29:50):
  • Scrutinize the data source behind any algorithm.
  • Quote (Jennifer Log, 29:50): “If I'm getting advice that seems to be coming from an algorithm, I think more questions come up for me. Is this data complete? ... The algorithm is going to magnify whatever patterns in the input data.”

---

Broader Reflections & Societal Implications

Scaling Trust and the Rise of Generative AI

• Growing Algorithm Adoption (31:08):
  • The more an algorithm is used in a domain, the more trust (“algorithm appreciation”) grows in that context.
  • Cited research (John Bogard, Suzanne Hsu): widespread adoption predicts rising algorithm appreciation.

Cautions: Bias & Deskilling

• Bias Risks:

  • Algorithms can perpetuate racism, sexism, cognitive biases—especially LLMs (large language models).
  • Quote (Katy Milkman, 31:37): “Biases like sexism and racism can be baked into algorithms if they're trained on flawed data. And like humans, ... LLMs exhibit many biases.”

• Deskilling Hazard:

  • Over-reliance can erode human expertise—doctors, for example, may lose their edge if aided too exclusively by AI.
  • Quote (Katy Milkman): “If we outsource learning and thinking to algorithms, that may be okay in some arenas but catastrophic in others.”

“Trust, But Calibrate”

• Final Takeaway (32:00):
  • Appreciate and incorporate high-quality algorithmic advice—but remain vigilant, critical, and aware of underlying data and context.
  • Quote (Katy Milkman): “We should lean into our algorithm appreciation, but with care and calibration... blind trust in any powerful tool can lead to missteps.”

---

Notable Quotes

• Katy Milkman (Opening, 01:06):
  “I find that recipe suggestions are usually pretty good. I guess the algorithm, because it's faster... I just want dinner on the table.”

• Dean Oliver (Celtics context, 04:04):
  “There are guys I can see for five minutes and I never have to see him again because I know right then and that's obviously not the way a lot of data goes.”

• Jennifer Log (Algorithm Appreciation, 18:31):
  “People incorporate advice more from an algorithm than from people. ... We call that algorithm appreciation.”

• Dr. Katy Milkman (Impact, 22:50):
  “The implications of that are a little scary, right? ... maybe don't listen to us, but listen to the algorithms and you'll make better judgments.”

• Jennifer Log (Data Skepticism, 29:50):
  “If I'm getting advice that seems to be coming from an algorithm, I think ... is this data complete? ... The algorithm is going to magnify whatever patterns in the input data.”

---

Key Timestamps

• 01:06: Everyday human vs. algorithm advice example
• 02:19 – 16:00: Deep dive into the Celtics’ analytics transformation
• 18:16 – 31:08: Jennifer Log interview—algorithm appreciation research and implications
• 29:16: Practical advice for applying algorithmic guidance
• 31:08 – End: Societal consequences, bias, deskilling, calibrated trust

---

Episode Tone and Style

Engaging, clear, and anchored in behavioral science. Katy Milkman blends vivid storytelling (Celtics, medical anecdotes) with research insights. The episode balances warning and optimism, urging listeners to use algorithmic tools thoughtfully.

---

Useful for Listeners Who Missed the Episode

This episode reveals why we increasingly gravitate toward algorithmic advice—even preferring it over human judgment—and when that bias can help (or backfire). Drawing lessons from the world of sports, business, medicine, and AI, the show offers both inspiration and caution for those striving to make better choices in a data-driven world.

Transcript

[Dr. Katie Milkman]

Okay, what am I supposed to make with this? Imagine you're cooking dinner on a Thursday night. You open your fridge and you see a random collection of groceries. Butter, ketchup, a bag of shredded cheddar cheese, a jar of salsa, and some chicken. So you text your foodie friend asking for ideas about what to do with the ingredients you have on hand. While you wait, you also type a list of your refrigerator's limited offerings into ChatGPT to see what it recommends. Which cooking advice are you more likely to follow and why? Honestly, probably ChatGPT because it gives you something concrete right away.

[Jennifer Log]

I think I trust my friend. She knows what I like and is a good cook.

[Dean Oliver]

I. I don't know. I wouldn't ask a friend, a foodie

[Dr. Katie Milkman]

friend, over chatgpt because I trust the authenticity of a foodie friend's advice.

[Jennifer Log]

I think I would ask ChatGPT.

[Dr. Katie Milkman]

I find that recipe suggestions are usually pretty good. I guess the algorithm, because it's faster

[Dean Oliver]

and you know, at that point, I

[Dr. Katie Milkman]

just want dinner on the table. Today we'll dig into what happens when people face choices between trusting another person's expertise versus the advice produced by a computer. We'll look at why it matters in the kitchen, at the hospital, and on the basketball court. I'm Dr. Katie Milkman and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.

[Dean Oliver]

The Boston Celtics are iconic. They have won a very significant percentage of all the NBA championships in history over the 80 year history or so of the NBA.

[Dr. Katie Milkman]

But the team had been in a rut for years and hit a Low Point in 2006 back when I was a graduate student living in nearby Cambridge, Massachusetts. So I remember it well.

[Dean Oliver]

They were among the worst teams in the NBA. They had the beginnings of a young core. They had Paul Pierce, who was already an established star, who missed a good part of the year. And then some of their young players got hurt. I remember going to one of their games that year and the Celtics people apologizing that they really didn't have their full roster that night. So their devoted fans were rather critical and they had to kind of reinvent themselves.

[Dr. Katie Milkman]

This is Dean Oliver. He's a data scientist at espn and he helped create the Sports Network's analytics department in 2006. The Celtics needed to rebuild. And to do that meant a big shakeup of their roster, which required ruthless scouting. These recruiting decisions traditionally relied heavily on scouts experience, on their intuition.

[Dean Oliver]

An old school scout is someone who goes and watches players play and uses their eye. They're watching the game. They have responsibility of contributing to making a judgment on whether a team should acquire that player, Whether through the draft or whether through trades or free agency.

[Dr. Katie Milkman]

Scouts were respected because they'd been around the game forever. Some of them believed they could size up a player just by looking at him.

[Dean Oliver]

I heard this from scouts, actually. There are guys I can see for five minutes and I never have to see him again because I know right then and that's obviously not the way a lot of data goes. There are players that have great games and there are players who have bad games that go against their typical pattern.

[Dr. Katie Milkman]

But the Celtics were starting to question whether a scout seeing was the same as a scout knowing. Going into the 0708 season, the Boston Celtics knew they needed a new approach, even if they weren't entirely sure what that would look like.

[Dean Oliver]

Yet they hired Mike Zarin, who came out of law school to work on legal aspects in the NBA. But Mike came in with a knowledge of analytics and trusting numbers. By showing this interest in using numbers, some of what came out of Moneyball. He was hired about the same time that Moneyball came out.

[Dr. Katie Milkman]

You might be familiar with Moneyball. It's the famous book by Michael Lewis about the struggling Oakland A's baseball team and how they used Advanced analytics in 2002 to to hire new talent and turn their team around.

[Dean Oliver]

Well, his on base percentage is all we're looking at now.

[Dr. Katie Milkman]

The Moneyball story served as inspiration for other baseball teams and eventually other sports leagues to adopt a more analytical approach to scouting. With Mike Zarin's influence, analytics became a key part of the decisions that would be used to rebuild the 0708 Celtics roster.

[Dean Oliver]

He is pretty close lipped about the details of it, but I know generally from our conversations on how he thinks about players and what they've had to do to build say a draft model. And they're trying to predict how good a player is going to be in year four or year three or throughout their career. They have metrics for that.

[Dr. Katie Milkman]

The predictive player model that Zarin used offered the Celtics insight that wasn't visible to the public. And it helped them make key trade decisions to build a stronger team. As the season approached, Boston started making moves, big ones.

[Dean Oliver]

The two new players that came to the team were Kevin Garnett and Ray Allen. These are players who had established careers. They were good for sure. They hadn't played necessarily quite as well in the last season.

[Dr. Katie Milkman]

These players weren't unknowns, they were stars, but aging ones.

[Dean Oliver]

There was some sense that they may decline, but the Celtics were able to identify that they had a long way to go before they would decline to a poor level or even an average level. Kevin Garnett, who was not a very big point scorer but was a great defender by all the metrics. He was a genius on the defensive side of the ball. And if you watch highlights around basketball, you will see that 90 plus percent of the highlights are about offense. They don't show what Kevin Garnett does when he's off the ball. And even the old school scouts, old school scouts are supposed to look for that and they're supposed to see it. But it is not as prominent as offense.

[Dr. Katie Milkman]

Mike Zarin and the Celtics general manager Danny Ainge brought together players whose reputations weren't flashy, but the data said they were undervalued. Paired with longtime Celtic Paul Pierce, Allen and Garnett became a force to be reckoned with. Known as the Big Three.

[Dean Oliver]

Celtic buzz is back in a big way. With the additions, of course, of Kevin Garnett and Ray Allen, along with all star Paul Pierce, that Celtics season felt like a season of inevitability. They were so good on a regular basis, it just felt like, okay, you're at the top. It's going to be a mountain for everybody else to climb up to touch them.

[Dr. Katie Milkman]

The Celtics went 66 and 16 in the regular season. That's 66 games won and 16 games lost over the 82 game season. They shut down teams that usually run up the score. They looked composed even when things got tight. By spring, the Celtics had one of the best records in NBA history and they cruised into the Eastern Conference final playoffs. But of course, regular season success doesn't guarantee anything. And the playoffs exposed some weaknesses.

[Dean Oliver]

They struggled a bit in their playoffs. It became clear that, okay, teams may have figured some of their juice out. They all of a sudden had to play teams that were generally better. There was no easy ones for the Celtics anymore. When you get to the playoffs, when you're playing the the top half of the league, okay, every night is work.

[Dr. Katie Milkman]

In the Eastern conference semifinals against LeBron James and the Cleveland Cavaliers, the Celtics pushed it to a Game seven and won by just five points.

[Dean Oliver]

And the Cavaliers fall just short in an absolute game seven thriller. And the Celtics win that'll do it. The Celtics are going back.

[Dr. Katie Milkman]

Then, for just the second time since 1988, Boston reached the Eastern Conference finals. They faced the Detroit Pistons, a team that had dominated the east for much of the decade. After a back and forth series, the Celtics prevailed 4 2, becoming Eastern Conference champions and NBA finalists. But the work was just beginning. They'd have to defeat the LA Lakers in the playoff finals. And no one was calling this a sure thing anymore.

[Dean Oliver]

The Lakers hate the Celtics. The Celtics hate the Lakers. Those two teams, they faced each other enough over time. And the Celtics, with their most titles on record, they were getting challenged by the Lakers. Lakers were getting close, so the rivalry was very real.

[Dr. Katie Milkman]

The Lakers were the Western Conference champions and the Celtics biggest rivals, meeting them in the finals for the 11th time in NBA history. They were led by superstar coach Phil Jackson and the incredible Kobe Bryant. But unlike the Celtics, the Lakers hadn't relied on analytics in crafting their roster.

[Dean Oliver]

What the Lakers frankly had as an advantage is that they were the Lakers. They were Showtime players, wanted to play for them. If you are working for a team in a smaller market and you think you're going to get a player that the Lakers want, you're not going to get them. Basically all the rest of the teams realized that they needed the analytics weapon to deal with someone who stood at the top with their branding like the Lakers.

[Dr. Katie Milkman]

The series was exhilarating to watch. It started at home in Boston with over 18,000 fans in TD Bank North Garden, decked out in green and white and cheering through the rafters.

[Dean Oliver]

NBA Finals games. They are incredibly loud. If you are sitting up near the top of a dome and such like that, you can't hear. Your ears will be ringing at the end. Especially in a place like Boston, people waving the signs, wearing the green when they were facing the Lakers. It was loud.

[Dr. Katie Milkman]

It was the first time the Celtics had been this close to the championship since 1986.

[Dean Oliver]

Their best players, their big three, definitely played well in those games and they ended up winning by around 10. They went all the way across country to face the Lakers and in game three, the Lakers kind of turned it around. They were able to slow down some of the big three. So at that point, what they say is that the series only becomes real when a road team wins. And so the Celtics were facing game four in la and that was when they were able to kind of take the close win. They were able to win by six points in game four. So they had at that point a 31 lead. So. So finally you had a Road team win. The Lakers were then desperate for game five.

[Dr. Katie Milkman]

Game after game, the series swung back and forth. Momentum shifted. Hope wavered. By the time the Celtics returned home for game six, they led three games to two. They were desperate to close out the series.

[Dean Oliver]

The Lakers. Actually, the Lakers did come out a little bit flat. By halftime, the Celtics had a 23 point lead. And so there was a sense that, okay, we've done a lot of the work that we need to do. We just need to maintain so that lead then kind of grows more. In the second half. They didn't take their foot off the pedal.

[Dr. Katie Milkman]

In the final minutes, the energy in the garden was erupting. Fans envisioning the NBA championship banner hanging from the rafters. The Celtics were winning 131 to 91. With just 30 seconds left courtside, Paul Pierce doused coach Doc Rivers with orange Gatorade. Splashing news cameras.

[Dean Oliver]

They end up winning by 39 points.

[Dr. Katie Milkman]

Green and white confetti filled the stadium. Players cried and gave passionate interviews. Garnett shouted, anything is possible. And bowed down on center court to thank Lucky the leprechaun. The Celtics logo. Finally, after 22 years, the Celtics had won the NBA championships. But what few people in the arena could see was that many of the most important choices leading to this remarkable victory were made long before the opening tip. The 0708 Boston Celtics were lauded with awards. Kevin Garnett was named NBA Defensive Player of the Year. Paul Pierce was named NBA Finals MVP. General Manager Danny Ainge was named NBA Executive of the Year. And all of Boston's 41 regular home games sold out. For Mike Zarin, who led the charge on analytics, the win meant everything.

[Dean Oliver]

I know what it meant for Zarin. Zarin, he got the big ring. He was so proud. Being a fan for all his childhood and then being able to add a ring onto a finger. Zarin doesn't brag about a lot, but he definitely was able to show off how happy he was. That's the thing, right? When you're a fan, you feel like you earned it in a sense, that you supported your team. You were loud and things that like, like that. When you're Mike Zarin, you're proud because you actually made a decision that shaped that franchise, and he, he earned it.

[Dr. Katie Milkman]

For Dean, the Celtics win was a watershed moment in demonstrating how useful analytics could be in building a championship NBA team.

[Dean Oliver]

I don't think it's recognized as much as it should be for, for how it kind of changed the league. There were enough circumstances. Oh, yeah. Those players were already good, but it was a big analytical decision behind the scenes. Certainly if they hadn't changed their path, if they hadn't decided to go after Ray Allen and Kevin Garnett, they would have been in a rebuilding period for quite a while. They might have competed, they might have been playoff teams, but it's extremely unlikely. Without having made another deal at some point to get better players, it would have been practically impossible to win a championship.

[Dr. Katie Milkman]

Dean Oliver is a pioneering sports statistician at ESPN and the author of the books Basketball on Paper and Basketball Beyond Paper. You can find more details and links to his writing in the show notes [e-mail] podcast. The Boston Celtics 2008 NBA win is legendary. Of course, many factors contributed to the team's success in addition to the choice to rely on algorithms. Factors like the individual talent of the players, the chemistry of the Big Three, a relentless focus on defense, and coach Doc Rivers building a culture of trust and teamwork. But without insights from Zarin's player prediction models, it's unlikely the team would have succeeded in the same way today. Algorithms are ubiquitous in the NBA. Every team has an analytics department, and they all strive to project how good players will be in this season and in the future. Algorithms have shaped the game in other ways, too. They've guided teams towards shooting more three pointers, which were previously thought of as high risk and low reward compared to two pointers. Defensive switching has also become more common, where players don't guard the same person all the time. Of course, coaches and scouts still second guess algorithms, especially when a pick that looked perfect goes wrong. We explored people's tendency to lose greater faith in algorithms than in humans for equivalent missteps. In an earlier episode of Choiceology that was back in February of 2019 with my Wharton colleague Cade Massey. The episode looked at algorithm aversion, which describes our tendency after seeing algorithms and humans make the same mistakes, to trust flawed human judgment over superior machine logic. The question of when and how much we'll embrace algorithmic guidance, and when we'll turn away from it, is the focus of my next guest's research. Jennifer Log is an assistant professor of management at Georgetown University's McDonough School of Business, and she joins me to talk about her research on algorithm appreciation. Hi Jen, thank you so much for joining me today.

[Jennifer Log]

Thanks so much for having me. I'm excited to be here.

[Dr. Katie Milkman]

Well, I'm excited to talk about your work on algorithm appreciation. Could you start by just defining what that term is that you and your Collaborators coined to describe a consistent pattern you are noticing in human behavior.

[Jennifer Log]

So we find that people incorporate advice more from an algorithm than from people. What we found was that the people who received advice from an algorithm incorporated more of that advice into their final prediction than the people who thought that advice came from a person. And that's the result that we call algorithm appreciation.

[Dr. Katie Milkman]

I really love this work and I was hoping you could give us some examples of the kinds of algorithms that offer people advice. Just so that we're all on the same page about what we're talking about here when we say an algorithm or a human could offer someone advice.

[Jennifer Log]

I always kind of think of Netflix. Netflix is known for its algorithm generating recommendations. But I could also text my friend if it's a rainy day and ask her, hey, what TV shows have you been binging?

[Dr. Katie Milkman]

And in this era where LLMs are everywhere, would you consider an LLM like a ChatGPT or Claude giving advice to be an algorithm category? That could be compared with, you know, calling my doctor or phoning my neighbor?

[Jennifer Log]

Definitely. Algorithms from 10 years ago versus Genai today. It can do so many more things.

[Dr. Katie Milkman]

I'd love it if you could describe one or two of your favorite studies showing that algorithm appreciation is an important phenomenon.

[Jennifer Log]

So in one of our studies, our participants made predictions about the popularity of songs that came out. For example, what rank will Perfect by Ed Sheeran place on the Billboard magazine Hot 100 this week week. So how popular will it be? Then they gave their numeric answer. On the next page, they received advice that half of our participants read that the advice came from an algorithm, half of our participants read it came from a person. And they made their final prediction so they could decide how much they wanted to listen to or disregard that advice. And then we measured how much people updated from their time 1 prediction to their time 2 prediction. So we found that when people were making these predictions about songs, they listened more to advice when they thought it came from an algorithm than from a person.

[Dr. Katie Milkman]

So the same advice labeled differently as coming from an algorithm versus a person basically produces different decisions. Do you find that experts and novices are similar when they are choosing how to incorporate advice from an algorithm or a human advisor? Because I think you maybe have another study on that, right?

[Jennifer Log]

Yeah. So we also asked people to make business predictions. But in this experiment, we compared how non experts, people who didn't necessarily make forecasts or predictions for a living, compared to a sample of national security professionals, people who on a regular basis for their jobs make predictions. And what we found was that our non expert sample, seeing the same exact questions, responded quite differently than our national security professionals. We found that our non experts were quite happy to listen to algorithmic advice relative to advice from people. However, when the experts, these national security professionals were deciding how much they wanted to listen to the advice that they received in the study, it actually didn't matter if they were the group of people that read that the advice came from the algorithm or advice came from other people. What the national security professionals did was discount advice from everyone. So I expected that maybe the experts knew something about these predictions that I could never expect. And maybe them discounting advice was a good thing for their accuracy. But we found that when we scored all of these predictions on accuracy, our sample of non experts actually made more accurate predictions than our expert sample. And that's because they were willing to listen to the algorithmic advice while the experts discounted all advice.

[Dr. Katie Milkman]

Yeah, the implications of that are a little scary, right? As two experts speaking to one another in a moment when algorithms are taking over sounds like maybe don't listen to us, but listen to the algorithms and you'll make better judgments.

[Jennifer Log]

It's a pretty humbling result.

[Dr. Katie Milkman]

Yeah. Yeah, that's really interesting. Could you tell us a little bit about why it is that people listen to advice from algorithms more than people when the advice is the same? It's not totally intuitive to me. Why getting an estimate that is labeled this came from an algorithm would lead me to say, oh, wow. Okay. Well, the probability forecast I just made of whether or not this geopolitical event will happen, I should really update now, now that I've got an algorithm telling me something more than if I'm told. Here's a piece of advice from a group of wise humans or from one wise human.

[Jennifer Log]

So my answer might not be very satisfying.

[Dr. Katie Milkman]

That's okay.

[Jennifer Log]

One of the things that I found was when we gave our participants 7 simple questions related to just simple mathematical questions, that was our way of capturing their comfort with numbers. So researchers have called it their numeracy. But the more comfortable people were with numerics, the more they relied on algorithmic advice. Now, why my answer might not be totally satisfying is because I tested a lot of other things and they did not actually drive the effect. So one of the things that I tested that didn't explain the effect was age. My prediction was that people who were older might have less experience with algorithms, say Netflix algorithms, compared to younger people. So age I used as a proxy for familiarity with algorithms. And it turns out in Our data age did not correlate with people's responsiveness to algorithmic advice, their willingness to update to algorithmic advice.

[Dr. Katie Milkman]

Is there any sense in which giving people a deeper understanding of how an algorithm works can affect the degree of algorithm appreciation? If we sort of unpack the black box, does that exacerbate algorithm appreciation, or actually, does it reduce it?

[Jennifer Log]

Yeah. So some of the first studies that I ran, I was kind of obsessed with the idea that you just brought up. And I tried to present the algorithm, which ultimately, for a lot of our lives, the algorithms and Gen AI we deal with, they are black box. But I wanted to try to unpack that and understand if people knew different pieces of information about the algorithm, would knowing more encourage them to trust it more than they already seem to do? Or might knowing more actually make them more hesitant? Because now that they understand it more, they might kind of consider how things could go wrong. So I compared if people were shown calculations of what was kind of behind the algorithm, the gears of the clock, if you will, and if it was presented in a simpler way, people didn't respond differently than if the algorithm was presented in a more complex way. And I would like to kind of mention Dan Goldstein's work. So Dan has a paper presenting people with different information about an algorithm, also simple and complex, and he similarly found no difference. People were quite happy to listen more to the algorithm than person, and it didn't matter if people thought that algorithm was complex or not. And I kind of think that's a nice bridge from thinking about algorithms to Gen AI, too. So hopefully the research that we've done on algorithms to some extent also can speak to how people think about Gen AI.

[Dr. Katie Milkman]

What are some situations where you think people would be wisest to privilege algorithmic advice?

[Jennifer Log]

Kind of feels like a tightrope walk. So where and when is this useful? One of my favorite results is actually from the medical domain. And they found this is before Genai. They found that when a biopsy was shown to an algorithm versus a pathologist, the algorithm was better at predicting the severity of breast cancer than pathologists. And the reason why I love this result so much is that they unpacked why it was better. The algorithm that they used was actually identifying new cues that pathologists just didn't even have in their textbook. So there were cues that were otherwise overlooked. So regardless of how you feel about algorithms, even in the abstract, if we can learn from algorithms in Genai, cues that we've otherwise overlooked that are predictive of forecasts that we care about, well, then we can update textbooks. Even if you say, I want a person to have ultimate control, are there

[Dr. Katie Milkman]

situations though where you would say you want people to privilege human judgment over algorithmic judgment that are sort of an edge case to what we've been talking about where algorithmic advice is so powerful?

[Jennifer Log]

So I kind of keep thinking about what Daniel Kahneman wrote before his book on noise came out, that algorithms are a great antidote or solution to human judgment, which can be inconsistent because by definition algorithms are consistent. I think the boundary conditions are, which Kahneman notes, where the data is lower quality. So if the data you're using is based on historic data made by people that might be biased, it might be missing data which could lead to bias.

[Dr. Katie Milkman]

So you could end up with a really low quality algorithm and trust too much in it.

[Jennifer Log]

Right.

[Dr. Katie Milkman]

That's really helpful. What advice do you have for our listeners about how they can make better decisions now that they're familiar with your research on this topic?

[Jennifer Log]

The way I think about how people can improve their lives with generative AI is really to go back to classic research on the wisdom of the crowd. We know that getting multiple predictions and judgments from groups of people is really beneficial for improving our accuracy. And we can think of Gen AI as another source to add to that collective wisdom.

[Dr. Katie Milkman]

Jen, what do you do differently in your own life as a result of having studied algorithm appreciation?

[Jennifer Log]

I think a lot more about data. And if I'm getting advice that seems to be coming from an algorithm, I think more questions come up for me. Is this data complete? You might think you have data that's useful to make predictions for the future, but if you are missing data on so many other demographics, or it could be about people's preferences, you are going to be making the same mistakes over and over again. The algorithm is going to magnify whatever patterns in the input data. So I think think a lot more about where does this data come from. Is this data representative of the things that my organization and I care about? And that is kind of the driving factor behind why I created a class called the Psychology of Big Data, like thinking carefully about measures and what we're feeding into algorithms and Gen AI?

[Dr. Katie Milkman]

That's so helpful. Thank you so much for taking the time to talk to me today, Jen. I really appreciated it.

[Jennifer Log]

Thank you. This was so much fun.

[Dr. Katie Milkman]

Jennifer Log is an assistant professor of management at Georgetown University's McDonough School of Business. You can find her paper Algorithm Appreciation. People prefer algorithmic to human judgment in our show notes or just visit schwab.com choiceology. To hear more about the intersection of behavioral science and sports, check out the Financial Decoder episode Does yous Inner Scorekeeper Skew youw Judgment? You can find [e-mail] financialdecoder or just search for Financial Decoder in your podcast app. In the past few years, sophisticated algorithms that can offer us guidance on nearly every type of decision have gone from a futuristic concept to a normal part of modern life. Genlog and her collaborators have shown that unless we've just watched an algorithm make a bad decision, we tend to exhibit algorithm appreciation, trusting and incorporating the very same advice more when we're told it came from an algorithm than when we're told it came from another person. This unquestionably reflects our appreciation that algorithms can process fear far more data faster and more reliably than we can. And research by John Bogard and Suzanne Hsu has shown that the more common it's become for people to rely on algorithms in a given setting, be it on the basketball court or in the operating room, the more we see algorithm appreciation arise in that setting. Our trust in algorithms is exciting because it suggests a safer, healthier future is around the corner. Self driving cars will soon be able to prevent tens of thousands of annual traffic deaths if we trust them to take over. And algorithm powered scans will catch cancer, heart failure, and other treatable diseases earlier and more often than doctors ever could. But it's also important that we not blindly hand over our futures to algorithms without recognizing that they aren't all created equal. While many deserve our trust, biases like sexism and racism can be baked into algorithms if they're trained on flawed data. And like humans, recent research shows that large language models exhibit many biases. For instance, Stephen Lehrer and colleagues have shown that they exhibit cognitive dissonance, persistently adopting views they've been randomly assigned to articulate after the fact to maintain a semblance of internal consistency. What other biases have LLMs learned from us or evolved independently? Research on this is still in its infancy, but there is some evidence that prompting LLMs to make rational decisions reduces biases. Another risk of placing too much trust in algorithms is deskilling. Recent medical research found that doctors with many years of experience who performed colonoscopies for just three months with algorithmic assistance were less effective when they returned to performing unassisted operations. If we outsource learning and thinking to algorithms, that may be okay in some arenas but catastrophic in others. So we should lean into our algorithm appreciation, but with care and calibration, and with a recognition that deskilling is a risk, that model quality is limited by training, data, and algorithmic biases, and that blind trust in any powerful tool can lead to missteps. There are limits to the quality of any advice in a world without crystal balls. You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or Feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book how to Change, or sign up for my monthly newsletter, Milkman delivers on Substack. Next time I'll speak with Richard Thaler, Nobel laureate and emeritus professor at the University of Chicago's Booth School of Business. We'll explore a bias that helps explain why most of us polish off our plates at restaurants, even when we're too full to enjoy what we're eating, and why we feel compelled to finish boring books. I'm Dr. Katie Milkman. Talk to you soon. For important disclosures, see the show notes or visit schwab.com choiceology.