Choiceology's Guide to Better Decisions: With Guests James Korris, Carey Morewedge & Jack Soll

Summary

Choiceology’s Guide to Better Decisions

Podcast: Choiceology with Katy Milkman
Episode: Guide to Better Decisions: With Guests James Korris, Carey Morewedge & Jack Soll
Date: May 20, 2024
Host: Dr. Katy Milkman
Guests: James Korris, Carey Morewedge, Jack Soll

---

Overview of the Episode’s Main Theme

This special episode explores how we can genuinely improve our decision-making by understanding and counteracting common cognitive biases. Dr. Katy Milkman brings together experts who have developed and tested training approaches—especially immersive video games—that help break bad thinking habits. The episode not only delves into successful real-world applications (in fields like US intelligence analysis) but also distills actionable strategies into a practical checklist for listeners to apply in their daily choices.

---

Key Discussion Points & Insights

1. The Power of Serious Games in Debiasing (00:35–08:49)

• Introduction to "The Pursuit of Terry Hughes" Game

  • [00:35] The episode opens with a scene from an innovative video game used to train US intelligence agents to avoid common thinking errors while solving a fictional missing-persons case.
  • Purpose: To help analysts practice considering alternative hypotheses and not jump to conclusions.

• Game Design Strategies

  • [02:21–03:56] James Korris explains that the game’s narrative and clue-discovery mechanics are specifically designed to evoke biases—like making the player favor certain hypotheses or overlook disconfirming evidence.
  • Example: A clue about a missing suitcase versus a tipped-over chair—does the player only look for evidence supporting an abduction or also consider if the neighbor left voluntarily?

• Connection to Behavioral Research

  • [06:42–07:24] The game's structure challenges users to resist fundamental attribution error (blaming personality over situation) and to spot confirmation bias (seeking only evidence to confirm beliefs).

2. Debiasing Works: The Science and Findings (08:49–19:13)

• Carey Morewedge: Building and Testing Debiasing Games

  • [08:49] Carey Morewedge shares the backstory: He was skeptical at first about whether playing video games could reduce ingrained biases, but saw surprisingly strong results.
  • [11:24] Measurement: Developed new tools to assess individual susceptibility before, immediately after, and months after the intervention.
  • Findings: Games reduced measurable biases by ~30% initially and results persisted (~23% reduction) even after 2–3 months.

• Six Targeted Biases

  • [12:21–13:42] The program tackled:
    1. Bias blind spot (not seeing your own biases).
    2. Confirmation bias.
    3. Fundamental attribution error.
    4. Anchoring.
    5. Representativeness heuristic.
    6. Social projection.

• Why Games Outperform Passive Learning

  • [15:11] Video games provide definitions, directionality, concrete feedback, and coaching—unlike lectures, they force you to see your own errors and correct them in real time.
  • Notable Quote:

    “I think the video game basically throws the kitchen sink of what we think works for debiasing at the problem... The game actually is like, yeah, like, you showed this bias in this particular scenario, and like, here's what you would do to do that.” — Carey Morewedge [15:11–15:47]

• Reassuring Conclusion About Learning

  • [18:44]

    “For listeners who aren't in the science, I think that this suggests that doing things like listening to this podcast or reading a book or watching a talk online, those are all things that can help you improve your decision making, and it's not futile. Your biases are not all permanent. You can do something about them.” — Carey Morewedge [18:44]

3. The “Checklist” for Better Decision-Making (20:50–27:49)

• Jack Soll: Four-Point Practical Process

  • [21:09] Given that lists help lower error rates, Jack Soll offers a clear, concise, research-backed checklist:

  a. Be Decision Ready - Don’t make choices hungry, tired, rushed, or angry. Pause decisions if you’re distracted. -
  > “We often fall back on our intuition and can be biased if we are distracted, if we're tired, if we're hungry, if we're really rushed... So at those times it's best to put off the decision to another time.” — Jack Soll [21:09–22:09]

  b. Broaden the Frame - Challenge yourself to think beyond the obvious options and fully consider what you might care about in the future. > “A lot of biases actually could be attributed to thinking too narrowly, that we're only thinking about the one option that's in front of us... Sometimes what you actually ought to do is throw out all those options and invent some new ones.” — Jack Soll [22:12–23:40]

  c. Seek (Independent) Advice - Ask for advice but avoid telling people your own view first to prevent influencing their response. > “If you really want to get good advice is to get independent advice.” — Jack Soll [23:41]

  d. Experiment—Try to Prove Yourself Wrong - When possible, test your assumptions. Use tactics like A/B testing or have someone serve as a devil’s advocate. If experiments aren’t possible, get repeated or third-party judgments to reduce error. > “Try to generate disconfirmation, try to prove yourself wrong... Have somebody argue for an alternative point of view.” — Jack Soll [23:41–25:10]

• On Inconsistent Judgments

  • [27:08] Even experts are inconsistent—reviewing decisions (like grading a paper) twice or having multiple people weigh in can help control for internal “noise”.

---

Notable Quotes & Memorable Moments

• On Measuring the Impact of Training:

  “We basically see on average a reduction of their propensity to exhibit these biases of about 30% from immediately before the game to after. And then ... two or three months later, we see the game is still reducing their propensity by about 23%.” — Carey Morewedge [11:24–12:07]

• On the Promise of Education:

  “Just by learning about decision biases, you can improve your judgment...it's not futile. Your biases are not all permanent. You can do something about them.” — Dr. Katy Milkman, summarizing [18:44]

• On Feedback in Games vs. Videos:

  “There's a layer of bias blind spot to watching a video too, where you see other people commit these biases and you're like, yeah, maybe I do that, but not as much as that person... The game actually is like, yeah, you showed this bias in this particular scenario, and like, here's what you would do to do that.” — Carey Morewedge [15:47]

• Playfully Acknowledging Human Inconsistency:

  “It's like two independent perspectives. Aren't you lucky? Anyway, that's my framing trick.” — Katie Milkman (on sometimes giving her PhD students contradictory writing advice at different times) [27:32–27:37]

---

Timestamps for Important Segments

• 00:35 — Introduction to the video game debiasing story
• 02:21–03:56 — Game mechanics designed to challenge biases
• 08:49 — Carey Morewedge’s research background and skepticism
• 11:24 — How effectiveness was measured; results
• 12:21–13:42 — The six targeted biases defined
• 15:11 — What makes the video game approach strong
• 18:44 — Takeaway: Biases can be changed through learning
• 21:09 — Start of Jack Soll’s practical checklist
• 22:09–23:40 — Broaden the frame, invent better options
• 23:41 — Independent advice and experimentation
• 27:08 — On inconsistencies in expert judgment

---

Practical Takeaways & Checklist

Summed up succinctly at [21:09–27:49]:

1. Be Decision Ready:
   Wait if emotional, distracted, hungry, or tired.
2. Broaden the Frame:
   Don’t lock in on the first solution; identify unconsidered objectives and options.
3. Seek Independent Advice:
   Ask others before sharing your view—get uncolored insights.
4. Experiment and Disconfirm:
   Run low-cost tests, have a devil’s advocate, and repeat or blind evaluations for tough judgments.

---

Closing Notes & Overall Tone

The tone throughout the episode is encouraging, upbeat, and practical—asserting that while cognitive biases are real and sticky, we’re not powerless against them. Both research and real-world interventions show people can learn, improve, and transfer good decision-making habits across domains.

“We’re not just stuck with these biases. And just like we can learn calculus and algebra and addition, we can learn how to make better choices.” — Dr. Katy Milkman [18:44]

The message is clear: If you’re willing to learn, reflect, and deliberately question your own thinking, smarter decisions really are within reach.

Transcript

[Jack Saul]

At the tone, please record your message.

[Chris Hughes]

Hi, this is Chris Hughes. Look, I. I can't reach my sister, and since you're next door, could. Could you please check on Terry? I'm really worried. I called the super. He's got her apartment open. Just please look around. I've called Terry, but she's not picking up. I don't know where she is. Anything you could find would be a big help. Look, I owe you, okay? I.

[Dr. Katie Milkman]

That's an opening scene from a video game called the Pursuit of Terry Hughes. The game was designed to help train U.S. intelligence agents to avoid decision pitfalls by working through a fictitious missing persons case. Today on the show, we're bringing you something a little different. We're calling it Choiceology's Guide to Better Decisions. We've gathered some advice from expert guests who study the science of debiasing decisions, and we hope their insights will help you make ever more thoughtful choices. You'll also hear from two guests whose work on the video game Missing has had a lasting, measurable impact on its players, improving the decision making skills of people who work in one of the highest stakes sectors in the United States. Intelligence analysis. I'm Dr. Katie Milkman and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high stakes choices. And then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.

[James Corras]

This is James Corras. I'm the president and CEO of Creative Technologies Incorporated in Los Angeles, a woman owned small business.

[Dr. Katie Milkman]

Back in 2015, James was contracted to design a game for the government agency iarpa. That stands for the Intelligence Advanced Research.

[James Corras]

Project's Activity Missing Part one, the Pursuit of Terry Hughes. It's a character centric narrative that follows the first person player through a series of discoveries. And basically you are in the role of a person living in New York in a cool part of Manhattan, but you have a very boring job. Your neighbor is a very exciting, very interesting person whom you know only very slightly because even though she's a next door neighbor in New York, she's almost a total stranger. And the inciting incident that gets the narrative rolling is a phone call from your neighbor's brother who's very concerned because she's been missing for three days and he can't figure out what's happened to her. So basically, it's a mystery.

[Dr. Katie Milkman]

The game is set up to challenge the player's decision making skills.

[James Corras]

Okay, so the first thing that pops up here is the user's smartphone. We see that Chris has texted. He says, yeah, you're in the apartment. Please look around, take as many pictures as you like, but pick the best three that you think are good clues to help us figure out what happened to Terry. I hope nobody grabbed her.

[Dr. Katie Milkman]

The clues are cleverly designed to elicit a biased response. In this case, they trip up the player by leveraging people's tendency to jump to conclusions with limited data.

[James Corras]

By Chris saying, I hope nobody grabbed her, he's setting up the hypothesis that there were malign actors who broke into her apartment and took her away. So the question is, will you be looking for evidence to support that or would you consider a counterfactual hypothesis? And this is where we start to navigate around the apartment. So starting in the living room, there's a. See, there's a desk here. So we see there's some papers around. See this bookcase? We move to this document here and pick it up. And you can see that there are restaurants in Los Angeles. And if you think about it for a moment, if someone is generating a list of restaurants in another city, it does not support the hypothesis that criminals broke into the apartment and kidnapped a person. So if you were to take a picture of this, then I would suggest that you do have some interest in exploring counterfactual hypotheses. So we'll take a picture of that. Okay, so here's a chair that was turned on its side. Then we say, oh, well, that might support the hypothesis that she was taken by force. Bathroom's kind of interesting. So here's what's in the bathroom. Wet towel on the ground. You know, what does that tell us? Let me take a picture of that. And then over here, here's her closet. And one of the things we notice is that there is a suitcase missing. So again, if someone is taken by force, they don't usually pack a bag. So we got the chair that shows maybe by force, the towel, which suggests that she wasn't taken by force. And the missing suitcase sounds like she packed for a trip. And so we're going to send them to Chris.

[Dr. Katie Milkman]

As the game progresses, other characters start to share clues, like the apartment building superintendent who comes over to Terry's apartment and shows you some security footage.

[Apartment Superintendent]

Hi, you're from 12G. I got a call from Chris, Terry's brother. He told me you were in here looking. The guy is massively freaking. Did you find anything? Come with me.

[James Corras]

Then he takes us into her office. And there's more discovery here.

[Apartment Superintendent]

I've got the security footage from the elevator. Here we go. This was just a couple hours ago. Terry seemed like a nervous person to you.

[Dr. Katie Milkman]

This clue might stand out if you remember the Choiceology episode about the fundamental attribution error. In that episode, we talked about our tendency to attribute other people's behavior to their underlying attributes or personality and to disregard how dramatically their situation has likely influenced their behavior.

[James Corras]

We might assume that she's anxious because there's something seriously wrong going on in her life, or it just might be the fact that she's, you know, in a creaky old elevator in Manhattan. It's the kind of thing that would make anybody anxious.

[Apartment Superintendent]

Personally, I think Terry's in some kind of trouble. She always seems short of cash these days. Late on the rent last month.

[James Corras]

There's lots of stuff to explore. On her desk here is an electric bill that says it's past due. So this would support the hypothesis that she's having financial problems.

[Dr. Katie Milkman]

After clues are gathered and the first part of the game is over, users complete an after action review and hear lectures on decision biases that might have tripped them up during the game.

[Carrie]

Confirmation bias is a mistake that people make when they have one idea of what's right and then they search for evidence and interpret evidence in a way that confirms it. Because people are trying to confirm one idea, they don't evaluate all evidence objectively.

[James Corras]

So that's Carrie, who's speaking now.

[Carrie]

In the game, for example, you search for clues about why Terry wasn't in her apartment. If you only search for clues that Terry had a financial problem and you didn't check whether she was gone for other reasons, you committed confirmation bias.

[Dr. Katie Milkman]

That's Kerry Morwich. Kerry is a professor of marketing at Boston University who studies psychological biases and how to reduce them. Back in 2015, James and Kerry worked together on the project we've just described to marry behavioral science insights with game design and to make the game as effective as possible at improving people's decisions. Kerry joined me to talk about the game and about his research on strategies for debiasing decisions.

[Interviewer (Katie Milkman)]

Hi, Kerry.

[Dr. Katie Milkman]

Thank you so much for taking the.

[Interviewer (Katie Milkman)]

Time to be here today.

[Kerry Morwich]

Hi, Katie. It's such a pleasure to be here.

[Interviewer (Katie Milkman)]

I want to start by asking you if you could just describe your 2015 paper about two different debiasing training programs that you designed, which I think of as really the best work on debiasing. They had remarkable and durable effects on reducing decision biases. So I'd love it if you could just explain the trainings you designed and how you proved their value.

[Kerry Morwich]

The backstory is I got a call asking me if I'd be interested in building video games to reduce cognitive biases. And that sounded absolutely crazy to me. Iarpa, which is an intelligence research program of the United States government, was doing these high risk, moonshot kinds of programs where a bunch of teams compete to see if they can achieve some kind of goal that has flummoxed the scientific community. And the program manager at IARPA who was running this program was a gamer. She got the idea that maybe serious games, which are video games with a learning component, could be used to reduce a lot of the cognitive biases that she was reading about. And then we had an initial meeting with them, and I said, you know, the data that you can reduce cognitive biases is pretty thin. Right. When we think about biases in decision making, I think as a field, we've often described that as there's not much that we can do. I accepted that kind of conclusion, but it was also very discouraging because it feels like, why are we spending all this time discovering these kinds of biases, but feeling like we can't do that much about them? So this was exciting to me. I didn't think it would work, but I was game to give it a shot. The basic structure of what we did was they had identified six cognitive biases that they thought were really important for intelligence analysis. So, for example, one was confirmation bias, the idea that when we're testing a hypothesis, we tend to look for information that confirms that hypothesis rather than disconfirms it. So there were six of these different kinds of biases, and they basically broke them into two video games where each game was designed to treat three of the cognitive biases that they'd identified. And then we competed against six other teams to see if we could reliably reduce these kinds of biases both immediately and then after a delay. And that delay was two or three months, based on which game we were testing. Each game takes about 90 minutes to play. But the game's really surprisingly effective at reducing cognitive bias.

[Interviewer (Katie Milkman)]

So how do you measure the effectiveness of the games?

[Kerry Morwich]

Yeah, that was really interesting too. I looked around and there weren't amazing scales to measure these biases at an individual level. And so we developed these scales of each of the biases, and so we could go out and measure people's susceptibility to the six different kinds of biases before they played the game immediately after they played the game and then two or three months afterwards. And we basically see on average a reduction of their propensity to exhibit these biases of about 30% from immediately before the game to after. And then we see, two or three months later, we see the game is still reducing their propensity by about 23%.

[Interviewer (Katie Milkman)]

That's really interesting. And you mentioned that there are six biases. You gave one example. Could you talk a little bit about the full set of six and whether there were any differences in how well teaching people in this immersive way worked across the different biases?

[Kerry Morwich]

So the six biases were bias blind spot. That's seeing more bias in other people than in yourself. Confirmation bias, fundamental attribution error, which is that we tend to attribute our own actions to the situation and other people's actions to them. Or like dispositional effects like that was you versus that was your context. Anchoring representatives, heuristics. So those are kinds of cases where we often use the similarity of situations as a proxy for the probability of situations. And then the last one is social projection. You know, when people try to think about how much other people have a particular attitude or a particular preference, they often use their own attitudes and preferences to predict how other people feel. Right. So, for example, if you find crocs to be very comfortable footwear and like crocs, you might think that most people like crocs. Right. If you think crocs are horrifically unfashionable, then you would think that no one wears Crocs because they're terrible. Right. That's a sort of schlocky example. But you can see that politics often divides how people feel about different kinds of issues, and people tend to overestimate how many people agree with them on their particular kind of political rift issue. So those are the six biases and.

[Interviewer (Katie Milkman)]

P S. I have to say, those are great biases to try to tackle.

[Dr. Katie Milkman]

And I think we've covered four of them, maybe on the show. But you just gave me two more topics we'll have to cover in future episodes.

[Interviewer (Katie Milkman)]

So thank you for that too, and.

[Dr. Katie Milkman]

I'd love to hear how it all worked.

[Kerry Morwich]

So the biggest surprise to me is how effective these debiasing strategies are. The second thing is, in other papers, we used random YouTube clips of people's lectures as a debiasing intervention. And we've even done things where we just explain the biases to professionals like intelligence analysts or risk analysts, and they work. And I think actually the biggest problem.

[Interviewer (Katie Milkman)]

Was the measurement Ah, so you're saying our pessimism prior to this research about how easily we can debias people was because we didn't measure their success dodging biases very well and you came up with better measurement tools?

[Kerry Morwich]

Yeah.

[Interviewer (Katie Milkman)]

I will just say, as someone who teaches about bias and who obviously spends a lot of time through this podcast trying to teach the world, it's such an encouraging message that even though we'd seen a lot of failures before, debiasing through education really does work, and it was maybe a measurement issue. But I also want to point out that in your paper, you show that these really engaging video game trainings work better than watching lectures by some of the best researchers in our field. And I would love it if you could share a little bit of your insight about why the video game was particularly useful way to improve people's decision making.

[Kerry Morwich]

I think the video game basically throws the kitchen sink of what we think works for debiasing at the problem. To debias someone, they need to know what the bias is. They need to know what direction it affects judgment in. Then giving people feedback helps, and then they need coaching and strategies. And the video game does all of that in as much intensity as possible in 90 minutes.

[Interviewer (Katie Milkman)]

I love that. And of course, watching a passive video doesn't give you that feedback component, and it can't be personalized or engaging in the same way as an active video game.

[Kerry Morwich]

Yeah, and I think there's a layer of bias blind spot to watching a video too, where you see other people commit these biases and you're like, yeah, maybe I do that, but not as much as that person. Or, like, that's not as much a problem for me. And the game actually is like, yeah, like, you showed this bias in this particular scenario, and like, here's what you would do to do that. So there's kind of a effect of getting feedback on your receptivity to the training. And in our bias blind spot paper that describes the scales that we used for bias blind spot, we actually find that people who are higher in bias blind spot are less receptive to de biasing trainings than people who are lower. And so I think shattering that kind of bias blind spot is helpful in itself. The other aspect that our findings suggest is that we get this transfer effect. So learning about cognitive biases in one domain does seem to, like, help you in other kinds of related problems. The take home, I would say, is that I think we've been really underselling the value of a lot of the public outreach of decision scientists that people are really benefiting from a lot of this kind of work. And, like, part of it is that it's not as exciting for many people to measure individual differences in these biases that have been established as to, like, understand them or document them. Right. As we mature as a group and as a field, I think it's very useful for us to try to think about, okay, we've done, like, a lot of good trying to show all of these biases. How do we go from the descriptive to the prescriptive, and, like, try to improve social welfare that way?

[Interviewer (Katie Milkman)]

I love that. And obviously, birds of a feather, I think you and I both, we'd love for the incentives to change a little bit. So there's evaluation not only of the basic science, but also some of the applied work that can have a positive social impact. So I love this research you've done. Is there anything that you do differently in your life as a result of the research you've done and that you've read on de biasing?

[Kerry Morwich]

Yeah, I think I'm just more optimistic about our field. And also I think that it's important for people who do this kind of work to do more public outreach because it seems to work. I really admire people like you who are, like, going out and talking about the science to everyone and trying to get the idea out. And I think that we could do more as a science to try to think about ways to improve on many of the kinds of biases that we've identified. And there's a lot of important work to be done on what works best and for whom and how related are these biases, and is there something that would work for many of them? Right. And just understanding more about the structure of different biases and how they fit together, I think would be really helpful to move forward. And for listeners who aren't in the science, I think that this suggests that doing things like listening to this podcast or reading a book or watching a talk online, those are all things that can help you improve your decision making, and it's not futile. Your biases are not all permanent. You can do something about them.

[Interviewer (Katie Milkman)]

Kerry, thank you for the very kind words and also, what an incredibly optimistic note to end on. It's so encouraging that the research shows learning works and that we're not just stuck with these biases. And just like, we can learn calculus and algebra and addition, we can learn how to make better choices. So thank you for the incredible work you have done to teach us that it's not feudal. And thank you for taking the time today to talk to me and to our listeners about some of your findings and how we can better Debias ourselves.

[Kerry Morwich]

Thanks, Katie. It was such a pleasure to be here.

[Dr. Katie Milkman]

Kerry Morwich is a professor of Marketing and the Everett W. Lord Distinguished Faculty Scholar in the Questrom School of Business at Boston University. James Corus is President and CEO of Creative Technologies Incorporated and is a pioneer in immersive game based simulation for military learning. You can find links to Carey's research and to a video of the game he collaborated on with James in the show notes [e-mail] podcast. The video game developed by James Corus and Kerry Morwidge and their colleagues shows that learning about behavioral biases and developing strategies to counteract them may be more effective than we thought. The robust results point to the value of education around debiasing. With that in mind, we thought it would be useful to share some practical strategies to help you make better decisions. A kind of checklist for a broad range of choices that you might encounter at work or at home or anywhere really. You may have heard some of this advice before on choiceology, but I've asked my next guest to distill it into a checklist that you can take with you as you make your next big or small decision. Jack Saul is an expert in debiasing strategies. He's a professor of Management and organizations at the Fuqua School of Business at Duke University.

[Interviewer (Katie Milkman)]

Hi Jack, thanks so much for taking the time to talk to me today.

[Jack Saul]

Hi Katie, it's a pleasure to be here.

[Interviewer (Katie Milkman)]

Well, it's a pleasure to have you.

[Dr. Katie Milkman]

And I'm hoping we could dive right into some strategies for better decision making.

[Interviewer (Katie Milkman)]

We've done an episode on the effectiveness of checklists, so I'd actually love to.

[Dr. Katie Milkman]

Get your list of the best advice in that format.

[Jack Saul]

All right, checklist. Okay, so I have four pieces of advice here, probably incomplete, because as we know, anytime one person sits down and writes down a list, there are probably some things they're not thinking about. But here's four. So one is be decision ready. And so what do I mean by that? So we often fall back on our intuition and can be biased if we are distracted, if we're tired, if we're hungry, if we're really rushed, we don't really have time to deliberate. And we often make better decisions if we actually one know how to make the right decision. So kind of we know what principles to apply, but also we kind of have some time to think about and apply those principles. So there's a lot of situations where we're not decision ready. Maybe we're angry at somebody or maybe we're trying to do multiple things at once that we're distracted. And so at those times it's best to put off the decision to another time.

[Interviewer (Katie Milkman)]

I love that, Jack. That's a great item, number one. What else?

[Jack Saul]

Yeah, I think it's really important, you know, we miss opportunities because we didn't think broadly enough. And so a second piece of advice is to broaden the frame. A lot of biases actually could be attributed to thinking too narrowly, that we're only thinking about the one option that's in front of us. Should I go to this college? Should I rent this apartment? And when we do that, we only often have one objective in mind. We're not thinking about all the things we care about. So imagine a student choosing a college and wants to be a software engineer. They might focus on which school has the best computer science department and that might turn out to be the best choice for them. But it's also the case that if they stop and think about it, they might realize, well, in the future I might prefer a different career path or there's other things I care about about a college, you know, like the social dimensions and things like that. And so I think the advice is to try to think more broadly about what can happen, how might our preferences change what can happen in the world. And at the end of the day, decisions are limited by the option set. Right? A lot of research has shown that people are not necessarily so great at choosing well, or they might be okay at choosing between A, B and C. But sometimes none of those options are the right option. Right? Sometimes what you actually ought to do is throw out all those options and invent some new ones. And so that's what I mean by broadening the frame. Number three on the checklist. Take advantage of the wisdom of others. So seek advice. And when doing this, make sure to get independent advice. And so don't tell the other person what you think the answer is, because this is going to influence what they tell you. They'll either think that you're asking for confirmation and they'll tell you what they think you want to hear, or their thinking will be influenced by your thinking. Either way you're going to be better off. If you really want to get good advice is to get independent advice. And then fourth experiment, try things that you think won't work. Do this in a low cost way. The idea here is to try to generate, disconfirmation try to prove yourself wrong. One way to do this is empirical. So if you think you know what a website should look like, you could do some A B testing and you could create variations that are different from what you think are best to kind of test whether or not your idea is correct. But in other situations, experimentation is difficult, and there we have to rely on the wisdom of others. And one way to do this is to have a devil's advocate, have somebody argue for an alternative point of view.

[Interviewer (Katie Milkman)]

Yeah, I love that. I often also tell my students to try to find somebody who will play the role of devil's advocate and just argue the opposite to when I'm worried that I might be too attached to a given path or argument and not thinking broadly enough.

[Jack Saul]

Oh, for sure. You want to think about reasons why you might be wrong. We often think about why we're going to succeed or why our answer is right. And you want to think about reasons why the answer might be much different from what you thought. And, you know, sometimes we're all on our own and, you know, this is a little bit challenging, but ask yourself why you might be wrong. There is one other way to do this kind of this, I guess fourth point of trying to seek this confirmation is to tackle the same problem, but on different occasions. So one trick actually that a lot of professors actually use when they're grading papers, number one, is you don't want to have the name of the student on the paper. And it's very tempting to know whose paper you're grading. But if the name's not there, then you could come up with an assessment of the work which is not tainted by your views about the person. So this is called blinding. But in addition to that, the advice here would be to do it twice. And it takes extra time. But oftentimes what happens is at any given moment there's some noise in our judgment, or we. We have different things in mind. And so if we look at the same decision problem twice or are grading the same essay twice, we might come up with different answers. And in those cases, if they're different from each other, you could either if it's quantitative, you can average the grades together. If it's more discrete or if it's categorical, what you can do is look at it a third time or have somebody else look at it. And so there's a lot of work along these lines in medicine where if you ask a radiologist to look at the same X ray on multiple occasions, they might come up with different answers. And this isn't really a critique of their expertise. This is just symptomatic of how difficult the task is. But for those kind of tricky ones where they're giving different answers on different occasions, then maybe a third look or bringing in another expert can be helpful.

[Interviewer (Katie Milkman)]

I love that, Jack. And P.S. it also makes me feel better about the fact that I'm that dissertation advisor who, after six months reads a new draft from my student of their, say, dissertation and says, I really don't like the third paragraph of the intro. You should cut it. And they tell me, actually, I added it because you told me to. Anyway, I'm often inconsistent in my advice to my students, but I'm like, well, you're getting the wisdom of the crowd within.

[Jack Saul]

Exactly.

[Interviewer (Katie Milkman)]

It's like two independent perspectives. Aren't you lucky? Anyway, that's my framing trick.

[Jack Saul]

There's more than one Katie Milkman.

[Interviewer (Katie Milkman)]

That's right. That's right. You caught me hungry today, Jack. This has been so enlightening and helpful, and I really appreciate you taking the time to talk with me and share these insights.

[Dr. Katie Milkman]

Thank you so much.

[Jack Saul]

Sure. You're welcome. It was a pleasure.

[Dr. Katie Milkman]

Jack Saul is the Gregory Mario and Jeremy Mario Distinguished professor of Management and Organizations at the Fuqua School of Business at Duke University. You can find links to Jack's research, as well as an article we wrote together with John Payne titled Outsmart your own biases in the show notes [e-mail] podcast cognitive and emotional biases can have a big impact on your financial life. To learn more about debiasing techniques and how to approach a variety of specific financial decisions, like how to save for college or what to do with an old 401k, check out the Financial Decoder podcast. You can find it at schwab.com financialdecoder com. Or wherever you get your podcasts. This episode has been a bit atypical. We've shared the great news that just by learning about decision biases, you can improve your judgment. I think we need to run an experiment to test whether listening to this show successfully debiases people. But there are a lot of different mistakes we make that we need to be aware of. We often seek confirmation of our existing beliefs rather than than searching in a balanced way for information. We misattribute people's behavior to fixed features of their personality instead of their situation and think too narrowly about important problems, and that's just scratching the surface. The good news is that cures exist for these types of biases, as Jack shared. You can set yourself up for success by being decision ready. Part of that, of course, is being familiar with decision biases. Kerry explained just how valuable that can be. But also you should make sure you're well rested, calm, and well informed before making a consequential decision. It can also be valuable to broaden your thinking by deliberately considering a wider scope of possibilities and future preferences than you would naturally. And you can be intentional about collecting information and seeking independent advice from other people with relevant expertise. Finally, try to prove yourself wrong by testing. That might mean a B testing, or just having a friend play devil's advocate. I hope that the more you listen to Choiceology, the more reliably you'll be able to dodge decision biases. You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book how to Change, or sign up for my monthly newsletter, Milkman delivers on Substack. Next time, we'll bring you stories about two very different battles from the American Revolution, and we'll share how people make decisions in the face of real versus hypothetical losses. I'm Dr. Katie Milkman. Talk to you soon. For important disclosures, see the show notes or visit schwab.com podcast.