Annie Duke — Why We Make the Wrong Decisions (Ep. 296)

Summary

Infinite Loops: Annie Duke — Why We Make the Wrong Decisions (Ep. 296)

Release Date: January 8, 2026
Host: Jim O'Shaughnessy
Guest: Annie Duke

---

Overview

In this enlightening episode, Jim O’Shaughnessy sits down with poker champion and author Annie Duke to discuss why we often make erroneous decisions, how we misinterpret data, and the importance of probabilistic thinking in today’s uncertain world. The conversation centers on Duke’s upcoming book, which explores the distinction between misleading data and outright misinformation, the cognitive biases that shape our interpretation, and ways to improve the quality of both everyday and high-stakes decisions.

---

Key Discussion Points & Insights

1. The Two Pillars of Life Outcomes: Luck and Decision Quality

• Annie opens by distilling life’s outcomes down to two factors: luck and the quality of our decisions.
• Quote: "One is luck, which, sorry, you can say I make my own luck all day. That literally is not a sentence of English that makes sense. The other thing is decision quality." (A, 00:00)

2. All Decisions Are Forecasts and Bets

• Every decision is fundamentally a forecast about possible outcomes, payoffs, and probabilities.
• Quote: "Everything is a bet, right?...That’s how we calculate the expected value toward your goals." (A, 00:17)
• Mistakes often arise from failing to interrogate the data inputs and being biased in interpreting our own or external information.

3. Misleading Data vs. Misinformation

• Annie distinguishes between outright falsehoods (misinformation) and misleading interpretations of true data.
• Refers to Duncan Watts’ research: misleading (but true) information is a much bigger problem than misinformation, at a 41:1 ratio.
• Example: COVID headlines and interpreting statistics without context.
• Quote: "The misleading thing, the misinterpretation, is actually the bigger problem." (A, 05:56)

4. Interrogating the Data: The Importance of Context

• Illustrates the Washington Post’s 2022 headline “Covid is No Longer a Pandemic of the Unvaccinated.”
• The article cited 58% of COVID deaths as vaccinated individuals—but failed to mention 80% of the population was vaccinated.
• Emphasizes always asking “Out of how many?” and seeking underlying context for any data presented.
• Quote: "I certainly think that this explanation, which is the headline, is not only unwarranted, but, like, dangerous." (A, 11:49)

5. From Description to Explanation: The "Explanatory Satisfaction" Trap

• People often jump from a description (data) to a satisfying explanation that fits their biases, skipping the necessary interrogation.
• Quote: "We don't like randomness. We want to know why things are occurring. So when we land on an explanation where it solves that discomfort, we will often just stop." (A, 15:24)
• Example: Attribution errors in business data, such as incorrectly crediting marketing strategies for improved outcomes.

6. Biases, Heuristics, and Probabilistic Thinking

• Probabilistic thinking is hard-wired; people prefer certainty and deterministic explanations.
• The world has always been probabilistic, but rapid change now more harshly punishes deterministic thinking.
• Quote: "We are deterministic thinkers living in a probabilistic world, and hilarity or tragedy often ensue." (B, 27:06)

7. Education for a Probabilistic World

• Annie proposes revamping education to focus less on rote facts and more on decision-based, probabilistic, and statistical reasoning.
• Initiatives like forecasting competitions for students and embedding decision-making into all curricula (e.g., exploring what-if scenarios in literature or science).
• Quote: "I think it would probably be better to be teaching people statistics and probability as kind of a core decision skill." (A, 33:49)

8. Agency and Socioeconomic Disparity

• Discusses how sense of agency can differ based on socioeconomic background—those with higher status often question and interact more assertively (e.g., with doctors).
• Agency is a teachable skill, not just innate.

9. Base Rates, Sampling, and Data Interpretation

• Use base rates as your starting point unless you have strong causal reasons to deviate.
• Beware of additional descriptive (often irrelevant) information that can distract from base rates (Linda problem, engineer/lawyer problem).
• Quote: "Start with the base rate, assume equilibrium...You better have a really good reason for why you think things are so different now." (A, 78:40)

10. Survivor Bias and Self-Selection

• Warns against simplistic success studies (e.g., "The Millionaire Next Door"): need to consider both successful and unsuccessful cases.
• Quote: "It's a natural proclivity...to look at the survivors and what people again, compared to what? What about people who do those exact same things and die?" (A, 88:31)

11. AI, LLMs, and Information Hygiene

• Treat LLMs and social media like potentially unreliable sources—be skeptical and learn how to prompt for better, more contextual answers.
• Quote: "You have to take control over this...you have to view social media...LLMs the same way." (A, 84:10)
• Annie mentions the Alliance for Decision Education working on decision copilots for navigating information.

---

Notable Quotes & Memorable Moments (with Timestamps)

• "You got two things that determine how your life turns out. One is luck...The other thing is decision quality." (A, 00:00)
• On data abuse: "The conclusion or the explanation that we're jumping to is inaccurate, and it's because we don't know how to interrogate the data." (A, 05:49)
• "If I could change the world, that earnings estimate would be lower bound, upper bound, with some context." (A, 65:44)
• On survivor bias: "We want to believe that we can come up with an answer. And the natural way to do that is to say, well, let's look at the successful ones...What about people who do those exact same things and die?" (A, 88:31)
• "If I could actually get people to think probabilistically, like, please, that would be so great. Imagine how much better it would be to watch the news." (A, 102:20)
• "The first explanation that comes to your mind...is not necessarily the right one." (A, 102:34)

---

Important Timestamps & Segment Highlights

• [00:00–06:00] Introduction to decision quality, forecast-based thinking, and misleading data.
• [10:24–15:50] Case study: COVID vaccination statistics and media misinterpretation.
• [22:55–26:13] Real-world outcomes versus anecdotal data; hormone replacement therapy reporting.
• [38:45–45:35] Reimagining education for probabilistic thinking and decision skills.
• [62:16–66:34] Teaching confidence intervals and communicating uncertainty.
• [86:36–95:36] Problems of self-selected samples in popular business books and self-improvement literature.
• [102:20–104:24] Annie’s “magic wand” wishes: instilling probabilistic thinking, skepticism about immediate explanations, and practical advice for happiness.

---

Actionable Takeaways & Closing Thoughts

• Always question data and check context: Key questions such as “Out of how many?” can prevent major misinterpretations.
• Start with base rates: Deviate only with strong, causal evidence.
• Embrace probabilistic thinking: Accept uncertainty, and avoid overconfidence in explanations.
• Update educational paradigms: Prioritize decision-making, forecasting, and statistical literacy from a young age.
• Practice agency: Ask questions, push for information, and be skeptical of both others and your own explanations.
• Utilize and prompt AI thoughtfully: Recognize its limitations and remember that outputs depend on your inputs.

---

Final “Inceptions” (Annie’s Two Key Behaviors for the World)

1. Think Probabilistically: Understand that the world is driven by probabilities, not certainties.
2. Be Skeptical of First Explanations: The most immediate, satisfying answer is often not the correct one.
   • Bonus from Lori Santos: Get out in nature and leave your phone behind for happiness.

---

This episode offers a practical, sometimes humorous toolkit for interrogating data, resisting narrative traps, and making sharper decisions in business, health, and life. Annie Duke’s new book will expand on these themes with checklists and actionable stories—stay tuned for its release in late 2026 or early 2027.

Transcript

[A]

You got two things that determine how your life turns out. One is luck, which, sorry, you can say I make my own luck all day. That literally is not a sentence of English that makes sense. The other thing is decision quality. At the core of every single decision is a forecast.

[B]

Everything is a bet, right?

[A]

Here's the set of possible outcomes. There's a payoff associated with each of those outcomes. That's how we calculate the expected value toward your goals. But the explanation that we're jumping to is inaccurate and it's because we don't know how to interrogate the data.

[B]

This one is particularly special. I just had an incredible conversation with my friend Annie Duke, the former world class professional poker player. Most of our conversation was about her new book, but she's written several books on decision making and I recommend all of them. Please enjoy my incredible conversation with Annie Duke. Annie Duke, welcome back to Infinite Loops. It's so good to see you in person.

[A]

In person this time, which is so amazing.

[B]

Last time I was on Zoom, I was reminiscing about. Do you remember when we were at Friend of a Farmer and we were talking about your last book?

[A]

Yeah.

[B]

And that turned out to be so great. But before we started recording, you started telling me about your new book, which absolutely fascinates me. We're not going to ask you the whole thing, but first off, tell our listeners and viewers a little bit about what it's about because I think, boy, is the time right for this book.

[A]

Oh, well, thank you. Yeah. So a lot of what I think about, obviously I'm trying to help people to make more effective decisions. And part of the way that I think about that is that at the core of every single decision is a forecast. So, I mean, if you think about any decision you make, right, like you're considering different options. Should I take this way to work or this way to work?

[B]

Everything is a bet, right?

[A]

What you have to do for any of those options that you're considering is make a forecast of how you know how much that option that you select is going to on average help you gain ground toward your goals in alignment with whatever your values are, right? So obviously when we get into the value thing, we get into utility. But so those forecasts, though, as you're thinking about that, right, Like I'm thinking about this option, here's the set of possible outcomes. There's a payoff associated with each of those outcomes. There's a probabilities of those occurring. That's how we calculate the expected value or ground gained toward your Goals, those forecasts can only be as good as the inputs. How do we think about those inputs? Well, we have experiences that we might learn from. A lot of what my past writing has been about is how the way that we interpret those experiences will be biased in a way to cause us to come to inaccurate conclusions or lessons learned about the experiences that we have. So for example, if you think about the centerpiece of thinking in bats, it's really this resulting problem that makes it so that when we have an experience, things go poorly, that we draw the wrong conclusions about what the decision quality was and then that causes us to make worse decisions going forward. If you think about that, that's really a forecasting problem, that it's changing the way you think about the probability of a bad outcome occurring under the same decision circumstances. So that's kind of like how are we learning from our own experience? But there's this other source of input, there's this other input into the decisions that we make, which is the information that we come across. And we're going to come across that information or data, either because it exists out in the wild, we find that data or that information somewhere else. Like a user on social media might offer it to us, or we might self generate. So we could be doing analyses of our portfolio performance, for example, or we might be looking over all the founders we've ever invested in and trying to signal detect what founders are good or what founders are bad, or we're tracking how our sales are doing given maybe different marketing strategies that we're trying. And this is all data generation and we might be creating data visualizations and things like that, and then we're trying to come to conclusions about what is causing us to observe those things that will then help us to make decisions about how to get better outcomes for ourselves in the future. So we might be looking at, can we come to the conclusion, is there an explanation for what we're seeing in the data that says that the marketing strategies that we're trying are actually improving sales? And if we think that that's true, then we may invest more money in those marketing strategies? As an example, we might see data about the efficacy of vaccines and that might drive decisions for ourselves about whether we ought to get a vaccine or not, depending on what we're looking for in terms of our own health outcomes. As an example, that's this other source of input, separate and apart from our own experience that is driving the types of decisions that we make. Anyway, I was thinking about that particular problem and I said it's interesting. Everybody really seems to be very worried about misinformation in this world. But I come across some work from Duncan Watts, who's at Penn, and he had actually been looking at let's divide it into two worlds, kind of like misinformation, let's call it someone just making something up in line to you.

[B]

Right. Propaganda motivated reasoning bots, that kind of.

[A]

Stuff, versus information that is either offered in a way that's been misinterpreted by the person who's offering it. So this is. They're not lying. Right. So the core fact that they're offering you is true. So that's the difference. So in misinformation, the fact that's being offered is not true. Someone says 17,000 people died in New York City last year, you can go look that up. It's not true. Right. Versus I offer a fact that's true. Let's say it were true that 340 people were murdered in New York City last year. Let's just say that's true. And then I draw a conclusion from that number that is misinterpreted. So it ends up being misleading. Right. And it could be that the person offering it is actually offering that interpretation that's misleading or misinterpreted. Or you could just misinterpret it yourself. Okay. So let's sort of divide it in those two worlds. And what Duck and Watts found is that it's 41 to 1. The misleading thing, the misinterpretation, is actually the bigger problem. So. So when I saw that this actually had aligned with something that had made me really mad, which was an article that I read in the Washington Post in 2022, where I really felt this problem of. Look, if you think about it, I don't think that the majority of people are trying to lie to you. I really don't. Yes. Do we all have our biases? Sure. But I think what the biases are doing is kind of under the hood, driving the types of explanations that we're jumping to. When we see data, so you have some sort of piece of data, and that data is merely. And I think it's hard for people to wrap their heads around it. But it's just a description. It's just a description of something that's been observed. So we can take a simple bar chart. It's a description. There were this many at this time, this many at this time, this many at this time, it's just a description. But when we see it's small, this year, and then it's bigger this year and it's bigger the next year. We think it's actually giving us why we sort of jump to that why. And the why that we're going to jump to is certainly under the hood, going to be driven by bias. But I don't think that people are purposely lying to you. I don't think they're purposely trying to mislead you. And I think that when you're looking at data that you might have generated, say in your own investment firm or whatever, that you're certainly not trying to lie to yourself there. Right. But we do end up in some sense lying in the sense that the conclusion or the explanation that we're jumping to is inaccurate, and it's because we don't know how to interrogate the data. So this is what I'm trying to address, and it aligned with this work that I had seen from Duncan Watts, because I was already really itching sort of in this space to write this book because of this Washington Post article. So here's what happened in 2022. It was like October 2022, I read this article in the Washington Post. And at that time, the Washington Post would have been considered liberal bias for sure. And at that time, 2022, certainly pro vaccine bias, right? So the headline of this article caught my eye because of that, because it was basically like in friction with what I knew their bias was. And the title of the article was Covid is no Longer a Pandemic of the Unvaccinated. So, I mean, I really perked up. I was like, whoa, what kind of data is in this? Because the Washington Post is certainly not, you know, we're not talking about a paper that's anti vax where I would have all sorts of reasons to be skeptical of that. In this case, I'm like, wow, something must have happened. So I read the article and the main data, and again, this is a description. This is what we have to remember. It's just a description. The main data that they're citing is that in August of that year, 58% of the people who had died of COVID were vaccinated, 42% were not. Okay, so the reporter, having seen this and go fact check it, it's true. You can fact check for August of 2022. You'll see that that's true. And this is the problem.

[B]

But the old quantity is just red flags.

[A]

Your head is floating, right? So this is what happened. It's like, this is bigger than this. 58% is more than 42. So that means more vaccinated people died of COVID than not vaccinated. So vaccines must not work. Clearly, the editor had not seen that there might be an issue with this explanation. So I said, well, okay, wait, I'm going to keep repeating because they're going to tell me what percentage of the population are vaccinated. Obviously. No, they did not. So I was like, all right, let me go look that up. So I went and looked it up because it wasn't in the article, which made me very upset. So I went and looked it up and they had a definition in there. And according to their definition, at that time, 80% of the population was vaccinated. So I was like, okay, well, wait, just. Right. Let's just start here. 80% of the population is vaccinated. 58% of the people who died of COVID are vaccinated. Let's look at it from the other frame. 20% of the population is unvaccinated, and they accounted for 42% of the deaths. Right. So I was like, okay, wait. Okay. I certainly think that this explanation, which is the headline, is not only unwarranted, but, like, dangerous.

[B]

Yeah.

[A]

So then I was like, well, okay, like, if I really wanted to take a step further, I ought to age match, right? Because obviously older people are both more likely to be vaccinated, more likely to die of COVID than younger people. There was a statistician who had died. Somebody else had gotten upset about the article, and they did do the age matching, and it turned out that you were five times more likely to die if you were unvaccinated of COVID In the month that the Washington Post declared that Covid was no longer a pandemic of the unvaccinated. And I just said, like, I don't think this reporter's trying to lie. I just think they don't know how to interrogate data. So they see this data and they don't know the questions you ask. The most simple of which would be out of how many? Which was the first question I asked. Right. Well, what. Wait, how many people are vaccinated versus unvaccinated? Like, I need to know that thing right before I can even interpret this on another stream. I've been seeing my clients making these type of errors too. So a very simple example would be, I had a client who was trying to increase their inbound top of funnel. This was an investment client trying to increase their inbound top of funnel investment opportunities. And they showed me a histogram and it was just like, here's the size of inbound top of funnel in 2016-2017-2018-2019, so on so forth. And all of a sudden, like around 2020, this thing hockey sticks and there's a little arrow there. There's an arrow that says implemented experimental marketing strategies. So they said, well, we just want you to look at this because this sort of part of my job. Right, we want you to look at this because what we really want to do is invest heavily in these marketing strategies because you can see that they caused this to go up. So notice that again, we're talking about this big jump, right? So I sort of call it crossing the chasm from description. Okay, you've shown me a description. This is how many in each year to explanation these marketing strategies, cause this thing to occur. That's an explanation. But there's this big thing in between that you've just crossed the chasm. Right. Interrogation. How would we interrogate that data in order to understand that? Now, I'm guessing to a lot of people that when I sort of describe this situation where they're like, well, isn't that obvious? But I can show you why it's not obvious pretty easily. I could say, what if I had an arrow there and it said russia invaded Ukraine? Would you fall for this? And the answer, of course, is no. Because there's this great concept in cognitive psychology. Tanya Lombroso talks about this called explanatory satisfaction. And it's when an explanations, it's exactly what the name is, feels satisfying. And that could be because it confirms a bias that you have. So if I have some sort of bias and the explanation confirms that bias or beliefs that I already have, I'm probably going to feel pretty satisfied by it. At which point, just like when you're satisfied by your meal, you don't go look for dessert. You're not going to go and try to do other things with the data. Another way that it can be satisfying is that, and this happens, I think, a lot in the investment world, it feels insightful and contrarian because then you feel like you discovered something that other people don't know. Right, right. And so that could be another way that like. Or sometimes it's just simply that it makes sense.

[B]

Yeah, right.

[A]

That it kind of.

[B]

Or that your reasoning is being motivated by some other external factor.

[A]

Yeah, it could be that. I think the makes sense thing, though, is a good thing to sort of just linger on, which is we don't like randomness. We want to know why things are occurring. So when we kind of land on an explanation where it sort of solves that discomfort, that we don't know why something happened, we'll often just stop just because. Oh, that makes sense. Right. So that actually can create explanatory satisfaction. It's just that feeling of like, yeah, that makes sense. So that would be like, when you don't necessarily have, like, a dog in the race. Right? Like, it's like, okay, I've got no dog in this fight now. I don't have to feel uncomfortable that I don't know why things are happening in the world. Okay, so we can take this example and say, well, why are they jumping to that conclusion? Well, doesn't it feel good that you experimented with some marketing strategies and then you're inbound, top of hockey stick. So anyway, I looked at this and I said, oh, okay, that's interesting. Before you do that, could you just make me the literally an identical histogram that's just total opportunities in each year. That's all I want. And so they did that. And then marketing strategies never got implemented because it just turned out that at that there were just like a whole bunch of extra opportunities that had been created partly because of COVID Right? Sure. So it could have been that the opportunities were flat and it really worked. Right. Like, I didn't have an opinion, and I want to make this clear, I didn't have an opinion about whether those strategies did work or didn't work. There'd be no reason for me to have the opinion. My opinion was whether you could draw that conclusion from the data that I had been shown. That was all that I cared about at that time. And I said, well, I don't know until I know how many total opportunities there were. And if you can show me that, then you could say, well, what proportion of the total opportunities did you actually capture? And if it's true that you were capturing a larger proportion of the total opportunities, then I'm going to support your conclusion because. So I just want to make it clear it's not just like, oh, the media is telling you things and not giving you context or whatever, which is a problem if you don't know how to take that into your own hands and actually ask the type of questions that I did of that article. But it's also within your own business, particularly in this world of data insights and AI is generating data for you, and how do you even know how to interpret it? Because the AI isn't going to interpret it for you. You have to know how to ask the right questions. Whether you're asking the questions of an AI or your own data or your insights team or whatever it might be, you have to take that into your own hands because that's going to determine choices you make about your health, right where you live. It's going to help you decide. If you're trying to compare the performance of one person versus another, it's going to tell you what you think is causing the outcomes that you observe in the world. And then sometimes that means that you're going to say this is signal for something to occur. Or sometimes it's, if I do this, I can cause a greater likelihood of X occurring or a lower likelihood of X occurring. It really, really matters. So the book is really just trying to get people to say I have to approach data with enough skepticism that I'm always going back to the, I'm never skipping the interrogations step. And then I need to know what the things are that I need to interrogate. And they're super simple things like out of how many, what am I comparing it to? Is the sample any good? And so I'm just building that out through fun narratives and like the COVID story which is in there, obviously there's a lot of like sports stories in there and then there's stories from like my own clients and trying to get people to understand you need to not allow information to happen to you. You have to look at the information and say, if I were to think about how I explain what I am observing. Because again, data is just a description of an observation, something you've observed in the world. If I'm going to explain it, what are the questions that I need to ask of this in order to actually get to the closest to the right explanation that I could.

[B]

Yeah. And you know, as, as you were talking about it, like some data is just so consistent with people's priors that they just, even if you tell them, I, I had an experience where I put it in my book what works on Wall street for a five year period. There was an investment strategy which was really simple. It was one line buy the 50 stocks with the highest gain in sales year over year. Makes a lot of intuitive sense to people. Wow. Their sales are cocky sticking and they're. And then you show them the actual results of that strategy over a five year period and it kills the market. It like triples the S&P 500. The problem was it was for five years and we were looking at a data set of nearly 50 years. When you run that Strategy over the entire data set. It's worse than T bills. And yet I could have, and I've been duplicitous and not honest, I could have gone out and raised a shit ton of money on that.

[A]

That's exactly right.

[B]

And people just fall in love with that idea.

[A]

One of my favorite examples that I have always used is when someone says to you that this player has had a hit six out of their seven last at bats, that you always know that on the eighth one they did not get a hit. Right. And that's that problem. Like, why are you showing me these five years? Because I think sometimes that can be disingenuous. Right. Like sometimes someone can literally be trying to sell you on something that isn't true. Again, even in that case, it's up to you. Because if you look at those five years and you fact check it, it's going to be right. So it's up to you to say, well, wait, what else should I be asking about this? Like, is this the right sample to be looking at, for example? Or why are they showing me these exact five years? Or whatever it might be. Right. It's up to you to say, hey, hold on a second. But then if you're, and this is a very strong tendency within business, you have to understand that you're actually doing that to yourself all the time.

[B]

Time.

[A]

Because what people do a lot is they kind of like, you know, massage. They don't massage the data, they just data mine. Right. They're just looking for which comparison is going to tell me that this thing that I did is better. And I think that most people don't understand that there's something wrong with that. They think that they actually find the truth when they find out that like X is better than Y. And they don't understand. Well, if you try a hundred different ways, you're just randomly going to find out that X is better than. Yeah. And it doesn't necessarily mean that that's going to hold going forward.

[B]

Yeah.

[A]

Right. So I don't really address so much of that in this book of things like, you know, just hygiene around data. You know, like you have to segregate some of the data so you can do all the massaging you want on 60% of the data, but then you have to see if it's predictive of the other 40%, for example, like, I don't care if you massage 60%. Like, you just have to make sure it predicts the other 40% that got quarantined.

[B]

Yeah, you have to have an out of sample holdout.

[A]

Exactly. You know, you, you can obviously, within the social sciences now and cognitive science, you have to preregister your hypotheses. You have to preregister the analyses that you're going to do. That helps you with that type of hygiene. I don't, you know, that's not. The everyday person doesn't necessarily need to know that. What the everyday person needs to know is, well, am I seeing the whole sample? That's a good question. Right. Like, you know, what's the denominator? That's my favorite one. You know, just like, what, how, how out of how many? There's a famous example on hormone replacement therapy that now has just become quite well known. Partly through Peter Attia's book Outlive. They looked at a large sample of people who were taking hormone therapy versus not. These are women who are perimenopausal, menopausal, or postmenopausal. And in the group of people who were taking hormone therapy, five ended up with cancers, reproductive cancers. And in the group that weren't, four did. Now, the way that that number was reported in the media was that you had a 25% increase in your risk of cancer. That's true if you're looking at 4 versus 5. Right. But you have to say, well, wait a minute, but out of how many people total, Because I need to know what the absolute risk is before I compare the relative risk. So it turned out it was out of 1,000. So it was four out of a thousand versus five out of a thousand. Now you're smiling because, like, immediately you're like, well, then that's not even a difference. Right. I think it ends up, I can't remember exactly. I have it in the book.

[B]

I think it's tiny, it's minuscule, it's under 1.

[A]

I think it's 0.06. Yeah. I mean, it's really small.

[B]

Yeah.

[A]

It's not a 25% increase. Right. But because it was reported that way, a generation of women were not given hormone replacement therapy and suffered through what are, for some people, really debilitating symptoms of menopause. So these questions that you have to ask are like, really high stakes. I'm not just being like a nitpicky, like, I know the scientific method and you need to know how to interpret data. You, as an everyday person in your life are making decisions about things like what medical treatments are going to work for me, what are the things that I'm going to do, what type of exercise am I going to do, what type of marketing strategies am I going to invest in? What type of signals do I think that I'm detecting in terms of what makes a good investment versus a not so good investment, so on and so forth, where it could be as simple as which baseball player is better than the other one, which I have an example of in the book. So these things matter. That's the thing. They matter for government policy, they matter for your own health issues, they matter for your business. And we have to start paying attention to how do we get good at this thing that feels like in some ways, like egg heading. And don't tell me about your math. Right. But you don't need to think. These questions are actually really not very. As, you know, while I write, all of my books are about mathematical concepts. They're not what I would call mathy. No, right. It's just more of these general concepts of like, well, why don't you just ask out of how many. Like, and you're. If you just. If that was the only question you ever learned, your life would be a lot better.

[B]

Totally agree. And the. There's a couple of things, right. So, you know, I did this series on the Great Reshuffle, right. And one of the things that I was thinking about as I was getting ready to chat with you was like the world we came from was kind of a clockwork world, right? A implies B implies C implies D. You know, you climb the ladder, you get your gold watch, you get a pension. You know, very simple, very deterministic. Right. And what's a deterministic type game? Chess. Right. If you know the rules, you can get pretty good at chess. The world we're going into is a probabilistic world. And what game is best for that? High stakes poker, Right. You have to make decisions on incomplete information. You have to understand that people might be lying to you. That is bluffing. You have to be able to update your priors immediately when the new card turns over. Right. So I just don't think your average person is built for that. You know, it's something I've banged on and on and on about. You know, we are deterministic thinkers living in a probabilistic world and hilarity or tragedy often ensue. And so my question to you is one who is, like, deeply in this domain. Is it just part of our human nature that we don't grok this stuff?

[A]

So I think, like, here's. Here's the thing. Well, first of all, let me just A slight quibble. There has been no time in our history where we weren't living in a probabilistic world. The question is, how big an error was it to act like it was deterministic?

[B]

Deterministic.

[A]

So the slower the cycle of change, the less of an error it is to act like the world is deterministic. But I can show you that even back then, the world was probabilistic because we did have big sea changes. Like, you lived your life thinking that you were going to be an ag. Right. And then you shifted to everything all of a sudden. Shifted to manufacturing.

[B]

Yep.

[A]

And it's like, whoa, my whole life just got turned upside down. And that's where you got, for example, the Luddite movement, where they're saying, whoa, we don't like that this changed. We thought the world was going to be this way. We thought that was a good prediction to make. And now, ooh, technology. And stop that. So I think that now the rapidity with which it's shifting makes it so that it's much more obvious that it's an error to behave as if the world is deterministic. You're going to get punished much more quickly for thinking that. Right. And I think that that's particularly true over, say, the last 25 years, that that cycle has sped up. Because even back in 1985 or whatever, I would say even in 1990, the early 90s, you still could believe that if I go do this, then these things will happen and these opportunities will be available to me. So let me just kind of start there. But let's think about an evolving human who lives their life not believing that things will be the same. Right. That, for example, so you have to figure out your migration pattern. Right. If you're a nomad, if you're not a nomad, you have to figure out, like, when are you planting, when are you harvesting, when are you doing? Don't you have to believe the world is deterministic in that case? Right. I mean, that's why we have rituals. Right. That's where Thanksgiving or harvest comes from. Right. We're planting at a certain time during the year. Now, obviously, they understood there were ups and downs in terms of rain. And when that happened, that was very uncomfortable. And so they would do rain dances, or in the worst cases, they might sacrifice virgin or two. Right. Which is like, you know, but they're where they're trying to sort of overcome some of the randomness that occurs. But of course, like, they must believe that because they have to plan they have to plant at a certain time, they have to harvest at a certain time. They have to. And that they have to believe that that's going to be the same over time. Right. As the world becomes more complex because we're more global, our social groups are bigger, and as technology advances at a faster pace, then what happens is that our minds that were built to be deterministic, I think for relatively good reason, to keep structure in society and structure in our day and structure in our year and structure in our lives now all of a sudden is butting up against the complexity and the speed of change. Right. Where it then becomes a much bigger error. I can take that to a broader view of just kind of bias in general. Right. Which is when we think about a lot of the sort of, you know, when we think about the heuristic side, right. These shortcuts that we take, they mostly work.

[B]

Yeah.

[A]

Which is why we have them. But then when they don't work, it's really bad. And the more sort of complex the world is, the worse it is. So you can think about something like the availability bias, for example. Well, if we're living in a very small territory in a social group that isn't more than 300, if we see things more, they probably do occur more in the world. It's actually a pretty reasonable shortcut under those circumstances. But when the world becomes global and it's not just what we're seeing in our area, and we're looking at the news and they show fires but not drownings or I live in America, so I think John is a more popular name. When it's actually globally not that popular right now, it becomes a problem. That's a way to add complexity into the problem. So most of the ways in which we think about cognitive error, a lot of them develop because they were shortcuts that worked most of the time in the world in which we evolved. And I think that that's true of this thinking about things deterministically in the world that we evolved, I think that that was probably mostly okay, and you mostly weren't punishing for it unless you were the person who got sacrificed because you didn't have enough rain that year. Then you got super punished for. You got super punished for that.

[B]

But I think another thing at play, and by the way, I agree with you, the world was always probabilistic, but in the current environment, it's really much more important to understand that very high uncertainty. And, and, and yet another thing is this illusion of certainty we seek, we we prefer the candidate who is certain in his or her pronouncements. And one of the things I was thinking about, you've seen the news on, on the prediction markets getting, doing deals with cnn, cnbc, and I was thinking how cool to be able to use prediction markets to speak, spin them up on and have a ticker on. The CEO says we're going to. He says with great certainty we are going to double sales next year. Wouldn't it be great if you had a decently liquid. Because you were good to call out liquidity, a decently liquid prediction market where the bets could go right underneath the statement.

[A]

Well, I think that would be great as long as at the same time we were educating people about how to think probabilistically. Obviously, this is part of the reason why I co founded the alliance for Decision Education, because I think trigonometry is such a silly thing to teach the average person. I don't think we're sailing by sextant anymore. So I don't exactly know why. Why people need to know trigonometry if you're raising a barn, I don't know. But I think that that's something that if you're actually going to be a structural engineer, you can probably take as an elective later. And it would probably be better to be teaching people statistics and probability as kind of a core decision skill. You can teach a lot of stuff a lot earlier than at which point that would be appropriate. But the reason why I mention that is because we all know that you can go back to the very famous 2016 election where I think Hillary Clinton was like 60% to win, according to the. I mean, Nate Silver was essentially sort of running something that would have looked like a prediction market. I think that had you. If you had done a prediction market, it would have been 60, 65, probably wouldn't have been relatively in line with that. And when Trump won, people were like, you were wrong. So there we go. Right? So, yes, I think it'd be wonderful to have that just as sort of for people in the know. Certainly. And I think that if it went below 50%, it might be helpful if someone was declaring it to be 100%, would it be helpful if it was like the market was saying it was 78% and the person was declaring it to be 100%? I don't think it would because I think that people would sort of interpret that as like, direction. Okay, well, that's definitely a sure thing.

[B]

Yeah, yeah, yeah.

[A]

As opposed to, like, okay, that's like Three to one. Ish. Right? Like, so I'm going to put one bullet in a four chamber gun. Russian roulette, anybody? Right. And I think that most people would be like, whoa, no, if you put it that way, no. Right. But I think you have to get it into those concrete terms for people to actually get it. And likewise, I think that there are things that people can say where they could even add some nuance to it. And the prediction market would say this is 35%. But actually given what the payoff for that would be, it'd be like, you would be willing to say, okay, yeah, I would take the 35% because the payoff's gonna be 5 to 1 or something like that. And I think that these concepts are actually just very hard for people to grasp. Obviously, as a poker player, that's the whole gig, right? It's okay. I remember once I was actually live streaming way back. This was in the early days of it. I was live streaming, playing in a poker tournament, and I had a hand and I was getting 11 to 1 from the pot. And this was on a final bet, so I wasn't going to have to invest any more money. There was nothing implied that was going to happen in the future. It was just a straight call on the end and I had a terrible hand. I was getting 11 to 1 from the pot and I said, well, I'm profitable here at like 9%. And I just better than that, actually. It's like 8%. And I was like, well, I've never been that sure about anything. I guess I should just call, right? Because I haven't. Right. I said I have to be. Well, like I have to be over 90% sure that this is not the best hand. And I just went, okay. I'm not sure about anything. I mean, I'm sure about my own name, but. But about that I'm not. And I called and I won the pot. And it was literally like I was trying to sort of take the magic out of it. It's like I don't have a dead read on this person. I do not. I would not. I'm not sure that they're bluffing. I'm not declaring anything with certainty. I'm just saying, like, I don't know, I've never been over 90% on anything I'm calling, right? And I won the pot. And I think that that type of thinking, right, it's. It's like you have to get to people early to start to get them to be able to view the world through that lens. So I think it would be great as like, ooh, what's this? And why should I care? And then if we back that up with really good education, I think that would be like the most amazing thing ever.

[B]

Yeah. So talk to me about education because that's another hobby horse of mine. Right. We are using an antiquated educational system for a world that no longer exists. Right. It was designed for the industrial era where essentially you wanted to be able to have people sit in a room for eight hours and take instruction. Right. If, if you were, if you were put in charge, if you were the new secretary. Do we still have an education department? I don't know whether they accidentally.

[A]

I don't know if they're doing anything at the moment, but I don't think it's been, I don't think it's been canceled yet.

[B]

But tell me what it would look like, because I agree with you, by the way, that early education especially would be able to mold the way that the younger people are thinking on their way up. But like, what would you change about the curriculum?

[A]

I think that there's a lot of people who are working in ed reform and a lot of the work is like, we're going to create this special type of school where, you know, people are doing something completely different. And I think that that's fine and I think it's great to have those kinds of things, but I think that we do need to live in the reality that whatever you're doing, 80% of kids are going to public school. And I'm including charter schools in that. Right. Like 80% of kids are going in public schools or charter schools. And so you need to be thinking about how do we change education within that system. I just think that's, it's very important to acknowledge that because I could say all sorts of things about here's the kind of school that I would create. Right. But that's great. But you're not, you're, you're not getting to the, you know, most of the population.

[B]

Thus is why I made you the head of the Department of Education.

[A]

Well, thank you.

[B]

So keeping, keeping that as stipulated, keep in mind, keeping that in mind for public school, since 80% of kids go to those.

[A]

Right. So basically what I would be doing is changing the curriculum out of something which is very fact based into something which is very decision based. This shouldn't surprise you. So there's some interesting things that you can do, actually. So at the alliance, we've actually developed a tool and it's it's AI. It's an AI tool where you as a teacher can input a lesson plan. So let's, let's imagine that you're teaching Macbeth or, or you're teaching about the water cycle or pick your. Pick your topic, right? You can input it into this tool. And all that the tool will do is embed decision education, decision skills into. Into the, into the lesson plan. So we could think about something like Macbeth, for example, or Eisenhower. Right. Or that. And let's think about a skill that we can teach people. Let's explore the counterfactuals. What if Macbeth had done this? Oh, that's cool, right? Like, how do you think that this would have turned out in the water table? We could say, okay, here's the average rainfall in this area per year. What if we put a mountain here? Because really, what are you teaching people in a fact base? You're like, here's the rainfall in an area where this mountain is here. And mountains affect things. This. But you're not actually asking people to put that into an actual forecast of what would happen to the rain if you stuck a mountain in a place, right. Where they're having to take into account all the other things about the climate in that area. As an example, right? Going back to Macbeth, it's like, well, what were all the options? Why do you think they chose this option? How did that affect Lady Macbeth? How was Lady Macbeth's decisions taking into account Macbeth's desires versus her desires versus so and so forth? What's interesting about it is I think it actually makes the material stickier anyway because you're not just memorizing facts, right? They're now thinking for themselves. So whatever decision skill you want to think about, like teaching agency or optionality or forecasting or bias, right? What were their different biases? You can start to teach the cognitive biases through that lens. For example, this idea of counterfactual thinking, right? You can start to embed that into any lesson. And I think it just becomes much more interesting than fact based. On top of that, I think that you have to start skilling kids up on how are you thinking about getting good inputs from AI into your decisions. So just a concrete example of what we're doing. We have a forecasting competition. Super fun. So you can think about good judgment project, which is Phil Tetlock, Superforecasters and Barb Millers. They've got good judgment. People go on there. You as an adult can go on there and you can enter a forecasting competition. It's Very fun. So we said, well, why aren't we doing this with kids? We have a forecasting competition. We actually are able to give some kids training and some not. Preliminary results are showing that the training is quite effective. The training is actually based on the work from the my dissertation which showed that training novices on some simple concepts for forecasting is very effective. Things like what's the base rate is a really good thing to teach people. So we do that with middle schoolers and high schoolers and we encourage them to use AI because that's the world we're living in.

[B]

Totally, right? Could not agree more.

[A]

But what we're trying to teach them is AI is a big fat liar unless you, you know how to prompt it correctly. So what we ask them to do is show your work. If you're using AI to help you with, try to figure out how you're going to forecast. And we have kids forecast things they care about, right? Like in the first week of the Taylor Swift new album drops, how many copies do you think it will sell? Right? So they care about that kind of thing. If you're going to use AI to help you with that, show your work and show the prompts. And we're going to also give you training on how to prompt AI, which is a decision skill in and of itself, right? How do you prompt it to get you to give you an answer that's more accurate? How do you check your sources? How do you get it to be skeptical? How do you get it to give you counter arguments? How do you get the right what ifs? And what they learn is that the better you are at prompting the AI, the better the forecast that you're going to be able to make because your inputs are going to to be better. And then you're more likely to win the prizes that we're offering along with that. So I think that there's just like starting in kindergarten, you can start to teach this stuff. You can start to teach very simple probability. And we used to, by the way, it's gotten dropped out of the education system until about eighth grade now. We used to start teaching probability in kindergarten. So let's bring that back in. A kindergartner could learn the concept of agency. They could learn the concept of options. Well, what are the different choices that you have for those different choices? What do you think would happen if you chose this thing? Why do you think that would happen? Why is that a conversation that we're not having with kindergarteners? Instead we're teaching trigonometry. It's so weird to me.

[B]

Again, you had me at hello there. Because I agree we're living in a world where I think when I was coming up, your memory was super important.

[A]

Oh, my gosh. Right.

[B]

Your ability to recall facts. And when you're talking about Macbeth, I immediately went, the lady doth protest too much.

[A]

There was a soliloquy I had to learn from King Lear. Cordelia. It was one from Cordelia. Because we all had to memorize something. I can still say it, right? Like, why is this living?

[B]

Exactly.

[A]

Whatever.

[B]

Exactly. But that isn't the case anymore. Right. We have ready.

[A]

Exactly. We had a set of encyclopedias on our.

[B]

Same with me.

[A]

I mean, I'm sure that at the time they were like 25 years old. I mean, I don't think any of the information. It was particularly correct anymore.

[B]

No.

[A]

If I wanted to know something, I had to look it up there.

[B]

Yeah, exactly. And. And so you. It. It's not like the world we were coming from was some bizarro world. No, it's. It's. That's what we had available.

[A]

That's right.

[B]

To us. That's right. And so you had. If you had a good memory, good for you. You're going to go a lot further. If you were a high agency person, sort of by your natural disposition, good for you. You're going to go a lot further.

[A]

But.

[B]

But I don't think that these things are necessarily. Well, the memory might be innate, but I don't think agency is innate. I think that you can learn agency.

[A]

Exactly. That's exactly right. You can see, actually there are socioeconomic differences in the way that people approach agency. So when you see people of higher socioeconomic status, which generally means that you sort of just live in a world where you have more agency. Right. And you see how they're interacting with a doctor, for example, they're asking, they're acting like, hey, I'm your customer. Right. And they're asking all sorts of questions and they're just not accepting what they say. But when you see people of lower socioeconomic status and generally, what obviously correlates with that is the parents tend to be not as well educated. For example, again, 10, not always, but. And they're living in a world where it's a little bit more happening to you, or at least it feels like that when they go to the doctor, the doctor says that that's it. They don't even ask any questions and they're told not to. Don't be rude to that doctor. Right. Like they're the authority in the room. Whereas when, you know, when you've been living a life where you understand, you know, a lot of doctors, like, you're like, well, you're an idiot. I mean, like, you're the dumbass at the party, right? Like, oh, my God, would you shut up? You're just more likely to be asking questions and skeptical and getting second opinions and things like that, which, of course, is an act of agency. So I think it's incredibly important to teach very early on this idea that you're an agent of your own decisions. And we can get back to this core idea that's really throughout all of my writing, which is, look, you got two things. Only two. That's it. That determine how your life turns out. One is luck, which. Sorry, it's exogenous force that acts upon you. You have no control. You can say, I make my own luck all day. It's like, that literally is not a sentence of English that makes sense. I mean, it's grammatically correct, but that's about it. It has words in it that have meanings, but that's it. You know, it's not like you're reading a Lewis Carroll poem or something like that. Like, it's actual words of English, but as a string of English, it doesn't make any sense because you can't make your own luck.

[B]

Definitionally, I actually want to push back. I don't believe you make your own luck, but I believe that if you have your reticular activating system, your aperture set very wide, you might be able to always be scanning for opportunities that are lucky.

[A]

I'm 100% with you on that.

[B]

Okay.

[A]

I think that when people say, I make my own luck, what they mean is I make decisions that increase the probability that I'm going to make out there.

[B]

I'm totally with you.

[A]

Right. That doesn't mean that you've made any luck.

[B]

No, wrong.

[A]

And by the way, nobody who's had bad things happen to them has ever said, I make my own luck. No, I just want to point that out, which is why, you know, it's dumb, but also.

[B]

But also, the person who is forever going on and on about how everything in the world is due to luck is probably one of the most unlucky people in the world.

[A]

Well, of course. I mean, that person obviously didn't have good outcomes. That is for sure. And that's the point, right? Like, we want to take credit. So, like, you know, it's always the most successful people who are like, I make my own Luck.

[B]

Right.

[A]

And it's like, no. I mean, you may make better than average decisions, but you do not make your own luck. And obviously, at sort of the start of anybody's life, life is like, look, what if I had been born in 1600?

[B]

What my life would, you know, actually, I love playing that game.

[A]

Yeah.

[B]

Because it. It totally immediately informs you of how I. I say this often. I am so lucky that I was born in America.

[A]

Right.

[B]

In 1960. Because, like, if. If it was 1870. Salon.

[A]

Exactly.

[B]

My opportunity set is gonna be a lot smaller.

[A]

Exactly. And the thing is, people will be like, oh, Jim, you're being so humble. It's like, no, it's just true.

[B]

Yes.

[A]

Right, right. It's like if Bill Gates had been born 30 years later, no Microsoft, no billions, because the computer already would have happened.

[B]

I make this point all the time about my own book, what works on Wall Street. The only reason I got to write that book was because chance luck made me born in 1960. If Ben Graham had access to the computers that I had and the databases that I had, he would have written that book. It would not have been an opportunity for me.

[A]

That's exactly right. And the thing is that that doesn't take away from the accomplishment. You saw, the opportunity you put in, the hard work, you did the research, you wrote the great book. That is. That's the thing. It's like a blend. Right. And it doesn't discount your accomplishment to say, hey, there was a lot of luck in what happened. I mean, LeBron James is tall. I'm pretty sure he didn't have any control over that. Right. Like, he's a great basketball player, but he's tall.

[B]

That's right.

[A]

What if he were five, four? Right. I don't think he ends up being taller. LeBron James.

[B]

I had a statistician on the podcast who wrote a book about the NBA, and he said it may not surprise you that your chances of being selected for the NBA go drastically higher if you're over 7ft tall.

[A]

That's exactly right. Right. And, like, a little coordinated, and then you're pretty good. But you don't even need to be that coordinated because you just go like this. I've got a basket. So, like, you have to. You have to. To be smart enough to know the rules. So that's thing number one is luck, which you do not make. It's just the thing. It happens. You have no control. The other thing is decision, quality. When we think about it, I'm born into the world, and when I'M born into the world to whatever parents I'm born to, with whatever talents or lack of talent that I might have, physical characteristics, so on and so forth. There's some distribution of outcomes that's available to me. That's true. Now the question is, what's the probability that I end up at the right tail? And that's going to be determined by the decisions that I make. The better decisions I make, the more that I can get to that right tail of the distribution on average. And obviously, that doesn't mean. Right.

[B]

That's mean, everybody.

[A]

I can go through a green light and a car can hit me.

[B]

Yep.

[A]

What am I going to do? But if I'm consistently going through green lights and not going through red lights, the probability that I have better outcomes is going to be higher than someone who's consistently going through red lights. Right. And that's all we're trying to do. The person going through the red lights could just literally have that for their whole life, be unscathed, but it's a much lower probability that that will occur for them.

[B]

And many times people will confuse in their minds probability versus possibility. Right. They're two very different things could happen, right? Yeah, exactly. It's how they market the New York Lotto, right? Hey, you never know.

[A]

Yeah, it could happen.

[B]

Well, you do kind of know.

[A]

You do kind of know. Look, every once in a while, you're going to get more than a dollar back for the dollar you spend.

[B]

Yeah.

[A]

It's rare, but it occasionally will happen. And even when you calculate that, it would be really good if you calculated in the fact that the more people play, the more likely you're sharing, which then makes it really hard.

[B]

Fran Leibowitz had a great line which was, I think my chances of winning the lottery are the same whether I buy a ticket or not.

[A]

Close to it.

[B]

But it is one of those. I love this framing, because you're absolutely right. The quality of your decision making, on average, it does not mean you could make all the best decisions in the world. And just sadly, you are in the cohort that bad shit happens to. But the two together, why is it so hard for people?

[A]

Well, I mean, let's go back to explanatory satisfaction. Right. Like, people are so uncomfortable, comfortable with randomness.

[B]

They really hate it.

[A]

They really hate it. And obviously, in its worst form, it becomes a conspiracy theory, of course. Right. And this concept of, like, we're sampling a lot of stuff all the time, and sometimes things will happen at the same time together. There was a study where. It's basically just like you're given sort of a toy example, right? And it's like, hey Jim, you know, there's this museum and we just found out again, data, which is a description. We just found out that people who visit the portrait room, the portrait gallery, are more likely to give a donation on their way out. You're sort of asked why? And people are kind of like a little bit uncomfortable, like, I don't know why, and they're sort of making things up, but they don't necessarily have a lot of confidence in what they make up. But then when you show them a scientific paper, and this is true, that when people are being observed, they're more pro social. Then all of a sudden they're like, oh, it's because there are eyes in the room and that's why they're doing it. And they're like 100% confident because it's taken away this, like, I don't know why these two things are crying together, which like, even on that small little bit and that thing that has nothing to do with you still makes you uncomfortable. Now how do you get them out of that? Right. You'd say to them, well, here are a whole bunch of other explanations how then all of a sudden they'll sort of get off that, like this is 100%. It's. If you give them this one piece of scientific data, which is true, and you let them make the connection, they actually don't self generate explanations. But when you're like, oh, did you know that the portrait room is shabbier than the other galleries as an example? So you start to generate other things that could be true. People who visit the portrait gallery are of higher net worth. Right. Like, so now I can start to give you a lot of other things and then you go back to not being, you know, you go back to that, but you're sort of back to being uncomfortable and you kind of want to know, but if I give you a pathway to certainty, you're going to take it. And I would go back to like, you're living in a small tribe in this small area and like, how are you supposed to survive if you think the world is just random? Right, right. Like you have to sort of.

[B]

Well, that's the illusion of certainty.

[A]

Right, right.

[B]

It, it I think is very much an evolutionary aspect of our behavior. And it's one of the. I had an example that I put in a couple of my books, which is this professor did this study where it was Mr. Smith and Mr. Jones and he told them that the object of the study was to see how good, by getting feedback, they could get at judging whether a picture of a cell was cancerous or not. What he didn't tell them was that Mr. Smith was getting true feedback. So in other words, if he guessed right, he got a green light. If he guessed wrong, he got a red light. Mr. Jones was getting feedback based on Mr. Smith's guesses. Right. Okay, so if Mr. Smith guessed wrong, but Jones guessed right, he. He'd still get a red light. What's really interesting, and they did this with multiple participants. And what is really, really interesting is the guy getting the true feedback is pretty soon improving his batting average to about. He's getting seven out of 10. Right, right. This poor guy over here is not. But what's really interesting is when they come and they ask them, well, how do you make your prediction? The guy who's getting real feedback offers concise, straightforward examples. Well, if the cell seems to have a broken edge, or if it's fuzzy over here, I guess that it's sick. And if it doesn't have those things, I guess that it's healthy. Mr. Jones, who's getting the bad feedback, launches. He looks at this guy and he's like, that's the most pedestrian example I've ever heard in my life. No, no, no. Sometimes it's this and sometimes it's that. And he just weaves this tapestry, very complicated tapestry. What's really interesting, all of the guys getting it right are in awe of the guy who is getting it right.

[A]

Oh, my God, you're so smart.

[B]

You're. And when they go and do it again, his performance materially declines.

[A]

That's interesting.

[B]

And so it's almost like we over index on people who wave their hands a lot.

[A]

I really try to teach people that when you're communicating information, this is so hard because of the point that you just made, that your job is to communicate at the exact same time what you know and what you don't know, literally at the exact same time. Now, what we tend to want to do is be like, I know this thing for sure, which is a very bad representation of what you know and what you don't know.

[B]

Very bad.

[A]

It's a very bad representation. So what I try to tell people to do is like, let's imagine that I'm working with somebody and their job is they're making sales forecasts. Okay, that's fine. Give me a point estimate. That's great, because there's some useful information I can derive from your point estimate. But then what I'd like you to do is give me a lower bound and an upper bound, and I can set 90% confidence interval, 80% confidence interval. And just for those who don't know, a 90% confidence interval just means that the true answer, if I were omniscient, would land within that boundary 90% of the time. Okay, so give me a lower bound and an upper bound. Now, what you've actually done is told me what you know and what you don't know. So there's a very simple example that I do in my classes, which is, so I have a dog. The people in my class have not seen a picture of my dog. So I always start with, what's the weight of my dog? Very often they'll be like, I don't know. And I'm like, okay, that's fine. But like, give me your best guess. Okay? So I'm just trying to get them to give me their best guess. And usually they'll say somewhere between 25 and 45, maybe 50 pounds somewhere in there. So let's imagine they say 35 pounds. Okay? So I'm like, okay, that's your best guess. What do you think the smallest amount my dog weighs is? And usually they'll say like £10. Because I didn't say puppy. Right? So let's say £10. I'll go, what do you think the biggest amount my dog weighs is? And they'll. They'll often say something like around 120, sometimes as high as 150. I say, okay, that's interesting. And then I start to ask them some question. So I can ask you a question. Why do you think they don't say £500?

[B]

Why do I think, yeah, yeah. Because there would be a very rare dog that weighs 500 pounds.

[A]

I think it would be dead, actually.

[B]

It could be a prehistoric dog that.

[A]

Was gonna be a dog that I own. Own today. Right. Because dogs don't weigh that much.

[B]

That's right.

[A]

So when I say to them, like, your initial instinct was, well, I don't know what your dog weighs, but, like, oh, actually, you know a lot about what my dog weighs, because things can weigh 10,000 pounds or a million pounds. And you're. You just gave me, like a very small range in the. In the context of what the possibilities are that they.

[B]

And what you know about the dog.

[A]

But you gave me 10 to 120 pounds. Right. So, you know, they don't come in one pound sizes. If I'm calling it A dog. And, you know, it doesn't come in 500 pound sizes. That's amazing. So now I ask them another question, which I'll ask you. You can pretend to be the person. All right, so it's interesting because you gave me £10 to 125. £120, but your point estimate was £35, which is a lot closer to 10 than it is to 120. Why is that?

[B]

I wouldn't make that estimate, though.

[A]

Well, so what they say, and I would argue they're correct to do this, is that larger dogs are rarer, so they're skew in the distribution. The majority of dogs that people have are on the small end of the scale. Right. Think about people in New York City who have apartments. They're going to tend to have smaller dogs.

[B]

Well, sorry to interrupt, but that was where my mind immediately went.

[A]

Oh, yeah, that you would have.

[B]

Where? I would like, say, where are you living?

[A]

Yes. Okay. So that's my next thing that I point out. Okay, so this is. I'm refusing to give you any information except it's a dog. Yeah, right. Okay. So I say, oh, that's interesting. So the fact that the point estimate is closer to the lower end than the higher end actually tells me something about the shape of the distribution. Right? How common are small dogs compared to, wow, you really know a lot about dogs. And then I say, okay, so let's imagine that I'm asking you to do this, right? I'm saying, like, give me a point estimate, lower bound and upper bound. Tell me, what are the things that you want to ask me? And they'll say things like, where do you live? What's the breed? Right? And I say, oh, okay. So that's interesting. So if I actually allow you to build, the uncertainty in it causes you to start to generate questions that will help you because it opens you. But here's the even better thing. If all I do is declare to you £40, if you say something different to me, you now are at risk of me thinking you are wrong. But if instead I say, well, I think Annie's dog is 35 pounds, maybe lowest 10, highest 120 pounds, I'm included in that is a request for information, right? Included in that is, hey, if you know anything, can you help me? So now maybe you'll be like, you know what? Annie has a toy poodle. I actually think you should lop off that topic. I think it's probably going to be lower. But the point is that it invites you into the conversation to Help me. And now when we're thinking about, I want to express both what I know and what I don't know. Now all of a sudden you find out, oh, but that doesn't mean I don't know anything.

[B]

Right? That's what I love. Yeah. Actually, that's what I love about the framing of this.

[A]

I'm more likely to end up knowing more at the end of the process because first of all, I'm going to be more inquisitive, but also other people are going to be more freely offer me information. So if I could change the world, that earnings estimate would be lower bound. This upper bound. This. With some context, here's the circumstances under which we would see the lower bound simple example with sales estimates. Let's imagine that someone calls a number like a million net new ARR at the end of Q4, lower bound 800, upper bound, 2 million. You immediately know that there's a big account that might slip to Q1. It's like, it's so easy to figure that out. And you can just say to the person, tell me why now? As someone who's trying to budget for the organization, isn't that way better?

[B]

It absolutely is. And. And yet I'm also thinking about another example that you're. What you're. By the way, I love that exercise because what I love about it is you are showing people that they know a lot more intuitively than they think they do. And they're learning that through asking questions, which is great. But there is an example of another professor. I love what these guys get up to, but essentially they give you one base rate. They say that there is a town of 100,000 people, 70,000 of them are lawyers, 30,000 are engineers. And then your job is to guess from random sampling that we take out of a big bag full of names how many of the 10 that I pull out are lawyers and engineers. So I pull out 10 names and they're just a name. Annie, Jim, Jean, Marc, Richard. Right. Just the name. What do you think most people say when asked how many of these are lawyers and how many are engineers?

[A]

I don't know. I mean, I would hope they just go, well, you gave me the base rate, so I guess I'm going to say seven of them.

[B]

Right. They either do that, and this is actually, again, through iterative trials of this, they use one of two strategies.

[A]

Okay.

[B]

They either go with the base rate and say that seven are our lawyers and three are engineers.

[A]

Yeah. Given that I literally don't know anything else, that's Right.

[B]

That's all. You know. And the other one they go with is they say they're all lawyers. They just keep saying they're all lawyers to try because they know they got to do it multiple times. Right. But then they add a little twist. The next round with a different group. Of course, the next round adds descriptive but meaningless information. Annie is 44 years old and wears glasses. Jim is 65 years old and does this, this and this. When they add descriptive information, people begin to largely deviate from the base rate. Then.

[A]

Well, I mean, obviously the guy who wears glasses is an engineer.

[B]

Right. Well, but then they add stereotypical.

[A]

Yeah.

[B]

And when they add stereotypical, people completely ignore the base rate. Frank is shy, likes mathematical puzzles and wears glasses. You could jack up the number of lawyers to 99.

[A]

It doesn't matter.

[B]

It doesn't matter.

[A]

That one's the.

[B]

And, and, and so how would you solve for that? Because that's such a natural tendency.

[A]

As a mathematician, you ought to just be ignoring that information and you should just go with the base rate as your first best guess. Unless there's something causal. Right. So obviously wearing glasses doesn't cause you to be anything.

[B]

Right?

[A]

Right. Okay. If you're going to deviate from the base rate as your forecast, you have to believe that there's some sort of dislocation location that's occurred. So there has to be some causal difference. So the example I give is like, don't. If you have to predict the probability that a Cat 3 or higher hurricane is going to make landfall in the US in a given year, don't use statistics from the 60s because something has changed. Right. So that I'm really trying to sort of pound that home that your best guess should always be the base rate unless you have a really good reason to deviate from it. That is actually part of the training from my dissertation work. And we do get people. We can get people to do that. Right. It's part of the training that we're giving to the high schoolers and the middle schoolers. Now I'm going to put on my linguistics app. There's a part of linguistics which is called pragmatics. And it just has to do with like, well, we know what pragmatics are. Right? Like, what is pragmatically? So, okay, so let's imagine that there's only one pencil on the table. There's only one, and it's red. There's a very big difference in how you think about the redness of the pencil. If I say, can you hand me that pencil versus can you hand me that red pencil? So you can intuit this immediately. Right. Because there's only one pencil on the table. So it must be like a red pencil. Like, this must be what it's called. The redness is a thing about it. Right. Like. So it's a correcting pencil or something.

[B]

Yeah, yeah.

[A]

Okay. This is just a thing about language, is that when we offer information to people that is outside of what needs to be offered in the situation, we automatically assume that that information is incredibly important. So there's an old study that they did which was on similarity, and it was basically, how do you interpret North America is similar to China versus China is similar to North America. And it turns out that people interpret those two things very differently because there are rules about what is being compared to what the son is like the father, not the father is like the son as an example. So there's things that have to do with social status, size, whatever. So let's imagine that I said to you, the bicycle is next to the house. I'm sure that what you're imagining is a normal bicycle and kind of a normal house. Right. But what if I said to you, the house is next to the bicycle. Right. It's like, you're like, what's the deal with that bicycle? Is it like a statue?

[B]

Like a big, big statue.

[A]

Right. So you could think about this. I don't know.

[B]

Or is it a little model house?

[A]

Right.

[B]

That you could build?

[A]

You're like, what's going on? So, like, a real life example of this is in Philadelphia. There's a very big clothespin.

[B]

Yeah, I've seen it.

[A]

Right. Which is a really big statue. So if you're saying, like, I'm standing next to the clothespin.

[B]

Clothespin.

[A]

It makes sense in the context of that. Right. But it doesn't make any sense if it's a legit clothespin. Right. Because you're bigger.

[B]

Yes.

[A]

Okay. So I do think that when we're doing those types of things, we have to take into account pragmatics. So there's the classic base rate neglect. Right. Linda is mousy and she's quiet and she has whatever. And, like, what's the chances she's a librarian? Right. And it has to do with this, what they call the conjunction fallacy.

[B]

Yeah, that's what I was actually thinking of. Right, yeah. Yep.

[A]

And. But as a linguist, what I say is, but why are you offering me all this information about Linda? Right, Right. Like, so, yes, we don't want to fall for it because it is a way of trickery. And you can see that trickery occurring in people who are actually trying to sell you or fool you or something like that. So we would like not to fall for it, but it's not that surprising that you are. Because when you're saying, what are the chances that Linda is both a D and D fan and a librarian or whatever it is. Right. That when you're giving all of this extra detail, as someone who speaks the language, what's going through your head is, why are you giving me all this detail? It must be meaningful. In the same way that you told me that the pencil was read, my.

[B]

Mind sort of automatically goes to. If I'm being fed a ton of additional information that they're. They're trying to sell me on something.

[A]

That is exactly right.

[B]

So I immediately get a little more suspicious.

[A]

Yes. So you intuitively know it.

[B]

Yeah.

[A]

See, that's the thing. Because you're like, what? Wait, what? Why. Why are you telling me all these completely unnecessary details?

[B]

I love the overlaying linguistics and the. And the different rules of language that I had never really actually thought about that. How they can make things that seem stupid, like, statistically they are. I love that. I've never actually thought about.

[A]

You know what I think is interesting about it? I bet you didn't think the conversation was going to go here.

[B]

I love it.

[A]

What I think is really interesting about it is that if you think about the framing effect, it's in the same vibe world, right?

[B]

Yep.

[A]

And they do recognize the framing effect. Right. So we should understand that there's just like a grammatical effect also. Right. Or a pragmatic effect of like, I'm offering you information. Why am I doing that?

[B]

Right.

[A]

Where people. Yes. In reality, you should just say 70%. So 70% chance it's a lawyer. Except why are you giving me all this information? Right, Right. Like, weird. So maybe I should be paying attention to it. So even they understood it when it was like, okay, 1,000 people will die of this cancer. I've got a new experimental treatment. If we give it to these thousand people, 600 of them will live. Everybody's like, yes, if you say 400 will die, they're like, oh, I don't know. So they're understanding that. Right. That. Okay, there's these issues of just sort of how things are framed, but then we also have to take into account the grammatical context in which you're saying them. And why are you being inefficient? Because that's what you're doing when you're giving the extra information is that you're adding inefficiency into the piece.

[B]

It's being very inefficient. I love that. This is absolutely.

[A]

The strange twists and turns of the conversation. Conversation that nobody was expecting.

[B]

I think that's actually great, though, because, you know, I think immediately about Claude Shannon and information theory. Right. And what he said was pretty straightforward. Right. Information is unexpected. Right. And so built into our processing of that is this natural. Why is Annie giving me this additional information? I love that.

[A]

Right. So should it change my probability? Well, like in most normal communications. Yes. That is what I would say is because you are assuming that I am an honest broker who is trying to help you to understand the world in some way. So I don't think it's unreasonable for you to say we're just having a conversation. I just told you a whole bunch of extra stuff. And it should probably change your forecast. Now, when a politician is. Don't fall for it, please. Right. Or a salesperson is adding in all of this extra context.

[B]

Well, you know, Shannon himself said the amount of information in a stump political speech is probably zero, whereas a. The amount of information in a short poem could be kind of off the charts. Right. Because of the honest. The. His notion of the unexpected. But I love the way that feeds into. Why are you giving me this information, this linguistically? Why are you adding red pencil? There must be something about that particular type of pencil that is meaningful or you would not have been inefficient in a normal exchange.

[A]

I mean, again, going back to evolution, how could we as humans have survived in social groups that we needed in order to be able to overcome the fact that we're physical weaklings if we did not believe that people were generally telling us the truth?

[B]

Right.

[A]

You have to. For social cohesion.

[B]

Unless you're a psychopath or a sociopath, you default to truth.

[A]

Exactly. So if we're just like normal people, I'm not a politician, I'm not a salesperson, I'm not whatever. Like we assume we're just normal people. And I say, can you hand me that red pencil when it's the only pencil on there? You're going to assume that I've added that in for a reason because I'm trying to give. Give you some sort of like. Otherwise I wouldn't do it.

[B]

Right.

[A]

The reason why I tell people in any training that I ever do, you have to start with the base rate. And then you better have a damn good reason that you're deviating from it. It better be super. I mean, and obviously, like, in the market, this is huge. People think dislocations are happening all the time. Right. And it's like, why are you assuming there's a dislocation? They're rare. Like, they're really rare. And you better have a really good reason for. For why you think that things are so different now. Words to live by, man.

[B]

But I'm thinking of the cartoon from the New Yorker with the guy at the payphone. So that dates the cartoon.

[A]

Sure.

[B]

And he's shouting into the payphone, I don't care what your earnings estimates are, sell. And in the background is a burning factory.

[A]

Right, right, right, right, right, right.

[B]

Exactly.

[A]

So you better have a good reason.

[B]

He had a good reason.

[A]

He had a good reason. He was looking at the factory legit burning down. So I think that you don't want to live in one world or the other. The one world is things will always stay the same. And that goes back to the beginning of the conversation, Right. Like, there's a pace of change. Right. And we don't want to be one of those people who's just like, it's all as it has been, it always will be. That's going to be a really bad place to live. But you also don't want to be like, it's different now, it's different now, it's different now, it's different now, it's different now. So that's why it's always, start with the base rate, assume equilibrium. Which just means that that base rate is going to hold. Obviously there's variation around it, but that should be your best guess. Right. And this is one of the things that I always caution people about, particularly when we're thinking about, for example, valuations. Right. Like, okay, you're looking at a company that's valued three times higher in terms of multiple, which is a lot higher, three times higher in terms of multiple to earnings than any company that's been like it before. Just because people are like, yo, we're so excited about this company, you have to understand that there's just a lot of contagion and social proof that could be happening there. You better be able to tell me why. Why is this company different than all these other companies? Why does it deserve a different valuation? And the example that I give to try to illustrate that is if you were looking at the price to earnings ratio for Amazon, and you were saying, well, okay, it's a retail store. Walmart is 4 to 6 times earnings. And here's Amazon, which is like 25 times earnings or whatever, right? Yeah. Then you may say, well, that's just wrong. But if you're like, well, maybe there's something different about this company, but you have to understand that, yes, in that particular case, I'm giving you an example where there was something different. Right. It's not bound by physical space and distribution is different. It's online, it can do all sorts of other stuff, so on and so forth, that it may be actually deserving of that higher ratio. In that case, you'd be correct. You also have to understand that under those circumstances, you have a bias toward thinking that it's different. So whatever explanations you're coming up with for why it's different, you should be suspicious of. And what I would really recommend, if you really want to get a good answer about it, is to ask six very, very smart people to answer that question independently. Don't all be in a room together, just have them do it independently. Why do you think this is different? And then you should actually add some context onto it. Imagine it's a year from now and the stock is cratered. Why do you think that is? Why do you think people got it wrong? Imagine it's a year from now and that price seemed to be reasonable. Why do you think that is? What do you think people got right about the price? And have six really smart people independently answer that question and you're more likely to get to whether that's really a dislocation or not.

[B]

That's actually a good AI prompt too, by the way. Rather than tell the AI, no matter which model you're using, say, what type of people should I ask about this problem? And then it generates all the different characters, attaches different probabilities to them. That's another little trick that is very helpful. What? Please attach a probability to this and you get the different actors, you get cognitive diversity and you get much closer to an interesting answer from the AI because it defaults to the dead middle right. And hallucinates anything that is inconvenient.

[A]

Yeah, I would actually add a little color to that, depending on the model that you're using, is that it defaults to whatever is going to get you to use it more. To be fair, that's not true of all of them, but for many of them, for the majority of them, I think. Is it perplexity that doesn't do that?

[B]

Perplexity generally does not. We have our own AI lab and we have them all in there. And perplexity Generally doesn't do it.

[A]

So I think that tends to be.

[B]

Better on that terribly. However, you can prompt even the most sycophantish.

[A]

But that's why we want to teach about prompts to kids, right? Because that's the thing. It's like you have to take control over this.

[B]

Oh, absolutely.

[A]

What people don't understand, and this is true of any social media use, is you have to think about AI the same way as social media. Right? It wants you to engage with it, right.

[B]

It's max. It's maximizing the objective function for. Or you keeping doing it.

[A]

You keeping doing it, which means that it's gonna tell you whatever's gonna keep you doing it, right? So in the same way that if I see some random, like, you know, you see little clips of things all the time on social media where they're clipping out like 20 seconds of an interview if it's not particularly high stakes, whatever. But if you're gonna change your behavior or your opinion or your something based on, on that 20 second clip, I would really highly recommend you go look at the whole interview, right? Like, please go look at the whole interview because there is no context around this thing. Right. And I think that you have to view social media the same. I mean, you have to view LLMs the same way, right? When you ask it something and it tells you something, you better try to get the whole interview right. Okay? Only cite scientific journals. Give me the citations with the links so that I can go look at them. Show me exactly where you're drawing this conclusion from. What if this conclusion that you're giving, what if what you're telling me is completely wrong? Why do you think that could happen? I've actually just done this with the Google AI where I've prompted it twice differently. I actually did it recently. I was trying to find out if a celebrity was still married to a another celebrity. And I asked it in a certain way and it was like, yes, they're still married. And then I asked it in a different way and it was like they got divorced two years ago. The actual answer was they were still married. Which, which I actually knew, but it was like amazing. It was within like a second of each other, you know, that it gave me a completely different answer. And so you, you should just act like if I asked in a different way, it would give me a different answer. So what would be the way that I would ask it that might get me a different answer? Because I want to see the whole clip it.

[B]

Yeah, I love that the last Thing that I want to talk about is another little bugaboo of mine and that is self selected samples. When that book came out, the Millionaire Next Door people, this is the greatest book like this is. And so I read it. I'm like, well, his sample was completely self selected.

[A]

Of course there's like, you know, it's like how many books are like the five Habits?

[B]

Exactly. From good to great.

[A]

Oh my God, Literally, like, I'm not for burning books. I'm just. Please don't read them.

[B]

But why does it work so well? Why did they become bestsellers?

[A]

Can I tell you something? This is why I'm writing this book. Because this problem is in the book and it's not just when people are trying to sell you. So it's a natural proclivity of us as human beings to say, if I want to have a good outcome, I should look at things that the good outcomes and try to figure out why it goes to that explanation. Right, sure. So I'm going to look at all these successful people. I'm going to figure out the things that they're doing and then if I do those things, I'll be successful too. Now, separate and apart from. Well, you have to understand there was a lot of luck in there. Like Bill Gates could do all the things he does today. And if he was born 30 years later, I'm sorry, bud. Right. I'm sure you're a very smart and wonderful person, but you're not having that result. And if I ran Bill Gates, if I Monte Carlo'd Bill Gates what percentage of the time.

[B]

Yeah, I love doing.

[A]

Right. Like let's Monte Carlo Bill Gates when he's born at the exact same time. So it's really hard for people to wrap their head around them again because we're determinists. So we think you will always have the result that you have and that the things that you do will create that result. Right. Again, looping right back to the beginning of the conversation. So it's this natural thing. It's this natural thing to look at the survivors and what people again, compared to what? What about people who do those exact same things and die? We have to look at the whole thing. Yeah.

[B]

No, no, no, no.

[A]

I completely agree. I want to look at 50 years, not five.

[B]

Right.

[A]

So working with a client and they were like, well, okay. We generally don't like to hire people that we would call just like difficult to work with. Okay. So let's think like you don't really like their Ocean score, for example. Right. So they were like, but you know what? We looked at our most successful engineers and 80% of those engineers were difficult. We were looking like our top 20% performers for engineers, and 80% of them were difficult. So they were like, we want to actually start changing sort of signal detection for hiring engineers. And we want to add this to sort of say, like, if. If they're difficult, we want to be more likely to hire them. Right? Okay, so this is what I'm saying. It's not just someone who's trying to sell a book. Okay. So anyway, so they say this to me because I was working on the hiring process with them and I said, oh, that's interesting. Did you ask the same question about how Difficult your bottom 20% engineering hires were? And they were like, what? And I said, well, look, I just don't know whether this is a quality of successful engineers or just.

[B]

Or engineers.

[A]

So anyway, they went and did that and of course it was literally the same. Just 80% had this quality. Right. If we actually get over our bias, remember I said that one of the things that brings explanatory satisfaction is you think, oh, I've learned something that other people don't know. If we get over our bias, that someone who tends to be disagreeable actually does better in this job and we should overcome our bias. And they're trying really hard and they don't understand that. They just went, wrote good to great. Right? Like, they don't get it. Oh, that's the millionaire next door. So, you know, I think that this is part of the problem is we want to believe that we can come up with an answer. And the natural way to do that is to say, well, let's look at the successful ones. Right? And you have to again say, well, I don't want to just look at these five years. I want to look at all 50. And that's actually going to help me because God forbid I start waking up at 4 in the morning because I think that that's going to make me a billionaire. Right.

[B]

It is one of my hottest buttons. I was sitting here on that couch and I saw one of those listicles. This is like 2018, 2019, and I just lost it. Yeah, like your head explodes. Big thread on Twitter. Like, don't, please, please just stop. Don't do this.

[A]

Yeah, There's a famous example that is like in this world about vitamin E. E. So in the 90s, it was like, oh, like people who take vitamin E are super healthy, so you should start taking vitamin E. And then that result was published Right. And more people were taking vitamin E. Anyway. When you do a randomized controlled trial, vitamin E taking vitamin E supplements is net bad, Right? So the question is why? And it has to do with the sampling error problem, right? Like well, the people who are taking vitamin E are self selected. So if they're taking a vitamin supplement, it means that on average they're healthier because they are more interested in health. So even if you reduce their health a tiny bit without knowing it, they're still going to appear healthier than the average person who's not taking vitamin E.

[B]

Because of the self selection bias.

[A]

Exactly. So you have to do a randomized controlled trial where you're like you're taking vitamin E, you're taking not, and then let's see if it's net good or bad. Right? And that gets you to the appropriate comparison. And then it turns out like vitamin E supplementation is actually net bad. But we only know that, right, if we do a randomized control trial. And this is how we end up like doing things like taking hydroxychloroquine because there's some random small sample, whatever, without a control group. And then people are like, I'm going for it. And here's the thing is that I'm fine if it's a free roll, right? I don't care if you're going off a small sample. That is suggestive data. If there are no negative side effects to the thing that you're doing. I'm really not. But we know that that's not true of hydroxychloroquine. Hydroxychloroquine has very, very bad adverse effects. And so the people who do take hydroxychloroquine are basically saying, I'm willing to bear those adverse effects in order for the benefit that this drug is going to offer me right now there are some things that you can do that really just don't have any downside. One of them that is very, very close to no downside are statins. Although I just want to say for the record, I'm not a medical doctor, so please don't go take statins because I told you to go talk to your doctor about it. But my understanding as a non medical professional is that hydroxychloroquine has very bad side effects and statins can have some muscle weaknesses, some issues that are pretty rare. It tends to be in people who are sicker. So if you're a healthy individual, it's a relatively free.

[B]

And I've read recently actually just on Statins, the newer class of, of statins has a lower incidence of the muscle weakening.

[A]

Yes. And you can take Qanol and things like that in order to resolve it. So the point is that if you're on the fence and you're like, oh my cholesterol's high, should I take this thing or not? And I would give this advice for any drug is say, well, what are the side effects? What are the things that can go wrong? And when the doctor says not too much, you can be like, okay, whatever. Right. But if the doctor says, ooh, a whole hell of a lot, you should be like, well, can you show me the randomized control trial, please?

[B]

Exactly.

[A]

Because I would actually like to understand if this is effective and then I would like to understand if it is effective. How effective is it? What's the probability of the adverse side effect? What's the probability that I'm going to get the benefits that I see? And now I can actually say, am I lengthening my life or shortening it? So you better go through all of that. But when the answer is meh, not too much, then I don't really care. It's actually a similar problem to what you're asking about the millionaire next door. Well, what about all the people who weren't millionaires, right? Did they have the same habits? Did they do the same things? What's the probability, given that you're doing those things, that you. Is it better than average? There's all these questions that you have to ask of that and you should have been asking that of that drug as well, given that there were bad things that could occur from it. So if somebody says if you work out, let's say if you take a brisk 30 minute walk five times a week, you're more likely to become a millionaire, then go do it. Because actually, independent of any millionaireness, taking walks is good for you. But I would really like it if you didn't believe it was going to make you a millionaire. That's the only thing. But go and walk. Maybe it'll help, I don't know, but it will help you in other ways.

[B]

I love both the theme and when is the book coming out?

[A]

Well, so the manuscript isn't even due until June, so it's going to be out in Q4 of 2026 or maybe Q1 of 2027 would be my guess. So you have a little bit to go.

[B]

I really want you send me the manuscript.

[A]

I will.

[B]

I would love to read. Sounds like I've loved all your books as you Know, well, thank you. But I love this book because I think it's incredibly necessary.

[A]

Well, particularly in the information environment we live in.

[B]

Exactly. And, and I just think anything that can be helpful to people who are drowning, that's a good thing with very little downside.

[A]

Yeah. Like your earlier argument, I just want to be clear, like there are no equations in the book. It's literally just, hey, ask this question. And it's, it's just a checklist of questions that you can ask that will allow you to avoid what I always say, impaling yourself on the data. Please don't impale yourself.

[B]

Well, like back to evolution though, we, we evolved to make stories click. It is I, you know, when I was selling quant investing, what did I do? I told stories about it. I, and I used to joke, I'm going to tell you stories about why you shouldn't listen to stories when making investment decisions. Why you. But I never put numbers like, no, because like people just like totally glaze over.

[A]

Right. And I think that the important thing is that this is the mind where, that we're born with and the idea that you can overcome the types of errors that we're going to make by sheer. Well, I know about it now and so I'm fine or whatever. I mean it's hilarious. And so you have to start to think about how do I build structure into the way that I'm thinking so that I can mitigate the errors that I might be making. That's always been my approach in my writing. There is nowhere in the book corporate quit that I say, oh, you know about the sunk cost fallacy, so you're fine. In fact, I say quite the opposite instead I say, look, you have to start making pre commitment contracts. You have to set kill criteria which are just stopping rules, right? You have to follow them. You need to write them down on a piece of paper. You've got to say, if I observe this signal, I will stop doing the thing that I'm doing. And that's going to really help you to get to these decisions to stop things faster. Because you can't just. The one that was always hilarious to me was investors who would say, oh yeah, I know about the sunk cost fallacy. So every morning I just imagine, what if I didn't own these things, would I buy them again? And I'm like, well that's nice, but you do own them, so that doesn't work. So if there were no transaction costs, then I would say, if there were no transaction costs, I would just tell you well, actually sell it. Just sell everything and then decide whether you want to rebuy it. Now, obviously, unfortunately, there are transaction costs, so that becomes impractical. But you can set rules around when would you sell and when would you not that are going to be much more helpful than your nice little thought experiment, which is cute, but it's not actually going to help you overcome anything. And I think that with this idea of this jumping to explanation that we do and then getting settled into that explanatory satisfaction, I can give you this conceptually all I want. I can tell you that these are errors that we're making left and right. I can tell you really fun stories about baseball players and orthopedists and vitamin E and education policy. I actually have one about school size and things like that. I can tell you this all the time. It's going to be obvious when I tell you this, that these errors are being made and you're going to be like, cool, done. I'm fixed now. And no, the book is actually saying you need to literally go through this checklist. And these are the set of questions that you have to ask of any piece of information that you see. And until you've asked all of these questions, you cannot draw a conclusion from it.

[B]

I love it. I love it. And can you program your personal AI to go through everything for you?

[A]

Well, you should be able to. And I think that that's actually something that AI ought to be helpful with.

[B]

Yeah.

[A]

You know, you can input a checklist into the. The AI and then the AI can actually tick it off for you and make sure that you're actually going through that. Now, I wouldn't necessarily trust the AI to answer the question that you're asking unless you're a very good prompter of AI and you do it. But it can at least get you to ask the question. Exactly right.

[B]

So which is kind of a forcing function. Right. It's like, why is it saying that? Right. Because when you were going earlier about getting people about learning about how much they actually do know about dogs. Right. It kind of the same thing. Right. If you have that on your AI, I just kind of think, how cool would it be if I had that and I'm reading an article and it's running in the background, that would be really cool.

[A]

So that's actually that type of sort of decision copilot kind of thing is something that we're actually working on at the Alliance.

[B]

Oh, very cool.

[A]

We're very lucky to have had Eric Horvitz just join the board from Microsoft. And he actually is working on a, he's been working on a tool like that that we're hoping to be able to translate for younger people as well.

[B]

Fantastic. Annie. I always have so much fun when I'm talking to you because you know so much about so many things. I love the linguistic thing.

[A]

That was just, that was.

[B]

I'm going to be just thinking about for the rest of the night.

[A]

I like to surprise. I like to surprise.

[B]

Well, you get one last chance. I don't know if you remember from the last time you were on because it was a long time ago, but we ask. We're going to wave a wand. We're going to live in magic world. We're going to make you empress of the world. You can't kill anyone. You can't put anyone in a re education camp, but what you can do is speak into a magic microphone and you can incept the entire population of the world whenever their morning is. They're going to wake up with the two things you're about to incept for me and our audience. That they're going to wake up and they're going to say, you know, I, I, I've never really acted on these ideas, these morning ideas that I get, but this time I'm going to be different. These two ideas that you've incepted in them, they're going to think are their own and they're going to act on them. What two I thinks are you going to incept into the.

[A]

Oh my gosh, this is so hard. I don't know what I answered last time. If I could actually get people to think probabilistically, like, please, that would be so great. Imagine how much better it would be to watch the news, right? Like, oh my gosh, that'd be so much better. Just view the world as it is. It's probable, isn't it? First of all, you won't be so freaking surprised and probably not as angry all the time. You think would be good. And I think the first explanation that comes to your mind for what you see is not necessarily the right one. Right? Like, you have to be much more skeptical. And I think that that would, I think that that's a really big one because we, that all this information is floating around and you just think you know what it means. It's like unless you actually know how to ask a question of that data, you just don't think about how many political arguments are based on the exact same piece of information and you're interpreting it one way, and I'm interpreting it another. And we never stop to think, like, well, maybe we're just not interpreting it right.

[B]

Yeah, maybe we're wrong.

[A]

Maybe we're wrong. I'm just gonna. I just wanna add a third one in. Just. But this is not me. I'm gonna attribute this to some. Somebody else.

[B]

Okay.

[A]

I had a conversation with Lori Santos, who's wonderful. She does all. You know, she's the happiness person over at Yale. I asked her, like, so sort of, what's the best thing you think that somebody can do to be happier? And her answer was, when you're with other people, don't even have the phones anywhere near where anybody can see them, because just having it here makes the conversations with worse. Which I've actually implemented in my life. But the one that really changed my life was get out in nature. I'll go out with my dog for two hours, and I literally don't touch my phone. I don't think anything has had such a profound impact on my happiness. More than that.

[B]

Now you got to tell me, what kind of dog do you have?

[A]

Oh, I'm trying to hide it because it was. Maybe people will take my class on Maven, and I know it's fine. I have a mini Bernadoodle named Otis who is one of the loves of my life. I think that my children and husband might say the love of my life. I'm not sure, but.

[B]

Annie, thank you so much. So great to see you. And this is so much fun to do in person.

[A]

Yeah, it's so nice to do it not on zoom. I was so happy that we could make this work.

[B]

Yeah. Thanks for coming.

[A]

Well, thank you for having me. It's always a joy.