Podcast Summary: 3 Takeaways – The Science and Skill of Superforecasting (#230) with Warren Hatch

Introduction

In the episode titled "The Science and Skill of Superforecasting" (Episode #230) of the 3 Takeaways podcast, host Lynn Thoman engages in a compelling conversation with Warren Hatch, CEO of the Good Judgment Project and one of its top superforecasters. The discussion delves into the methodologies behind superforecasting, the shortcomings of traditional expert predictions, and practical applications of forecasting techniques in everyday life and significant global events.

1. The Importance of Forecasting

Warren Hatch begins by emphasizing the ubiquitous nature of forecasting in daily decision-making. He states:

"In a sense, every decision that we make is a forecast because we're going to be taking action and doing things to improve the odds that we're going to get our desired outcome, whatever that might be." [02:53]

Forecasting is presented not just as a tool for major life choices but as an integral process underpinning thousands of everyday decisions.

2. Common Forecasting Mistakes

Hatch critiques the prevalent use of ambiguous language in forecasting, which often leads to misinterpretation. He cites a historical example involving President Kennedy:

"When Kennedy came into office and inherited a plan to invade Cuba and topple the regime, he asked his advisors, 'Will this succeed?' And they said, 'There's a fair chance it will succeed.' Now, it turns out Kennedy had in mind north of 50%, the analysts had in mind something more like 25%..." [03:24]

This anecdote illustrates how vague probabilities can create significant misunderstandings, highlighting the need for precise numerical probabilities in forecasting.

3. Experts vs. Superforecasters

A critical comparison is made between traditional experts and superforecasters. Hatch explains that expertise does not inherently translate to forecasting accuracy:

"Most experts do not go through that process... when you ask them for a probabilistic forecast, experts generally tend to assign higher probabilities to events in their area of expertise than actually occur." [05:40]

He underscores that superforecasters, through their systematic approach, often outperform experts by maintaining calibrated probabilities and relying on crowd-sourced insights.

4. Crafting Good Forecastable Questions

Effective forecasting starts with well-defined questions. Hatch stresses the importance of clarity and verifiability in question formulation:

"You want to make sure that everybody understands it in the same way. You want to be sure that it's actionable, it's useful... It needs to be crisp and tight, it's actionable, it's verifiable, and we all can agree if it happened or it did not." [07:00]

He provides an example of a poorly framed question about canal construction in Nicaragua, illustrating how ambiguity can hinder accurate forecasting.

5. Applying Superforecasting in Personal Life

Hatch offers practical advice on incorporating superforecasting techniques into personal decisions. He suggests breaking down complex decisions into smaller, forecastable components and utilizing the "outside view" or base rates to inform predictions:

"Instead of going to the particulars right away, we want to zoom out. How do things like this usually go? What Daniel Kahneman calls the outside view." [10:46]

This approach allows individuals to anchor their forecasts in historical data before refining them with specific details.

6. Steps to Better Forecasts

To enhance forecasting accuracy, Hatch outlines a four-step process:

1. Start with a Base Rate: Utilize historical data to inform initial estimates.
2. Make a Comment: Document the rationale behind your forecast to clarify and refine your thinking.
3. Exchange Views: Share and discuss your reasoning with others to uncover additional insights.
4. Update Your Forecast: Adjust your predictions based on new information and perspectives.

He emphasizes that this iterative process leads to more accurate and reliable forecasts.

7. Lessons from Real-World Forecasting Events

Reflecting on the COVID-19 pandemic, Hatch illustrates how superforecasting outperformed traditional expert models:

"Our forecasters... did much better at identifying how quickly the pandemic would spread. They also did very well at identifying when a vaccine would become available." [12:23]

He points out that static models falter in times of rapid change, whereas superforecasters adapt by recognizing subtle shifts and emerging patterns.

8. Handling Black Swans and Dark Gray Swans

Hatch introduces the concept of "dark gray swans"—low-probability, high-impact events that are challenging to predict but critical to monitor:

"These things may be small probability, but high impact. And somehow getting those small probability items on your radar can help you be better prepared." [14:37]

Examples include wildfires, AI developments, and geopolitical tensions in regions like Korea. Superforecasting techniques are particularly effective in anticipating such events, allowing for proactive preparation.

9. The Role of AI in Forecasting

Addressing the intersection of artificial intelligence and forecasting, Hatch acknowledges the advancements in AI but maintains that human superforecasters currently outperform AI models:

"Phil Tetlock did a big research project... none of them did better than the superforecasters. And the best of them still lag by 20%." [17:54]

He advocates for a collaborative approach where AI aids humans in refining forecasts, rather than viewing AI as a competitor.

10. Three Key Takeaways

Concluding the episode, Warren Hatch presents three actionable takeaways for the audience:

1. Slow Down: Engage in deliberate, analytical thinking ("system two thinking") for important decisions to challenge assumptions and reduce overconfidence.

   "Start having a dialogue with yourself. You can challenge your thinking yourself." [18:58]

2. Change Your Mind: Remain open to updating your forecasts as new information emerges, fostering adaptability and continuous improvement.

3. Keep Score: Track the accuracy of your forecasts to receive feedback, enhance calibration, and improve decision-making over time.

   "You need to see if your forecasts occur with the frequency that you think they do." [18:58]

Conclusion

The episode provides a deep dive into the science and methodology of superforecasting, contrasting it with traditional expert predictions and highlighting its applicability in both personal and global contexts. Warren Hatch's insights offer listeners practical strategies to enhance their forecasting skills, ultimately leading to better-informed decisions in various aspects of life and career.