Nudge Podcast Episode Summary
Title: The Experiment Every Marketer Should Know
Host: Phil Agnew
Release Date: January 10, 2025
Introduction to the Pioneering Experiment
In the latest episode of Nudge, host Phil Agnew delves into the transformative power of randomized controlled trials (RCTs) and their profound impact on both science and business. Drawing inspiration from Alex Edmonds' insightful book, May Contain Lies, Phil sets the stage by highlighting how RCTs have revolutionized our understanding of causation versus correlation.
Phil Agnew [00:00]: "From curing scurvy to shaping billion dollar business strategies, this is the story of the most important experiment in science and its profound impact on our world."
The Historical Significance of RCTs: James Lind's Scurvy Experiment
Phil begins by recounting the harrowing struggle against scurvy during European exploration in the 1500s. The disease claimed millions of lives, prompting captains to experiment with various, often ineffective remedies.
Phil Agnew [00:58]: "There was a desperate search for a cure... These remedies were chosen without any system or logic."
Enter James Lind, a ship doctor who conducted the first documented RCT aboard the HMS Salisbury. By randomly assigning different treatments to scurvy patients, Lind discovered that citrus fruits significantly accelerated recovery, laying the foundation for modern scientific experimentation.
Phil Agnew [04:30]: "Lind's approach was to make the treatment exogenous, so randomly assigned, and this would eliminate the bias."
Advancing Scientific Rigor: The Introduction of Placebos
Phil transitions to the evolution of RCTs with the introduction of placebos. Highlighting Austin Flint's 1863 trial on rheumatism, Phil explains how the use of dummy treatments revealed the placebo effect, ensuring fair comparisons in experiments.
Phil Agnew [10:15]: "Flint's work pioneered the concept of using a placebo in experiments, leading to the development of modern blind RCTs."
Modern Applications of RCTs: Uncovering Discrimination and Beyond
The discussion shifts to contemporary applications of RCTs, emphasizing their role in uncovering systemic issues such as racial discrimination in employment. Phil cites the groundbreaking study by Marianne Bertrand and Sendhil Mullainathan, where racially stereotyped resumes received fewer callbacks, demonstrating clear evidence of discrimination.
Phil Agnew [20:45]: "This RCT showed causation and it demonstrated clear evidence for racial discrimination."
Overcoming Ethical Constraints: The Use of Natural Instruments
Addressing the ethical limitations of traditional RCTs, Phil introduces the concept of instrumental variables. Through Caroline Hoxby's educational study, he illustrates how natural phenomena like rivers can serve as instruments to analyze the impact of school district configurations on educational outcomes without direct intervention.
Phil Agnew [31:20]: "Rivers serve as an exogenous variable affecting school choice, but it's unrelated to modern factors like parental engagement."
Similarly, Phil discusses Morten Benderson's research on CEO succession, where the gender of a departing CEO's firstborn child was used as an instrument to assess the performance impact of family versus external hires.
Phil Agnew [38:50]: "The instrument she used was the gender of the departing CEO's firstborn child... it makes for a perfect instrument."
RCTs in Business: From Theory to Practice
Post-advertisement break, Phil bridges the historical and scientific discussions to practical business applications. He likens RCTs to A/B testing in marketing, underscoring their value in enabling data-driven decisions and mitigating risks.
Examples Highlighted:
-
Subway's 5-Foot Long Sandwich Promotion: Subway conducted an RCT by introducing the promotion in select locations before nationwide rollout, which resulted in a substantial increase in sales.
Phil Agnew [48:30]: "Subway conducted an RCT by offering the promotion in select locations and comparing the sales where the promotion was held to control locations without the promotion."
-
Wawa Food Markets' New Product Introductions: Wawa utilized RCTs to assess the impact of new products in specific stores, ensuring successful and informed product rollouts based on sales and customer feedback.
Phil Agnew [51:15]: "This approach allowed the company to make informed decisions about product rollouts and ensured successful additions to their product lines."
Conclusion: The Enduring Legacy of RCTs
Phil concludes by reaffirming the indispensable role of RCTs in both scientific inquiry and business strategy. He emphasizes that RCTs not only enhance our understanding of human behavior but also drive effective and efficient decision-making in various fields.
Phil Agnew [54:40]: "Randomized control trials has changed the way business operates and they're vital for understanding how the human mind works and for understanding behavioural science."
Phil encourages listeners to explore further by engaging with May Contain Lies and shares his intention to invite Alex Edmonds for a future episode, inviting feedback through the podcast's newsletter and LinkedIn.
Phil Agnew [57:00]: "I'll be trying to get Alex on the main podcast hopefully later on this year and let me know what you think of this episode."
This episode of Nudge offers a comprehensive exploration of randomized controlled trials, tracing their historical roots, methodological advancements, and practical applications in modern business. Phil Agnew effectively illustrates how the principles of RCTs empower marketers and business leaders to make informed, evidence-based decisions that drive success.