Phil Agnew (0:00)

From curing scurvy to shaping billion dollar business strategies, this is the story of the most important experiment in science and its profound impact on our world. Today's episode is based on the insights and storytelling of May Contain Lies, how stories, statistics and studies exploit our biases and what we can do about it. It's authored by Alex Edmonds and is a fantastic book. All of that coming up.

Alex Edmonds (0:26)

So you want to be a marketer? It's easy. You just have to score a ton of leads and figure out a way to turn them all into customers. Plus manage a dozen channels, write a million blogs and launch a hundred campaigns all at once. When that's done, simply make your socials go viral and bring in record profits. No sweat. Okay, fine, it's a lot of sweat. But with HubSpot's AI powered marketing tools, launching benchmark breaking campaigns is easier than ever. Get started@HubSpot.com marketers.

Phil Agnew (0:58)

Hello everyone, and welcome to today's first Friday episode of Nudge. As I said in the main pod on Monday, this is a new sort of bonus episode in the week and it's a bit different from the typical episodes. It covers a specific topic, there are no guests, it's hopefully a bit shorter, and it's much more conversational as well. I don't have a full script to read through, so I'll just be talking about some of these topics more generally and hopefully providing a bit more anecdotes and information. And today we're talking about the experiment that every marketer should know. And the story really starts back in the 1500s, which was a time of real European exploration with iconic figures like Ferdinand Magellan, Vasco da Gama, Sir Francis Drake. These explorers crossed vast oceans like the Atlantic, Pacific and Indian, and they discovered and colonised many new lands. But these voyages brought wealth and knowledge. But it came at the cost of thousands of lives, primarily due to scurvy. Scurvy. So scurvy is a pretty horrible illness. It leads to lethargy, bleeding gums, losing teeth, ulcers and gangrene. If you get scurvy, you'll have excruciating pain in your muscles, joints and bones, and you'll eventually die from heart or brain haemorrhages. This had a devastating toll, killing over 2 million sailors between the 1500s and 1800s. And captains who went on these voyages expected to lose half their crew. There was a desperate search for a cure. Vasco da Gama's sailors rinsed their mouths out with urine. That was one of the expected Ways to try and treat this. Other treatments involved the elixir of vitarole. This was a sulfuric acid based remedy. There was another proposed treatment which was Electraary. This was garlic mustard seeds in the balsam of a Peruvian tree. These remedies were chosen without any system or logic. They were random, basically, often influenced by the resources available or or the severity of the disease. In other words, there was no real science behind any of the cures. And it's not a surprise that none of these early remedies worked. That was until James Lind came in. So James Lind was a ship doctor on HMS Salisbury. He conducted the first ever documented randomised control trial. This is an RCT. For his trial, he divided 12 scurvy patients into six pairs, giving each a different remedy. So some received an elixir of Vitarol, others received the electre. Some received cider, others received vinegar, seawater. And then the final group of six received citrus fruits, two oranges and one lemon. James Lin actually found a successful result. The pair who were assigned the citrus fruits recovered rapidly. One of them returned to duty and the other was assigned for caring Lyn's other patients. This was the first evidence that citrus cured scurvy. And this evidence would go on to save millions of lives. This was the Power of Iran randomised controlled trial. So before Lind came in, all of the remedies were endogenous. The decisions were influenced by factors like the severity of the illness or the available resources. This created some confounding variables that obscured the true efficacies of treatment. People could stumble upon the fact that lemons might help them, but they weren't able to find that systematically. Lind's approach was to make the treatment exogenous, so randomly assigned, and this would eliminate the bias. So, for example, he drew cards to assign the different remedies to these six different groups. And that ensured that there was no link between the assignment and the recovery likelihood. The randomization isolates the cause effect relationship between the remedy and the recovery. So this is the first documented rct, and it shifted observational studies which looked at correlation to intervention studies which could find causation. And when inputs are randomly assigned, common causes are eliminated, making a data source far more reliable. And this RCT was a bit of a phenomenon. It sort of changed science and started to be used in lots of different cases. And today it's used in fields like medicine, economics and social sciences. But there are limitations. Of course, causation isn't always enough, and other factors like the Placebo effect can also influence outcomes. So let's cover placebos. Austin Flint in 1863 addressed the problem of placebo effects in an early trial on rheumatism. He conducted a trial which used a dummy treatment and this was a diluted Quassia plant extract with no medicinal properties. He found that the placebo group who were given this dummy treatment improved just as much as those treated with the drug, which supposedly improved the treatment as well. And it suggested of course, that the drug had no effect, that people were only improving because of this dummy treatment, the placebo effect. Flint's worked pioneered the concept of using a placebo in experiments. And that meant there would be fair comparisons. It led to the development of modern blind RCTs where participants don't know if they're receiving the real treatment or a placebo. Now, these randomised controlled trials have helped humans learn an awful lot about how we operate. One of the most probably famous is an experiment on discrimination. In the US, African Americans face higher unemployment rates and earn 20% less than Caucasians. Now there are of course different theories that people could use to describe this. One could be that discrimination explains the gap. Another could be that there are differences in qualifications. So how do you figure out what is actually causing these higher unemployment rates? Well, using a randomised control trial, economists Marianne Bertrand and Sendhil Melainathan created an experiment to test this. They created a bank of resumes classified as high or low quality based on the education and experience. Then they randomly assigned names to these resumes. So some would get a Caucasian sounding name like Emily Walsh or Greg Barker, and others would get get an African American sounding name like Lukashi Washington or Jamel Jones. They sent these 5,000 resumes to 1,300 job ads in Boston and Chicago. And importantly, these resumes were random. The qualifications were consistent. The only thing that varied was the name. And the results were fascinating and a bit horrifying. The Caucasian names received one callback for every 10 resumes they sent, while the African American names received one callback for every 15 resumes they sent. Essentially, a Caucasian name yielded as many extra callbacks as having an additional eight years of work experience. This RCT showed causation and it demonstrated clear evidence for racial discrimination. In Alex Edmond's book, he talks about the problem with education and trying to figure out what types of schools are best. So for example, one of the problems is trying to figure out if merging school districts is better or keeping separate schools is better. So what is better? Having lots of schools merged into one Mega school or having lots of smaller schools which parents can choose between. Now obviously one easy way to do this is to do a randomized control trial, right? Take all the schools in a state, merge half of them, keep half of them separate and see how performance changes. But there are real problems with this type of experiment. Merging districts is very, very expensive. And, and if competition, for example, does improve performance, then the merged districts could have thousands of students who are negatively impacted just by this experiment. So there are real ethical concerns here. This is similar to smoking and cancer studies. If you force harmful conditions like forcing someone to smoke or denying the medical treatment, this is seen as unethical and it's not allowed. So here a randomised control trial isn't feasible. So what scientists do is they use something called an instrument. An instrument is a naturally occurring phenomenon that acts as a shock to the system. So what the instrument does is it causes a random change in the input without being influenced by the output. It enables observational studies without directly intervening. And this is something that Caroline Hoxby did with education. So she wanted to find out if different types of schools, so merged schools or multiple different schools, effective students performance. So is being an emerged district better than being in one with separate. And obviously she can't do a randomised controlled trial, but she could use an instrument. And the instrument she used was rivers. See, in the 18th century, rivers were big barriers to travel and they led to the formation of multiple school districts in areas with many different rivers. So if an area had lots of different rivers, it would have lots of different districts, lots of separate schools, and these districts still remain largely unchanged today. Whereas areas without rivers have a lot of merged schools because people can travel freely. Rivers serve as an exogenous variable affecting school choice, but it's unrelated to modern factors like parental engagement, for example. So for her experiment, Hoxbee analysed 30,000 schools in 316 metropolitan areas. She decomposed school choice into exogenous factors, which is attributed to rivers, and endogenous factors influenced by modern causes like parental involvements. Her findings were pretty conclusive. Areas with more districts, due to the fact that there are more rivers, had better short term and long term educational outcomes. So in the short term, children at areas with more districts improved their 8th grade reading scores and 10th grade math scores. And there were long term factors as well. Children who went to schools with more districts achieved higher levels of education and increased their income by the age of 32. The rivers allowed Hoxbee to prove causation and not just correlation between school choice and performance. These instruments can be used in other ways as well. There's a really interesting study with succession debates at companies. So the study looks to answer the question which is do family CEOs perform worse than external hires? So if you have a family run company, should you bring in your child to run the business or should you get an external hire? And there's a real challenge here because there are confounding variables like the company morale or brand issues that could explain poor performance and that could trump any difference that the CEO makes. So researcher Morten Benderson used an instrument to see if he could settle the succession debate. The instrument was the gender of the departing CEO's first born child. The relevance here is that having a male firstborn made it more likely for the CEO to choose a family successor due to gender bias and the primogentia that's unfortunately common. The smart thing about using the gender of the firstborn child is it's determined randomly during conception and is unrelated to firm performance. So it makes for a perfect instrument. The findings are quite fascinating. Companies with a male firstborn child are more likely to have lower profits due to the fact that they're more likely to pick a family CEO. This suggests that hiring a family CEO causes worse performance than hiring an external CEO. These two experiments are great examples of finding a valid instrument that's both relevant and random and allows the researcher to figure out causation without doing an amoral study. So that is the history of randomised control trials up to now. After this break, we'll explain why they are so important to marketers. Create like the Greats hosted by Ross Simmons, is brought to you by the HubSpot Podcast Network, the Go to audio destination for business professionals. In each episode, Ross dives into the stories behind some of history's greatest creations and creators. He unpacks the strategies, processes and lessons that shaped them. His episodes are engaging, his insights are practical, and he's been living these principles that he shares for over a decade. If you enjoy exploring creativity, the history of creators and actionable advice, this podcast is for you. Listen to Create like the Greats wherever you get your podcasts. Now, most of you working within businesses will know that randomized controlled trials are often just simply known as a B testing in the business world. And we know that they are invaluable tools for marketers and business leaders. They enable data driven decisions by comparing two different strategies against each other. And there are some fantastic examples of businesses that use RCTs to improve their work and grow. Subway famously did this before launching their 5 foot long sandwich promotion worldwide. Subway conducted an RCT by offering the promotion in select locations and comparing the sales where the promotion was held to control locations without the promotion. The test demonstrated that there was a substantial increase in the sales in the test locations and this validated the effectiveness of the promotion and justified its nationwide rollout. And another example is Wawa Food Markets now. Wara employed RCTs to evaluate the effect of introducing new products in select stores and comparing the sales and customer feedback to control stores without the new offerings. This approach allowed the company to make informed decision about product rollouts and ensured successful additions to their product lines. The takeaway here is that RCTs can be, well, very important to business. They allow for data driven decisions, they allow you to mitigate risks, they allow for resource optimizations and to continue improving Randomized control trials has changed the way business operates and they're vital for understanding how the human mind works and for understanding behavioural science. Whenever I look for studies to share on this show, I always try to make sure the study has a randomized control trial in it because it is one of the most effective ways of truly understanding human behaviour. So that is the history of the experiment, which I think has changed science and has changed many of the businesses that we work at and many of the businesses that we use. I really do hope you enjoyed this episode again. It was inspired by the fantastic book May Contain Lies by Alex Edmonds. I'll be trying to get Alex on the main podcast hopefully later on in this year and let me know what you think of this episode. It's a bit different from the normal one, so please do let me know. The easiest way to do that is to subscribe to the newsletter. Go to nudgepodcast.com and click Newsletter in the menu. You'll get an email from me which you can respond to with any of your feedback or you can find me on LinkedIn. I'm Phil Agnew on there. Thank you for listening folks. I'll be back on Monday for another episode of Nudge. Cheers.