Policymaking Is Not a Science — Yet (Update)

Summary

Freakonomics Radio Episode Summary: "Policymaking Is Not a Science — Yet (Update)"

Episode Information

• Title: Policymaking Is Not a Science — Yet (Update)
• Release Date: April 9, 2025
• Host: Stephen J. Dubner
• Network: Freakonomics Radio + Stitcher
• Description: An exploration of the challenges in translating scientific research into effective public policy, focusing on the scalability of interventions across various domains such as medicine, education, and economic policy.

---

Introduction

In this bonus episode of Freakonomics Radio, host Stephen Dubner revisits the theme "Policymaking Is Not a Science" with updated insights and data. Building upon a two-part series discussing "sludge"—the frictions that impede effective policy implementation—this episode delves into the complexities of scaling research-based interventions into real-world applications.

---

Key Contributors

• Stephen Dubner (Host)
• Dana Susskind (Physician Scientist at the University of Chicago): Specializes in cochlear implants and early childhood education.
• John List (Economist at the University of Chicago): Pioneer in field experiments and the credibility revolution in economics.
• Patti Chamberlain (Senior Research Scientist at Oregon's Social Learning Center): Founder of Treatment Foster Care Oregon.
• Lauren Suplee (Deputy Chief Operating Officer of Child Trends): Expert in implementation science.

---

Understanding the Problem: Scaling Scientific Interventions

Medical Adherence Challenges

• Dana Susskind discusses the remarkable success of cochlear implants in restoring hearing to profoundly deaf children. (02:12)

  • "The cochlear implant is a remarkable piece of technology... but how often we fail to take advantage of these advances." (02:41)

• Despite technological advancements, adherence remains a significant issue:

  • "About a third of all Americans have high blood pressure... Only 50% actually are controlled." (03:39)

• John List highlights human behavior as a critical barrier to adherence:

  • "Prescription adherence is a very difficult nut to crack." (04:18)

Scaling Failures Across Sectors

• John List and Dana Susskind examine why interventions that work in controlled research settings often falter when scaled:
  • "Solutions that look foolproof in the research stage are failing to scale up." (05:59)

---

Case Studies in Scaling Interventions

Parent Academy Initiative

• John List describes the Parent Academy program in Chicago Heights, which successfully improved children's cognitive and executive function skills within months. (10:17)

• Failure to Scale: When the program was introduced in London, parental uptake was minimal, leading to its failure despite initial success. (10:27)

Treatment Foster Care Oregon (TFCO)

• Patti Chamberlain explains the development and scaling of TFCO, which places individual children in family homes rather than group settings, resulting in better outcomes and lower costs. (18:44)

• Scaling Challenges: Initial attempts to implement TFCO across 15 sites faced systemic barriers due to conflicting policies across child welfare, juvenile justice, and mental health systems. (20:24)

• Resolution through Fidelity Standards: TFCO overcame scaling issues by developing strict fidelity standards and training protocols, ensuring consistency across multiple sites. (42:17)

---

The Science of Scaling: Implementation Science

• Lauren Suplee introduces "implementation science" as a field dedicated to studying how programs are integrated into real-world settings and how implementation quality affects outcomes. (22:54)

• Definition: "It's the study of how programs get implemented into practice and how the quality of that implementation may affect how well that program works or doesn't work." (22:48)

• Challenges Identified:

  • Voltage Drop: The reduction in program effectiveness when scaled up.

    • "Voltage drop essentially means I found a really good result in my original research study. But then when they do it at scale, that voltage drop ends up being, for example, a tenth of the original result." (30:33)

  • Fidelity: Maintaining the integrity of the original program during scaling.

    • "Measures of fidelity is a really critical part of the implementation process." (41:21)

---

Three Buckets of Scaling Failures

John List categorizes scaling failures into three primary buckets:

1. Lack of Evidence for Scaling

   • "Only 8% of those programs were actually backed by research evidence." (27:34)
   • False positives: Initial research findings may not hold without robust evidence.

2. Wrong People Studied

   • Studies may focus on populations that do not represent the broader target group.
   • "These are studies that have a particular sample of people that show really large program effect sizes. But when you scale, that effect disappears." (28:20)

3. Wrong Situation Used

   • Environmental and contextual changes alter program effectiveness.
   • "When you go from the original research to the scaled research program, we don't understand what properties of the situation or features of the environment will matter." (29:53)

---

Proposed Solutions for Effective Scaling

Rigorous Replication

• John List advocates for scaling programs only after multiple, well-powered independent replications confirm the original findings:

  • "We do not believe that we should scale a program until you're 95% certain the result is true." (34:52)

• Encouraging Replication: Rewarding scholars for replicating studies to ensure reliability.

Fidelity Standards

• Patti Chamberlain emphasizes the importance of maintaining fidelity through secure session uploads and rigorous coding:
  • "If they're not meeting the fidelity standards, then we offer a fidelity recovery plan." (42:40)

Adaptability and Humility

• Researchers and policymakers must remain flexible and willing to adjust programs based on real-world feedback:
  • "You shouldn't assume your original program was perfect; it will need adjustments." (43:12)

---

Quotes Highlighting Key Insights

• Dana Susskind (02:12):

  "Someone who is severely to completely profoundly deaf after implantation can have normal levels of hearing. And it is pretty phenomenal."

• John List (04:18):

  "Prescription adherence is a very difficult nut to crack."

• John List (07:32):

  "Now, my contribution in the credibility revolution was instead of working with secondary data, I actually went to the world and used the world as my lab and generated new data to test theories and estimate program effects."

• Patti Chamberlain (18:17):

  "We have to figure out how to use our own science to make better policies."

• John List (34:52):

  "We need to know what is the magic sauce."

• Dana Susskind (44:08):

  "Everybody's motivation at the end of the day is about trying to do good for the people they serve."

• John List (45:06):

  "So I do think inherently it is about people."

---

Conclusions and Forward Path

• Integration of Implementation Science: Essential for bridging the gap between research and policy by ensuring that programs can be effectively scaled without losing their intended impact.

• Cultural Shift in Academia and Policy: Encouraging replication and valuing fidelity over rapid implementation can lead to more reliable and effective policies.

• Collaborative Efforts: Researchers, policymakers, and practitioners must work together to understand and overcome the human-centric barriers to scaling.

• Vision for the Future: As John List eloquently puts it:

  "The world is imperfect because we haven't used science in policymaking. If we add science to it, we have a chance to make an imperfect world a little bit more perfect."

---

Additional Resources

For those interested in delving deeper into the research and methodologies discussed, Freakonomics Radio provides links to academic papers and further reading materials on Freakonomics.com. The full transcript of this episode is also available for comprehensive review.

---

Notable Audio Clips with Timestamps

• Dana Susskind on Cochlear Implants: (02:12)

  "My job is to implant this incredible piece of technology which bypasses these defective hair cells..."

• John List on Scaling Challenges: (10:17)

  "So if you want your program to work at higher levels, you have to figure out how to get the right people into the program."

• Patti Chamberlain on Implementation Barriers: (20:33)

  "When we tried to implement, we ran into tremendous barriers because if we satisfied the policies and procedures of one system, we were at odds with the policies and procedures in the other system."

• Lauren Suplee on Implementation Science: (22:54)

  "It's the study of how programs get implemented into practice and how the quality of that implementation may affect how well that program works or doesn't work."

---

Closing Remarks

This episode of Freakonomics Radio provides a comprehensive examination of why effective policymaking remains elusive despite robust scientific research. Through insightful discussions with leading experts like Dana Susskind, John List, and Patti Chamberlain, listeners gain a deeper understanding of the intricate challenges in scaling interventions and the critical role of implementation science in bridging the gap between research and real-world application.

---

Transcript

[Bank of America]

As America's leading business lender, bank of America is on your corner and in your corner. With $215 billion in business loans and over 3,700 business specialists across the nation, we help businesses thrive so communities prosper. What would you like the power to do? Learn [e-mail] LOCALBUSINESS bank of America Official bank of FIFA Club World Cup 2025 Copyright 2025 bank of America Corporation. All rights reserved.

[Amica Insurance]

At Ameca Insurance, we know it's more than just a house. It's your home. The place that's filled with memories. The early days of figuring it out to the later years of still figuring it out. For the place you've put down roots. Trust Amica Home Insurance AMEC Empathy is our best policy.

[Stephen Dubner]

Hey there, it's Stephen Dubner. We just published a two part series on what some people call sludge, meaning all the frictions that make it hard to fill out tax forms or find a healthcare provider or even cancel a subscription. One part of our series involved government sludge and how it interferes with getting policy done. The series reminded me of another episode we once made that I thought was worth hearing again. So we're playing it for you here as a bonus episode. It is called Policymaking is Not a Science. Yet we have updated facts and figures as necessary. As always, thanks for listening.

[Dana Susskind]

Usually when children are born deaf, they call it nerve deafness, but it's really not the actual nerve. It's little tiny hair cells in the cochlea.

[Stephen Dubner]

Dana Susskind is a physician scientist at the University of Chicago. And more dramatically, she is a pediatric surgeon who specializes in cochlear implants.

[Dana Susskind]

My job is to implant this incredible piece of technology which bypasses these defective hair cells and takes the sound from the environment, the acoustic sound, and transforms it into electrical energy, which then stimulates the nerve. And somebody who is severe to completely profoundly deaf after implantation can have normal levels of hearing. And it is pretty phenomenal.

[Stephen Dubner]

It is pretty phenomenal. If you ever need a good cry, a happy cry, just type in cochlear implant activation on YouTube. You'll see little kids hearing sound for the first time and their parents flipping out with joy. She lifts her up.

[John List]

Good job. She's smiley.

[Stephen Dubner]

Oh, that's great. She's so smiley.

[Dana Susskind]

Yeah, that's your ears.

[John List]

Yeah.

[Stephen Dubner]

The cochlear implant is a remarkable piece of technology, but really it's just one of many remarkable advances in medicine and elsewhere created by devoted researchers and technologists and sundry smart people. You Know what's even more remarkable? How often we fail to take advantage of these advances.

[Dana Susskind]

One of the most compelling examples is the issue of hypertension. About a third of all Americans have high blood pressure. First of all, the awareness rate is about only 80% of the total amount. Only 50% actually are controlled. We have great drugs, right? But you can see the cascade of issues when you have to disseminate, you have to adhere, et cetera, and the public health ramifications of that.

[Stephen Dubner]

Those blood pressure numbers are even worse today than they were when we first published this episode in 2020. Clearly, we still have not figured out how to get the science to the people who need it.

[John List]

Prescription adherence is a very difficult nut to crack.

[Stephen Dubner]

That's John List. He's an economist at the University of Chicago.

[John List]

They actually have to go and get the medicines, which a lot of people have a very hard time doing. Even though it's sitting next to your bed every night, people don't take it. And they don't take it because they forget. They don't take it because the side effect is a lot worse than the benefit they think they're getting. All of these types of problems. As humans, including myself, we do a really bad job in trying to solve.

[Dana Susskind]

All of us, our lives get busy, we forget.

[Stephen Dubner]

You wouldn't think you'd have an adherence issue with something like the cochlear implant. It has such an obvious upside, and.

[Dana Susskind]

Yet when I put the internal device in, it stays there, but it actually requires an external portion as well, sort of like a hearing aid. And that is the part where you see issues related to adherence. Just because I put the internal part doesn't mean that an individual or a child will be wearing the external part.

[Stephen Dubner]

In one study, only half of the participants wore their device full time.

[Dana Susskind]

I mean, we have figured through randomized control trials to understand causation, real impact in the small scale. But the next step is understanding the science of how to use this science, because you know, how you do it on the small scale in perfect conditions is very different than the messy real world. And that is a very real issue.

[Stephen Dubner]

Today on Freakonomics Radio. What to do about that very real issue. Because you see the same thing not just in medicine, but in education and economic policy and elsewhere. Solutions that look foolproof in the research stage are failing to scale up.

[Lauren Suplee]

People said, let's just put it out there. And then we quickly realized that it's far more complicated.

[Patti Chamberlain]

There might be something that you think would be great, but it's never going to be able to be implemented in the real world.

[John List]

We need to know what is the magic sauce.

[Stephen Dubner]

We'll go in search of that magic sauce right after this.

[Freakonomics Radio]

This is Freakonomics Radio, the podcast that explores the hidden side of everything with.

[Patti Chamberlain]

Your host, Stephen Dubner.

[Stephen Dubner]

John List is a pioneer in the relatively recent movement to give economic research more credibility in the real world.

[John List]

If you turn back the clock to the 1990s, there was a credibility revolution in economics, focusing on what data and modeling assumptions are necessary to go from correlation to causality.

[Stephen Dubner]

List responded by running dozens and dozens of field experiments.

[John List]

Now, my contribution in the credibility revolution was instead of working with secondary data, I actually went to the world and used the world as my lab and generated new data to test theories and estimate program effects.

[Stephen Dubner]

Okay, so you and others moved experiments out of the lab and into the real world, but have you been able to successfully translate those experimental findings into, let's say, good policy?

[John List]

I think moving our work into policymaking circles and having a very strong impact has just not been there. And I think one of the most important questions is how are we going to make that natural progression of field experiments within the social sciences to more keenly talk to policymakers, the broader public, and actually the scientific community as a whole.

[Stephen Dubner]

The way Lis sees it, academics like him work hard to come up with evidence for some intervention that's supposed to help alleviate poverty or improve education, to help people quit smoking or take their blood pressure medicine. The academic then writes up their paper for an incredibly impressive looking academic journal. Impressive at least to fellow academics. The rest of us, it's jargony and indecipherable. But then with paper in hand, the academic goes out proselytizing to policymakers. He might say, you politicians always talk about making evidence based policy. Well, here's some new evidence for an effective and cost effective way of addressing that problem you say you care so much about. And then the policymaker may say, well, the last time we listened to an academic like you, we did just what they told us, but it didn't work. And it cost three times what they said it would. And we got hammered in the press. And here's the thing, the politician and the academic may both be right. John List has seen this from both sides.

[John List]

Now, in a past life, I worked in the White House advising the president on environmental and resource issues within economics.

[Stephen Dubner]

This was in the early 2000s under George W. Bush.

[John List]

A harsh lesson that I learned was you have to evaluate the effects of public policy as opposed to its intentions.

[Stephen Dubner]

Because the intentions are obviously good. For instance, improving literacy for grade schoolers or helping low income high schoolers get to college.

[John List]

When you step back and look at the amount of policies that we put in place that don't work, it's just a travesty.

[Stephen Dubner]

List has firsthand experience with the failure to scale.

[John List]

So down in Chicago Heights, I ran a series of interventions and one of the more powerful interventions was called the Parent Academy. That was a program that brought in parents every few weeks. And we taught them what are the best mechanisms and approaches that they can use with their three, four and five year old children to push both their cognitive skills and their executive function skills. Things like self control. What we found was within three to six months, we can move a child in very short order to have very strong cognitive test scores and very strong executive function skills. So of course we're very optimistic after getting this type of result and we want the whole world to now do parent academies. The UK approaches us and said, we want to roll it out across London and the boroughs around London. What we found is that it failed miserably. It wasn't that the program was bad, it failed miserably because no parents actually signed up. So if you want your program to work at higher levels, you have to figure out how to get the right people and all the people, of course, into the program.

[Stephen Dubner]

Wow. If you had asked me to guess all the ways that a program like that could fail, it would have taken me a while to guess that you simply didn't get parental uptake.

[John List]

The main problem is we just don't understand the science of scaling.

[Stephen Dubner]

If you had to attach a noun to what this is the scalability blank, is it a problem? Is it a dilemma? Is it a crisis?

[John List]

I do think it's a crisis in that if we don't take care of it as scientists, I think everything we do can be undermined in the eyes of the policymaker and the broader public. We don't understand how to use our own science to make better policies.

[Stephen Dubner]

So John List and Dana Susskind and some other researchers are on a quest to address this scalability crisis. They've been writing a series of papers, for instance, the Science of Using Science towards an Understanding of the Threats to Scaling Experiments. A lot of their focus is on early education, since that is a particular passion of Susskind's.

[Dana Susskind]

I guess you could say I'm a surgeon by day and social scientist by night. My clinical work is about taking care of one child at a time. My research really comes out of the fact that not all children do as well as others. After surgery and trying to figure out the best ways to allow all my patients and really children born into low income backgrounds to reach their educational potentials.

[Stephen Dubner]

It is kind of like a superhero in reverse. During the day, you're doing the big, dramatic stuff. Then at night, you're going home to analyze the data and figure out what's happening.

[Dana Susskind]

I think that really the hard part is the night part. I love doing surgery. I adore my patients. But it's actually not as hard as many of the complex issues in this world.

[Stephen Dubner]

And was that a recognition that some kids after the surgery sort of zoomed up the education ladder and others didn't?

[Dana Susskind]

Yeah. It's not simply about hearing loss. It's because language is the food for the developing brain. Before surgery, they all look like they'd have the same potential to, as you say, zoom up the educational ladder. After surgery, there were very different outcomes. And too often that difference fell along socioeconomic lines. That made me start searching outside the operating room for understanding why and what I could do about it. And it has taken me on a journey.

[John List]

So Dana and I met back in 2012, and we were introduced by a mutual friend. And we did the usual ignore each other for a few years because we're too busy. And push came to shove. Dane and I started to work on early childhood research. And after that, research turned to love.

[Dana Susskind]

I always joke that I was wooed with spreadsheets and hypotheses.

[Stephen Dubner]

Is that true?

[Dana Susskind]

Yes. Yes. In fact, the reason I decided to marry him was because I wanted this area of scaling to be a robust area of research for him, because it really is a major issue.

[Stephen Dubner]

Susskind started what was then called the 30 million words initiative. 30 million being an estimate of how many fewer words a child from a low income home will have heard than an affluent child by the time they turn four. But these days, the project is called the TMW center for Early Learning and Public Health.

[Dana Susskind]

We've actually moved away from the term 30 million words because it's such a hot button issue.

[Stephen Dubner]

Hot button? Because it's so hard to believe that the number is legit?

[Dana Susskind]

Well, no. I mean, some people say, look, it's a deficit mentality. You're talking about what's not there. And then the replication, somebody did another study that said, oh, it's only 4 million. And it really isn't actually even the point, because it's not even about words. It's about the interaction. So I just made the decision. I'd rather Be focusing on developing the research and fighting a naming battle.

[Stephen Dubner]

So you didn't make TMW stand for something else?

[Dana Susskind]

Well, that's what everybody gives me trouble for. But it stands for 30 million words. But only I know that.

[Stephen Dubner]

Okay, now you all know it too. Anyway, they started the center with this.

[Dana Susskind]

Idea, with this idea that, you know, we need to take a public health or a population level approach during the early years to optimize early foundational brain development. Because the research is pretty clear that coherent talk and interaction in the first three years of life are the catalyst for brain development. And so that's basically our work.

[Stephen Dubner]

Okay, so far so good. The research is clear that heavy exposure to language is good for the developing brain. But how do you turn that research finding into action and how do you scale it up?

[Dana Susskind]

Initially, we started with an intensive home visiting program. But understanding that to reach population level impact, you need to develop programs both with an eye for scaling as well as an eye for understanding where parents go regularly. Because healthcare, unlike the education system, the first three years of life really don't have any infrastructure in which to disseminate programs. So we actually expanded our model. We have this multifaceted program that reached parents where they were from maternity wards, into pediatrics offices, into the homes as well as group sessions. Those programs that are most vulnerable to the issues of scale are the complex sort of service delivery interventions. You know, anything that takes human service delivery scaling isn't an end, it's really just a continuation.

[Patti Chamberlain]

You know, it's a hard one, that is.

[Stephen Dubner]

Patti Chamberlain, senior research scientist at Oregon.

[Patti Chamberlain]

Social Learning center, and I do research and implementation of evidence based practices in child welfare, juvenile justice, mental health and education systems.

[Stephen Dubner]

Chamberlain also looks at scaling as a process.

[Patti Chamberlain]

So it's almost like there's stages that you have to go through.

[Stephen Dubner]

And if the first stage is research that involves an rct, a randomized controlled trial, there's already an important choice to make.

[Patti Chamberlain]

You're far better off to situate your RCT in a real world setting than a university clinic, so that you're learning from the beginning what's feasible and what's not feasible. There might be something that you think would be great, but it's never going to be able to be implemented in the real world. I've been at this now for, oh, probably 25 years, and I learned sort of through failing.

[Stephen Dubner]

One program Chamberlain founded is called Treatment Foster Care.

[Patti Chamberlain]

Oregon kids tend to commit crimes together. It's a team sport. But then oddly, the way that we are set up to deal with kids who, you know, reach the level where they're really being unsafe to themselves and to the community is we put them in group homes together. We're putting kids in a situation where they're more likely to commit crimes. So we decided, what if we placed a child singly in a family that was completely devoted to using evidence based parenting skills to help that child do well with peers in school and in the family setting? What if we gave the parents, the biological parents of that kid, the same kind of skills that the treatment foster care family had? What if we gave the kid individual therapy, the biological family was getting family therapy, we were giving the kids support at school. So we were basically wrapping all these services around an individual child in a family home. What we found was, yeah, the kids do a lot better. They have a lot fewer rests, they spend less days in institutions, they use fewer drugs, and guess what, it costs a lot less as well because you do not have a facility. You do not have 24, seven staff that you're paying in shifts. You do not have, you know, all of the stuff that it takes to run an institution. You have a family.

[Stephen Dubner]

The success of Chamberlain's program caught the eye of researchers who were working on a program for a federal agency called the Office of Juvenile justice and delinquency Prevention.

[Patti Chamberlain]

And so we got this call saying, you know, we want you to implement your program in 15 sites.

[Stephen Dubner]

If the program was successful at one site, how hard could it be to make it work at 15?

[Patti Chamberlain]

I went in thinking that it wouldn't be that hard because we had good outcomes. We showed that we could save money, and yet we were absolutely not ready. It wasn't because we didn't have enough data. We had at that point, plenty of data, but we didn't have the know how of how to put this thing down in the real world. And it blew up.

[Stephen Dubner]

One reason, systemic complication.

[Patti Chamberlain]

The three systems, child welfare, juvenile justice and mental health, all put some money in the pot to fund this implementation. I was completely delighted. I thought, oh, this is going to be great, because we have all the relevant systems buying into this. Well, what happened was when we tried to implement, we ran into tremendous barriers because if we satisfied the policies and procedures of one system, we were at odds with the policies and procedures in the other system.

[Stephen Dubner]

Patti Chamberlain had run up against something that Dana Susskind had come to see as an inherent disconnect. When you try to scale up a.

[Dana Susskind]

Research finding, there's obviously the implementation, everybody focusing on adherence, but there's also sort of the infrastructure delivery mechanism, which I think is an issue, whether it's government or health care, that they're just not set up for interventions which are sort of like innovations. So you've got these researchers who think of themselves as, you know, scientific entrepreneurs developing the next best thing, you know, thinking, you know, you build it and they will come. And then you've got organizations that are sort of built for efficiency rather than effectiveness, that can't uptake it.

[Stephen Dubner]

If only there were another science, a science to help these scientific entrepreneurs and institutions come together to implement this new research. Maybe something that could be called the implementation science.

[Lauren Suplee]

Implementation science.

[Patti Chamberlain]

Implementation science.

[John List]

Implementation science.

[Stephen Dubner]

Okay, let's define implementation science.

[Lauren Suplee]

It's the study of how programs get implemented into practice and how the quality of that implementation may affect how well that program works or doesn't work.

[Stephen Dubner]

That is Lauren Suplee. When we spoke with her, Suplee was the deputy chief operating officer of a nonprofit called Child Trends, which promotes evidence based policy to improve children's lives.

[Lauren Suplee]

This whole science is maybe 15 years old. It's really coming out of this movement of evidence based policy and programs where people said, well, we have this program. It appears to change important outcomes. Let's just put it out there. And then we quickly realized that there are a lot of issues and actually that put it out there is far more complicated. A lot of the evidence based programs we have were designed by academic researchers who were testing it in the maybe more. More ideal circumstances that they had available to them that might have included graduate students. It might have been a school district that was very amenable to research. And then you take the results of that and trying to put that into another location is where the challenge happened.

[Stephen Dubner]

So coming up after the break, can implementation science really help?

[John List]

You know, I want policy science not to be an oxymoron.

[Stephen Dubner]

You're listening to Freakonomics Radio. I'm Stephen Dubner. We will be right back.

[Amica Insurance]

Every day, our world gets a little more connected, but a little further apart. But then there are moments that remind us to be more human.

[Dana Susskind]

Thank you for calling Amica Insurance.

[Stephen Dubner]

Hey, I was just in an accident.

[Freakonomics Radio]

Don't worry, we'll get you taken care of.

[Amica Insurance]

At Amica, we understand that looking out for each other isn't new or groundbreaking.

[Stephen Dubner]

It's human.

[Amica Insurance]

Amica empathy is our best policy.

[Lowe's]

Lowe's knows how to freshen up your yard. And it all starts with mulch. That's why we're introducing Mulch Week during Springfest with can't miss savings right now. Get five bags of Stay Green Premium Color 2 Cubic Foot Mulch for only $10 plus members earn five times the points on select mulch purchases all week long. Lowes we help you Save valid through 49 loyalty programs subject to terms and conditions. Points are awarded on eligible purchases. [e-mail] Terms subject to change. Excludes Alaska and Hawaii.

[McDonald's]

If you've been having your McDonald's sausage McMuffin with an iced coffee from somewhere else, now is a great time to reconsider.

[Freakonomics Radio]

In the Pacific Northwest, it's never too cold for an iced coffee in the morning. Grab yourself a medium caramel, French vanilla or classic iced coffee for just $2.29. Beverage may cause craving for McMuffin or hash browns. Prices and participation may vary. Cannot be combined with any other offer or combo Me.

[Dana Susskind]

What randomized control trials tell us about an intervention is what that actual intervention does in a particular population, in a particular context. It doesn't mean that it's generalizable.

[Stephen Dubner]

That, again, is Dana Susskind from the University of Chicago.

[Dana Susskind]

But you have to continue the science so you can understand how it's going to work in a different place, in a different context, in a different population and have the same effect. And that's part of the scaling science.

[Stephen Dubner]

The scaling science. That is what Susskind and her economist collaborator John List, who's also her husband, and other researchers have been working on. They've been systematically examining why interventions that work well in experimental or research settings often fail to scale up. You can see why this is an important puzzle to solve. Scaling up a new intervention, like a medical procedure or a teaching method, has the potential to help thousands, millions, maybe billions of people. But what if it simply fails at scale? What if it ends up costing way more than anticipated or create serious unintended consequences that'll make it that much harder for the next set of researchers to persuade the next set of policymakers to listen to them. So List and Susskind have been looking at scaling failures from the past and trying to categorize what went wrong.

[John List]

You can kind of put what we've learned into three general buckets that seem to encompass the failures. Bucket number one is that the evidence was just not there to justify scaling the program in the first place. The Department of Education did this broad survey on prevention programs attempting to attenuate youth, substance and crime and aspects like that, and what they found is that only 8% of those programs were actually backed by research evidence. Many programs that we put in place really don't have the research findings to support them. And this is what a scientist would call a false positive.

[Stephen Dubner]

So are we talking about bad research? Are we talking about cherry picking? Are we talking about publication bias?

[John List]

So here we're talking about none of those. We're talking about a small scale research finding that was the truth in that finding, but because of the mechanics of statistical inference, and it just won't be right. What you were getting into is what I would call the second bucket of why things fell. And that's what I call the wrong people were studied. You know, these are studies that have a particular sample of people that shows really large program effect sizes. But when you program is gone to general populations, that effect disappears. So essentially we were looking at the wrong people and scaling to the wrong people.

[Stephen Dubner]

And when you say the wrong people, the people that are being studied then.

[John List]

Are too what they are the people who are the fraction or the group of people who receive the largest program benefits.

[Stephen Dubner]

So I think of some of the experiments that are done on college campuses, right, where there's a professor who's looking to find out something about, let's say, altruism. And the experimental setting is a classroom where 20 college students will come in and they're a pretty homogeneous population and they're pretty motivated, maybe they're very disciplined. And that may not represent what the world actually is. Is that what you're talking about?

[John List]

That's one piece of it. Another piece is who will sign their kids up for Head Start or for a program in a neighborhood that advances the reading skills of the child. Who's going to be first in line? The people who really care about education and the people who think their child will receive the most benefits from the program. Now, another way to get it is sort of along the lines that you talked about. It could be the researcher knows something about the population that other people don't know. Like I want to give my program its best shot of working.

[Stephen Dubner]

Okay, and what's in your third bucket of scaling failures?

[John List]

The third bucket is something that we call the wrong situation was used. And what I mean by that is that certain aspects of the situation change. When you go from the original research to the scaled research program, we don't understand what properties of the situation or features of the environment will matter. There are a really large group of implementation scientists who have explored this question for years now. What they emphasize and focus on is something called voltage drop. And voltage drop essentially means I found a really good result in my original research study. But then when they do it at scale, that voltage drop ends up being, for example, a tenth of the original result or a quarter of the original result. An example of this is when you look at Head Start's home visiting services. What they do there is this is an early childhood intervention that found huge improvements in both child and parent outcomes in the original study, except when they tried to scale that up into home visits at a much larger scale. What they found is that, for example, home visits for at risk families involved a lot more distractions in the house and there was less time on child focused activities. So this is sort of the wrong dosage or the wrong program is given at scale.

[Stephen Dubner]

There are many factors that contribute to this voltage drop, including the admirably high standards set by the original researchers.

[Dana Susskind]

When the researcher starts his or her experiment, the inclination is, I'm going to get the best tutors in the world, so I'm going to be able to show how effective my intervention is.

[Stephen Dubner]

Dana Susskind, again, you only needed 10.

[Dana Susskind]

Math tutors and you happen to get the PhD students from the University of Chicago. And then what happens is you show this tremendous effect size and in the scaling, all of a sudden you need 100 or 1,000 and you no longer have that access to those individuals. And you go either down the supply chain with individuals who are not quite as well trained, or you end up having to pay a whole lot more money to maintain the train tutor program. And one way or the other, either the impacts of the intervention go down or your costs go up significantly.

[Stephen Dubner]

Another problem in this third bucket, it's a big bucket, is when the person who designed the intervention and masterminded the initial trial can no longer be so involved once the program scales up to multiple locations. Imagine if instead of talking about an educational or medical program, we were talking about a successful restaurant and the original chef.

[John List]

When you think about the chef, if a restaurant succeeds because of the magical work of the chef and you think about scaling that, if you can't scale the magic in the chef, that's not scalable. Now, if the magic is because of the mix of ingredients and the secret sauce, like Domino's, for example, the secret sauce or Papa John's is the actual ingredients, then that will be scalable.

[Stephen Dubner]

Now, if you are the kind of pizza eater who doesn't think Domino's or Papa John's is good pizza, well, welcome to the scaling dilemma. Going big means you have to be many things to many people. Going big means you will face a lot of trade offs. Going big means you'll have a lot of people asking you, do you want this done fast or do you want it done right? Once you peer inside these failure buckets that List and Susskind describe, it's not so surprising that so many good ideas fail to scale up. So what did they propose that could help?

[John List]

Now, our proposal is that we do not believe that we should scale a program until you're 95% certain the result is true. So essentially what that means is we need the original research and then three or four well powered independent replications of the original findings.

[Stephen Dubner]

And how often is that already happening in the real world of, let's say, education reform research?

[John List]

I can't name one.

[Stephen Dubner]

Wow. How about in the realm of medical compliance research?

[John List]

My intuition is that they're probably not far away from three or four well powered independent replications. In the hard sciences, in many cases, you not only have the original research, but you have a first replication also published in science. You know, the current credibility crisis in science is a serious one that major results are not replicating. The reason why is because we weren't serious about replication in the first place. So this sort of puts the onus on policymakers and funding agencies in a sense of saying we need to change the equilibrium.

[Stephen Dubner]

So that suggests that policymakers or decision makers, they are being, what, overeager? Premature in accepting a finding that looks good to them and want to rush it into play. Or is it that the researchers are overconfident themselves or maybe pushing this research too hard? Where is this failure really happening?

[John List]

Well, I think it's sort of a mix. I think it's fair to say that some policymakers are out looking for evidence to base their preferred program on. What this will do is slow that down. If you have a pet project that you want to get through, fund the replications and let's make sure the science is correct. We think we should actually be rewarding scholars for attempting to replicate. You know, right now, in my community, if I try to replicate someone else, guess what I've just made? I've just made a mortal enemy for life. If you find a publishable result, what result is that? You're refuting previous research. Now I've doubled down on my enemy. So that's like a first step in terms of rewarding scholars who are attempting to replicate. Now, to complement that, I think we should also reward scholars who have produced results that are independently replicated. You know, and I'm talking about tying tenure decisions, grant money, and the like to people who have given us credible research that replicates.

[Stephen Dubner]

After the break. How can researchers make sure that the science they are replicating works when it scales up.

[Amica Insurance]

At Ameca Insurance, we know it's more than just a house. It's your home. The place that's filled with memories. The early days of figuring it out to the later years of still figuring.

[Stephen Dubner]

It out.

[Amica Insurance]

For the place you've put down roots. Trust Amica Home Insurance Amica Empathy is our best policy.

[Lowe's]

Lowe's knows how to freshen up your yard. And it all starts with mulch. That's why we're introducing Mulch Week during springfest with can't miss savings right now. Get five bags of stay green premium color two cubic foot mulch for only $10 plus members earn five times the points on select mulch purchases all week long. Lowe's we help you Save valid through 49 loyalty programs subject to terms and conditions. Points are awarded on eligible purchases. Details@lowe's.com Terms subject to change. Excludes Alaska and Hawaii.

[McDonald's]

If you've been having your McDonald's sausage McMuffin with an iced coffee from somewhere else, now is a great time to reconsider.

[Freakonomics Radio]

In the Pacific Northwest, it's never too cold for an iced coffee. Coffee in the morning. Grab yourself a medium caramel, French vanilla or classic iced coffee for just 2.29 warning beverage may cause craving for McMuffin or hash browns. Prices and participation may vary. Cannot be combined with any other offer or combo meal.

[Stephen Dubner]

Before the break, we were talking with the University of Chicago economist John Listener about the challenges of turning good research into good policy. One challenge is making sure that the research findings are in fact robust enough to scale up.

[John List]

Say I'm doing an experiment in Chicago Heights on early childhood and I find a great result? How confident should I be that when we take that result to all of Illinois or all of the Midwest or all of America, is that result still going to find that important benefit cost profile that we found in Chicago Heights? We need to know what is the magic sauce? Was it the 20 teachers you hired down in Chicago Heights where if we go nationally, we need 20,000? So it should behoove me as an original researcher to say, look, if this scales up, we're going to need many more teachers. I know teachers are an important input. Is the average teacher in the 20,000 the same as the average teacher in the 20?

[Stephen Dubner]

This is the dreaded voltage drop that implementation scientists talk about and the implementation.

[John List]

Scientists have focused on. Fidelity is a core component behind the voltage drop.

[Stephen Dubner]

Fidelity, meaning that the scaled up program reflects the integrity of the original program.

[Patti Chamberlain]

Measures of fidelity that's a really critical part of the implementation process.

[Stephen Dubner]

That again, is Patty Chamberlain, founder of Treatment Foster Care Oregon.

[Patti Chamberlain]

You've got to be able to measure is this thing that's down in the real world the same? You know, does it have the same components that produce the outcomes in the RCTs?

[Stephen Dubner]

Remember, it was Chamberlain's good outcomes with young people in foster care that made federal officials want to scale up her program in the first place.

[Patti Chamberlain]

We got this call saying we want you to implement your program in 15 sites.

[Stephen Dubner]

She found the scaling up initially very challenging.

[Patti Chamberlain]

It wasn't the kumbaya moment that we thought it was going to be.

[Stephen Dubner]

But in time, Treatment Foster Care Oregon became a very well regarded program. It's been around for roughly 30 years now, and the model has spread well beyond Oregon. One key to this success has been developing fidelity standards.

[Patti Chamberlain]

So the way that we do it is we have people upload all of their sessions onto a HIPAA secure website and then we code those, and if they're not meeting the fidelity standards, then we offer a fidelity recovery plan. You know, we haven't had to drop a site, but we have had to have some of the people in the site retrained or not continue.

[Stephen Dubner]

Being able to measure fidelity well from afar provides another benefit to scaling up. It allows the people who developed the original program to ultimately step back so they don't become a bottleneck, which is a common scaling problem.

[Patti Chamberlain]

There can be sort of an orderly process whereby you step back in increments as people become more and more competent doing what they're doing. And that's what you want because you don't want to have this tied to the developer forever. Otherwise you can't get any kind of reasonable reach.

[Stephen Dubner]

That said, you also need to have some humility when you're scaling up. You shouldn't assume your original program was perfect, that it won't need adjustment, and you need to be willing to make adjustments.

[Patti Chamberlain]

For example, we recognized that when we were in real world communities, kids needed something that wasn't therapy per se. They needed skills because the kids had often been excluded from normal socializing sort of things like sports teams and clubs. And so we needed what we call a skills coach to help those kids learn the moves that they needed to be able to participate in these pro social activities that are normal kind of things. So you have research, you have a theory, and then you have the implementation. And that feeds into more research, more theory, more implementation.

[Dana Susskind]

Look, everybody's motivation at the end of the day is about trying to do Good for the people they serve.

[Stephen Dubner]

Dana Susskind, again, there are many children.

[Dana Susskind]

Out there and there are a lot of injustices, so we need to move. But I don't know. The science is slower than you'd like. People have wanted things before. I thought they were ready. And finding a way to deal with that dance of people wanting information but also wanting to continue to build the evidence. I think we can figure out how to do it.

[John List]

I think that's exactly right.

[Stephen Dubner]

And John List, again, I think too.

[John List]

Many times, whether it's in public policy, whether it's a for profit or a not for profit, we tend to only focus on one side of the market when we have problems. And you really need to take account of both sides because your optimal solutions, the best solutions are only going to come when you look at both sides of the market.

[Stephen Dubner]

I'm probably getting this wrong, or at least being way too reductive, but to me it sounds like the chief barrier to scaling up programs to help people is people. The people are the problem.

[John List]

Yeah. So I do think inherently it is about people. That said, this is not a fatal flaw that causes us to throw up our arms and say, well, this isn't physics, this isn't chemistry. We have to deal with people, so we can't use science. I think that's wrong because there are some very, very neat advantages of scaling. You know, think about on the cost side. Economists always talk about, you know, when things get bigger and bigger, guess what happens? The per unit cost goes down. It's called increasing returns to scale. The problem that kind of we're thinking about is let's make sure that those policymakers who really want to do the right thing in youth science, let's make sure that they have the right programs to implement.

[Stephen Dubner]

So one of your papers includes this quote from Bill Clinton, or at least something that Clinton may have said, which is essentially that nearly every problem has been solved by someone somewhere, but we just can't seem to replicate those solutions anywhere else. So what makes you think that you've got the keys to success here where others may not have been able to do it?

[John List]

You know, I view what we've done is put forward a set of modest proposals is only a start to tackle what I think is a most vexing problem in evidence based policy making, which is scaling. I think we're just taking some small steps theoretically and empirically. But I do think that these first set of steps are important because if you go in the right direction, what I've learned is that literature will follow that direction. If you go in the wrong direction, sometimes the literature follows that wrong direction for several years, and we really don't have the time right now. The opportunity cost of time is very high. You know, in the end, I want policy science not to be an oxymoron, and I think that's what this research agenda is about. The way that I would view it is that the world is imperfect because we haven't used science in policymaking. And if we add science to it, we have a chance to make an imperfect world a little bit more perfect.

[Stephen Dubner]

If you want to read the papers that John List and Dana Susskind and their collaborators have been working on, you will find links on Freakonomics.com as well as links to Patty Chamberlain's work with treatment, foster care, OR and much more, including, as always, a complete transcript of this episode and we will be back soon with another new episode of Freakonomics Radio. Until then, take care of yourself and if you can, someone else too. Freakonomics Radio is produced by Stitcher and Renbud Radio. You can find our entire archive on any podcast apple [e-mail] where we publish transcripts and show notes. This episode was produced by Matt Hickey with an update by Augusta Chapman. The Freakonomics Radio Network staff also includes Alina Coleman, Dalvin Abuaji, Eleanor Osborne, Ellen Frankman, Elsa Hernandez, Gabriel Roth, Greg Rippon, Jasmine Klinger, Jeremy Johnston, John Schnarz, Morgan Levy, Neal Carruth, Sarah Lilly, Teo Jacobs, and Zach Lipinski. Our theme song is Mr. Fortune by the Hitchhikers, and our composer is Luis Guerra. As always, thanks for listening. So you want to talk scaling?

[John List]

Well, it's a heavy paper, right?

[Stephen Dubner]

It's great. I thought it was about scaling fish initially, so that was all my background reading. Yeah, I don't know anything about what we're going to talk about today.

[John List]

Neither do I. So we can just both wing it.

[Freakonomics Radio]

The Freakonomics Radio Network the Hidden side of everything.

[Dana Susskind]

Stitcher.

[Stephen Dubner]

For 140 years, MultiCare has been in Washington prioritizing long term solutions, partnering with local communities and expanding access to care. Together, we're building a healthier future. Learn [e-mail].

[Amica Insurance]

Every day, our world gets a little more connected, but a little further apart. But then there are moments that remind us to be more human.

[Dana Susskind]

Thank you for calling Amica Insurance.

[Stephen Dubner]

Hey, I was just in an accident.

[Freakonomics Radio]

Don't worry, we'll get you taken care of.

[Amica Insurance]

At Amica, we understand that looking out for each other isn't new or groundbreaking. It's human Amica. Empathy is our best policy.

[McDonald's]

If you've been having your McDonald's sausage McMuffin with an iced coffee from somewhere else, now is a great time to reconsider.

[Freakonomics Radio]

In the Pacific North Northwest, it's never too cold for an iced coffee in the morning. Grab yourself a medium caramel, French vanilla or classic iced coffee for just $2.29. Beverage may cause craving for McMuffin or hash browns. Prices and participation may vary. Cannot be combined with any other offer or combo meal.