Stereothreat

Summary

---

Radiolab – "Stereothreat" (November 23, 2017)

Main Theme / Purpose

This episode of Radiolab, hosted by Jad Abumrad and Robert Krulwich, probes the origins, science, and recent challenges surrounding "stereotype threat"—the idea that being reminded of a negative stereotype about your social group can undermine your performance, especially in academic settings. The episode traces how this idea revolutionized social psychology, was championed as a force for social justice and educational reform, then weathered questions about its robustness during social psychology’s “replication crisis.” The hosts and guests offer a nuanced view of how stereotype threat stands up to modern scientific scrutiny and what the debates reveal about science itself.

---

Key Discussion Points & Insights

1. The Discovery of Stereotype Threat (00:14 – 13:01)

• Claude Steele’s Background: Raised by civil rights activists, Steele was acutely aware of race and its social consequences (01:41).
• The Achievement Gap: At the University of Michigan, Steele puzzled over why black and white students with identical academic credentials had very different outcomes after one semester. Black students’ grades were strikingly lower than their white peers' (02:34).
• Not Just Race: A similar gap existed between women and men in advanced math courses (03:00).
• Initial Explanations: While systemic racism and opportunity gaps were recognized, these didn’t explain performance differences among students entering college with equivalent academic histories (05:43).
• Influence of "The Bell Curve": The controversial 1994 book fueled genetic explanations for group differences, which deeply troubled Steele, both scientifically and personally (04:39, 05:08).
• Steele’s Experiments: He and colleagues designed experiments to test whether performance gaps could be induced or eliminated by altering test conditions (06:22). Notably, simple reassurances before a challenging test—telling women that a math test does “not show gender differences” or telling black students that a test is “not diagnostic of ability”—erased the performance gap between groups (07:38).

2. Stereotype Threat in Action (08:42 – 13:55)

• "Puzzle" vs. "Test": A nonverbal IQ test framed as a "puzzle" (instead of an "IQ test") led to black students matching white students’ performance (09:15).
• Where It Appears: Stereotype threat only kicks in when tasks are sufficiently hard: “As soon as the test gets difficult, that’s where the voices kick in.” (10:16).
• The "Gremlin": This metaphorical gremlin “whispers” distracting, self-doubt-inducing thoughts, even if the student doesn’t personally believe the negative stereotype (10:45–11:16).
• Distraction, Not Inability: The stereotype doesn’t “prevent” achievement, but “distracts you for just a beat,” which can make all the difference (11:22).

3. Stereotype Threat Goes Mainstream (12:02 – 14:11)

• Academic Fame: The theory was widely replicated and inspired a new wave of social psychology research into bias, discrimination, and academic achievement (12:45).
• For Social Justice: For researchers—including Michael Inzlicht, featured guest—the appeal was that stereotype threat offered a hopeful, actionable way to address inequities (13:19).

4. The Replication Crisis (14:11 – 25:10)

• Research Wobbles: Inzlicht struggled to reproduce stereotype threat effects at the University of Toronto, perhaps due to a more diverse, less stereotype-aware student body (16:18–17:36).
• False Positive Psychology (18:09): A pivotal 2011 paper showed how standard research practices could skew results—selectively reporting only those measures that confirmed the hypothesis (19:03).
• Emergency Meetings: Social psychologists realized these practices could “turn up spurious findings” and called urgent meetings to re-evaluate their work (19:52).
• Replication Attempts: The field moved toward “high powered replication” studies—larger samples, strict methods, pre-declared analyses (21:03).
• Early Results: Some previously celebrated effects (willpower depletion from resisting cookies, "power poses" changing hormones) failed to replicate (22:00–25:15).
• Civil War in Psychology: The field split, with some researchers defending their theories and others insisting on new standards and skepticism (26:00).

5. Stereotype Threat Under Scrutiny (26:20 – 34:12)

• Mixed Replication Results: No massive, multi-site preregistered replications of stereotype threat have taken place, but smaller efforts have had mixed results—occasionally even reversing the effect (“students did better” after being exposed to stereotype threat) (26:45–27:04).
• Claude Steele’s Defense: Steele points out that stereotype threat has been “dramatically well replicated,” though not in every context. “If you can’t replicate one of them or six of them, I wouldn’t—that doesn’t surprise me.” (27:29).
• Evolving Contexts: Steele cautions that stereotypes—and who they affect—change over time, so sticking to 25-year-old experimental protocols may not reflect current reality. “Social psychology… the meanings come from the contemporary moment, the state of the culture at that time” (28:13).
• Publication Bias: Concerns persist that repeated “trials” until a positive result is found, then published, could bias the entire field—though Steele views wholesale fabrication as unlikely (29:13, 29:49).

6. Real-World Impact & Openness (30:17 – 32:49)

• Beyond the Lab: Stereotype threat interventions have been tested in schools, producing, in some cases, dramatic, lasting gains in achievement (30:28).
• Replication Failures: Still, some large, real-world replications have failed (31:02).
• Transparency: Steve Spencer, an early collaborator, is preparing to publish all his data and questionable research practices and is participating in adversarial collaborations—cooperating with skeptics on large replication projects (32:00–32:29).
• Emotional Toll: For some, the replication crisis provokes unsettling doubts: “Was I doing good work? Was I part of the problem? Was I chasing signal...or chasing noise?... It might be there, but it might be so small as to not be meaningfully important.” (33:13)

7. Growing Up, Moving Forward (34:12 – End)

• Simple Fixes Fade: Radiolab’s hosts reflect on their own desire for “a very simple fix that would work everywhere…and sort of save the world.” The growing realization is that solutions may have to be smaller, targeted, and more context-dependent (34:55–35:04).
• Mature Uncertainty: “Maybe that’s exactly what you need in order to be able to find the smaller places where you can have an effect, you know, right here, right now, with this person trying to do this particular thing.” (35:04)

---

Notable Quotes & Memorable Moments

• On the Discovery of Stereotype Threat:
  • “So if going forward and taking a test, the black kid gets a lower score...then something must be happening right there. Right there.” — Claude Steele (05:43)
• On the Subtlety of the Effect:
  • “The real subtle power of a stereotype isn’t that it prevents you from doing the thing you want to do. It distracts you for just a beat from doing the thing you want to do. And that may be all the difference.” — Jad Abumrad (11:22)
• On the Replication Crisis:
  • “My jaw dropped. I sent this paper, I circulated it...we called an emergency meeting.” — Michael Inzlicht, on reading "False Positive Psychology" (19:35)
  • “You don’t really know how many things were tried in each individual lab.” — Dan Engber (24:55)
• On Shifting Standards:
  • “Maybe that’s exactly what you need in order to be able to find the smaller places where you can have an effect.” — Robert Krulwich (35:04)
• On Scientific Humility:
  • “I want the truth out there more than I want anything else.” — Steve Spencer (31:35)
  • “It might be there, but it might be so small as to not be meaningfully important.” — Michael Inzlicht (33:13)

---

Timestamps for Key Segments

• 00:14 – 06:12: Claude Steele’s background and the origins of stereotype threat research
• 07:06 – 08:34: The experiment—removing threat from test instructions, performance gap disappears
• 09:02 – 09:35: "Puzzle" vs. "IQ test" framing, and group performance equalizes
• 10:04 – 11:22: How and when stereotype threat manifests; "the gremlin"
• 12:02 – 13:19: Stereotype threat goes mainstream in social psychology
• 16:18 – 17:36: Inzlicht’s failures to replicate; possible reasons
• 18:09 – 19:35: The "False Positive Psychology" paper and its impact
• 21:03 – 22:14: High-powered replications and the replication crisis
• 26:20 – 27:00: Replicating stereotype threat; sometimes results reverse
• 28:13 – 29:49: Claude Steele on shifting cultural contexts and bias
• 30:28 – 31:16: Real-world interventions and mixed replication results
• 32:00 – 32:45: Steve Spencer’s openness to scrutiny and replication
• 33:09 – 33:54: Michael Inzlicht reflects on the emotional burden of doubt
• 34:12 – 35:32: The hosts’ reflections—growing up, letting go of “big, simple promises”

---

Tone and Style:
Conversational, intellectually curious, self-reflective, occasionally rueful but generally hopeful. The episode balances personal stories (from researchers and hosts) with careful explanations of science and its pitfalls.

---

Summary Takeaway

Radiolab’s "Stereothreat" unpacks a watershed theory in social psychology—not just for its explanatory power, but as a lens on how science, ambition, and culture shape what we think we know. Stereotype threat remains real for many researchers, but its scope, mechanisms, and interventions are more complex and context-dependent than initially hoped. The scientific process—messy, self-correcting, and human—continues, moving from grand solutions toward humility and precision.

Transcript

[Jad Abumrad]

Oh, wait, you're listening. Okay.

[Dan Engber]

All right.

[Jad Abumrad]

Okay.

[Robert Krulwich]

All right. You're listening to Radio Lab Radio from wny.

[Jad Abumrad]

See y. Do you. Can I ask you, do you. Do you just want to, like, lay out for us the chronology of your obsession? I think I feel comfortable saying it was. Has it become an obsession or just a. Just a dalliance?

[Robert Krulwich]

I mean, or maybe you just noticed a crumbling building and ran over to stick your pen.

[Dan Engber]

I mean, I am, I'm a contrarian and I'm interested in alternative facts about science, let's say.

[Jad Abumrad]

Hey, I'm Jad Abumrad.

[Michael Inslick]

I'm Robert Krulwich, this is Radiolab.

[Jad Abumrad]

And a little while back, our editor Soren Wheeler and I, we. We talked to a science journalist named Dan Engber who got us kind of tangled up about something that we thought we knew about the world, about ourselves.

[Narrator/Additional Commentator]

Something beautiful, as I recall.

[Jad Abumrad]

Yeah. That we talked about at great length on this show.

[Claude Steele]

Okay.

[Jad Abumrad]

Hello.

[Claude Steele]

Hello.

[Robert Krulwich]

And to set that conversation up, we're gonna start with this guy.

[Jad Abumrad]

Hi, is this Claude Steele?

[Robert Krulwich]

Yes, Professor Claude Steele.

[Claude Steele]

I'm the Lucy Stearns Emeritus professor of Psychology at Stanford University.

[Robert Krulwich]

We actually had him on the show a number of years ago, some time ago.

[Claude Steele]

I can't remember how many years ago.

[Jad Abumrad]

Yeah, it's been a long time. I'm looking at the sheet here and it says 2000.

[Claude Steele]

Whoa, that blows my mind.

[Jad Abumrad]

Yeah, that was a little.

[Claude Steele]

He was a lot younger man then.

[Robert Krulwich]

I know. And the reason we had him on the show back then was because he had done a study in the mid-90s that just completely changed the way that we thought about the power of stereotypes.

[Claude Steele]

Well, my mother was white, my father was African American, and they were very active in the civil rights movement. So you can imagine that race was no distant or passing thing in our life and family.

[Robert Krulwich]

For Claude growing up, that was just everyday dinner table talk.

[Claude Steele]

Yes, exactly. So the topic is a sort of family birthright.

[Robert Krulwich]

But then years and years later, midway through his career as a psychology professor, Claude ran into a demonstration of the power of race that really surprised him.

[Claude Steele]

I got a job offer. This is in the 80s at the university of Michigan. And it was part psychology and part to administer a minority student program there.

[Robert Krulwich]

This is Claude Steele, and the original program we ran back in 2009.

[Claude Steele]

And in the process I saw data that surprised me.

[Jad Abumrad]

What he saw was a troubling trend. Two kids would enter Michigan. One was black, one was white. They'd come in at the exact same.

[Claude Steele]

Level, same skills, same SAT score.

[Jad Abumrad]

So theoretically, they should do the same when they get to Michigan, but without fail or almost without fail. After one semester, the black kid was.

[Claude Steele]

Winding up with lower grades.

[Jad Abumrad]

How much lower?

[Claude Steele]

Pretty, pretty, pretty dramatic. At least 2/3 of the letter grades.

[Jad Abumrad]

Meaning if the white kid got an A, the black kid who should be getting an A too, is instead getting a B or a B. That's significant.

[Claude Steele]

That's significant. That's significant.

[Jad Abumrad]

And he also, by the way, saw this performance gap between women and men when it came to math. To the same degree, to the same.

[Claude Steele]

Degree in advanced math courses, it was comparable. I learned this is a national phenomenon. If I was to walk into almost any college class in the United States, I'd have a very high probability of finding exactly that.

[Dan Engber]

So I think it's important to put it in context of what was going on at the time.

[Robert Krulwich]

This is Dan Engber again, and he says that that gap in achievement between black and white students that Claude had noticed, that was actually a huge topic of conversation at the time.

[Dan Engber]

There was a lot of discussion of what to do about the achievement gap. And the familiar argument is, well, this has to do with systemic racism and systemic differences in opportunities that play out through an individual lifespan. Now that seems right. It's also daunting because how are we ever going to, like, cure all of the socioeconomic disparities in this country?

[Robert Krulwich]

But then in 1994, a different and in many ways very dangerous idea was being tossed into this debate. Charles Murray, co author of the book the Bell Curve. When the Bell Curve came out, the.

[Narrator/Additional Commentator]

Bell Curve Intelligence and Class Structure in American Life.

[Robert Krulwich]

Bell Curve argued that one explanation for the achievement gap, among others, was genetic and IQ based. Now, of course, it's not. But even though there is no scientific backing at all for the idea of genetic differences like this, the Bell Curve was still significant just because of the kinds of conversations it was creating at the time and the effect that it had on researchers in this field.

[Claude Steele]

Well, the Bell Curve is one point in a long history of that kind of argument that the difference between groups really is rooted in genetic differences. Let's just be frank and honest. And if you really can't admit that, then you don't have the courage needed to be a real scientist. That's a thumbnail way of describing this experience that I've had of being confronted with that notion.

[Robert Krulwich]

And this was obviously disturbing to Claude. Well, first, on a personal level, in.

[Claude Steele]

Order to be a scientist, are you supposed to actually be open to the possibility that you and your family and your whole race have Some genetic limitation.

[Robert Krulwich]

But also because it was so weak, you know, scientifically it's been very difficult.

[Claude Steele]

Impossible to produce anything like definitive data that the differences in test scores between groups is genetic.

[Robert Krulwich]

But while the differences between these students that Claude was seeing in Michigan clearly wouldn't be explained by genetic differences, it also didn't seem to him like it could be explained simply by their backgrounds or their opportunities.

[Claude Steele]

Because you take, let's say a black kid and a white kid in Michigan, they both have extraordinary scores. Like, you know, they're in the 98th percentile in their SATs. So the background between the two kids, whatever it is, has not resulted in a difference to that moment in time. So if going forward and taking a test, the black kid gets a lower score or lower performance in a chorus of some sort of. Then something must be happening right there. Right there.

[Robert Krulwich]

Something must be happening in the moment.

[Claude Steele]

There was something there that people didn't understand and that we certainly didn't understand.

[Jad Abumrad]

So he figured he would start with the woman in math issue. He brought a bunch of women in and a bunch of men.

[Claude Steele]

Sophomores, brought them into the laboratory one at a time. Gave them a half an hour section of the graduate record exam you take if you're a math major. Very, very difficult math. And sure enough, the women who had all the same credentials coming into that situation performed dramatically worse than the men.

[Jad Abumrad]

Worse as in it'd be a couple.

[Claude Steele]

Hundred points on an SAT test.

[Jad Abumrad]

Big difference.

[Claude Steele]

It was a big effect.

[Jad Abumrad]

So Claude Steele thought, all right, step one complete. I've got a lab situation that resembles the real world. Good. Now the next step is to tweak things a little bit. See if I can mess around with. Now normally in these situations, the test.

[Claude Steele]

Giver'S got a white lab coat on and he brings in a big stack of cellophane wrapped tests and he puts a clock on the table. It's all, it's all, you know, it's like that's gonna intimidate almost anybody.

[Jad Abumrad]

Maybe that's what's happening. He thought, what if I took away the clock, took away the coat? And most importantly, right before the test, I had the test giver, instead of saying the normal, I'm going to give you a test pre test thing, maybe instead say something like this.

[Claude Steele]

Look, you may have heard that women don't do as well as men on difficult standardized math test. You may have heard that, but that is not true for this particular test. This particular test does not show gender differences. Never has, never will.

[Jad Abumrad]

He wondered if maybe saying that simple sentence before giving the test would have an effect.

[Claude Steele]

And sure enough, I wouldn't be here if their performance didn't go up and go up to match that of the equally skilled men.

[Jad Abumrad]

That performance gap vanished.

[Claude Steele]

Gee, look at this thing. So we raced and did it very quickly. The same kind of an experiment with African Americans there.

[Jad Abumrad]

The pre test disclaimer went like this.

[Claude Steele]

This is an instrument that we use to study problem solving and it is not diagnostic of individual's intellectual ability.

[Jad Abumrad]

In other words, this is not a test of your intelligence. I repeat, not an IQ test.

[Claude Steele]

So just do the best you can.

[Jad Abumrad]

With that simple disclaimer at the start.

[Claude Steele]

Same kind of an effect.

[Jad Abumrad]

The black students and the white students were now equal.

[Claude Steele]

Just recently, Ryan Brown and Eric Day did an even cleverer treatment. There is an IQ test which is non verbal.

[Additional Researcher/Commentator]

It's called the Advanced Progressive Matrices.

[Robert Krulwich]

It has figures, very abstract, they got.

[Claude Steele]

Lines crossing that you have to match and so on.

[Jad Abumrad]

Checks.

[Narrator/Additional Commentator]

It's essentially pattern matching diamonds with dots in them, totally visual.

[Claude Steele]

And so they could represent that test as it is as an IQ test. It's in fact seen as the gold standard of IQ tests because it's quote culture free.

[Narrator/Additional Commentator]

There's no math, there's no reading because.

[Claude Steele]

It doesn't involve language. Or you could represent the exact same test as a puzzle.

[Jad Abumrad]

Puzzle, meaning you can give an IQ test to a bunch of kids and the black students will perform worse. But if you give that same test, lose the word test, lose the word IQ and just call it a puzzle.

[Additional Researcher/Commentator]

The black participants suddenly jump up in their performance.

[Narrator/Additional Commentator]

Basically we got a reversal when you.

[Claude Steele]

Represent it as a puzzle. Blacks perform as well as whites.

[Dan Engber]

They did.

[Robert Krulwich]

Yeah.

[Jad Abumrad]

That's all it takes, just change a few words. Stereotypes are powerful. Okay, that makes sense. But in terms of understanding how this works, can you make this tactile for me? Like if the stereotype that's having all these effects is like a thing, like a, like a little gremlin that bites, like when in the test taking process does it actually like do its damage?

[Narrator/Additional Commentator]

That's gonna be way open to debate.

[Jad Abumrad]

What does seem to be clear from the data according to Eric Day and Ryan Brown and Claude Steele, is that the gremlin only seems to appear when the test is sufficiently hard.

[Claude Steele]

If the test is easy, it's important to point out these effects don't happen.

[Jad Abumrad]

It's not that the gremlin is not there.

[Additional Researcher/Commentator]

Well, he walks in with you, but he doesn't speak necessarily until things get challenging.

[Claude Steele]

As soon as the test Gets difficult.

[Additional Researcher/Commentator]

That's where the voices kick in.

[Jad Abumrad]

Which means that for most of the tests, everybody's doing about the same. It's only at problem number 17, the one about cosines and factorials and whatever, where things start to go wrong. At least that's the theory. At that problem, the black student starts to stiffen up a little bit.

[Claude Steele]

That's right.

[Jad Abumrad]

And Claude Steele's measured this.

[Claude Steele]

Their blood pressure's elevated. Their short term memory is impaired.

[Jad Abumrad]

It's that flicker of frustration through their body that wakes up the gremlin who starts to whisper in their ear.

[Additional Researcher/Commentator]

I don't know if you can do this.

[Claude Steele]

Oh, shit. Is what they say about us true?

[Additional Researcher/Commentator]

They don't think you can do it.

[Jad Abumrad]

All the usual stuff. And even if the student doesn't believe.

[Narrator/Additional Commentator]

It, which is likely, see, you don't.

[Additional Researcher/Commentator]

Have to believe it. That's the kind of insidious thing here.

[Jad Abumrad]

Just the fact that he has now this extra bit of mental chatter, that.

[Narrator/Additional Commentator]

Little guy whispering, well, it's a distraction.

[Claude Steele]

And that makes their performance go down.

[Jad Abumrad]

Just a little bit.

[Additional Researcher/Commentator]

All of this dialogue is keeping you from being 100% focused on the task at hand, which is solving these problems.

[Jad Abumrad]

So the real subtle power of a stereotype isn't that it prevents you from doing the thing you want to do. It distracts you for just a beat from doing the thing you want to do. And that may be all the difference.

[Robert Krulwich]

So that's how we ended the piece that we did back in 2009. But in the years after Claude did that original study, the effect which he called stereotype threat became one of the biggest stereotype threat and most important ideas, stereotype threat in all of social psychology.

[Steve Spencer]

But now some psychologists say stereotypes can.

[Claude Steele]

Become self fulfilling prophecy.

[Robert Krulwich]

And Claude Steele, ladies and gentlemen, it's.

[Claude Steele]

My great pleasure to present Claude Steele.

[Robert Krulwich]

He became a sort of academic rock star.

[Dan Engber]

Professor Claude Steele.

[Robert Krulwich]

Dr. Claude Steele speaking to overflowing audiences at places like Columbia and Cornell. Welcome, Dr. Steele. And this idea of a stereotype threat was shown to be relevant in cases that had to do with age, socioeconomic status. There were studies about women playing chess, men being tested on like social sensitivity. I mean, Claude's work ended up inspiring sort of a whole generation of social psychologists.

[Michael Inslick]

Yeah, I mean, I would, I would say that the original stereotype threat paper by Steele and Aronson, you know, blew me away, just spoke to me and it was beautiful. And it seemed to offer answers.

[Dan Engber]

You.

[Michael Inslick]

Know, to questions that, you know, troubled me.

[Robert Krulwich]

So this is Michael Inslick and I'M.

[Michael Inslick]

A professor of psychology at the University of Toronto.

[Jad Abumrad]

So when did you begin? If you could give us sort of the Cliff Notes version of your history with stereotype threat in particular?

[Michael Inslick]

I mean, I think I was certainly attracted to. That's that part of social psychology that dealt with prejudice and discrimination. So I'm Jewish. I went to, you know, Jewish day school and high school and kind of perhaps baked into me was a desire to. For social justice and, you know, seeing, you know, the evils of prejudice and, you know, how those evils taken to their logical extreme, what could happen. So I was passionate about the topic. So it seemed like a very hopeful sort of explanation that also offered relatively easy solutions to fix.

[Jad Abumrad]

Sounds like you really came into it with a very social, political sort of bent to it.

[Michael Inslick]

Yes, that's right. I wanted my work to have an impact. I wanted it to. Yeah, to change the world.

[Jad Abumrad]

Coming up after the break. Michael tries to change the world, but the world kind of changes him. Him. Yeah.

[Additional Researcher/Commentator]

Hi, this is Vincent Rojas from Norman, Oklahoma. Radiolab is supported in part by the Alfred P. Sloan Foundation. Enhancing public understanding of science and technology in the modern world. More information about [e-mail].

[Jad Abumrad]

JAD Robert, Radiolab. We're back and we're talking about Claude Steele's seminal research on stereotype threat. How the threat of a negative stereotype is, can impact a person's academic performance.

[Robert Krulwich]

Right. And just before the break, we were talking to Michael Inslick, who after getting into stereotype threat research, he went on to grad school and really focused in on stereotype threat as his field of choice.

[Michael Inslick]

Yeah.

[Dan Engber]

So how many studies of stereotype threat did you end up doing?

[Michael Inslick]

I would say in the order of, like, in terms of number of studies, you know, 20, maybe 15 to 20, something like that.

[Robert Krulwich]

He did a lot of studies on women's performance in math, but also just looking at different environments and how they create or encourage these stereotypes. And over the course of his career, he ended up editing a book about stereotype threat.

[Michael Inslick]

Oh, yeah, yeah, yeah. I mean, there are.

[Robert Krulwich]

Gave numerous talks on the subject all over the world. He even signed a brief kind of explaining stereotype threat to the Supreme Court.

[Michael Inslick]

Of the United States.

[Narrator/Additional Commentator]

The Supreme Court heard the oral argument in Fisher v. University of Texas at.

[Robert Krulwich]

Austin as it applied to the question of affirmative action. The court will decide whether race used.

[Narrator/Additional Commentator]

In university and college admission policies. It's constitutional.

[Michael Inslick]

So I, you know, put my name to this.

[Robert Krulwich]

But then. He moved to Toronto.

[Michael Inslick]

I started my, my current job at the University of Toronto in 2005.

[Robert Krulwich]

And while he was there, he was running a lot of his stereotype threat research on women in math. You know, giving women a test and then doing some intervention to reduce the stereotype and seeing if there'd be a difference in their test scores. And simply put, for the first time.

[Michael Inslick]

I was not able to consistently show any effect of stereotype threat.

[Robert Krulwich]

In other words, the women who had received this stereotype threat reducing intervention performed just about the same as the women who hadn't.

[Michael Inslick]

So I think when I first got here, I, you know, failed to replicate some of these effects. I'm like, let me, you know, go back to the drawing board. Let me think about, you know, how is my population different?

[Robert Krulwich]

And his first thought was, maybe it has something to do with Toronto.

[Michael Inslick]

So Toronto, incredibly diverse place.

[Robert Krulwich]

Like, take the freshman psych classes he was teaching.

[Michael Inslick]

We're talking about, you know, a third East Asian, a third South Asian, and only about 15% Caucasian or European Canadian, and then a little bit of everything else.

[Robert Krulwich]

So he started wondering, to what extent.

[Michael Inslick]

Do our students even have, Are they aware of these stereotypes about, you know, women in math? And remarkably, when I would ask, you.

[Robert Krulwich]

Know, our students this, like, he'd ask, you know, who here has heard of the stereotype that went are bad at math?

[Michael Inslick]

I would say no more than like a quarter to a third had, you know, a strong awareness of this stereotype.

[Robert Krulwich]

So he started running the experiments again.

[Michael Inslick]

But I would only pick those women and men who actually had awareness. But even then, I still then couldn't get the effect. So it's like, okay, well, maybe I'm doing something wrong.

[Robert Krulwich]

Or maybe, like, maybe the interventions he was designing weren't reducing the threat. Or with these students, maybe the threat just wasn't that much of a threat in the first place. But by this time, Michael had already done a ton of research on stereotype threat.

[Michael Inslick]

And, you know, I'm a person who gets bored rather quickly and I just, you know, started. Started losing interest.

[Robert Krulwich]

But just a couple years later, well, so things changed. On October 17, 2011, this one paper was published. It was a paper called False Positive Psychology.

[Michael Inslick]

And this paper detailed how, just by.

[Robert Krulwich]

Doing some very standard practices in psychology.

[Michael Inslick]

Research using this technique, some of which were taught explicit grad school, you could.

[Robert Krulwich]

Sometimes end up with these sort of ludicrous conclusions.

[Jad Abumrad]

Yeah, so you're saying there were. This paper was showing ways in which experimenters could subtly, unknowingly juice their results. Is that essentially what it was?

[Michael Inslick]

Yeah, essentially.

[Robert Krulwich]

So the paper was pointing out that it Becomes sort of standard practice that, you know, when you were researching for some effect like, oh, how happy people were or how well they did on a test, that you would measure that thing in a couple different ways, which.

[Michael Inslick]

To some extent is considered good practice. Right.

[Dan Engber]

You want to.

[Michael Inslick]

No one measure captures your construct perfectly. So you should measure that thing in as many ways as possible. But now, what if you find that it, quote, unquote, works? Your hypothesis is confirmed with one of those measures, but it's not confirmed with three or four of the other ones.

[Robert Krulwich]

And it was not uncommon practice at that point to just report the one place where you got an effect, only.

[Michael Inslick]

Report the one that worked.

[Robert Krulwich]

You could even argue that you were just dialing in exactly where this effect happens. But the paper, this false psychology paper, pointed out that if you ignore the places where it didn't work, that's not.

[Michael Inslick]

Really a full picture of what that data actually looked like.

[Robert Krulwich]

The data as a whole, if you looked at all of it, might not actually support your conclusion.

[Michael Inslick]

I remember reading that my jaw dropped. I sent this paper, I circulated it to the other faculty in my department, and all of us, or many of us, saw the importance of this paper and we called it an emergency meeting.

[Jad Abumrad]

You called an emergency meeting?

[Michael Inslick]

Yeah, I'll never forget it.

[Jad Abumrad]

I mean, it was, what does an.

[Robert Krulwich]

Emergency meeting look like of the faculty.

[Michael Inslick]

And of the graduate students to just discuss the contents of the paper, to see what it meant, what the implications were.

[Robert Krulwich]

And as they talked it over, they realized that in some ways, probably some of their own work had fallen prey to the problems that this paper was pointing out.

[Michael Inslick]

Yeah, I did see myself in some of this. And I thought, you know, wow, like, you know, what has been implicated, what papers of mine, what papers more generally have been implicated in the field, you know, writ large.

[Robert Krulwich]

And in fact, meetings like the one that Michael found himself in started happening at universities and conferences all over the world.

[Dan Engber]

Yeah, people looked at that paper and everyone thought, oh.

[Robert Krulwich]

This is Dan Engber again.

[Dan Engber]

You know, this is what we're all doing this stuff. And now we know from this one paper that it's very, very easy to turn up spurious findings this way.

[Michael Inslick]

Yeah, it changed everything in part because.

[Robert Krulwich]

A group of scientists started thinking, wow, maybe we really need to go back and reinvestigate some of the key findings in our field.

[Dan Engber]

They call them high powered replication. So you're just kind of doing the same thing, but just with more people.

[Robert Krulwich]

So they started doing these studies with bigger sample sizes and with strict rules about what data you Were looking at.

[Dan Engber]

They say ahead of time, exactly what they're going to do and how they're going to analyze the data. So, you know, there's data, no possibility of monkeying around to get the answer you want.

[Robert Krulwich]

And these attempts to replicate or reproduce, you know, major findings in social psychology and the sort of panic that went along with it came to be known as the replication crisis.

[Narrator/Additional Commentator]

Replication, it's the cornerstone of science. News articles this week are talking about the reproducibility crisis in science.

[Michael Inslick]

It's like, oh man, what, you know, what is going on here?

[Robert Krulwich]

Because as they reexamined some of these.

[Narrator/Additional Commentator]

Studies, the facial feedback effect studies that.

[Robert Krulwich]

Got lots of coverage in newspapers and.

[Narrator/Additional Commentator]

Magazines, the experiment you'll be taking part.

[Jad Abumrad]

In today involves the.

[Robert Krulwich]

Some of these replications were failing.

[Dan Engber]

So specifically, what got me really into covering the replication crisis was news about ego depletion.

[Robert Krulwich]

So this is a whole literature of studies that were all about how we sort of use up our willpower.

[Dan Engber]

The original study is you go into a lab and you're presented with a dish of delicious fresh baked chocolate chip cookies. I love the method section of that paper. They describe baking the cookies in the lab so that the smell would be around the subjects when they come in and they put these cookies out and they say, you can't have any cookies. And then they leave you alone with the cookies.

[Robert Krulwich]

And what they found was that, you know, if someone has to sit there resisting a cookie with it right in front of them and the smell is wafting up their nose as they sit there, if they have to go through that, it'll actually be much harder for them to complete certain kinds of logic puzzles. And the argument was you use willpower.

[Dan Engber]

On task A, then you try to do task B and you just won't have a store of willpower unless later studies found you drank some lemonade in.

[Robert Krulwich]

The meantime, because sugar, the argument goes, replenishes your willpower.

[Dan Engber]

So there are sort of like increasingly bizarre elaborations of this original theory. It ended up working with M&Ms. It ended up working with cookies over the years.

[Robert Krulwich]

Dan says this idea that you use up willpower and place and have less in another just started entering all different corners of our lives.

[Dan Engber]

The insight of that original study was replicated again and again and again for decades.

[Robert Krulwich]

But then this group of scientists did this massive effort to replicate the original study. They had over 2,000 subjects. They followed these rigorous rules about, like, what data they were going to look at.

[Dan Engber]

This is as rigorous a replication as you can get. And they just found like no effect.

[Robert Krulwich]

Basically no effect.

[Jad Abumrad]

But I guess I still don't understand, like, how is it that they are finding nothing now, but before they had a study that was then replicated a bunch of times in a bunch of different labs? I still don't get what's going on.

[Dan Engber]

Well, so you have that original cookie study. I think it's notable that no other studies that I know of used cookies now. I found studies where M&Ms. Were used. So that just makes me wonder. I have no idea how that lab did their study, but it makes me wonder what would happen if I ran a lab and I wanted to reproduce this cookie finding and extend it in a new direction. And I kept trying it with cookies and it just never worked. Let me try it with Charleston Chews.

[Robert Krulwich]

Doesn't work because it's not hard to resist a Charleston Chew.

[Dan Engber]

Maybe Charleston Chews are not good candy. And I end up, I'm like, I do it with M&Ms. And it works. Boom, there's my dissertation. I publish a paper out of it. So now that's in literature. And so now ego depletion seems like an even stronger, more valid thing because it's not just about cookies now it's about cookies OR M&MS. My point is that you don't really know how many things were tried in each individual lab.

[Jad Abumrad]

Do you think that might be what's happening in these labs is that there's a lot of trial and error and the error is sort of swept aside and the successes are offered up and then suddenly you have one more success that bolsters the idea is that what you think might be happening.

[Dan Engber]

I think that is the heart of it.

[Robert Krulwich]

But whatever the problems are, not with those follow up studies. The big thing was that scientists were continuing to fail at replicating these big and fairly well known studies. Like the idea that when you smile, it changes your mood.

[Narrator/Additional Commentator]

Or so I want to start by offering you a free no tech life hack.

[Robert Krulwich]

The idea presented in one of the most watched TED talks ever, that the way you sort of hold yourself or.

[Narrator/Additional Commentator]

Stand so you make yourself big, you stretch out, you take up the idea.

[Robert Krulwich]

That that could have a measurable impact on your behavior or even your hormone levels, that one also failed to replicate. And that kept happening with study after study. And of course, you know, other people would come back and say, oh, the replication effort wasn't done right or you didn't really design it.

[Dan Engber]

Well, you're seeing a lot of the researchers who have made their careers studying certain effects. They're just not budging. A few of them are, but most of them are not budging. So you just have a split forming. One researcher described it to me as like a civil war within social psychology.

[Jad Abumrad]

So is stereotype threat now itself under threat? Is it one of these bodies of research that's being rethought?

[Dan Engber]

Well, no one has yet done the big multi site pre registered replication that they did for ego depletion, the one that like really woke me up to this.

[Robert Krulwich]

But Diane says there have been sort of smaller scale here and there kind of attempts to replicate some of the studies. And you know, some of those have failed.

[Dan Engber]

Some studies came out that found that, you know what, I tried to do the stereotype threat thing on a big group of students and I found that sometimes the opposite happened. When I tried to induce stereotype threat, the students did better.

[Robert Krulwich]

It's the you effect.

[Dan Engber]

Exactly. I'll show you.

[Claude Steele]

Well, let me say this, maybe this will help there, because this is something I've thought a lot about from the very beginning.

[Robert Krulwich]

So when we talked to Claude Steele about all this, he had a couple things to say.

[Claude Steele]

First of all, this research has been dramatically, well replicated.

[Robert Krulwich]

The stereotype a threat effect has been demonstrated, you know, way more times, in way more different contexts than really any of those other social psychology studies.

[Claude Steele]

I don't know if there's another, if there's another phenomenon that has produced so many demonstrations and if you can't replicate one of them or six of them, I don't know, I wouldn't. That doesn't surprise me.

[Robert Krulwich]

And he says, you know, you could even see the failures as just information about where the effect really applies and where it doesn't.

[Claude Steele]

This is science gradually getting sophisticated enough to help apply it in appropriate places.

[Robert Krulwich]

For example, he says he would only expect the effect to appear at times when the person is really invested in what they're trying to do. And thus the negative stereotype, you know, really is threatening. On top of that, he says the kinds of stereotypes that are actually threatening to a given group might change over.

[Claude Steele]

Time if you just exactly replicated what we did 25 years ago. I'm not sure that the stimuli and the procedures would have the same meaning with today's college students that they had then. Social psychology is the meanings come from the contemporary moment, the way the state of the culture at that time.

[Jad Abumrad]

Yeah, I mean that is interesting actually, that, I mean if what this research is doing is studying like an individual's relationship to like threat and like that's gonna be different Depending on who you are. Because you're gonna find different things threatening where you are or even when you are.

[Claude Steele]

Yeah, I don't have. I don't think I know enough about, quote, the culture of Black students today versus the culture of Black students 25 years ago. But it wouldn't surprise me that there are some real differences. So I don't put as much stock in the exact replication of experiment A or B as I do in the conceptual replication.

[Dan Engber]

Is it possible, I mean, you mentioned there are just hundreds of studies that kind of circle around the same idea in different ways. That is, on its face, very compelling evidence that, you know, this is a robust phenomenon. But at the same time, you know, there's people who gather all the data together and they say, look, you know, maybe there's some kind of bias that slips in when people are doing this research. They just keep trying different versions until they get something that looks like a result. Does that seem plausible to you?

[Claude Steele]

Boy, that's a deeply, I guess, cynical, you know, account of a scientific literature this big. It would seem to. That doesn't seem highly probable to me. It doesn't seem highly likely to me. It's clearly real and replicable under these circumstances. Just because they're not everything doesn't mean they're not extremely important to the progress of this society.

[Robert Krulwich]

And in fact, Claude points out that the stereotypes threat is pretty unique in the fact that many of the studies in this literature are not just in the lab. They've been taken out into the streets with real people.

[Claude Steele]

That's really where the, you know, the tire meets the road is, can you actually move the educational performance of real people in real school situations?

[Robert Krulwich]

And Claude actually sent us a list of several dozen studies showing that intervention.

[Claude Steele]

Is designed to reduce stereotype threat can have dramatic and long lasting effects on achievement.

[Robert Krulwich]

Now, we should say there was at least one case. There was a study done in 2006 that a researcher tried to replicate in 2011.

[Dan Engber]

It's just much smaller than the original 2006 effect.

[Robert Krulwich]

And then just last year, he did.

[Dan Engber]

It again with more students this time, and he got pretty much nothing. This is the same guy in the same school system trying to do the same study with, you know, hundreds and hundreds of kids, and he came up with nothing.

[Jad Abumrad]

I'm curious to know, given that replication has become a conversation that you have to unfortunately contend with. I'm just curious if it's changed your opinion of the work.

[Claude Steele]

No, I mean, I don't think there's anything that could make me go uh oh, this whole thing is not true.

[Steve Spencer]

I mean, I want the truth out there more than I want anything else.

[Robert Krulwich]

This is Steve Spencer. He was an early collaborator with Claude Steele, especially on those studies involving women and math tests.

[Steve Spencer]

Recognize in some of the critiques real issues that we need to deal with.

[Robert Krulwich]

But Steve, like Claude, is very confident in the results of his studies in the effect of the stereotype threat in general. So much so that I'm writing right.

[Steve Spencer]

Now an article where I'm going to disclose every single study I've ever done, what the results are and put the data up for everybody to look at. I will admit to the questionable research practices that I've done and be as forthcoming and honest with everything in my own lab. In addition to doing that, I've entered into an adversarial collaboration, it's called, which.

[Robert Krulwich]

Just means he's gonna do a big scale reproduction study with people who have.

[Steve Spencer]

Serious doubts about whether stereotype threat is real. You know, I can't say ahead of time what my reaction will be. I think what I can promise is that I will take the findings very seriously and I will do my utmost not to be defensive about them.

[Dan Engber]

Are you nervous about this? No, I mean, the stakes sound pretty high.

[Steve Spencer]

Oh, no, I mean, you know what you mean the stakes are high. I mean, I'm a full professor, I have tenure. What are the real stakes for me in this? Not really much, but.

[Robert Krulwich]

Well, not everyone in the field is handling these kinds of niggling doubts so well.

[Michael Inslick]

There are so many pieces of evidence that things are not all right.

[Robert Krulwich]

This again is Michael Inslick, the professor from Toronto.

[Michael Inslick]

To be faced with the probability, the likelihood that all this might have been for naught, or much of it might have been for naught. You know, it's unsettling, right? It's a loss of meaning. You know, was I doing good work? Was I contributing to knowledge? Was I part of the problem? Was I chasing, you know, signal? Was I chasing noise? I mean, I think the effect is, you know, it might be there, but it might be so small as to not be. Not be meaningfully important.

[Jad Abumrad]

That's interesting.

[Robert Krulwich]

I don't.

[Jad Abumrad]

It's hard to know where to stand on this.

[Robert Krulwich]

Well, one thing that you know, should make clear is that stereotypes can be really damaging. I mean, having someone tell you you suck at something when you're under the gun to do it, that's always gonna, you know, that's gonna have an effect. I guess the question is, yeah, how, you know, what effect exactly and when and how and what can you do about it? And. And those things feel like maybe. I guess it feels like the part of me that wanted this to be a kind of very simple fix that would work everywhere and sort of save the world, you know, that part is, you know, feeling a little bit of a lot like, worrying that this whole thing is shrinking on me a bit.

[Jad Abumrad]

Yeah, totally. It does feel weirdly like we're all growing up a little bit because you kind of just have to walk away from those big, simple promises, you know.

[Robert Krulwich]

Maybe that's exactly what you need in order to be able to find the smaller places where you can have an effect, you know, right here, right now, with this person trying to do this particular thing.

[Jad Abumrad]

Producer editor soren wheeler. Thanks to him. Thanks to dan angber.

[Michael Inslick]

This piece was produced by simon adler and amanda aronchek.

[Jad Abumrad]

Okay, I guess we should go.

[Michael Inslick]

Yeah.

[Jad Abumrad]

I'm jad abumrad.

[Robert Krulwich]

I'm robert grill.

[Jad Abumrad]

Which. Thanks for listening.

[Liza]

Oh, hey, Liza. Here on a Monday morning with staff credits and a strong cup of joe. Radiolab was created by Jad Abumrad and is produced by Cern Wheeler. Dylan Keefe is our director of sound design. Maria Matisse. Arpedilla is our managing director. Our staff includes Semi Nadler, Becca Bresler, Rachel Cusick, David Gable, Bethel Habte, Tracy Hunt, Matt Kielty, Robert Crowge, Annie McEwan, Malad Dev Nasser, Melissa O', Donnell, Arian Wack, and Molly Webster, with help from Amanda Aronchik, Shima Oliai, David Fox, Nagar Fatali, Phoebe Wayne, and Katie Ferguson. Our fact checker is Michelle Harris.