Loading summary
Transcript78 lines
- [00:02]
Brian Phillips
Did you know that scientific studies have found most people lie once every 10 minutes? In my new podcast, Truthless, I'm talking to people about the lies they tell. From faking illnesses in high pressure moments to making up stories on national TV from Spotify and the Ringer Podcast Network. I'm Brian Phillips. Listen to Truthless on Spotify or wherever.
- [00:29]
Derek Thompson
You get your podcasts.
- [00:32]
Brian Phillips
Foreign this episode is brought to you by Amazon. Sometimes the most painful part of getting.
- [00:44]
Derek Thompson
Sick is the getting better part.
- [00:46]
Brian Phillips
Waiting on hold for an appointment, sitting in crowded waiting rooms, standing in line at the pharmacy. That's painful. Amazon One Medical and Amazon Pharmacy remove those painful parts of getting better with things like 247 virtual visits and prescriptions delivered to your door thanks to Amazon Pharmacy and AmazonOne Medical Healthcare just got less painful. This episode is brought to you by Lifelock. The New year brings new health goals and wealth goals Protecting your identity is an important step. LifeLock monitors millions of data points per second. If your identity is stolen, LifeLock's restoration specialist will fix it, guaranteed or your money back. Resolve to make identity, health and wealth part of your New Year's goals. With Lifelock, save up to 40% your first year. Visit lifelocked.com podcast terms apply today's episode.
- [01:39]
Derek Thompson
Must start with a confession. I adore psychology. I wish I had majored in it. Some of my favorite episodes that we've done on this show have been with psychologists who I think do interesting, important and true work. But for a long time, my love of psychology has been intertwined with this other feeling, a darker feeling that sometimes when I'm reading a book or a paper that is filling me with that best feeling, that gosh, wow. Sense of wonder at the majesty of the human mind. The thing that I'm reading is not real. In the last decade, several major findings in social psychology have turned out to be hogwash, according to a phenomenon known in the field as the replication crisis. For example, perhaps you've heard of power poses based on a 2010 study finding that subjects reported stronger feelings of power after they posed, say, with their hands on their hips for several minutes. When psychologists tried that experiment again, it did not replicate. Or perhaps you've heard of ego depletion. Ego depletion? This even more famous assertion that when people make a bunch of decisions, it exhausts their ability to make future decisions. In follow up research, that finding too did not replicate. Now science is hard. Digging out fundamental truths about how our brains work is difficult stuff, and I think psychologists deserve a certain amount of latitude as all scientists do. Researchers who are doing their best can reach conclusions that as solid in psychology as, say, the laws of physics. But the problems in this field go deeper than replication. In the last few years, a crisis has been billowing at some of America's most prestigious business schools, where the work of academics that has made its way into TED talks and best selling books is slowly being revealed to be based on not just mistakes, but in some cases, outright fraud. Adam Mastroianni, a psychologist who writes at the site Experimental History, put the crisis in psychology this There's a thought that's haunted me for years. We're doing all this research in psychology, but are we learning anything? We run these studies and publish these papers and then what? The stack of papers just gets taller? I've never come up with satisfying answers, but now I finally understand why. Today's episode features two interviews. First, I talk to Adam, a social psychologist who does research and also writes at this newsletter, this wonderful newsletter, Experimental History, about his big picture critique of the field, how psychology too often fails as a science and what it could do better. And second, we speak with the Atlantic journalist Dan Engber. We talk about his reporting on a billowing scandal in psychology that has enveloped several business school stars and raised big questions about the future of the field. What is psychology for? What would progress in psychology actually mean? And how can this field, which might be the discipline that I like more than any other in academia, how can it become more of a science? I'm Derek Thompson. This is Plain English. Adam Mastroianni, welcome to the show.
- [05:47]
Brian Phillips
Thanks for having me.
- [05:48]
Derek Thompson
So we are about to engage in what I hope will be a really fun and adventurous critique of the field of psychology. But I want to begin by saying some nice things about psychology, starting with why did you become a psychologist? Was there a moment in a classroom, reading a book, watching people at a high school party where you thought, holy shit, this is it. This is how I have to spend the rest of my life?
- [06:14]
Brian Phillips
Yes, I remember the moment very specifically. I was taking a social psychology class. We were reviewing for the midterm and playing a Jeopardy Game to review. And I answered one of the bottom row questions, the ones that are supposed to be the hardest. And the graduate student who was leading this review session looked at me and was like, you're pretty good at this. And I was like, me good? And then, and then I thought like, oh, I do work pretty hard at this. That must mean that I like it. Like, oh, that's what self perception theory would say that you can infer your preferences from looking at your behaviors as if you're observing yourself from the outside. I was like, oh, maybe I should. Maybe I should be a psychologist. And around that time, I got an email that, like, there's this psychologist named Dan Gilbert who's looking for research assistants for the summer. And I just read his book, which is about how we're really bad at predicting what makes us happy. And I was like, this guy lives on Earth. You can go hang out with him. You can do this stuff with him. And. And I did. And it was so much fun to, you know, to run studies about. Like, oh, you think this is how you're going to feel? Well, we'll find out if that is really how you feel. I mean, it felt like. Like opening a back door to the mind. Like, who? I don't understand how anybody does anything else.
- [07:27]
Derek Thompson
What do you think is the most important thing that the field of psychology has done for human knowledge?
- [07:34]
Brian Phillips
I think it's done a really good job at overturning our folk psychology. So before there was science, humans had to make sense of the world somehow. And we made sense of it with what psychologists now call folk intuitions, folk theories. So you got your folk psychology, your folk biology, your folk physics. And I think our folk psychology of all of these is the thickest and the deepest because we need to know the most about how other humans behave and how they work and how ourselves work as well. You don't need to know that much physics to get through life. You need to know enough to not fall off a cliff or run your SUV into a bunch of pedestrians. But other people are really complicated, and there's a lot you need to know about them. And so we develop all of these rules of thumb or all these sort of general theories about other people that now that we have a thing called science, we can test. And this is where I think psychology has been really successful, is taking these things, is eliciting these beliefs that we have and putting them to the test. I mean, that's what brought me in in the first place, is like, oh, we have this theory that when bad things happen to us, we'll be sad forever. And then it turns out that's not true, that remarkably quickly you go back to feeling the way that you always feel. That, to me, felt like a revelation, and it felt like a permanent upgrade to my life to know that. And so I think that's what we've done really well, and that's how we've benefited humanity.
- [09:02]
Derek Thompson
You write that if we had to Boil down everything that psychology, and in particular social psychology, has figured out about the human mind. All that knowledge really fits into two buckets that you call proto paradigms. The first proto paradigm is. Is cognitive bias. What is it that we fundamentally uncovered here and how long have we been at it?
- [09:24]
Brian Phillips
So I think we've been at this generously for, like, 50 years. So a lot of people would start this history with Danny Kahneman and Amos Tversky publishing their landmark papers in the 70s about heuristics and biases, and that created a whole field. And that went on to win basically two different Nobel Prizes, not even in psychology, because there isn't one. They won Nobel Prize. These ideas won Nobel Prizes in economics because that's the closest thing. And that work, I mean, is really cool. And you can tell because everybody knows about it, right? Like, people know those words. People know about, like, the availability heuristic. Like, these are things that if you wanted to entertain people by telling them about them, they'd be like, we've heard all this before.
- [10:11]
Derek Thompson
Do you have a favorite? Do you have a favorite Kahneman, Tversky bias?
- [10:15]
Brian Phillips
Yeah, the ones that I like the most are the ones that you can replicate in the course of a conversation. So, like, availability. Like, do you think there are more words that start with R or that have R in the third position? And people are like, more words that start with R, but it actually turns out there's more words that have R in the third position, and it's because it's easier to think of words that start with R than it is to think of words that have R as the third letter. And this is a nice little demonstration that one of the rules of thumb that we have for estimating how common something is is by how easily it comes to mind. And this shows up not just in something silly like estimating the number of words, but also estimating the danger of various things, like how safe is it to get on an airplane versus to get into a car? It's easy to think of plane crashes, hard to think of car crashes, even though there are many, many, many more of the second one than there are of the first.
- [11:08]
Derek Thompson
The second proto paradigm that you say psychology has gifted us in the last half century is the concept that situations matter. What does this mean?
- [11:18]
Brian Phillips
It's funny, because now it seems almost silly that that would be interesting, and I think it's because of the success of this work. So there was a moment, I mean, maybe the entirety of human history up until we started doing this work, where people had A reasonable theory that there are just certain kinds of people in the world and certain kinds of people are capable of something and other people aren't. And what this work did was essentially through a series of elaborate pantomimes, show that you can create situations where people do things that you thought that only the craziest kind of people were capable of doing. The classic one here is the Milgram shock experiments from the 1960s, where if people aren't familiar with these, basically you come in and you think you're teaching someone in another room how to learn words and you shock them when they get them wrong. And through some stage managing and sleight of hand, it actually turns out you're not delivering shocks. But 2/3 of people in this study shocked someone basically to death, or so they thought. There have, by the way, been some attempted debunkings of that work that although many lots of that work from that time has been debunked, these, I think, have survived those attempts. And so those, I think, really stuck around because they are such incredible demonstrations of the power of the situation. Oh, you thought that only an evil person was capable of shocking a stranger to death? No, it's your neighbors, it's you in the right situation, you could do that. And the fact that now that doesn't seem so surprising is because that work did so well at demonstrating that.
- [12:55]
Derek Thompson
So in addition to having some nice things to say about psychology, one really fascinating theme of your work is, is that you think this field has actually failed in many ways to achieve the kind of progress that other fields of science have achieved. And from reading your work, I would say that there are three strikes that you make against the field of psychology. Three big picture critiques. And I want to go through those. One, two, three. The first is that many of the most famous findings, including findings in these two buckets we just talked about cognitive bias and situations matter just don't seem to replicate. That is to say that when you do these studies over and over again, they come to different conclusions in terms of what happens or what the strength of the finding is. And the proper context here to set you up is that nine years ago in 2015, Brian Nosek at the University of Virginia published, or excuse me, persuaded several hundred psychologists to redo 100 published psychological experiments to see if they would get the same result. And many of the studies failed to replicate. And this became famously known as the replication crisis in psychology. We've been living in the world of the replication crisis in psychology now for nine, almost 10 years. What do you think is the most underrated consequence of the replication crisis for psychology? Why is it important?
- [14:14]
Brian Phillips
I think there are. If I can do two, there are two. One good and one bad. The. The good underrated consequence, and you can see some people saying this now, but it takes a while for everyone to catch up. That's not watching this very closely, is that we're much better at doing this now. That there are things that you could do even 10 years ago that would get you laughed out of a room now, if you tried to put them up in terms of dropping people from your studies because, oh, they're outliers or testing your effect as it goes along and then stopping if it's significant. These are things that nobody. I shouldn't say nobody, that most people thought were fine things to do. And through some really effective demonstrations, we know that now we shouldn't do them because it inflates basically how easy it is to show that something is statistically significant. That's the good news. The underrated bad news is I think we've learned the wrong lesson from the replication crisis, that now the first question that we ask of any work is, is it true? Which is a sensible question to ask, but we should instead ask a different question first, which is, would it matter if it were true? And the thing that I think we haven't learned about the replication crisis is most of the work that doesn't replicate didn't matter in the first place. Like, I didn't care that it happened the first time, so I don't care if it happens the second time. And I think a piece of evidence for this is in that study you just cited, you know, these hundred studies we tried to replicate. Nobody can tell you which ones they were, which you might think would be like, well, wouldn't we be interested to be like, oh, man, is it one that I rely on? And it's not, because nobody really relies on anything. Like, we each create these little research worlds unto ourselves. And that is, I think, the underrated bad consequence that we've learned to ask the questions in the wrong order.
- [16:04]
Derek Thompson
This is such a subtly important point, and it was fairly mind blowing to read it in an essay you published several months ago. And I just want to pause and dilate on it because it is really significant. Significant. If in the field of physics or chemistry, there was an attempt to replicate 100 of the most important studies and 40 to 60 of them failed to replicate, the world would change, right? We would say, like, oh, my God, like, chemistry doesn't work. The way we thought it did. We thought that we were poisoning the atmosphere with carbon dioxide. It turns out that we were just actually emitting stuff that wasn't carbon dioxide at all into the atmosphere. There's no global warming or. Oh, all of the ideas from chemistry have been totally changed. That's not steel at all. There's no difference between steel and cardboard. They're actually the exact same material. These are the kind of things that would follow a revelation that half of what we assumed in the fields of chemistry and physics turned out to be false. But instead, dozens and dozens of conclusions from the field of psychology were found not to be true. Things like social priming or subliminal exposure. Right. Like having little flashes of the word like rats in an ad that's trying to get you to hate Democrats and get people to associate Democrats with rats. Ego depletion. The idea that we have a limited supply of willpower. So if I go to the grocery store and my wife makes me order 17 things rather than two things, I'll have less energy when I come back to the house. All of these things turned out to not be necessarily real. And it seems to have changed the average person's life not one bit. And that has to, I think, at some level, create some kind of crisis of confidence or existential despair in psychologists to think, what if our entire life's work is zeroed out and no one's life has changed because of it? Do you buy into this rant that I just did? Do you think that it is a little bit of an existential crisis for psychology to have this replication crisis happen and most people's lives just move on exactly as they did before?
- [18:11]
Brian Phillips
I think it should be, but it generally isn't. I mean, I grew up basically as a psychologist while this crisis was unfolding. And certainly the feeling that we all had was like, oh, wow, the stuff we Learned in psych 101. We were lied to, basically. How dare they? But I think no one ended up asking the second question, which was like, why did we care in the first place? A lot of these are cool little stories, but what is riding on the fact of. Does holding a pen in a certain way in your mouth make you think jokes are funnier versus less funny? Which is one that people have really spent some time trying to figure out whether it's true or not. What were the stakes?
- [18:48]
Derek Thompson
I just want to pause because this is a real study. It was a real study that was done of whether or not a joke is funnier depending on whether the pen is Held in your teeth or by your lips and not touching your teeth. And this was overturned?
- [19:05]
Brian Phillips
Yes and no. I think the most recent version, there may be something more updated. But the last I checked on that finding, it was, we think it might be there if you're not recording people. So they ran a study where they had a camera or not, and like, well, maybe we're making people self conscious. And then other people were like, actually you look at that after the fact. This is just trying to make the effects show up. So I don't know if we've settled this one and I'm not that interested to know how it settles, because what is this really about? This isn't some parlor trick. This was meant to be about, well, can people's physical posture or the way your face moves influence the way you feel about something? Well, I certainly believe it can. Does it in this situation? Like, maybe, maybe not. But that doesn't really change whether or not I think it is at all possible to construct any situation in which someone's physical arrangement could change how they feel about something. It's like, it's not. It makes no sense to put so much on that one demonstration. If we tried everything we could and could never find a situation in which like moving someone's body around makes them feel different, I would both be very surprised. But I'd also be like, well, I guess, I guess that reasonable sounding hypothesis is wrong, but it can't be solved by being really sure about whether this pen thing works or not.
- [20:32]
Derek Thompson
The second major strike that you make against the field of psychology is that in addition to this replication meaning crisis, many assumptions that we hold deeply in psychology actually don't seem to hold up against what you call folk wisdom. You, you have a very entertaining example of personality tests. There's a theory of personality in psychology, the big five theory, that has been used in, I don't know, hundreds, thousands of papers. But when you put it to the test against personality tests that people just pull out of their butts, it doesn't seem to do that much better in terms of predicting behavior, predicting personality. Can you unpack that point a little bit for us? First, explain what is the big five theory, and then also for people who might hold on to the big five theory as being their way of expanding personality to themselves, what is it that we've started to learn about the big five theory? Not actually working?
- [21:31]
Brian Phillips
Yeah, the big five is like the Honda Civic of personality psychology. It's solid. Everybody thinks it's great. Some people would like a different model That's a little fancier or a little less fancy, but this one satisfies. And the idea is that everything about human personality comes down to these five factors. Openness to experience, conscientiousness, extraversion, agreeableness, and what used to be called neuroticism and is now more politically, correctly referred to as emotional stability. So that no one has to feel bad about scoring high on it. And the idea here, and the thing that worked out empirically over and over again in studies is that whatever you measure about someone's personality ends up falling on one of these five factors. So you can reduce all of that data down to these five things. And you know, we've, we've run these studies for decades. This has been like a great achievement, as you said, thousands of studies, probably millions of dollars. And one reason it stuck around is because it does a good job at predicting other things. That if I get you to take one of these big five questionnaires, I am, I'm good at figuring, finding out like, oh, are you married? Are you likely to have this kind of job versus that kind of job? Are you likely to commit a crime? So that's one way of testing whether it works. Another way is testing it against basically a personality test that like, was not developed scientifically. Like for instance, the Enneagram, which comes out of a spiritual tradition. And no offense to what they, they're trying to do a different thing, right? They're trying to help people better understand themselves. They weren't trying to run like make an empirical scientific instrument, but they have this questionnaire and it'll tell you you're one of nine types. And so this group called Clear Thinking earlier this year tested these against each other. They had people take a Big 5 questionnaire and also take an Enneagram. And by the way, take a few other Myers Briggs inspired test because the Myers Briggs is actually proprietary as well as they figured out their astrological sun sign and they just tested like, okay, which of these does the best at predicting other things like are you married? Have you bought a house? Have you been convicted of a crime? These life outcomes that we might be interested. And when you put them head to head, the big five and the Enneagram performed just as well as each other, which is a humbling moment, or at least it should be for us that this thing that we worked on really hard and poured a lot into it should have blown the other test out of the water. But in fact we are performing on par with people who had no intention even of performing well, in this way, and certainly didn't refine this with years of study and evidence. So that should make us sweat a little bit.
- [24:18]
Derek Thompson
So up to now we've established that too much psychology, A, doesn't replicate, B, ask questions that don't matter, C, fail to predict human behavior better than ideas, more or less made up by spiritual teachers. And God bless spiritual teachers. But that's not the field of academic psychology. The final strike that you make against the field of psychology is that it has struggled to turn its knowledge into technology. And that might seem a little bit ethereal to some listeners at first, but I do think it's important to remember that the reason to study something, the reason to glean science from the cosmos, should ultimately be to make things that wouldn't otherwise be possible, right? That you build bridges with actual physics. This is your point. I think I'm stealing. You build bridges with actual physics, not with folk physics. That's why the bridge stands up. But you showed that it doesn't always work out that psychology produces so called technology that works. Tell us about nudges, what nudges are and what evidence you have that they haven't worked out in the real world as much as some psychologists and behavioral economists would like.
- [25:32]
Brian Phillips
So a nudge is the idea that you can make a small change to someone's environment and get a big change in their behavior. And so like a classic example is you change how the food is laid out in the cafeteria and as a result people eat more vegetables because you put them in the closer to where people enter or something like that. This has been hot for now like 10, 15 years and there's been a lot of research on it and a lot of people showing like, look, we can nudge people this way and we can nudge people that way. And I think really the way to see like whether this approach works, whether we've nailed it, whether we've figured it out, is to look at like, well, what's the state of the art? The state of the art right now is this approach called the mega study, where they test a bunch of interventions at the same time to see which ones work. There was a paper that came out two years ago that was trying to do this to get people to go to the gym more often, which is already like, people already go to the gym. People want to go to the gym, right? This isn't like, this isn't smoking cessation. This isn't trying to get people to switch their vote. Like, this isn't trying to get people to levitate or go to the moon. This is like very low level, like trying to help people do something they already do and want to do more.
- [26:46]
Derek Thompson
It's sort of like if nudges should be able to do anything, they should allow people to align their preferred life. I go to the gym all the time. With their lived life. I actually do go to the gym all the time. Right?
- [26:58]
Brian Phillips
Yes, yes, yes. We should be able to unblock like the, the, the dam between people's intentions and their behavior. And like, and these are people who did this, who are, who are like, they are top of the field, like, this is as good as it gets. And what approach did they take? They were like, look, we don't know which approach is going to work. So let's try 53 different interventions. You know, based on our best guesses, half of those interventions did not produce detectable effects. They did not help people go to the gym more often. And in that same study, they had experts in the field, behavioral scientists and public health experts, try to predict which interventions were going to work and they couldn't do it. Not only did they dramatically overestimate how effective the interventions would be, they couldn't predict which ones would work and which ones wouldn't. And in fact, in another mega study that was about vaccine uptake, actually non experts outperformed experts in terms of predicting which interventions would work. So, like, is it, can we in any situation use nudges to like, help people live better lives? Yes. But when it really comes time to like, okay, in this situation, what should we do? We really don't know. And not even to mention, like, okay, we did this with this gym. Would it work for getting people to return their library books on time? Would it work for a yoga studio instead of a gym? Like, would it work for helping people stop smoking? Like, like for all those we don't know, we'd have to run another mega study.
- [28:30]
Derek Thompson
You have a very funny line in the piece that says, this isn't super reassuring. This is your reaction to the mega study where only half of the 53 nudges worked and it was difficult to predict which ones. You say, this isn't super reassuring. If I hired a plumber to install a toilet in my house and he was like, sure thing. I'll just install 53 different toilets and then check which one's flush, end quote. That's a very good relation of. Here's a physical world property that we understand. Water go down. And here's another physical world property we don't entirely understand. Which is behavioral nudges to get people to go to the gym and lift weights. I think we've done a really good job here of establishing the baseline critique that too much psychology doesn't replicate. Ask questions that fundamentally don't matter, fails often to predict human behavior better than folk wisdom, and finally fails to translate into effective policy at better than a coin flip. 50, 50 rate. So you publish these articles, they're read by psychologists, by people who have, like you, devoted their life to hopefully not entirely failed enterprise, but at least a difficult field to push forward in terms of human progress. What's been the feedback? I mean, surely some folks have said, you're a jerk, you don't know what you're talking about. What's the feedback been like, yeah, I.
- [29:54]
Brian Phillips
Gave a talk at a conference of social psychologists a couple weeks ago, and the reaction spanned from people nodding along and going like, yes, yes, it's all a lie. And be like, hey, man, what are you gonna do? You gotta make a living somehow. And other people who were livid, people would come up to me afterward and go like, no, no, we're really good at making technology with psychology. I mean, just look at, you know, social media. Those are like, those are made by psychologists and they're controlling our minds. And I'm like, I under, I understand this critique. What knowledge from psychology are they using? Like, can you point to the papers that they are, like, then taking and feeding into the algorithm, or they're using to, to like, control our minds? And he's like, well, we can't know because they won't tell us. And I'm like, but these are, we trained these students, we taught them everything they knew. Like, what is it that they know that we don't know? And I think, in fact, that's been very much overblown as to the usefulness of findings from social psychology for making social media more addictive. That I bet. I mean, we don't know for sure. But I bet that most of what makes social media work they've discovered from their own testing. Because again, these people come from our field. We would know. They publish papers. So if it was something like that, we would know. Just as an example here, years ago, there was this Cambridge Analytica scandal where basically the upshot was like, there's this company that's scraping your Facebook data illegally, and they're using it to micro target ads to you that then mind control you and make you vote for Trump. And I think this was like a really useful scapegoat for people who were surprised and dismayed that Donald Trump won the election. And then some people tried, some academic psychologists were like, hey, can we do this, by the way, this whole micro targeting thing? And they published a paper where they're like, yeah, it doesn't. Well, they couldn't really get it to work. Very micro targeting helped a little bit, but it actually turns out to be really hard to persuade people of things. And, yeah, it doesn't matter that you know which Hogwarts house they're in because they took a Facebook test and now you, like, stole their data that it just turns out, like, yeah, knowing someone is a Slytherin versus a Hufflepuff, like, is maybe a little bit better than just guessing, but, like, it doesn't open up their skull and allow you to, like, muck around in there.
- [32:24]
Derek Thompson
So let me give you my own feedback. I am persuaded by so much of what you've said, but there's two bumps that I have when I hear your argument. The first is that even if psychology isn't a hard science that produces mathematical truths, like in physics, that you can use to design structures like nuclear power plants, the truths of psychology are still true enough. That is to say, an advertising department that understands cognitive bias really can make a commercial that's 5 to 10% more effective. And a salesperson who understands social psychology can be a 5 to 10% better salesperson, or a listener who understands psychology can be a 10% better friend. And so even if psychology is doomed to messy truths, those messy truths are true enough to be useful. The second is that in clinical psychology, because my wife's a clinical psychologist, I know that, for example, cognitive behavioral therapy, cbt, has been shown to work very well, not just in the US weird Western educated, industrialized, rich countries, but also around the world. In 2022, Christopher Blattman and several other researchers tried CBT in Liberia and found that it had strong effects among the population there in terms of reducing anxiety and depression. So how do you feel about those two fronts of a critique? Number one, that psychology might not be physics, but it's certainly better than nothing. And so it can absolutely help some people. And number two, that there seems to be some advances in terms of turning the science of psychology into the technology of something like cbt.
- [34:06]
Brian Phillips
Yeah, for both of these, I would say totally possible, but we're not sure enough that I think the costs are clearly outweighed by the benefits. So, to the first one, yeah, it's possible that advertisers and marketers do a little bit better because they know about things like social norms or social proof, or they've read some studies about persuasion at the same time, like we actually have. I mean, this is purely an anecdote, but I think a representative one where this year there was this big project trying to basically get Democrats and Republicans to like each other more. It was called the Strengthening Democracy Challenge. And people could send in the interventions they thought were going to do this to, like, you know, get people after a short intervention to say that they like the opposite party more. One of the interventions that worked the best was a Heineken commercial from 2017 that was, like, not made as far as I'm aware. Like, based on. They read a bunch of literature. They just, like, showed people making and, like, a piece of Ikea furniture together and then having a beer. It's a great commercial. It makes me want to buy Heineken beer. It also, I guess, makes people feel a little nicer toward the other side. But that was not made by psychologists trying to do that. That was made by beer marketers trying to move units of beer. And so, yeah, it's possible they had a head start because they had read a book of psychology. But the head start doesn't seem to be very big to your point about cbt. So I'm a social psychologist, so I don't treat patients. And so this is outside of my direct area of expertise. But as some of what I read for writing this, this piece, there's an ongoing. There is a, I would say, significant minority of folks who. Who make the point that, like, it kind of seems like all psychotherapies work equally well, it's at least hard to tell the difference between them. In fact, there is one paper I read where the author was like, we should stop developing new therapies because it seems like they all work. And what we really need to do is just get them to more people. And look, I don't know enough to weigh in conclusively on one side of that or the other, but it does seem like the margins are close enough that it's not a slam dunk. And so, again, I don't think it's that psychology has made zero progress, but if what we're arguing about is whether the confidence interval includes zero, I think we should be worried about the state of the field.
- [36:42]
Derek Thompson
What is your grand theory here? Do you think social psychologists are dumb? Or do you think that social psychology is young, that this is a difficult field and people aren't atoms? And psychological truths are in many ways much more complicated than those in chemistry or physics. Right. Because people are incomprehensible, including to themselves. And so this field is like my daughter just learning to walk and talk and use utensils. This is just an infant discipline.
- [37:15]
Brian Phillips
Yeah, it's exactly the way I feel that I know all this can read as, like, psychology is so dumb. You guys are such idiots for doing, doing it. It's not the way I feel at all. I actually feel like we're just getting started. But the real mistake that we could make is thinking that we're done thinking that we're just crossing the T's, dotting the I's. And like, what we figured out so far is like, you know, we'll discover some, a few more things, but, like, this is pretty much the way that it is. And you know, like, paradigm shifts, that's a thing that used to happen and doesn't happen anymore. Like, that's the reason why I care so much about making people dissatisfied with it. Because you need to be dissatisfied before you start to entertain the sort of strange ideas that might bring about that paradigm shift. So, yeah, this isn't like, you know, lecturing a 13 year old that they should be 40. Like, it's fine to be 13 years old. It's not fine to be 13 years old and to think that you're done growing, because that makes it a lot harder to grow.
- [38:14]
Derek Thompson
I think a good way to move a field forward can sometimes be to ask questions that are so fundamental that they might even seem stupid. And you have a couple of these questions at the end of a recent essay, including the following. The Average American watches 2.7 hours of television a day. We write this off as leisure, as if that's an explanation. But why is it fun to watch someone make a salad on tv? Why are you interested in this question of why it's fun for people to watch salads being made on tv?
- [38:50]
Brian Phillips
Because it's weird. Why do people like doing that? What are they getting out of it? You can't eat the salad and they spend so much time doing it. Obviously not everybody's watching a salad get made. But, you know, when people are bored at work, what they long to do is come home and watch people do things on a screen. Why? What is that? Like, any good explanation in psychology will have to explain how people spend the majority of what we call their leisure time. When people can do whatever they want to do, they like to do this. Why this? And this is the kind of mindset that I want to inculcate on myself and others. Because I think it's how psychology got started in the first place. Like, you know, there's a lot of things that make psychology complicated. You know, you don't have to ask the permission of protons to, like, smash them together. You do have to ask that of people, whatever. But I think it's really interesting that it took us so long to start at all. Like, you know, if you think of Galileo as the first physicist, we're talking like 1600, the first experimental psychologist, like, the first actual person that we might consider a psychologist. Not until 1860, 250 years later. Why is that? I think it's because people actually felt pretty satisfied with what they thought they knew about how their minds work and how other people's minds work. And, like, the first guy who, like, got this started, he. He only got started because, like, it's sort of a long story, but he stared at the sun for too long because he was trying to study, like, after images. He was a. He was a physicist, but first he stared at the sun too long. He went blind, get really depressed because he's like, bedridden. He, you know, his sight's gone. At one point, he undergoes moxibustion, which is where they put this root called. Or this. This weed called mugwort on your skin and set it on fire. It's maybe a psychedelic. Maybe he's like, tripping. He eventually gets his sight back and. And he's like, what I need to do is start a new religion. He writes these books about his new religion, and they don't go anywhere. And he's like, what I really need to do is provide a scientific basis for my new religion. And what he starts doing is what we now call psychophysics, which is, like, you show people a flickering candle and you see, can you tell that it's there or not and from what distance and how much intensity? And he lays down the first laws of psychology. This guy, Gustav Fechner, who, by the way, trains him and his colleagues, train the first generation of psychologists. I can trace my academic lineage back to that guy who went blind, got high, invented a religion, and then started doing experimental psychology. Why did it take that? I think it took that because he had to be alienated from what he thought he knew about how the mind works. And I'm not going to stare into the sun or set myself on fire. So instead I have to ask, why do people make salads on tv?
- [41:37]
Derek Thompson
I do love this idea that sometimes the most interesting questions can be asked. When you try to see the world, you say like an alien. Sometimes I think of it as like a tourist, right? As if all these things that are familiar to you are instead pieces of a foreign country. And what if you went about the world thinking like a tourist? It can be a really interesting way, I think, to come up with novel questions or actually to see the novelty and familiar questions. But one question that I'm very interested in, and I would love to close on any thoughts that you have here, is I have always been so interested in the science of creativity, while nonetheless recognizing that the science of creativity is not much of a science, it is an attempt at a science. I think people are doing their best. But how people come up with new ideas and new scientific paradigms, new novels, new salad recipes, to use one of your favorite examples. This is an amazingly interesting mystery. And the implications are so profound, not only because it would be amazing if there were some kind of formula for creativity, but also because some of the most powerful people in the world right now are trying to build creative machines. And I think in many ways failing to do it in part because what they understand how to do is how to take a synthesis of all of the knowledge that exists and mix and match it in a kind of pastiche to create next tokens. And if there were some signs of creativity, the things that we could do with these machines would be really, really fascinating and magnificent. How would you even think about pushing forward the frontier on something as over subscribed and yet difficult to make progress on as the science of creativity?
- [43:37]
Brian Phillips
I would start by banning the word creativity from my mind. I mean, a great piece of advice that I got from my advisor was study verbs, not nouns. There's no thing in the mind called creativity. It's a thing, thing we use to describe what people do. So focus on what is it they do. Well, what they do is they come out of dreams and they write the song yesterday. At least that's what Paul McCartney said he did. He said it came to him in a dream. We got mathematicians saying the same thing. And so I would start with the fact that it really seems like whatever is going on here is beyond the reach of people's individual consciousness. And so what's going on there is it. And like an offline subconscious recombination of experiences that people have had in what sorts of situations? Like, can we make those dreams happen more often? What was it that caused that to happen? Like, did you need to be thinking of songs all day before you go to sleep? And then sometimes you come out with yesterday? Or is it you need to hear some song and kind of half forget it. And then you rearrange it. Yeah. I don't know how to solve the mystery other than to begin by embracing it and to look for the parts of it that are the most, most mysterious, which is we appear to be riding atop of this vast subconscious that seems to be doing all the recombinating and generating, and then it feeds it back up to the conscious mind. And that this is both true for musicians and for mathematicians. And so maybe those things that result in very different outputs are, in fact, the same process, which would be really interesting if, in fact, the mind treats those things similarly, although we think of them as totally different. So that's where I would start.
- [45:25]
Derek Thompson
Adam, thank you so much. This was really fun.
- [45:27]
Brian Phillips
Thanks for having me.
- [45:33]
Derek Thompson
That was the social psychologist Adam Mastroianni. Next up is our conversation with the Atlantic journalist Dan Engber and his reporting on a billowing scandal in business school psychology. Dan Engber, welcome to the show.
- [45:48]
Dan Engber
Thanks, Eric.
- [45:49]
Derek Thompson
You have a story to tell us, and while I think it's a great and important story, it's not a simple one. So I want to try to go as simply and chronologically as we can here. Why don't you start us off by telling us about Francesca Gino and the blog post that began this whole sordid affair?
- [46:09]
Dan Engber
Sure. So Francesca Gino is one of those very famous people you may never have heard of. She is, or was, a superstar business school professor at Harvard. She wrote a bunch of books. One of them was called Rebel Talent. Her research was on dishonesty and rule breaking and how those relate to creativity and success in business and life. She had a very successful career doing, you know, public speaking, corporate gigs, TED talks, that kind of thing. And in summer of 2023, two things happened. One, she was placed on administrative leave by Harvard Business School. Behind the scenes, there had been a long investigation of her work. And the other thing that happened was a group of her business school colleagues published a blog post. That group is called Data Colada. That's just their kind of blog name, where they laid out evidence of fraud in four of Gino's papers. And it turned out that the internal investigation at Harvard Business School was looking at the same four papers. In fact, the Data Colada bloggers had tipped off Harvard to their work, and then they'd held back the blog post until the investigation was complete. Okay, so this was a huge story in the world of scientific fraud. Scientific fraud is discovered on a fairly regular basis. But this one was so Juicy. Mostly because of the fact that Gino was herself an expert on, as I said, breaking rules and lying and also, you know, very successful. So this was, it made international headlines. It was a big deal. And you're right, Derek, this is such a complicated story because there's like another, there's like another twist.
- [48:14]
Derek Thompson
We haven't scratched the surface here.
- [48:15]
Dan Engber
Yeah, there's another twist just in the setup here, which, I mean I could go right into, which is that one of the four papers that had been flagged for apparent fraud, the same group of bloggers had already identified potential fraud from another data set used within the same paper from another famous superstar, business school professor, Dan Ariely, who's written bestselling books of his own. So you had this totally just unprecedented situation where here was a very important paper in the business school psychology literature that had many prominent academics as co authors and two separate studies within that paper had separately been identified as being potentially fraudulent. So it was, I mean, for me, and I follow these kinds of scientific misconduct stories for a long time, it was mind blowing. So that was June of 2023. That's the starting point.
- [49:19]
Derek Thompson
People will be listening to this show initially during Thanksgiving week. And it occurs to me that there's a famous dish popularized by the NFL coach and announcer John Madden called the turducken, which is turkey that has within it a duck that has within it. Ch. And this is sort of like a turducken of fraud, right? Because the paper had fraud in it, but it contained within it data sets that had been separately identified as being fraudulent. And to your point, I'm using your language here. You summarized the scandal by saying she was accused of faking data for a study about when and how people might fake data. And Dan Ariely, one of the co authors or contributors to this piece, is a celebrated expert in the psychology of telling. So this scandal is already fully in the realm of if you turned in a fiction manuscript to a publisher, they might reject it for being too heavy handed. So this is already a famous scandal within academic psychology. And people who might have seen it reported, the New York Times, or for whatever reason are interested in scandals in academic psychology, they might have already begun to have heard about this chapter one of the story you're telling here. What did you initially want to add to the story? What made you initially interested in reporting out your own angle here?
- [50:43]
Dan Engber
So it was what came right after that, how the community responded to what had happened. So one of the things that was obvious from the start to people in the Know, people have thought about this kind of thing is, okay, if you've got, we know there's fraud in science, there has to be, just in the nature of things. If you had this coincidence of two potentially fraudulent data sets within the same paper, what does that tell you about the base rate of fraud in science generally within this field, within this particular community of academics who work together within this field? And that's a really important question. And you think it's one we might know the answer to, but we simply don't. And the reason we don't is because there's never any kind of broad audit of the scientific literature in any field to figure out what the base rate of fraud might be. So we had this tantalizing and disturbing clue that that base rate might be much higher than people generally believe. So I mean, that was just very interesting to me as someone who's followed these issues for a long time. But what happened next is that Gino's colleagues and Ariely's colleagues in business school psychology decided, okay, we've got a problem here, which is we have four papers from this one researcher that seem like they could be fraudulent. She published a lot. I mean, she was incredibly prolific. She has something like 138, I think, co authored papers with other stars in the field, but also young academics. And the problem is, okay, we just don't know how many of the other 134 are problematic. We don't know, you know, on some of those papers, maybe she was just like the fourth author out of seven authors or something and didn't really contribute anything meaningful to the actual research. Maybe we should consider that paper, you know, totally fine. We shouldn't be suspicious of it at all. But if there was another paper where she was, you know, a more significant co author, let's say she had actually run some of the studies herself. Well, what do we do with that? No one's gone to the trouble of investigating it for 18 months behind closed doors. Should we just assume that that study is sound? Unless we have specific reason to think otherwise, or what? So you can imagine how this is just a problem for everyone. It's especially a problem for Gino's younger colleagues, who each one of these papers could be playing a pivotal role in their shorter CVs in terms of getting grants and getting jobs. So it was kind of a crisis on a very practical level for a lot of people in this field. So what they decided to do was something. Again, I've never seen this before. They came together and they said, we're just going to do kind of a giant audit of all of our work. So it was called the Many Co Authors project and it referred to the many co authors of Francesca Gino. The idea was, okay, we're going to set up a central database. And actually this, this project was sort of spearheaded by one of the original Data Colada bloggers who had identified the signs of fraud in the original four papers. And we're just going to ask every single co author on every single paper to just tell some basic facts. Okay, did Francesca Gino run any of the studies? Did she have access to the data? Did the co authors have access to the data while the paper is being put together? So even just that step will get some basic questions answered about this full body of research. And then in a perfect world there would be further investigation of each of these papers. Like we want to know, can these data be trusted? Again from 134 papers. So this was both kind of just interesting and inspirational to see happen, how everyone came together to try to test and buttress the integrity of the field. But I also realized in a sense this was something like an audit of, you know, this was not the kind of audit you would want to understand the base rate of fraud within psychology broadly. This was decidedly non random. Right? This is only papers that have Gino as a co author. But it's kind of like a large scale look at scores of papers and maybe you could get a sense of, well, how deep do the problems go? So I thought that was kind of a side benefit of this, that it was almost like accidentally a way of testing a larger sample of academic papers to see how many of them were deeply flawed.
- [55:42]
Derek Thompson
So the reporting on Gino's potential fraud inspires this project to determine whether or not she's a bad apple or whether or not there's a rot inside the barrel here. And for folks thinking what kind of studies are in this Many Co Authors project, we're talking about research about whether, say, procrastination makes you more creative. Research on whether you're better off having fewer choices. An idea that's sometimes called the paradox of choice and is related to this other phenomenon that you've reported on called ego depletion. Research suggesting that you can buy happiness by giving things away. These are not studies that change the world necessarily, but they are often very famous studies. They're the subject of viral TED talks that have millions of views. They're the subjects of airport bestsellers. And as you point, speaking of self auditing, nearly every single Business academic named in this Many Co Authors project has either been quoted by or cited by the Atlantic, our employer. So these ideas are very much in the ether. And one of the people who's helping to run this Many Co Authors project, this huge audit of business school psychology is a woman named Juliana Schroeder. And when you start reporting the story and you reach out to Juliana Schroeder, she just seems to be a paragon of scientific rigor. I mean, she seems to have a fantastic reputation. And the folks you're talking to about her are calling her exceptional and a model researcher. But here's where we have one of the many interesting twists. Just days into the creation of this initiative, she discovers that a paper that she helped to write suffers from some major data problems. Tell us about this paper which is called for those who want to look it up, don't stop believing rituals improve performance by decreasing anxiety. What's this paper and what did she find?
- [57:42]
Dan Engber
Okay, I'm sorry, I want to just back up for one half step just to say that some of those big topics you talked about, Paradox of choice, for example, and procrastination creativity, those are not specifically represented in the Many Co Authors project. Those are representative of this field, which I'm calling in my story, Business School Psychology. So this is. These are people who come out of psychology programs, often are using the methods of behavioral science that are common in campus psychology departments. They're publishing in the same journals as regular psychology departments on campus, but they're located in business schools like Harvard Business School or Wharton or at Berkeley or the University of Chicago. And they're getting paid a lot more money to do the same kind of research. And it's from that sort of world of business school psychology that you get this hugely influential kind of gee whiz research that shows up so often in the Atlantic and the New York Times and many other publications and on television, et cetera. So I just wanted to be clear about what was actually included in the Many Co Authors project.
- [58:56]
Derek Thompson
I'm very grateful that we're practicing the art of self audits even within this interview. I'm not going to pretend, pretend that I screwed up that setup purposefully, but I'm glad that we have you as a paragon of keeping everything straight on the show. So, Dan, please continue.
- [59:14]
Dan Engber
Okay, so Juliana Schroeder is someone who had co authored a bunch of articles with Gino. So obviously she was very disturbed to find that there was now this evidence of fraud in Gino authored papers, right? So she immediately got involved in the Many Co Authors Project and became one of the leaders of it. Not only that, but she took the work of the Many Co Authors Project to a place that other people who got involved did not. So many people were, like, thought it was important. They gave the basic level of information about the papers they'd worked on. Oh, yeah, Study three of that paper. Francesca collected the data, but I've never seen the data. I don't have any particular reason to think that there are problems there, but they would just report those basic facts. What Schroeder did was she said, you know, I'm going to go back and I'm going to audit every study in every paper that I worked on with Gino and I'm going to post everything to osf, which is like a platform for open science and sharing and sharing data and methods. And, you know, one of the Yuri Simonson, one of the Data Colada bloggers, even told me he thought this was kind of overboard. Like, he thought the point here was just to, you know, get a handle on what Gino herself had done. Schroeder was treating it more like, no, this is a way of looking at the, you know, the field at large, really trying to see how much of this work stands up, how much of this work is, you know, potentially reproducible, but also how much of this work is just clean in the sense of being without, you know, obvious glaring flaws or potential corruption. So, yeah, so she's coming into this as a leader of this push to really test the literature for signs of rot, basically. Yeah.
- [61:16]
Derek Thompson
Yeah. So thank you for. I think that step back was really fantastic context. I'm particularly glad that you corrected me on exactly what papers are in the Many Co Authors project versus what papers are representative of the field that is being studied within the Many Co Authors project. That's very important. So let's go to the study specifically. Again, this is a study about the degree to which performing a ritual can decrease your anxiety. And in study 1B of this paper, they are measuring the heart rates of people before and after they perform a ritual. Tell us a little bit about Study 1B and what Schroeder and maybe other folks see in here that is so clearly representative of fraudulent data.
- [62:04]
Dan Engber
Okay, so, yeah, very quickly getting into it. Francesca Gino and Harvard Business School colleague named Michael Norton had done this important work. Seemed important at the time, let's say, about what Norton would later call the ritual effect. The idea that performing rituals is. Is sort of like performance enhancing and improves your life in Many different ways. So a lot of people were excited about that idea. In the mid 2010s, Juliana Schroeder was a graduate student at the time she started working on this. There was a graduate student at Wharton named Alison Wood Brooks. She was working on this. They kind of realized they were working on the same thing. They came together, they did this paper. And what Brooks in particular had been doing were these studies using a karaoke machine. And she would have people try to sing Don't Stop Believing, which is pretty hard to sing, I can say from personal experience. And then she would have them do a ritual before singing Don't Stop Believing to see if that helped them do a better job, made them less nervous, et cetera. So the study that we're talking about, Brooks apparently contributed, although there's some dispute about this later, these studies, to that paper where people would again perform this arbitrary ritual. They would draw scribbles on a piece of paper, crumple it up, and pour salt on it. And then they would perform Don't Stop Believing. And she would test their heart rate, or whoever ran the study would test their heart rate before and after they performed the ritual. And the basic finding of study 1B, which you mentioned, is that if you perform this ritual, it made your heart rate lower. Basically, the idea was just like, it helped you to stay calm, and then you could infer that this would then improve your performance. So that was the idea. This isn't going to change the world. But if you're speaking to, let's say, a business audience, you have people who are making presentations. Maybe that's something that they could use. They could could go to the bathroom, perform some ritual in front of the sink, and then go back and load up their PowerPoint. I don't know. Anyway, that was the paper. But when Schroeder started looking at it, she gathered what data they had. They didn't have the raw data anymore, but these data spreadsheets that had been produced after the fact, and they just looked wrong. Especially the Data spreadsheet for 1B, although there was other ones that looked problematic as well. But in particular, so this is all public now. So I was able to download these spreadsheets for myself and just be like, well, what looks wrong? And there's just one thing where you do not need a PhD to understand what is wrong with this data set. And it might be hard to describe in all.
- [64:56]
Derek Thompson
Can I try to describe it? I want to try to describe it. First of all, we're going to link to your Piece, obviously, in the show notes, because people should read the piece. But this is a graph showing the heart rates of a bunch of people that have just done this ritual of pouring salt on a piece of paper. And if you look at the middle of this distribution of heart rates, it's basically a straight line going up. Now, typically, you would think that if you measured a bunch of people's heart rates, you'd get a bunch of variability. You know, it's like it's 120 here, it's 50 here. If you line them up randomly, it should look very, very spiky. But this isn't spiky at all. And I was trying to think, like, what's a way to represent this for listeners? It's a little bit like, Dan, tell me if you agree with this. If there's a fire drill at the Atlantic and everybody at the Atlantic has to line up outside the office spontaneously, and there's a researcher there who comes by and measures our heights, and the results show that 70 of us just happen to line up in perfect order from short to tall. That's about the same as the odds of this data that you're looking at in Study 1B of Don't Stop believing being real. Like, there's just no way Way that 70 people at the Atlantic just happened to line up outside the office in perfect order of short to tall, with every difference being like the exact same one centimeter. So you look at this data, you look at this graph, and it's clear that someone is essentially just drawing a line and placing heart rate measurements on it, rather than studying a realistic distribution of heart rate measurements after some crazy ritual, does that sort of, in an audio way, kind of translate to a loose description of what we're looking at in this data set?
- [66:41]
Dan Engber
Yeah, I think that that's good. I think it's, you know, one thing that you have to keep in mind. It's very tricky. So this forensic data analysis is super subtle stuff. So. And I don't want to pretend to be an expert, but you could imagine a situation where you look at a data set like this, and you do see ordered data in this way. And one thing you have to be careful about is, well, maybe there was a reason why a research would sort by heart rate, and maybe they, after sorting by heart rate, assigned participant numbers to all the people who did the study. You could come up with a story where this sort of happens, but it doesn't make sense in the case of this data set because it's not like every Data point in the study was in this order. It's just a run of about 100 of them in the middle. So it's just one of the red flags that kept coming up when I was talking to data experts about this study was it's so hard, I would say impossible, to come up with a narrative of how this happens. Right. And so that's the people who, the fraud detection experts, that's what they look for. They're constantly thinking, I mean, of the null hypothesis maybe, if you want to use science language. They're constantly thinking about how could you explain this other than someone made up data? And this one is just a study where it is so hard to come up with an alternative explanation than data were made up. Right. But your description of the fire drill is good.
- [68:16]
Derek Thompson
So I was fishing for a compliment there. But I really did spend time looking at this graph and thinking, how the hell do I describe this on radio? That's what could possibly do. So I want to keep the story going because there's about to be another turn that's really, really important to the narrative. So Schroeder has already, in the process of combing through all of this work, revealed that there are major problems with this paper on the ritual effect. But that's not all. You report that as she continues to review papers associated with Francesca Gino, she comes around to looking at a paper that she's worked on that is also about the power of rituals, which came out in 2018 in the Journal of Personality and Social Psychology. And this is where story starts to get very knotty. Tell us what happens here.
- [69:02]
Dan Engber
Sure. Can I just make clear what the significance of the previous finding was to you?
- [69:08]
Derek Thompson
Oh, absolutely. Yeah, sure.
- [69:09]
Dan Engber
This is what, again, when they started this, people thought, oh, you know, Francesca Gino was associated with four papers that had, you know, suspicious data. I wonder if there are more that looks like a pattern of behavior. But when Schroeder found this problem that we're just talking about in the, the run of data and the Don't Stop Believing study, she had no reason to think that Gino had ever touched those data. And so just going back to where we started with the idea of, you know, this fraud turducken, it was, I guess we had like a fraud ter duck before, just a duck in a turkey. And now it's like, okay, we've just added another layer. There may be a third independent source of made up data just in this specific community of scholars within this specific field of research, within business, academia. So, I mean, it just starts to get weird, right? Like, we knew there were already two sources of suspect data. Now there's maybe a third independent source. Like, how far does this go? Okay, so that was when I started reporting on this. That was the point I was interested in. I was like, wow, we're up to three, right? How many more could there possibly be? So then the next turn in the story, I guess, is that Schroeder finds another paper with her name on it and Gino's name on it, as you say, about rituals, that has another kind of suspicious looking data set. In this case, though, it's Schroeder's own data set. There's no dispute with the Don't Stop Believing paper. There was disagreement among the co authors who actually handled those data, and they don't, to this day don't agree on that question. So it's hard to say if those data are flawed, who created the problems in it? That's an open question. But in this other paper that came out in 2018, Schroeder's like, these are my data. I did this analysis. But if you look at those data, they are quite suspicious. Basically, this was a study of whether you can use rituals. We already talked about, you can use rituals maybe to sing Don't Stop Believing Better. She wanted to know if you could use rituals essentially to lose weight. And the way the study was set up, women were recruited at a gymnasium at the University of Chicago. Women who wanted to lose weight, and then about half of them were told to do some kind of ritual before every meal. They were supposed to organize their food on their plate in some way, like cut the food up into pieces and then separate it on their plate. And then they wanted to see if merely doing this ritual helped these women who are all trying to consume fewer calories in general, to actually consume fewer calories. The study found that it worked, right? The women who did the ritual ate less. But what Schroeder discovered when she was auditing again, her own work, she found that a bunch of data points had been flipped, potentially from one condition to the other. So women who had been marked on the log sheet for the study as being in the ritual condition, somehow it seemed like maybe they had actually been in the control condition based on other data from the file. And then similarly, some data were flipped the other way. Okay, so you had some. This is often described fairly or euphemistically as a coding error. Okay, so you had some kind of coding error where some of the data was misattributed by condition, potentially. Now, if you flip it back, this is naturally what you would do and what Schroder did. Let's flip it back and see what happens to our findings. And actually, if you flip it back, the finding vanishes completely. If anything, the women who were doing the ritual ate a little bit, tiny bit more than the women who didn't do the ritual. So that's already disturbing. Right? There was a coding error that just happened to create the effect you were looking for. And if you. Again, Schroeder has made all this public, you could download the data. If you look at the data as I did, you see that the particular data points that flipped are not just. They don't appear to be a random set of people from the study. They seem really specific. Right? It's people who either ate a whole lot and were in the ritual condition or people who ate very little and were in the control condition. And so, just on first principles, it looks deliberate. Now, I talked to some data fraud experts who tried to sort of formally look at this. They evaluated the odds that if you were just making these coding errors at random, what are the chances you would end up with a publishable effect? And they were coming up with basically one in a million. So you've got this deeply suspicious, quote, unquote mistake in this paper. But I just want to emphasize Schroeder, who was responsible for this study, is 100% the one who identified this problem and brought it to everyone's attention. So this was, I guess, the first twist in the reporting of the story. I noticed this basically last spring when I had already been reporting on the other paper.
- [74:54]
Derek Thompson
There's a very interesting and very tense section of your essay where you confront Schroeder with a weighty accusation and an accusation that I think flows from a logical way of thinking about everything you've reported on. You ask her if she's engaged in purposeful data tampering itself. And even more tensely, you suggest that maybe she started this entire project, or you do it in a very clever way. You say, essentially, couldn't one potentially think that you've started this entire project as a way to own the narrative of fixing previous errors, rather than have outside critics like those from Data Colada identify the data tampering before you brought it to light? Tell me about that conversation and. Or conversations that you had with her where you're trying to work out, you know, Professor Schroeder, what really happened here? What were your motivations in making public an error that some data experts I'm talking to say, is a one in a million chance.
- [76:07]
Dan Engber
Yeah, I mean, it was. As I was reporting the story, this was really perplexing to me, Right? Like, it. It was inconceivable that Schroeder herself had been responsible for switching the data on purpose. Right. She's the one who brought it to light. She identified the problem and publicized the problem. On the other hand, her best explanation, or what she deems the most likely explanation for what happened is that her research assistants must have altered the data. And in her telling, they did this because maybe they'd made a mistake earlier and were trying to fix the mistake. It was like, well intentioned in some way. But of course, if that's true, what they did is so completely out of bounds for science to look at the data and start moving, you know, switching conditions. So, you know, that was her explanation. It's possible that that's true. RAs make mistakes, RAs commit fraud. On the other hand, it's also, and often what professors say when they're accused of fraud is they say, oh, I don't know what happened there. My RAs must have messed up the data. So I just had this really. I was faced with what seemed to me like a situation that was impossible to explain. Right. It didn't make any sense that Juliana Schroeder would have altered these data and then identified the alterations herself. On the other hand, I found the most likely explanation for what had happened very unsatisfying, especially because we don't know what these RAs have to say for themselves. I tried to find them and failed. Schroeder said she knows who they are, but didn't want to involve them in the story. So of their take is totally mysterious. So I just, at a certain point, I started to wonder, is there a narrative here that makes sense? Like, this goes back to what we were saying about. Can you come up with a story here that explains the facts? Is there a story here that could explain why someone might have done something like this and then called attention to it? Is that really so inconceivable as I had originally believed? And it occurred to me, as you said, that maybe there's a certain logic to it. If you think that, you know, you know, that this big audit of Gino's papers is going on, you know, that people are looking closely at this stuff. You have a sense that problems are going to emerge on lots of papers. Maybe there's a kind of logic to, okay, let me just be like the most rigorous, integrity oriented researcher there is. Let Me just make myself totally unimpeachable in this whole story. Let me just look closely at all of my own work, just throw it all out into the air and then let me just cop to whatever problems appear and try to move past them and maybe it'll be okay. I'm the one who identified the problem. Certainly that itself is laudable, right? And it's over and done with, the paper's retracted, or that there's a correction or whatever, and you move on with your life. It also occurred to me, what if this is something it was not a pattern of behavior, but just like a one off thing. So you can just kind of imagine getting it out there and behind you. So I'm not saying that this is necessarily what I believe happened. It's just something I began to think of this as an alternate narrative that could potentially explain what happened. And I just felt like I had to present that to Schroeder and see what she thought. So that's the setup for this final conversation we had. And she was very, you know, calm in hearing me out. She, you know, she's, she. Many business school professors, I feel like, are, are trained in and teach kind of these inter, these tense interpersonal exchanges, negotiation and stuff. And I felt like there was some of that going on too with like, she told me, she heard me and understood me. She said my name a lot while we were talking. And of course she absolutely denied ever having gone in and altered data herself. She acknowledged that it does look bad and how weird it is. But we really got to this point where we almost took a step back from the specifics and talked about what the field should do. And I thought it was quite compelling and moving even that she said at the end of that conversation that she thinks the whole field, that there could be a lot more empathy involved in how these situations are handled. And I won't try to interpret over. Interpret what specifically she meant by that. But that resonated with something that I definitely feel about this process, which is, again, let me just say this is not, I don't know that this is what Juliana Schroeder believes. So I'm fully in the range of my own take here. But I do think one major problem in business school psychology and in dealing with fraud in science more broadly, is that there is no room whatsoever for grace. Right? The stakes are so high. If you are found to have committed scientific fraud, you're done, right? Your PhD will be stripped, you'll lose your position, and that's it. And so there Is this incredibly high bar for concluding that that's what's happened in the Gino case. She was placed on administrative leave after this. The report is now public. It's like a 1200 page document. It is, you know, it's like the 911 Commission about this one Harvard Business School researcher and four of her papers. Right. And even that is just a total mess now because Gino has sued Harvard. She also sued, initially sued the bloggers who first published the evidence about those papers for defamation. So who even knows what the upshot of that is gonna be? But it's like, I mean, in taking a step back, it just seems like a mess, right? It's a gigantic mess. What do you do in a case where it's not someone who's potentially a prolific fraud, but one paper that might have a little bit of something fishy in it? There's just no middle ground where you could address that head on, expect people to be honest and have some kind of moderate consequences. If that paper, for example, were, if it were true that that paper contained fraudulent data and the University of Chicago agreed with that, again, it's the ultimate punishment. I think the PhD would be nullified. Whatever happened with Juliana Schroeder aside, I just became very aware of the fact that by the heavy constraints on action in dealing with the problem of Frau, like, there is just, there's no room to operate if someone did, you know, do something bad at one point in their career. There's no way for them to come clean. There's no way for them to say, even to serve as a role model of how you could, you know, stray as a graduate student and then become, you know, truly a hero of open science. That path is not available. And I guess, you know, maybe that's, maybe there are people who believe that's how it should be, that, that data tampering is the ultimate crime in science and so forth. But I just, I find it quite distressing. And the outcome of this super high stakes system is terrible, right? Like most cases of suspected fraud are just swept under the rug because again, it's such a mess to even open the door to looking at it.
- [84:58]
Derek Thompson
Ironically, you're posing an incredibly rich topic for something at the intersection of moral philosophy and psychology. Maybe it's organizational psychology, but how do you design a system that punishes bad behavior enough to discourage the bad behavior, but not so much that everyone is disincentivized to report on their own bad behavior? Right? You're trying to strike this incredibly knotty middle ground. Where you don't want people to commit fraud, but also you want to encourage light being shown on two frauds committed in such a way that people have some incentive to say, oh, I did something wrong and I'm sorry, and I hope this doesn't destroy my entire life. Because at the end of the day, you know, maybe misreporting a single cell in an Excel chart in a study on ego depletion should not in fact, ruin the rest of your career. I mean, I don't have a strong feeling about this, but I'm very willing to hear that argument. You make a really interesting comparison in the essay with steroids and baseball. And I'm reminded that Barry Bonds, if I recall this correctly, allegedly started using steroids when he realized that players who were clearly inferior to him were using steroids and putting up higher numbers, were hitting more home runs. And so the use of steroids justified more steroid use as a kind of social contagion, or maybe like a keeping up with the Joneses effect. If that in fact is an effect. Maybe it's subject to its own. To its own fraud. But you make the point that maybe if you're a young social psychologist and you want to be published in some of the most prestigious journals in social psychology, or quote unquote, business, school psychology, and everyone around you is putting up these, you know, steroidal numbers with these incredible whiz bang effects that are like, oh my God, didn't even realize that sprinkling salt on a piece of paper turned me into the lead singer of Journey. You become motivated to maybe mess with your own numbers just a little bit in order to get the whiz bang factor up to a point where a journal on social psychology will publish your work. So maybe just dial in on that last point a bit because I thought it was so interesting the degree to which the expectation of researchers that fraud exists in their field serves as a mechanism for them to create their own minorly or majorly fraudulent work. Because again, they're playing in a league where everyone is using steroids.
- [87:37]
Dan Engber
Yeah, I mean, I think that's right. It certainly does seem to me that fraud would be contagious in that sense. I think there's another issue that we're touching on here, which is super important, which is that below the level of fraud is this whole world of. Of what are called in the business questionable research practices. QRPS is the term of art. And that's. You know, your listeners probably have heard of P hacking, right? This is this big problem in psychology, social sciences and truly sciences broadly for a long time where there's just these ways of kind of just tiptoeing up to the edge of what is sub fraudulent statistical analysis. Look at your data. There's some data points that don't really help you and you come up with some reason why. Honestly, that guy, when he came into the lab, he was acting strange, maybe he was even drunk. I think we should discard that data. And you can, I mean, it's obvious, right, how this could be abused. But I think it was commonplace as recently as 10 or 15 years ago to have no guardrails whatsoever on data exclusion. Just to pick one example that comes up in my story. So that's common. If you can just kind of, in this freewheeling way exclude data, even if you kind of, of you don't think you're doing wrong, that itself is going to lead to this inflation of stats and that just pushes the incentives in the wrong direction already. So just to go back to the baseball analogy, you could imagine, short of taking steroids, there's a norm in baseball where it's okay to steal signs when you're on second base. I think this is the Astros got in a lot of trouble for a science dealing operation some years ago that involved technology. But I think just doing it by eyeballing is kind of like part of the game, right? So maybe that's analogous to B hacking. I might be stretching here, but I'm just imagining that there's ways to get kind of an advantage in science in terms of producing the results you want that everyone is doing and everyone except accepts. And so already you have like these unrealistic expectations on rigorous science to produce publishable findings. And then you can keep tiptoeing closer and closer to the line. Like P hacking can get more and more egregious until it's really in a gray area where you're like, you probably know that what you're doing isn't scientific per se, right? And then, then take another few steps and now you're actually just changing data. So the case is often made that maybe there's a wide swath of research practice where fraud is indistinguishable from questionable research practice. I don't really buy that myself so much. I think everyone knows intuitively that there's something different, but there's a meaningful difference between sloppy application of statistics and just making up numbers. But I do certainly think there is a lot of blurry stuff in between. And I do think the creation, the Existence of that blur is not going to help and contributes to that kind of spread of cheating. Right. That we were talking about. I think that makes it only that much harder to get published. So it's. It increases the incentive to cheat and also gives a sense that there's just kind of like, as I said, a lack of guardrails. Really. Truly.
- [91:41]
Derek Thompson
I want to round toward a conclusion in just a second, but you spoke to several academics for this story, a bunch of people in and around this business school psychology space. How would you describe their general attitude? Are they in denial? Are they disillusioned by the revelations the last few years? Are they in despair over the state of their industry? I guess I used 3D words there, but which would you choose? Denial, disillusionment, or despair?
- [92:15]
Dan Engber
Denial, disillusionment, or despair?
- [92:17]
Derek Thompson
Actually, feel free to use literally any word. You do not have to choose the three words that just randomly popped in my mind. Just generally, how would you describe.
- [92:24]
Dan Engber
I can only. This is a psychology experiment. I can only think of words that begin with D. I'm going to go with Demoral. Yeah. I mean, I think there was denial for a while. Right. I think the Geno scandal has made it harder to be in denial in this particular field. Right. But, yeah, I mean, I think Schroeder's experience itself is so demoralizing for her and for anyone who's watching it because. Because she has done so much to try to correct the record. Right. And even in just identifying the flaws in that first paper we talked about, she is running afoul of very powerful people. Right. Including a very senior academic who, as I described in the paper, kind of tried to tell me on the sly that he thinks Schroeder is a fraud. Just to give a sense. Like, this is someone who, you know, Schroeder's tenured academic at Berkeley, but like someone who is a much more senior person is telling a reporter that she's a fraud. Like, that's. That just shows, I think, how dangerous it is for anyone in this world to even try to, like, cast a light on this stuff. Now, we've talked about how Schroder's case is complicated, but I do think it's incredibly dispiriting. I'm just gonna lean into the Ds. It's incredibly dispiriting to see how in this. What I described earlier is this inspiring project, the Many Co Authors project, has both shed light on the scope of the problems, but also been so damaging for so many people. And, yeah, it's Brutal things are not set up to get better as they are right now.
- [94:25]
Derek Thompson
I do want to offer at least the possibility of an optimistic interpretation of this story, which is that 10 years ago, 20 years ago, business school psychology existed. The field of social psychology was a frothy field. Best selling books were being published. None of this was happening. On the positive side, there was no many co authors project, there was no data collada, there was no public awareness of p hacking such that there could be an ability to problematize the practice. Many of these iffy ways of dealing with data might have been widespread, but it existed entirely in the shadows. And the process of, of pulling these practices into the light, while an incredibly painful one for the people in the field, could optimistically lead to a social psychology discipline that is much more disciplined in the next decade, in the next few years. And so we are in a way, hopefully somewhere close to the emotional nadir of this general experience. We are realizing how much fraud has been in the system. But that very realization is going to lead to different outcomes. The same way that you know to jump back to the steroid experience of baseball. No matter how you feel about the evolution of baseball since the early 2000s and I have a lot of complicated feelings, nonetheless, you can't argue with the fact that there is a steroid testing regime and people are banned from the sport for taking substances that were probably, probably absolutely widespread 20 years ago. And theoretically something similar could happen to this field of psychology. To what extent do you think that optimistic interpretation is completely Pollyannish naive versus potentially realistic?
- [96:18]
Dan Engber
No, I mean things are changing quickly, so I don't think that's totally unreasonable. I think about. But in a way there's two approaches. What are we going to do about this? We can continue to try to diagnose the problem where we can intervene and obviously intervening fixing is better than diagnosing. Diagnosing doesn't get you anywhere on its own and some people are trying to fix it. Yuri Simonson from Data Colada is working to come up with a system that you can track the provenance of data basically. So not just share your raw data with people, but actually create a technological fix where people can see who worked with the data. Just the full think of it sort of like cryptocurrency. You can follow the life history of the data and that kind of openness, if it were adopted broadly would make a big difference because there would be, you know, there's no way to hide what happened if there's a full data trail on the data itself. Right. But in order to get those fixes adopted, those fixes are expensive in terms of time and money. And the costs of such interventions are disproportionately borne by young researchers and researchers at less well funded universities. Right. Because if you're at Harvard Business School or Wharton and you're a senior person, someone else can handle all this data tracking stuff for you or whatever else. What other interventions are put in place. If you're just starting out at a different school, that's on you. That administrative burden is fully on your head. So you have to be really careful about what fixes you put in place because there are these disproportionate harms from the fixes themselves, themselves. So there is both reasonable opposition to these interventions and there's unreasonable opposition. People don't want these fixes because they don't want to, as one person told me, they don't want to kill the golden goose. If there is no fraud or p hacking in business school psychology, that's not great for the people who are making successful careers doing this work. So you have this opposition to it. Reasonable and unreasonable. And I think to overcome that opposition, you need the diagnosis like you need to understand baseball. At a certain point in baseball, I remember this, the Mark McGwire, Sammy Soza home run race where you're just watching the game and you're just like, this is absurd. These people don't look normal, their performance is abnormal. There is just widespread agreement that there was just rampant cheating in baseball. I don't think you have that in science. And so these moments, like this moment of the Geno papers being identified in 2023 and now again with two more suspicious papers being identified, each one, I think it doesn't tell us the big picture. We still don't know the base rate of fraud in psychology or in science broadly, but it just sort of dials up the pressure a little more to do something about it. And I suspect, I have no idea, but I suspect that we still need more of that. We need more diagnosis before everyone acknowledges. Not that there is a problem, everyone does acknowledge that. But like the true scope of the problem, I think it's still possible to go, okay, well, this is one particular group of researchers. Maybe the fraud rate is high there or something. But there's so much business, academia, that's so wonderful. Granted, that's true. I'm just saying I think we do need to know how much fraud exists. I wish we could have some kind of broad random audit. That would be a Lot of work, but I think it could create the conditions that would be necessary for, for truly fixing things.
- [100:37]
Derek Thompson
I want to suggest one more conclusion of your really, really interesting reporting. And it points us back to an argument that Adam Mastroianni makes in the first interview that we're going to have on this show, where he talks about the fact that psychology hasn't produced workable technology the same way that, say, physics or chemistry have. And in part, psychology is a very different field because you're fundamentally working with people and people are messier than atoms and molecules. And whereas in a field like chemistry or physics, there's a paper and it reaches a conclusion and that conclusion becomes a technology, and the technology lives in the real world and it either works or it doesn't. Either quantum mechanics makes the transistor and the semiconductor possible, or it doesn't. And the transistor doesn't work and software doesn't exist because quantum mechanics, it turns out, was just wrong. Whereas in psychology, those falsifying feedback loops in the so called real world outside of the lab don't necessarily exist in the same way that they do in other so called hard sciences. And so maybe one, one thing that we're seeing here is that it's easier to get away with little flubs and little frauds in psychology because it's so much harder to falsify the fundamental question of does a ritual like pouring salt on a piece of paper make you better at presenting a PowerPoint? That's a very, very hard thing to say for the entire human population, given how many variables are present in each and every PowerPoint point. And this is simply a necessary correcting mechanism for a field where the so called world of technology, the physical world wasn't going to serve as that kind of correcting mechanism, the same way that it can serve as a correcting mechanism in chemistry, where you reach a conclusion, you try to build something based on that conclusion, and it just falls apart in the world of atoms because it turns out the chemistry was wrong. Am I making sense here? Basically, the physical world falsifies the hard sciences. We needed some other mechanism of falsification in a so called soft science like psychology. And a movement like this, therefore maybe just had to be inevitable to serve as that mechanism.
- [103:02]
Dan Engber
I mean, I agree with that. Basically. I think Adam is brilliant, by the way, and hilarious and everyone should read his substack. I do think these falsification loops may not be as robust in other fields as we would like them to be. Sure, of course, right? Like power posing if it doesn't work, who's going to notice? Whereas if a plane doesn't fly, you're going to have a cabin full of angry passengers. Yes, obviously, true. However, I think there's a lot of middle ground there. Where. Take for example, fraud scandals in Alzheimer's research. Right. Like a leading paradigm for Alzheimer's is false. Right. It's based on some key papers that are fully unreliable, has been revealed by reporting by Charles Pillar and science, among other people. Right. So I don't think that for me, at least, and I'm guessing for you, too, our experience isn't like. Well, of course this became obvious because a lot of people with Alzheimer's took a drug that didn't work and it was like the plane didn't fly. No. In fact, there was just huge quantities of research dollars. I mean, I could be wrong on the order of magnitude, but my understanding is hundreds of millions of dollars. At any rate, a lot. And years and years of research from pharmaceutical companies, those are people who are interested in the falsification feedback loops went into this work that was essentially for nothing. So I just don't. I don't think there is a clear demarcation between psychology research, where it's really hard to tell when something's bogus, and say, I'm going to biomedical research, where you can look at patients in the hospital and their outcomes. It would be nice if it was easy to tell when biomedical research was bogus, but it's just in practice. I don't think that's true. So I think the problem exists outside of psychology. And so you need the same kind of diagnosis and intervention across fields, basically.
- [105:26]
Derek Thompson
That reminds me, I really want to do an episode on the amyloid hypothesis in Alzheimer's because you glossed it, but it's so fascinating and to your point, billions of dollars were probably spent.
- [105:38]
Dan Engber
Billions.
- [105:39]
Derek Thompson
Billions of dollars were probably spent. I mean, when you think about the money that is sloshing around for pharmaceutical companies to cure a disease as widespread as Alzheimer's, my uninformed guess is we're talking about billions of dollars chasing a theory about neuroscience that turns out to be totally bunk. Science, it turns out, is very hard. And I think you're one of the best chroniclers that we have at exactly how hard it is. So, Dan Engber, thank you for your work and thanks for walking us through, through this particular story. We appreciate it.
- [106:05]
Dan Engber
It's my pleasure. Thank you so much for having me on the show.
- [106:09]
Derek Thompson
Thank you for listening. Today's episode was produced by Devin Beraldi Our schedule for plain English for the next few weeks will be one episode a week on Fridays. We'll see you next week.