Max Bazerman (5:43)

Certainly I felt a moral obligation to correct the record.

Steven Dubner (5:47)

Today on Freakonomics Radio, we'll hear from him, as well as the three data detectives who blew the whistle on Gino and Ariely.

Yuri Simonson (5:56)

So I would say I have. On the falsity of the findings, I don't have reasonable doubt, but this is.

Steven Dubner (6:02)

A much bigger story than two high profile cases in behavioral science, we will get into the incentives that produce academic fraud.

Brian Nosek (6:10)

If you were just a rational agent acting in the most self interested way possible as a researcher in academia, I think you would cheat.

Steven Dubner (6:17)

We will hear what's being done to change that and most important, why this matters because the research fraud in academia extends well beyond academia and it has consequences for all of us. The first of a two part series begins right now.

Brian Nosek (6:46)

This is Freakonomics Radio, the podcast that explores the hidden side of everything with your host Stephen Dubner.

Steven Dubner (7:03)

I rarely do this, but today I'm going to start by reading a couple sentences from Freakonomics, which Steve Levitt and I published in 2005. Cheating, we wrote, may or may not be human nature, but it is certainly a prominent feature in just about every human endeavor. Cheating is a primordial economic act, getting more for less. So when you think about it, why shouldn't we expect cheating even among scientific researchers? Consider this. Today it is thought that Ptolemy, the 2nd century Greek astronomer, faked his observations to fit his theories. And a new study in the journal Nature found that last year more than 10,000 research articles were retracted, easily breaking the old record. While a lot of the recent headlines are about scholars at big name American universities, the countries with the most retractions were Saudi Arabia, Pakistan, Russia and China.

Francesca Gino (8:02)

Fraud has existed since science has existed, and that's primarily because humans are doing the science. And people come with ideas, beliefs, motivations, reasons that they're doing the research that they do. And in some cases, people are so motivated to advance an idea or themselves that they are willing to change the evidence fraudulently to advance that idea or themselves.

Steven Dubner (8:30)

That is Brian Nosek, a psychology professor at the University of Virginia. In 2013, he founded the center for Open Science, a nonprofit that tries to improve the integrity of scientific research just to get it out of the way. I asked Nosek where his funding comes from.

Francesca Gino (8:47)

Our funders include nih, nsf, NASA and DARPA as federal sources, and then a variety of private sources such as the John Templeton Foundation, Arnold foundation and many others. And that diverse group of funders, and it's quite diverse, I think, share the recognition that the substantive things that they are trying to solve won't be solved very effectively if the work itself is not done credibly.

Steven Dubner (9:18)

In other words, the stakes here are high, higher than just individual academic researchers trying to advance their careers. If your goal is to improve medicine or transportation or immigration policy, any area where decisions are based on academic research, you don't want that research to be compromised.

Francesca Gino (9:39)

There are specific cases where a finding gets translated into public policy or into some type of activity that then ends up actually damaging people lives, treatments, solutions. One of the most prominent examples is the Wakefield scandal relating to development of autism and the notion that vaccines might contribute. And that has had an incredibly corrosive impact on public health, on people's beliefs about the sources of autism and the impacts of vaccines, et cetera. And that is very costly for the world. There's also a local cost in academic research, which is just a ton of waste. So even if it doesn't have public downstream consequences, if a false idea is in the literature and other people are trying to build on it, it's just waste, waste, waste, waste.

Steven Dubner (10:30)

There's also the idea that as much as universities worry about their students cheating, like using ChatGPT to write a paper, what kind of example are their professors setting? And there's one more big reason this story is so frustrating, and that has to do with the standards of academic research. The general view, at least this is the view that I have long held, is that academic research exists in a special category. It is a fact finding coalition that operates under a set of rules built around the accurate gathering and analysis of data, with the entire process subject to fact checking and peer review. Good journalism operates under similar rules. The New York Times has an old mission statement I've always liked. It goes to give the news impartially, without fear or favor, regardless of party, sect or interests involved. I've always thought this mission applies to academic research as well, that it's meant to be not only accurate, but free of personal or financial interests. These research papers aren't being written by some political official or management consultant or equity analyst. They're being written by someone so devoted to their field of research that they went through the hell of getting a PhD in order to spend their days doing that research. But the fact that Brian Nosek has been kept very busy with his center for Open Science suggests that my faith in academic research has been misplaced. I asked Nosek to walk me through how he went from being a researcher himself to being a new sort of referee.

Francesca Gino (12:14)

Yeah, I have always had an interest in how to do good science as a principled matter. And in doing that, we in the lab would work on developing tools and resources to be more transparent with our work, to try to be more rigorous with our work, to try to do higher powered, more sensitive research designs. So I wrote grant applications to say, can we make a repository where people can share their data? This is like 2007. And they would get polarized reviews where some reviewers would say, this would change everything. It'd be so useful to be more transparent with our work. And others saying, but researchers don't like sharing their data. Why would we do that?

Steven Dubner (13:00)

And why would researchers not want to share their data?

Francesca Gino (13:03)

Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career.

Steven Dubner (13:31)

Gosh, I do feel so naive because I've spent time in this ecosystem for many years now. And what you're saying just sounds wrong and it sounds selfish. And worst of all, it sounds like it goes against the mission of the scientific goal, which is to discover and distribute knowledge to the world. And that kind of offends me, I have to say.

Francesca Gino (13:54)

Yeah. And here's the irony is that almost every academic would say, of course science is supposed to be transparent. Of course we're doing research for the public good. Of course this is all to be shared. But come on, Pollyanna, we live in a world, right? The reality here is that there is a reward system and I have to have a career in order to do that research. And so yes, we can talk all about those ideals of transparency and sharing and rigor reproducibility, but if they're not part of the reward system, you're asking me to either behave by my ideals and not have a career, or have a career and sacrifice some of those ideals.

Steven Dubner (14:38)

You can see how these incentives create a problem. If a system has a built in bias against transparency, not only will there be less transparency, but also more opportunity to cheat. Nosek and his colleagues set out to address this by trying to replicate the result of papers that had already been published in academic journals. They called their idea the Reproducibility Project.

Francesca Gino (15:03)

And in the end of that 2015, we published the findings, which was a 270 co author paper of 100 replications of findings from three different journals in psychology. We got a little less than half of the findings successfully replicated.

Steven Dubner (15:19)

You did not mishear Brian Nosek. That's what he said.

Francesca Gino (15:23)

Little less than half of the findings successfully replicated.

Steven Dubner (15:27)

So he's been running large scale replications ever since. And not just in psychology.

Francesca Gino (15:32)

A year and a half ago, we published the results of the Reproducibility project in Cancer biology doing the same kind of process and found very similar results. Less than half of the findings in preclinical cancer research replicated successfully when we tried to do so.

Steven Dubner (15:47)

It's important to point out that what Nosek says here may not be as bad as it sounds. Here's how he thinks about that High number of studies that don't replicate.

Francesca Gino (15:56)

That doesn't mean that the original finding is necessarily wrong. We could have screwed something up in the replication. Successfully replicating doesn't mean the interpretation is right. It could be that both the findings have a confound in them, but we just are able to repeat the confound.

Steven Dubner (16:09)

There might be other explanations too, legitimate explanations. So we shouldn't equate a failure to replicate with fraud or even misconduct. Still, no six numbers do suggest that a lot of research happening today may not be producing science of lasting value. And we're talking here about the research being done at the most elite universities and academic journals. If you are a fan of science, any kind of science, this should concern you.

Francesca Gino (16:40)

Fraud is the ultimate corrosive element of the system of science. Because as much as transparency provides some replacement for trust, you can't be transparent about everything. So the ideal model in scholarship is that you can see how it is. They generated their evidence, how they interpreted their evidence, what the evidence actually is, and then independent people can interrogate that. And so to the extent that fraud intrudes and actually the evidence isn't there, it isn't actual evidence, then the whole edifice of that scholarly debate, entangling with ideas falls apart because you're actually tangling with ideas that aren't based on anything.

Steven Dubner (17:24)

How familiar are you with the Joachim Bolt situation?

Francesca Gino (17:28)

I'm not recalling that name, but I may know the case if you describe it.

Steven Dubner (17:32)

Yeah. This was the German anesthesiologist who had almost 200 papers retracted. And, oh, yeah, I gather there were people actually dying as a result of this faulty research. So I am curious to ask you, in which academic fields or disciplines do you think fraud or sloppiness is most prominent?

Francesca Gino (17:52)

We can't say with any confidence where it's most prominent. We can only say that the incentives for doing it are everywhere. And some of them gain more attention because, for example, Francesca's findings are interesting. They're interesting to everyone, so of course they're gonna get some attention to that. Whereas the anesthesiologist's findings are not interesting.

Steven Dubner (18:13)

They put people to sleep until they kill you.

Francesca Gino (18:15)

Well, yeah, I guess they put you to sleep and Then they kill you.

Steven Dubner (18:18)

But it does seem like your field of training, psychology, and especially social psychology, is a hotbed of, I wouldn't say fraud, but certainly controversy and overturned findings over the years.

Francesca Gino (18:31)

Yeah, I would say that there is a lot of attention to social psychology for two reasons. One is that it has public engagement, interest, value.

Steven Dubner (18:41)

That's a way of saying that people care about your findings.

Francesca Gino (18:43)

People care about, at least in the sense of, oh, that's interesting to learn. Right. But the other reason is that social psychology has bothered to look, and I think social psychology became a hotbed for this because the actual challenges in the social system of science that need to be addressed are social psychological problems.

Steven Dubner (19:03)

What do you mean by that?

Francesca Gino (19:04)

I mean like the reward system. How it is that people might rationalize and use motivated reasoning to come to findings that are less credible. A lot of these problems are ones that social psychologists spend every day thinking about.

Steven Dubner (19:20)

I asked Nosek why he thinks that presumably honest people might over time, come to behave dishonestly.

Francesca Gino (19:28)

The case that people make to say that this is a bigger problem now is that the competition for attention, jobs advancement, is very high, perhaps greater than it's ever been.

Steven Dubner (19:40)

Do you think that's been driven by the shrinking of tenure positions at universities?

Francesca Gino (19:43)

Yeah. So there are many more people and many fewer positions, which is an obvious challenge for a competitive marketplace. And there are now pathways for public attention that have even bigger impact. Academics, by and large, didn't think about ways to get rich. They looked for ways to have time to think about the problems that they want to think about. But now they have pathways to get rich.

Steven Dubner (20:07)

Those pathways have benefited many people, myself included. Even though I am not an academic. Thanks to my partnership with Steve Levitt, who's an economist at the University of Chicago, I have had more opportunities than I ever would have thought to possible, including this show. I have wondered why there seems to be so much less shady research in economics than in psychology and other fields. When you talk to economists, they'll give you several reasons. Economic research is very mathy, and it comes with a lot of what they call robustness checks. There is also a tradition of, let's say, aggressive debate within academic economics. Long before you publish a paper, you typically present it to your peers and elders at seminars, and they are only too happy to point out any possible flaw and call you an idiot if you disagree. I'm not saying this is the best way to conduct business, but it certainly makes shaky data more costly. Also, economists tend to work with big data sets much bigger than in the rest of the social sciences and it's often publicly available data. So no cheating opportunity there. This doesn't mean there is no controversy within economic research and no overturned results. There are a lot. And if you hang around with economists, as I tend to do, you will hear whispers about a few researchers who are suspected of fudging their data. But it does seem that most researchers in econ are mostly honest. And for all the honest researchers out there in whatever field, there's one more twist. When you are playing a game by the rules, but you see that the people winning the game are cheating, you feel like a sucker and no one enjoys feeling like a sucker. But it's bigger than that. If the cheaters are winning, that means non cheaters get smaller rewards. And it means all their hard work may also be viewed with suspicion. So what can be done about that problem? This might require something more invasive than a reproducibility study. This might require interrogating the data or research methodology on suspicious research papers and making public accusations of fraud. And this brings us to the whistleblowers we heard about earlier. They are co directors of the Credibility Lab at the University of Pennsylvania and collectively they run a blog called Data Colada.

Leif Nelson (22:46)

My name is Leif Nelson and I'm a professor of business administration at University of California, Berkeley.

Yuri Simonson (22:53)

Yuri Simonson. I'm a professor of behavioral science at the Azade Business School in Barcelona.

Joe Simmons (22:58)

Joe Simmons. I'm a faculty member at the Wharton School at the University of Pennsylvania.

Steven Dubner (23:03)

Nelson, Simonson and Simmons are, let's call it, mid career academic researchers. They've been at it for a while and they hold high status positions within the ecosystem. They are concerned. They've all published widely in top journals of psychology and decision science. So they're coming at this not only from the inside, but with inside knowledge of how academic research works and doesn't work. So they focus on examining the methodology used in this kind of research.

Yuri Simonson (23:35)

What motivated our whole journey into methodology was that we would go to conferences or read papers and, and not believe them. And we would find that whenever a finding didn't align with our intuition, we would trust our intuition over the finding. And that was it sort of defeated the whole purpose. Like if you're only believing things you already believe, then why bother?

Joe Simmons (23:54)

This guy, Darrell Bem published a nine study paper with eight studies worth of statistically significant evidence that people have esp. And most people were like, what is going on? Like this cannot be a true finding.

Yuri Simonson (24:05)

So the idea was like, how do we show people that you can really very easily produce evidence of anything. So we thought, let's start with something that's obviously false. We said, okay, something that's quite hard to do is to make people younger. We've been trying forever. We never succeeded. So let's show that we can do that in a silly way. So we decided to show that we can make people younger by listening to a song by the Beatles.

Steven Dubner (24:27)

The song was when I'm 64, correct?

Yuri Simonson (24:29)

That's right. And so the idea is if we can make anything significant, one way to prove it is to say, I'm going to show you with statistical significant evidence that people got younger after they listened to when I'm 64, will you still need me?

Steven Dubner (24:42)

Will you still feed me when I'm 64? They ran real lab experiments with real research subjects who had real birth dates and played them real songs. When I'm 64 and two others.

Yuri Simonson (24:57)

A control song. It's called I believe Kalumba by Mr. Scruff. And then we had another song that was meant to go in the other direction and it didn't work, so we just didn't report it. Which was Hot potato, Hot potato, Hot.

Steven Dubner (25:12)

Potato, Vayner, Hot potato. They essentially manipulated and cherry picked their data to produce the absurd finding. They wanted that listening to when I'm 64 does lower your age by a full year and a half. It turns out they published their article in Psychological Science, one of the top journals in the field. Their piece was called False Positive Psychology. Undisclosed flexibility in data collection and analysis allows presenting anything as significant, they wrote. These studies were conducted with real participants, employed legitimate statistical analyses, and are reported truthfully. Nevertheless, they seem to support hypotheses that are unlikely or necessarily false.

Joe Simmons (25:55)

So we were surprised when the paper got accepted. And then the immediate aftermath was shocking. Like there were so many people for whom this resonated. And then there were lots of people who were really not very happy. Like, why are you giving the field a bad name?

Steven Dubner (26:09)

But Simmons, Nelson and Simonson felt their field already had a bad name.

Yuri Simonson (26:14)

Yeah, we thought it was very bad.

Leif Nelson (26:16)

We started our blog in late 2013. We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part, but would definitely not be sending signals of Self seriousness. So we tried out a few things and somewhere in there, datacolota was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name, he thought it rhymed, which still tickles me and Joe because for him it's Dada Colada.

Steven Dubner (27:02)

The Data Colada team, or Dada Colada or maybe Data Kaleida, they weren't going to look just for cases of outright fraud. They were concerned, as Brian Nosek had been, that the pressure to publish interesting results might produce unreliable findings, even if the researcher had mostly followed the rules. Consider, for instance, a practice they came to call P hacking, with the P standing for probability. This is Leif Nelson.

Leif Nelson (27:32)

The classic forms of what we had characterized as P hacking. They're not quite errors, they're decisions that are accidentally self serving. It's if you measure multiple things but only report the one you like the most. Or you run a study where there's three treatments, Condition A, condition B and condition C. But in the end you drop condition B and you don't even talk about it. You just compare A to C. Hot.

Steven Dubner (27:56)

Potato, hot potato, hot potato. That, remember, is one of the things the Data Colada guys did in their pranky paper about when I'm 64, they just left out hot potato.

Leif Nelson (28:07)

And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers and you say we should eliminate those outliers, or we should windsorize the outliers, which is basically truncating them down to a lower high number. Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just if you have a consideration of reporting one variable or the other, and one variable makes your hypothesis look good and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self serving or honestly because you'd say, like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good and that one looks good. It's probably the better measure.

Steven Dubner (29:03)

And here's Simonson.

Yuri Simonson (29:04)

There's a few approaches, some of them that we've done like just do statistics and say this is statistically impossible. The other is you see associations in the data or lack associations with the data that they're not mathematical properties, but just anybody looking at data who's familiar with that would realize this is not right. Imagine that you have data on weight and height, and you correlate it and you find zero correlation. That cannot be right. People who are bigger are heavier. And so if you found zero correlation or negative correlation, you think, maybe these are not real weight measures. Another one is you see rounding or precision that is suspicious. You see rounded values where there shouldn't be any rounding or absence of rounding where there should be. So, for example, in one case that we worked on, there was data, supposedly when people were asked, how much will you pay for this T shirt? And the very curious thing is, there was no rounding. People were equally likely to say, $7 is $8 or $10. But if you've ever collected data like that, you know that people round people say 10 or 20. They don't say 17.

Steven Dubner (30:06)

There is another category that you might call convenient errors. Here is Nelson again, these will be.

Leif Nelson (30:12)

Things that can be as simple as a typo, where someone's writing up their report, and the means are actually 5.1 and 5.12. But instead someone writes it down as 51.2, and you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally, a typo might end up in print. And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data.

Steven Dubner (30:45)

Do you think that the first set of conditions that you described are quite likely to lead to fraud? I mean, how slippery is the slope? Is the kind of person who's willing to do those things, willing to go ahead and commit fraud, especially if they've gotten away with it for a while?

Leif Nelson (31:02)

Well, Steven, you were asking a question that is pretty heavy and one that I'm not particularly well equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer, and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect ten measures and I report one. That's slippery slope. But making up data feels qualitatively different, and I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop Some measures or drop a condition or you remove the outliers. And then also you take Participant 35 and you change their answer from a seven to a nine. You're like, Whoa, that last one doesn't sound the same, but maybe there's some psychology for that that it feels like it's an extension.

Steven Dubner (32:04)

So these three self appointed data detectives looked into the work done by their peers and they publicly posted their findings.

Leif Nelson (32:13)

The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual. I want to go read that paper. And so I read the paper and then looked at that data set and that one, it had collected Data on a nine point interval scale. So people can answer 1, 2, 3, up through 9. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy. Once you open the data set, you can then close it and say, it's broken.

Steven Dubner (32:59)

The paper Nelson was describing had been published by four Taiwanese researchers in 2012 in the Journal Judgment and Decision Making. After the Data Colada investigation, the paper was retracted, although as far as we can tell, the researchers weren't sanctioned or punished. So how much fraud is there? I asked Yuri Simonson.

Yuri Simonson (33:21)

I would estimate the share of fraud in the order of, say, 5% of articles.

Steven Dubner (33:25)

What's the difference between, let's say high profile academic journals versus mid tier or lower? Is fraud more likely to be prominent in the higher or the lower?

Yuri Simonson (33:33)

I don't read really low tier ones, so I don't know. Sometimes I will, but if I come across frothier, I will ignore it because the cost is so high of pursuing a case of fraud that it's just not worth it. If it's a paper that has, you know, seven citations after three years and it's published in a journal that nobody knows, I just let it be. And I'm sure other people do that too.

Steven Dubner (33:57)

Simonson, Nelson and Simmons kept on with their investigative work as a sideline to their regular research and teaching responsibilities. Within psychology and behavioral science circles, Data Colada became well known and a bit feared. But they didn't have much reach beyond those circles. That changed a couple years ago. They published a post called Evidence of Fraud in an influential field experiment about dishonesty the fraud they claimed to identify was in a paper published years earlier in a top journal known as PNAS or the Proceedings of the National Academy of Sciences. The paper was called Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self Reports in comparison to Signing at the End. Let's put that in English. The paper claimed that if you are asked to sign a form at the top of the form before you fill in the information, you are more likely to be truthful than if you sign at the bottom. You can see how that might work, right? There are four things you might want to know about this paper. One, the central finding had proved to be extraordinarily popular. A lot of firms and institutions started putting the signature line at the top of tax statements, insurance forms, things like that. Number two, the article had been edited for PNAS by Danny Kahneman, perhaps the best known living psychologist at that time and one of the most highly regarded. Number three. Two of the five co authors on the paper were among the best known people in this field, Dan Ariely and Francesca Gino. And four, there was already evidence that something was up with the original paper because its authors had published a second paper saying that their original findings didn't replicate. The failure to replicate, as we heard earlier, does not necessarily mean fraud. But now the data Colada investigators claimed to have proof that yes, there was fraud in the original paper. That original paper was written by Dan Ariely, Francesca Gino, along with Nina Mazar, Lisa Shue and this man.

Max Bazerman (36:09)

I'm Max Bazerman and I'm a professor at the Harvard Business School.

Steven Dubner (36:13)

After the break, Max Bazerman takes us through the collaboration that triggered a crisis. We will be right back. Freakonomics Radio is sponsored by Indiana Jones and the Great Circle. Indiana Jones and the Great Circle is available now. Uncover one of history's greatest mysteries in a first person single player adventure game set between the events of Raiders of the Lost Ark and the Last Crusade. The year is 1937. Sinister forces are scouring the globe for the secret to an ancient power connected to the Great Circle. And only one person can stop them. Indiana Jones Adventure calls on Xbox Series X and S Game Pass and PC rated T for Teen. Copyright and trademark Hallmark 2024 Lucasfilm Ltd. All rights reserved. Freakonomics Radio is sponsored by Hydro this holiday season. Give the gift of an immersive full body workout all from the comfort of home with the Hydro Rower. Hydro combines strength and cardio to give you an amazing full body workout all in 20 minutes. With the largest library of rowing workouts led by Olympians and world class athletes. Hydro helps you stay motivated and crush your fitness goals and they have you covered with free standard shipping, 30 day risk, free trial and a one year warranty. Give the gift of a full body workout all from the comfort of home with Hydro. Head over to hydrow.com and use code FREAKONOMICS to save up to $800 off your Hydro Pro Rower. That's H Y D R-O-W.com code FREAKONOMICS to save up to $800 hydro.com code FREAKONOMICICS FREAKONOMICS Radio is sponsored by Merrell with a dedicated Merrill Advisor. You get a personalized plan for your financial goals and when plans change, Merrill's with you every step of the way. Go to ML.com/bullish to learn more. Merrill, a Bank of America company what would you like the Power to Do Investing Involves Risk Merrill Lynch, Pierce, Fenner and Smith Incorporated Registered Broker Dealer Registered Investment Advisor Member SIPC Max Bazerman, a professor of Business Administration at the Harvard Business School, is considered an elder statesman in the field of behavioral science. For decades he has been publishing well regarded research papers and books and he's known as a wise and caring mentor to younger scholars. That last bit, it would seem, is the best explanation for how Bazerman wound up being a co author on what would turn out to be a very, very, very problematic research paper. This is the signing at the top paper we heard about before the break. The paper published in PNAS in 2012, which claimed that you are more likely to be honest if you sign a form at the top before you fill in the information than if you sign at the end. This paper actually started out as two separate research projects.

Max Bazerman (39:32)

Here's Bazerman so 2011 Lisa Xu and Francesca Gino and I had a working paper that got rejected from a couple of journals that basically claim to show if you sign a document before you fill it out, you're more likely to tell the truth. Our studies were done in the laboratory.

Steven Dubner (39:51)

Lisa Hsu was a doctoral student at the time. Bazerman was her advisor, one of the chairs of her dissertation committee. As for Francesca Gino, Francesca started visiting.

Max Bazerman (40:04)

The Harvard Business School as a doctoral student from Italy. By 2004 she was attending my doctoral seminar and we started to interact pretty regularly and eventually I was on her dissertation committee and played a pretty active role in advising her.

Steven Dubner (40:21)

Bazerman liked Gino a great deal. In fact, nearly everyone liked Gino and admired her intellect and work ethic.

Max Bazerman (40:28)

We once even Got to the point of our two families making an offer to a developer on a project to have houses connected to each other.

Steven Dubner (40:38)

Their signing at the top paper was based on data from two lab experiments from the University of North Carolina at Chapel Hill, where Gino taught before coming to Harvard. In those experiments, research subjects tried to solve a number of puzzles with a financial reward for every right answer. And afterward they filled out a form to tell the experimenter how much money they had earned. Some of the research subjects were asked to sign an honesty pledge at the top of of the form, others at the bottom. In the experiment, the ones who signed at the top were more truthful about their earnings. At least that's what these data said.

Max Bazerman (41:18)

They were brought to our threesome from Francesca, and the exact role of Francesca and the lab manager remains somewhat unclear, but it's certainly safe to say that Lisa Xu and I had little to do with the collection of the data. So this goes back to a time when Francesca was developing an excellent career, but she was not the superstar that she came to be. And I was a senior person and probably doing the least amount of work and working more after the document came to fruition.

Steven Dubner (41:54)

Do you think that most people, let's say the median American who maybe holds a decent opinion of university life and academic research, which maybe that's not the median person, maybe, maybe the median person doesn't hold such a decent opinion. But for someone who might read an article that's based on an academic study and say, oh, that's interesting, I'm going to, you know, file that away as a useful, probably true piece of information, how surprised do you think that person would be to learn that a senior colleague like you, who's co author on a lot of papers with junior colleagues, that you personally don't interact at all with the original data, how surprising do you think most people would find that?

Max Bazerman (42:35)

So I wouldn't say don't interact with all. I certainly would read the results section pretty carefully, but I would read it with the intent of seeing if there was any error along the way.

Steven Dubner (42:48)

But how can you tell if there's error if you're not in the, you know, it's like this whole issue reminds me a little bit of being, let's say I'm a chef in a restaurant and I'm given the ingredients to cook, but I'm not allowed to examine them. So I don't know if they're rancid or fresh or even fake.

Max Bazerman (43:06)

So I like that example. So instead of being a chef of a restaurant. Let's imagine that you're the owner of 12 different restaurants and you have a head chef in each. And that head chef is going to be what I think of as the most senior of my more junior colleagues on the project. And over time, I've come to trust that they're going to be doing a really good job of overseeing the ingredients that go into the research process. And by doing so, there's other things that I can do. I can work on making sure that we have the funds available. I could work on whatever particular problems come up administratively. I could work with more young scholars because my time is more available. So there's lots of good by this efficiency of trusting the assistant professor on the project or the head chef at a particular restaurant so that I'm not examining the specific ways in which the sausage is made.

Steven Dubner (44:08)

Now, remember, this original paper was rejected by multiple journals. Bazerman says they got feedback suggesting their own argument about signing at the top would be more believable if, in addition to the lab results that Francesca Gino provided, they had some results from the real world as well. That's what researchers would call a field experiment versus a lab experiment. As luck would have it, another researcher, a friend of Geno's, no less, apparently had some good field results.

Max Bazerman (44:38)

We collectively heard that Dan Ariely was presenting a very similar result based on a field experiment having to do with an insurance company. And Francesca reached out to Dan, and we basically combined efforts to pull the three studies into one paper.

Steven Dubner (44:58)

Ariely's data included the number of miles the customers of this insurance company reported having driven in a year. If you think about how insurance works, a customer might have an incentive to underreport their mileage in the hopes of lowering their insurance bill. Ariely's data showed that customers who were asked to sign this mileage statement at the top reported having driven more miles than customers who were asked to sign at the bottom, again suggesting that signing at the top makes people more honest. Max Bazerman now received the first draft of a new paper that combined the Ariely and Geno studies.

Max Bazerman (45:39)

I'm reading the insurance field experiment for the first time at this point, and as I'm reading it, I have some questions. I wasn't remotely thinking about fraud as an issue. I just thought there was something wrong. And what I saw as wrong was that we were reporting that the average driver in the database had driven between 24,000 and 27,000 miles per year. And I just looked at that and I said, that seems off.

Steven Dubner (46:10)

It seemed off. To Baserman, because the average American only drives around 13,000 miles a year.

Max Bazerman (46:16)

So I asked some questions about it, and Ariely, who was the point person for that data, the origins of which are not completely clear, sent back a very quick email saying the mileage is correct. And I continued to say, well, we need to clarify what's going on here. It seems off that people have driven so many miles, particularly when you're talking about tens of thousands of drivers. And eventually Ariely comes back with the drivers are senior citizens in Florida.

Steven Dubner (46:50)

Sounds like they should drive even less than 24,000 miles then.

Max Bazerman (46:53)

Exactly my thought. So my questions continue and I don't get very good answers. And literally this goes on for months. And I'm seriously considering taking my name off the paper at the time. Lisa Xu is a doctoral student on the job market and she's presenting this work.

Steven Dubner (47:09)

And how concerned were you about damaging her prospects?

Max Bazerman (47:12)

I was very concerned that if I dropped off the paper that there's something suspicious about Lisa's presentation. So I keep on asking questions, but I don't withdraw. And by early 2012, I'm attending a conference. And I arrive at the conference and in the big hallway, I run into Lisa, who's my advisee, my friend, my co author, somebody who I like a lot. And she's with Nina Mazar, who I had never met before. So Lisa introduces us, and I believe I was expressing my unhappiness with the lack of clarity on this mileage issue.

Steven Dubner (47:52)

Nina Mazar is Ariely's collaborator on the insurance study. Correct.

Max Bazerman (47:57)

So that's a little bit unclear. Okay. So when we asked Ariely to join forces, he said, fine, but Nina would be part of the project as well. So I always thought she was part of the insurance study. Later on in life, Nina claimed she had no more connection to the insurance study than I did, that she first connected to it when this five author paper comes together.

Steven Dubner (48:22)

But in any case, at this conference, she assuaged you to some degree.

Max Bazerman (48:26)

Yeah, exactly. So she basically pleasantly and openly pulled up the database on her computer. And I said, so what's going on? And she said, I think what's going on is that we don't know that the period between time one and time two for assessing the number of miles driven was one year. We know when time two was collected, but it may have been more than one year for when time one was collected. And in my mind, what becomes clear is that that makes our study noisier. But as long as a real experiment was run, this is actually Pretty good news. All we need to do is correct the presentation in the paper, which we did, the paper submitted, it's published. I develop a belief that this effect is true, and people love this result. And from a theoretical Standp point, it's a shockingly simple idea. From a practical standpoint, it's just perfect. It's so simple that organizations can easily implement it.

Steven Dubner (49:27)

And who did implement it?

Max Bazerman (49:28)

A lot of people implemented it. You know, I think Lemonade Insurance, under Ariely's advice, implemented it.

Steven Dubner (49:38)

Lemonade Insurance, by the way, didn't just implement Ariely's advice. They hired him as their chief behavioral officer.

Max Bazerman (49:47)

And many government agencies implemented, including the US Government.

Steven Dubner (49:51)

Indeed, the first sentence of the paper that Baserman and the Others published in 2012 reads like the annual tax gap between actual and claimed taxes due in the United States amounts to roughly $345 billion. Now imagine working for the IRS or any tax agency anywhere in the world and learning that these brilliant academic researchers from Harvard and Duke had found that if you simply have people sign their tax forms at the top rather than the bottom, that millions, maybe billions of extra dollars will suddenly flow your way.

Max Bazerman (50:32)

And by 2016, I get an email that's actually critical to the whole evolution of what happens later. The email is from Stuart Bazerman, only he spelled his name with an S instead of a Z. And he basically is a pretty low key guy who was working on an insurance startup to do insurance online. And his wife sue encouraged him to email me because he was working on the problem of how do we get people to tell the truth online? And he had read the 2012 signing first paper and he said, this looks like I might even be related to a guy who knows how to get people to tell the truth. So Stewie emails me and we develop a very good relationship. Turns out he's a fifth cousin, according to 23andMe. And I also develop a consulting relationship with Slice Insurance, the company that he was developing.

Steven Dubner (51:35)

Now, sorry to ask the rude question, but does that seem like a conflict of interest at all in retrospect, to.

Max Bazerman (51:42)

Work for my cousin?

Steven Dubner (51:43)

Well, to go be a consultant for an insurance company based on the findings of a paper which turns out to have been fraudulent that you were co author on.

Max Bazerman (51:52)

I didn't know it was fraudulent in 2016. I even believed it by now.

Steven Dubner (51:55)

Yeah. Are you still a consultant for Slice?

Max Bazerman (51:58)

No.

Steven Dubner (51:58)

Did you leave because the finding was fraudulent?

Max Bazerman (52:01)

No, no. I have a terrific relationship with Slice.

Steven Dubner (52:06)

Okay, so going back to 2016, Max Bazerman wanted to help his newfound cousin learn whether signing at the top would be as effective in an online setting the way it seemed to work with paper documents. So Baserman and some junior colleagues set out to test that question. And, and how was Baserman feeling at the time about the original sign at the top finding?

Max Bazerman (52:30)

We know it works. We know the effects are big. We know the world is intrigued by it. Seems perfect.

Steven Dubner (52:36)

Were you worried about a placebo effect at all? Which is to say, if enough people have heard through media descriptions of this signing first phenomenon, that if they encounter a forum where they're asked to sign first, they will now know that they are under some kind of scrutiny, perhaps, and therefore they're more likely to be honest because they know about it.

Max Bazerman (52:58)

That's a terrific methodological critique. So what you just said logically makes sense. I think as of the time we were doing these online studies, and there are many of them, I don't think that there was widespread public awareness of the signing of first effect. You know, Ariely and Gino and I talked to lots of executive classes, but I wouldn't say it was a well known social phenomena, but I could be wrong. So your methodological critique could be valuable. But anyhow, signing first with or without a placebo doesn't work online. We got nothing.

Steven Dubner (53:36)

How surprised were you?

Max Bazerman (53:37)

Very. And I kind of said, well, let's take a look at what we did. Let's see sort of how we might have messed up the design. Let's try again. So we make some changes, we do it a second time, we make some changes, we do it a third time. Still no effect. No effect. No effect. And recall, the 2012 paper not only has effects, it has effects across three different studies that are all statistically significant and the effects are large. So the project clearly transforms somewhere between replication failure 3 to 5. It's transforming from how do we get people to tell the truth online to a massive replication failure of a pretty visible academic effect. And so after we fail six times, we decide, well, let's go back and do a large scale replication of one of the original lab studies.

Steven Dubner (54:34)

Bazerman and the collaborators he had brought on to do the online replication work now convened with the authors of the original paper, including Dan Ariely and Francesca Gino, and they set out to replicate one of the that paper's lab studies, but using more than 10 times as many research subjects as in the original. One feature of many academic studies, especially in a field like psychology, is that they often use a small set of research subjects to run this kind of Experiment, often just students from their own university. A small research pool is cheaper and faster. And if the goal is to produce a lot of publishable research, fast is good. But small sample sets are more likely to return a skewed result. So now with a bigger sample and much more scrutiny, they get no effect. Signing at the top doesn't seem to do much of anything.

Max Bazerman (55:33)

So certainly I felt a moral obligation to correct the record.

Steven Dubner (55:37)

And not just moral obligation, but as you've told us, there are institutions, government institutions and firms that are using this research. Did you feel, I mean, I don't want to put a word in your mouth, but was it a sense of guilt or panic or fear, anything like that?

Max Bazerman (55:51)

Certainly I would feel some sense of maybe. Maybe guilt is the right word, that my name's on a paper that people are using when I no longer think that they should be using it. But you know, I didn't think that I was doing anything wrong. And quite honestly, I'm still not thinking fraud at this point. I just don't know what's going on. I'm thinking of cleaning up the record.

Steven Dubner (56:15)

In 2020, Baserman, along with all the original authors and his two more recent collaborators, published a follow up paper in PNAS, the same journal where the original piece was published in 2012. This paper was called Signing at the beginning versus at the end does not decrease dishonesty. Now, from the outside, you may think this is not a very brave position to take. You were simply putting out a new paper saying that the paper you put out eight years ago, the one that got so much attention and advanced so many careers, that that one didn't actually work. On the other hand, you could say this is how science is supposed to work. You have a hypothesis, you run experiments to test your hypothesis, you gather and analyze the data, and you present your findings. If new information comes along that overturns your finding, well, that's what needs to happen to correct the scientific record. But it's worth noting what the original authors didn't do. They didn't retract the original paper, nor did the journal retract it, at least not at this point. So from the outside, this looked like a story of science that was perhaps conducted sloppily, but not a story about fraud. And at least some of the original authors hadn't given up on the original finding.

Max Bazerman (57:43)

Ariely and Nina Mazar both continued to argue, well, it works some of the time, it doesn't work other times, and we need to do more studies to find out when it works. And when it doesn't. And my attitude, so I won't speak for other co authors, was to basically say we have plenty of evidence to conclude that we should tell the world that we have no faith that this effect works so we don't retract and life moves forward. And In June of 2021, I believe I get an email from one of the members of the Data Colada team saying, max, can the three of us meet with you on Zoom to talk about something important after the break?

Steven Dubner (58:31)

What it's like to be on the other end of a Data Colada Zoom call, and how much does academic fraud contribute to the public's ever sinking opinion of universities in general? I'm Stephen Dubner. This is Freakonomics Radio. We'll be right back. Freakonomics Radio is sponsored by Indiana Jones and the Great Circle. Indiana Jones and the Great Circle is available now. Uncover one of history's greatest mysteries in a first person single player adventure game. Set between the events of Raiders of the Lost Ark and the Last Crusade. The year is 1937. Sinister forces are scouring the globe for the secret to an ancient power connected to the Great Circle. And only one person can stop them. Indiana Jones Adventure calls on Xbox Series X and S Game Pass and PC rated T for Teen. Copyright and trademark 2024 Lucasfilm Ltd. All rights reserved. Freakonomics Radio is sponsored by Merrill with a dedicated Merrill Advisor. You get a personalized plan for your financial goals and when plans change, Merrill's with you every step of the way. Go to ML.combullish to learn more. Merrill, a Bank of America company. What would you like the power to do? Investing involves risk Merrill Lynch, Pierce, Fenner and Smith Incorporated. Registered Broker Dealer Registered Investment Advisor Member S Freakonomics Radio is sponsored by Range Rover Sport. What makes a leader? It's a tough question, but one thing is for sure. True leaders lead by example and take risks. They plunge into life with determination. For those who lead by example and who approach life with palpable passion, there's the Range Rover Sport. A new dimension of sporting luxury. Elevate your desires with the visceral, dramatic and uncompromising Range Rover Sport Explorer. Experience its world renowned off road capability with adaptive off road cruise control which monitors ground conditions and acclimates to the present terrain. Agility, control and composure are achieved with dynamic air suspension. While adaptive dynamics reduce unwanted body movements to deliver smooth and composed handling for focused on road performance. Rise to every occasion in the Range Rover Sport where sophisticated refinement meets visceral power. Build your Range rover sport@landroverusa.com that's land roverusa.com okay, so Max Bazerman, a well regarded senior researcher and professor at the Harvard Business School, gets an email asking him to meet with the Data Colada team. And they say it's important.

Max Bazerman (61:28)

So there's a zoom. And the first part of the zoom is the data Collada team showing me the evidence for fraud in the insurance paper. And it's kind of overwhelming. These guys are careful and they're thorough and like, they convinced me that there was fraud in the study.

Steven Dubner (61:49)

This insurance study was one of three studies in the Sign at the Top paper that Bazerman had co authored some years back. The Data Colada researchers had scrutinized the data that Dan Ariely had used and found several things suspicious. The most obvious one was a data chart called a histogram showing the number of miles driven each year by the people in his study. For data like this, a histogram will typically look like a bell curve with a lot of people clustered around the center and then some outliers sloping off toward the upper end and lower end. But this histogram showed a nearly uniform distribution of drivers from 0 miles to 50,000 miles. This is not what real data look like, the Data Kolata team wrote on their blog post, and we can't think of a plausible, benign explanation for it. And what's Max Baiserman thinking now?

Max Bazerman (62:46)

I'm just kind of overwhelmed with the fact that I'm the author of a fraudulent paper.

Steven Dubner (62:50)

And later, there was new information from the insurance company that had given Dan Ariely the data. They told the Planet Money podcast that the data Ariely published was significantly different from what they had given him. And in their original data, there was no difference between those who signed the forms at the top and those who signed at the end. Although Ariely declined to do an interview for this episode, he did send a written statement. As someone who spent many years studying dishonesty, he wrote, I appreciate the irony of being accused of dishonesty. There's no question that the data underlying the 2012 study I co authored with four other researchers about dishonesty was, well, dishonest. I have looked diligently to figure out what went wrong, but given that this took place more than 15 years ago, I can't tell for sure what happened, he added. All five co authors of the study in question participated in review sessions with people from the insurance company and asked questions about the data. Ultimately, we all felt satisfied with the Answers we got and collectively decided to move forward with the paper. Max Bazerman. When we shared with him, Ariella's statement replied, I am extremely confident that I never participated in any such review sessions and confirmed this with conversations with Lisa Hsu and a thorough search of my email records. The final part of Ariella's statement reads, the circumstances that led to the data being falsified are being investigated by Duke University. I am confident that the investigation will find no evidence to suggest I was responsible for any data manipulation. I am sure that this matter will be behind me very soon and I will be resuming my research at Duke at full speed. That's the end of Ariely's statement. Since we originally published this episode, Ariely claims that the investigation has concluded at Duke, where he remains an active professor. As for the PNAS journal, they did finally retract the original paper. Meanwhile, on that Zoom call, the datacollada team had some more news for Max Bazerman.

Max Bazerman (65:03)

So after they presented the insurance evidence to me, they said, and now for worse news. That's when they introduced the allegation of data fabrication in one of the lab studies.

Steven Dubner (65:17)

One of the lab studies, meaning one of the separate studies in the same paper whose data had come from Francesca Gino. Data Colada said they had found serious problems with her data too.

Max Bazerman (65:30)

And to make matters worse, there was evidence of data fabrication in three other projects that Gino was a co author of.

Steven Dubner (65:37)

And when you say they said, now for the worst news, I'm assuming, but tell me if I'm wrong. They're saying that because they know that you have had a long and close relationship with Francesca Gino.

Max Bazerman (65:48)

Yeah, and I was clearly more closely connected to those lab studies.

Steven Dubner (65:53)

And you're a co author on more than one of the papers.

Max Bazerman (65:56)

I was only a co author of one of the four papers that they were showing me, but I by then had been a co author of eight different papers, empirical papers that have Francesca Gino as a co author.

Steven Dubner (66:07)

So what happened next on the Zoom call?

Max Bazerman (66:10)

So they provide the evidence, and so I'm now aware that there's what datacologa later called clusterfake, that there's at least two frauds in the same paper, or that that was likely to be the case. And then Data Clada basically said, so Max, you're the Harvard professor. We think that Harvard should have access to this information. Are you the person who will take it to them? And I said, no, thank you.

Steven Dubner (66:40)

Because why?

Max Bazerman (66:41)

Because I didn't. I certainly thought that Harvard should be aware of what I was looking at, but I didn't want to play a central role in making that happen.

Steven Dubner (66:58)

Data Colada had begun investigating Francesca Gino after they received a tip from a graduate student named Zoe Zianni and another anonymous researcher. In addition to the sign at the top paper, the Data Colada team wrote, we believe that many more Geno authored papers contain fake data, perhaps dozens.

Leif Nelson (67:20)

Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud.

Steven Dubner (67:32)

That again, is Leif Nelson, one of the three data collateral researchers, along with Yuri Simonson and Joe Simmons.

Leif Nelson (67:39)

Bridging the gap between those two positions is this other entity, Harvard University. We only know what they've said outwardly, which is that they've put her on administrative leave and they've recommended the retraction of those four papers or the retraction of three, plus an amendment to a previously retracted paper.

Steven Dubner (67:59)

And how confident are the Data Colada researchers in the accuracy of their analysis? Here is Simonson.

Yuri Simonson (68:06)

So I would say I have on the falsity of the findings. I don't have reasonable doubt.

Steven Dubner (68:13)

Not long after Harvard Business School placed Francesca Gino on leave, she filed a lawsuit.

Yuri Simonson (68:19)

So we were sued together with Harvard for $25 million. We were sued for defamation.

Steven Dubner (68:26)

Again, Francesca Gino declined our request for an interview. On a website called Francesca v. Harvard, she wrote, I absolutely did not commit academic fraud. Harvard has ruined my career wrongfully. The only way to right this wrong is for me to sue Harvard. As for Data Colada, Gina wrote, the decision to sue Data Colada was more difficult. I have long admired Data Colada's work. I have particularly respected its commitment to sharing any negative findings with the author before going public with the charges. Yet in my case, Data Colada changed its procedure. Francesca Gino also claimed in her lawsuit that Harvard Business School discriminated against her on the basis of sex. And later, her allies wondered why Gino's punishment had been so prompt and severe when, for instance, Harvard President Claudine Gay hadn't been disciplined immediately for outright plagiarism in her research. But then, as you likely remember, Gay was forced to resign from the presidency. Yet more academic fraud in the headlines. I asked Leif Nelson how it felt when he heard that Francesca Gino was suing him and the other members of Data Colada.

Leif Nelson (69:44)

Certainly scary. Scary because it's so unfamiliar. I found out from talking. Basically, I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now, given the lawsuit, would you like to add a new comment? And I was basically like, what? What lawsuit are you talking about? And so it's like this devastating thing to be like, oh, my God. It's like the whole house is collapsing and no one told me.

Joe Simmons (70:12)

Look, I definitely had moments after this where I was scared for myself and my family.

Steven Dubner (70:16)

And that is Joe Simmons.

Joe Simmons (70:18)

Like, just the amount of money involved, I didn't quite appreciate that at the very first moments. I mean, this is not the way to adjudicate these things. There's a million chances to show we're wrong. A million. Like earlier, a lot. There's a lot of chances. And this is what you're gonna do. You're gonna sue three individuals for $25 million. That seems. That seems not great.

Steven Dubner (70:48)

The DataCoRoada team found out it's expensive to defend yourself in a lawsuit like this. Some colleagues set up a GoFundMe campaign. Here is Simonson.

Yuri Simonson (70:59)

Within 24 hours, they had $200,000. We found a First Amendment expert lawyer who's representing us. We've learned a lot of the boring stuff that happens with lawyers that you don't get from TV shows, like the timelines and the language and how long the judge, like, things take forever. I mean, makes academia seem expedient in comparison. It's not just nice to have money, but it's also nice to know that thousands of people are willing to at least somewhat publicly support what you're doing. So that was a big boost.

Joe Simmons (71:28)

My very first thoughts were like, oh, my God, how's anyone gonna be able to do this again if you can get sued for conducting these kinds of investigations? And we are business school professors at, you know, very good institutions that have a lot of resources. We're definitely in a position to bear the brunt of this more than, say, the average person in the field. And so just the chilling effect on scientific inquiry and criticism. It's like, is everything we've been working for for 10 years, is that now gone?

Steven Dubner (72:06)

Since we originally published this episode, a Massachusetts judge dismissed Francesca Gino's lawsuit against Data Colada, although he did allow her lawsuit against Harvard to proceed. Harvard is also reviewing Gino's tenure and may revoke it. But that kind of accountability for tenured professors is incredibly rare in academia.

Brian Nosek (72:26)

Our field doesn't have a culture of open criticism. It's not considered okay.

Steven Dubner (72:30)

And that is Samin Vizier.

Brian Nosek (72:33)

If I name a specific theory or finding, that's considered a personal attack on the people associated with that theory or finding, even if I don't talk about the people behind it. And that's considered not okay.

Steven Dubner (72:43)

Vazir is a psychology professor at the University of Melbourne. She's also the new editor in chief of the top journal Psychological Science. And she has already been a central figure in the push to reform behavioral science. It was Vizier who set up the GoFundMe campaign for the Data Colada team after Francesca Gino sued them.

Brian Nosek (73:04)

I think one thing we can say for sure is that humans are quite good at self deception. There's a lot of reasons why researchers want to believe that they've found the answers to these problems. Right. One is a pro social reason. They want to help solve these problems, they want to help people. Another one is more self interested, that they want a seat at the policy table. They want attention for themselves, they want to promote their theory and their brand. And it's also kind of a survival thing to stay in academia to be able to continue to do the research. You need to have successes. And those successes often mean selling your work and sometimes overselling your work. We want to be taken seriously as scientists and be scientific. And that means being calibrated and careful and not exaggerating. But at the same time, the people who do exaggerate are probably going to get more of those successes that get them attention, get them a seat at the table, get them the next grant, the next job and so on.

Steven Dubner (73:51)

When you describe the incentives like that, it sounds to me as if those incentives conspire against the scientific method, do they not?

Brian Nosek (74:01)

Yeah. So if you were just a rational agent acting in the most self interested way possible as a researcher in academia, I think you would cheat. I think that is absolutely the way the incentives are set up. I don't think most people do, but not because of the incentives.

Steven Dubner (74:15)

So the more I hear you talk about the, let's call them perverse or at least mixed incentives that conspire against the pure pursuit of knowledge. Even within academia. I just think like if, if people like you, if fellow scientists can't successfully evaluate these claims, what's the public supposed to do? What are we supposed to think when we hear a claim that is often amplified through media? Doesn't it just make everyone skeptical or even cynical about everything?

Brian Nosek (74:49)

I think that maybe that's appropriate for fields like psychology where we're tackling really messy, complicated things that are multiply determined, that have many causes. It's hard to measure even one of causes. And so I think if you hear a Claim that sounds too good to be true or sounds too simplistic. I think it's appropriate to use common sense to be skeptical of it. Generally, I think we're taught that science can overturn common sense and you shouldn't just not believe science just because your common sense goes against it. But I think that should be different for different sciences and psychology. Psychology is just a lot harder to do and it's a relatively young science. We haven't perfected even the measurement of many of these concepts, much less all the other steps to studying them.

Steven Dubner (75:29)

Do you think of your field as being in crisis?

Brian Nosek (75:33)

I do.

Steven Dubner (75:34)

What kind of crisis? I mean, we hear the phrase replication crisis, which is a specific response to a fear that research is garbage. But I assume it's broader than that.

Brian Nosek (75:44)

Yeah, I mean, I think to our credit, it's broader than that. I think that our field is really in a period of intense self examination. It's going to tell the world, I think, how committed we are to scientific values, how we deal with this crisis. There's so many branches to this crisis. And I think it's just a crisis of, I don't know, integrity or credibility or whatever the most fundamental word you could use to describe what it means to be scientific. There was a lot of debate in the early days of the crisis of whether we should air our dirty laundry in public. The people who were arguing against it, I lost a lot of respect for them. But the other side won. We did air it in public and I think that we deserve some credit for that. I don't think we should rest on our laurels there. Identifying the problem is not the same thing as changing our practices. We should do some soul searching about how we got there and how we can prevent that from happening again.

Steven Dubner (76:36)

Max Bazerman has got a head start on the soul searching.

Max Bazerman (76:41)

You know, I've been connected to so many research projects where as the most senior member of the team, I never looked at the database. So, you know, does that make me guilty of something? I think yes. I think that it makes me complicit that I didn't do more verification that I thoroughly trusted. And I think that I should be somewhat accountable for the fact that I didn't do a better job of verifying, not just in the signing first paper, but in my research more broadly. And when, you know, in the signing first paper I was bothered by some aspects of the data and I asked a lot of questions back in 2011, but I eventually got an answer that I wanted to be true. And I accepted it. And I never looked at that database at that time either. Do I wish I did? Absolutely. I'm not sure that this story would be unfolding today had I looked at that data in 2011. There's so many positive influences that social science can have, from getting us to eat healthier foods to exercising more, to saving for our retirement. And social scientists have been remarkably good at helping figure out how do we get people moving in the right direction. And the vast majority of that work is honest, credible research that we should pay attention to. And if we now end up having sort of the big story being fraud in social science, then all the credible stuff is going to have less value and less impact than it should.

Steven Dubner (78:27)

Coming up next time on the show, we are going to keep this conversation going, but from some different angles, including the money. You may have heard of diploma mills or puppy mills, but how about research paper mills? It could be anywhere from hundreds of dollars to even thousands of dollars per paper. And they're publishing tens of thousands and sometimes even more papers per year. So you can start to do that math. We will start it, maybe even finish it. That's next time on the show. Until then, take care of yourself. And if you can, someone else too. By the way, while we were finishing up this episode, someone very close to me turned 64, and in response to the questions posed by Paul McCartney in that song, the answer is yes. Freakonomics Radio is produced by Stitcher and Renbud Radio. You can find our entire archive on any podcast app. Also@freakonomics.com where we publish transcripts and show notes. This episode was produced by Alina Coleman. Our staff also includes Augusta Chapman, Dalvin Abuaji, Eleanor Osborne, Ellen Frankman, Elsa Hernandez, Gabriel Roth, Greg Rippon, Jasmine Klinger, Jason Gambrell, Jeremy Johnston, John Schnarz, Lyric Bowditch, Morgan Levy, Neal Carruth, Rebecca Lee Douglas, Sarah Lilly, Teo Jacobs, and Zach Lipinski. Our theme song is Mr. Fortune by the Hitchhikers and our composer is Luis Guerra. As always, thanks for listening. So if I were to rephrase your answer, you basically said podcasts are better than academic journals, correct?

Brian Nosek (80:05)

As a former podcast host, I have to agree. The Freakonomics Radio Network the Hidden side of Everything.

Francesca Gino (80:20)

Stitcher.

Steven Dubner (80:26)

Freeconomics Radio is sponsored by the State Department's Bureau of Consular Affairs. Maybe your idea of a trip abroad is an all inclusive cruise, or exploring museums and local delicacies, or climbing the planet's tallest peaks. There's a whole world of wonder out there and you need a valid passport to go. So check, apply for or renew yours at least six months before you travel. And visit travel.state.gov for State Department guidance on passports, country specific safety info, and more. If you could hear love, what would it sound like? Son, can we talk about your drinking? Yeah, Dad, I think we should helping those closest to you think about their excessive drinking. Maybe that's what love sounds like. More@rethinkthedrink.com an OHA initiative.

Francesca Gino (81:31)

If your day sounds like we need.

Steven Dubner (81:33)

To report asap.

Francesca Gino (81:36)

You deserve my delight. If you've persevered through, you deserve this rich golden lager with a crisp but refreshing taste. Or if you overcame.

Steven Dubner (81:51)

Two more reps. Two more.

Francesca Gino (81:53)

You deserve this ice cold reward. Model Remarkable fighter. Drink responsibly. Beer imported by Crown Lake Port, Chicago, Illinois.