← Freakonomics Radio
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
01:08:57
Probably not — the incentives are too strong. But a few reformers are trying. We check in on their progress, in an update to an episode originally published last year. (Part 2 of 2)
Loading summary
Transcript154 lines
- [00:01]
Stephen Dubner
Freakonomics Radio is sponsored by Progressive, where drivers who save by switching save nearly $750 on average. Plus auto customers qualify for an average of 7 discounts. Get a quote now@progressive.com to see if you could save Progressive Casualty Insurance Company and affiliates national average 12 month savings of $744 by new customers surveyed who saved with Progressive between June 2022 and May 2023. Potential savings will vary. Discounts not available in all states and situations. Freeconomics Radio is sponsored by the State Department's Bureau of Consular Affairs. Maybe your idea of a trip abroad is an all inclusive cruise or exploring museums and local delicacies or climbing the planet's tallest peaks. There's a whole world of wonder out there and you need a valid passport to go. So check, apply for or renew yours at least six months before you travel. And visit travel travel.state.gov for State Department guidance on passports, country specific safety info and more hey there, it's Stephen Dubner. This is the second and final part of a series we are revisiting from last year. Stick around for an update at the end of the episode. Last week's episode was called why Is There so Much Fraud in Academia? We heard about the alleged fraudsters, we heard about the whistleblowers and then a lawsuit against the whistleblowers.
- [01:42]
Unknown Speaker 1
My very first thoughts were like oh my God, how's anyone going to be able to do this again?
- [01:48]
Stephen Dubner
We heard about feelings of betrayal from a co author who was also a longtime friend of the accused.
- [01:55]
Max Bazerman
We once even got to the point of our two families making an offer to a developer on a project to have houses connected to each other.
- [02:06]
Stephen Dubner
We also heard an admission from inside the house that the house is on fire.
- [02:11]
Ivan Oransky
If you were just a rational agent acting in the most self interested way possible as a researcher in academia, I think you would cheat.
- [02:19]
Stephen Dubner
That episode was a little gossipy for us at least. Today we are back to wonky. But don't worry, it is still really interesting. Today we look into the academic research industry and believe me, it is an industry. And there is misconduct everywhere from the universities.
- [02:41]
Unknown Speaker 2
The most likely career path for anyone who has committed misconduct is a long and fruitful career because most people, if they're caught at all, they skate.
- [02:51]
Stephen Dubner
There's misconduct at academic journals, some of which are essentially fake.
- [02:57]
Unknown Speaker 2
May be something that sounds a lot less nefarious than what I just described, but that is actually what's happening and.
- [03:04]
Stephen Dubner
We'Ll hear how the rest of us contribute because after all, we love These research findings, you know, you wear red.
- [03:11]
Unknown Speaker 2
You must be angry. Or if it says that this is definitely a cure for cancer.
- [03:16]
Stephen Dubner
We'll also hear from the reformers who are trying to push back.
- [03:19]
Ivan Oransky
It was a tense few months, but in the end, I was allowed to continue doing what I was doing.
- [03:24]
Stephen Dubner
Can academic fraud be stopped? Let's find out.
- [03:37]
Ivan Oransky
This is Freakonomics Radio, the podcast that explores the hidden side of everything with your host, Stephen Dubner.
- [03:54]
Stephen Dubner
Last week, we heard about two alleged cases of data fraud from two separate researchers. In one paper, the paper claimed that if you ask people to sign a form at the top before they fill out the information, you will get more truthful answers than if they sign at the bottom. After many unsuccessful attempts to replicate this finding and allegations that the data supporting it had been faked, the original paper was finally retracted. The two alleged fraudsters are Dan Ariely of Duke and Francesca Gino, who has been suspended by Harvard Business School. Gino subsequently sued Harvard, as well as the three other academic researchers who blew the whistle. The lawsuit against the researchers was dismissed. The whistleblowers maintain a blog called Data Colada. They have written that they believe there is fake data in many of the papers that Francesca Gino co authored, perhaps dozens. Gino and Arielli, meanwhile, both maintain their innocence. They also both declined our request for an interview. On that one paper that caused all the trouble about signing at the top, there were three other co authors, Lisa Hsu, Nina Mazar, and Max Bazerman. None of them have been accused of any wrongdoing. So let's pick up where we left off with Max Bazerman, the most senior researcher on that paper. He also teaches at Harvard Business School.
- [05:24]
Max Bazerman
When there's somebody who engages in bad behavior, there's always people around who could have noticed more and acted more.
- [05:34]
Stephen Dubner
Bazerman was close with Francesca Gino. He had been her advisor, and he trusted her. So he has been spending a lot of time lately thinking. Thinking about the mess. He recently published a book called How We Enable the Unethical and how to Stop. And he's working on another book about social science fraud. This has led him to consider what makes people cheat. Let's take the case of Dieterik Stappel, a Dutch professor of social psychology who, after years of success, admitted to fabricating and manipulating data in dozens of studies.
- [06:12]
Max Bazerman
Part of the path toward data fabrication occurred in part because he liked complex ideas, and academia didn't like complex ideas as much as they liked the snappy sort of clickbait. And that moved him in that direction and also put him on a path toward fraudulent behavior.
- [06:33]
Stephen Dubner
So here's something that Stoppel wrote later when he wrote a book of confession essentially about his fraud. He wrote, I was doing fine, but then I became impatient, patient, over ambitious, reckless. I wanted to go faster and better and higher and smarter all the time. I thought it would help if I just took this one tiny little shortcut. But then I found myself more and more often in completely the wrong lane. In the end, I wasn't even on the road at all. What struck me about that, and I think about that with Dan Ariely and Francesca Gino as well, which is that the people who have been accused of having committed academic fraud are really successful already. And I'm curious what that tells you about either the stakes or the incentives or maybe the psychology of how this happens, because honestly, it surprises me.
- [07:22]
Max Bazerman
I would say we don't know that much about why the fraudsters do what they do. And the most interesting source you just mentioned. So Stapel wrote a book in Dutch called Outsporing, which means something like Derailed, where he provides his information and he goes on from the material you talked about to describing that he became like an alcoholic or a heroin addict and he got used to the easy successes and he began to believe that he wasn't doing any harm after all, he was just making it easier to publish information that was undoubtedly true. So this aspect of sort of being lured onto the path of unethical behavior followed by addictive like behavior becomes part of the story. And Stapel goes on to talk about lots of other aspects, like the need to score, ambition, laziness, wanting power, status. So he provides us good insight. But most of the admitted fraudsters or the people who have lost their university positions based on allegations of fraud have simply disappeared and have never talked about it. One of the interesting parts is that Mark Hauser, who resigned from Harvard, and Ariely and Gino, who are alleged to have committed fraud by some parties, all three of them wrote on the topic of moral behavior and specifically why people might engage in bad behavior.
- [08:50]
Stephen Dubner
That's right. A lot of the fraud and suspected fraud comes from researchers who explore fraud. In 2012, Francesca Gino and Dan Ariely collaborated on another paper called the Dark side of Original Thinkers can be More Decisive, dishonest, they wrote. We propose that a creative personality and a creative mindset promote individuals ability to justify their behavior, which in turn leads to unethical behavior. So just how much unethical behavior is there in the world of academic research? That's a hard question to answer precisely. But let's start with this man.
- [09:31]
Unknown Speaker 2
I essentially spend all of my nights and weekends thinking about scientific fraud, scientific misconduct, scientific integrity, for that matter.
- [09:39]
Stephen Dubner
Ivan Uransky is a medical doctor and editor of a neuroscience publication called the Transmitter. He's also a distinguished journalist in residence at nyu. And on the side, he runs a website called Retraction Watch.
- [09:54]
Unknown Speaker 2
We hear from whistleblowers all the time, people we call sleuths who are actually out there finding these problems. And often that's pre retraction, or they'll explain to us why retraction happened. We also do things like file public records requests.
- [10:08]
Stephen Dubner
He began retraction watch in 2010 with Adam Marcus, another science journalist. Marcus had broken a story about a Massachusetts anesthesiologist named Scott Rubin. Rubin had received funding from several drug companies to conduct clinical trials, but instead, he faked the data and published results without running the trials.
- [10:29]
Unknown Speaker 2
I went to Adam and I said, what if we created a blog about this? It seems like there are all these stories that are hiding in plain sight that essentially we and other journalists are leaving on the table. And when we looked at the actual retraction notices, the information was somewhere between misleading and opaque.
- [10:49]
Stephen Dubner
What do you mean by that?
- [10:50]
Unknown Speaker 2
So when a paper is retracted, and it's probably worth defining that a retraction is a signal to the scientific community, or really to any readers of a particular peer review journal article, that you should not rely on that anymore, that there's something about it that means you should. You know, you can not pretend it doesn't exist, but you shouldn't base any other work on it.
- [11:14]
Stephen Dubner
But when you call it misleading or opaque, you're saying the explanation for the retraction is often not transparent.
- [11:22]
Unknown Speaker 2
Right. So when you retract a paper, you're supposed to put a retraction notice on it. The same way when you correct an article in the newspaper, you're supposed to put a correction notice on it. But when you actually read these retraction notices, and to be fair, this has changed a fair amount in the 13 years that we've been doing this, sometimes they include no information at all. Sometimes they include information that is woefully incomplete. Sometimes it's some version of getting Al Capone on tax evasion, they fake the data, but we're going to say they forgot to fill out this form, which is still a reason to retract, but isn't the whole story.
- [12:00]
Stephen Dubner
So let's say we back up and I ask you how Significant or widespread is the problem of, we'll call it academic fraud.
- [12:08]
Unknown Speaker 2
So we think that probably 2% of papers should be retracted for something that would be considered either out and out fraud or maybe just severe bad mistake. According to our data, which we have the most retraction data of any database, about 0.1% of the world's literature is retracted. So one in a thousand papers, we think it should be about 20 times that, about 2%. There's a bunch of reasons, but they come down to 1. There was a survey back in 2009 which has been repeated and done differently and come up with roughly the same number, actually even higher numbers recently. That says 2% of researchers, if you ask them anonymously, they will say, yes, I've committed something that would be considered misconduct. Of course, when you ask them how many people they know who have committed misconduct, it goes much, much higher than that. And so that's one line of evidence, which is admittedly indirect. The other is that when you talk to the sleuths, the people doing the real work of figuring out what's wrong with literature and letting people know about it, they keep lists of papers they've flagged for publishers and for authors and journals, and routinely, most of them are not retracted. Again, we came to 2%. Is it exactly 2%? And is that even the right number? No, we're pretty sure that's the lower bound. Others say it should be even higher.
- [13:33]
Stephen Dubner
Retraction Watch has a searchable database that includes more than 45,000 retractions from journals in just about any academic field you can imagine. They also post a leaderboard, a ranking of the researchers with the most papers retracted. At the top of that list is another anesthesiologist, this one a German researcher named Joachim Bolt. He came up briefly in last week's episode, too. Bolt has had more than 200 papers retracted.
- [14:03]
Unknown Speaker 2
Bolt, an anesthesiology researcher, was studying something called Heterostarch, which was essentially a blood substitute. Not exactly blood, but something that when you were on, like a heart, lung pump, a machine during certain surgeries or you're in the ICU or something like that, it would basically cut down on the amount of blood transfusions people would need, and that's got obvious benefits. Now, he did a lot of the important work in that area, and his work was cited in all the guidelines. It turned out that he was faking data.
- [14:33]
Stephen Dubner
Bolt was caught in 2010 after an investigation by a German state medical association. The method he had been promoting was later found to be associated with a significant increased risk of mortality. So in this case, the fraud led to real danger. And what happened to Bolt?
- [14:53]
Unknown Speaker 2
He at one point was under criminal indictment, or at least criminal investigation. That didn't go anywhere. The hospital, the clinic also, which, to be fair, had actually identified a lot of problems. They came under pretty severe scrutiny, but in terms of actual sanctions, pretty minimal.
- [15:10]
Stephen Dubner
You've written that universities protect fraudsters. Can you give an example other than Bolt?
- [15:16]
Unknown Speaker 2
Let's say so universities, they protect fraudsters in a couple different ways. One is that they are very slow to act, they're very slow to investigate, and they keep all of those investigations hidden. The other is that because lawyers run universities like they frankly run everything else. They tell people who are involved in investigations. If someone calls for a reference letter, let's say someone leaves and they haven't been quite found guilty, but as a plea bargain sort of thing, they will leave and then they'll stop the investigation. Then when someone calls for a reference, and we actually have the receipts on this, because we filed public records requests for emails between different parties, we learned that they would be routinely told not to say anything about the misconduct.
- [16:04]
Stephen Dubner
Let's just take three of the most recent high profile cases of academic fraud or accusations of academic fraud. We've got Francesca Gino, who was a psychology researcher at Harvard Business School. Dan Ariella, who's a psychologist at Duke, and then Marc Tessier Levine, who was president of Stanford, a medical researcher. Three pretty different outcomes, right? Teski Levine was defenestrated from his presidency. Gino was suspended by hbs, and Dan Ariely, who's had accusations lobbed at him for years now, is just kind of going on. Can you just comment on that heterogeneity?
- [16:37]
Unknown Speaker 2
So, Dan, I would just not so much as a correction, but just to say that, yes, Marc deselevine was defenestrated as president. He remains, at least at the time of this discussion, tenured professor at Stanford, which is a pretty nice position to be in.
- [16:53]
Stephen Dubner
A Stanford report found that Tessier Levine didn't commit fraud or falsify any of his data, although work in his labs, quote, fell below customary standards of scientific rigor, and multiple members of his labs appear to have manipulated research data. Dan Ariely also remains a professor at Duke, and Duke has declined to comment publicly on whether an investigation into the allegations of fraud even occurred. As Ivan Aransky told us, universities are run by lawyers.
- [17:26]
Unknown Speaker 2
I've been quoted saying that the most likely career path or the most likely outcome for anyone who has committed misconduct is a long and fruitful career. And I mean that because it's true, because most people, if they're caught at all, they skate. The number of cases we write about, which grows every year, but is still a tiny fraction of what's really going on. Dan Ariely. We interviewed Dan years ago about some questions in his research. Duke is actually, I would argue, a little bit of a singular case. Duke in 2019 settled with the US government for $112.5 million because they had repeatedly alleged to have covered up really bad, significant misconduct.
- [18:16]
Stephen Dubner
Duke has had particular trouble with medical research. There was one physician researcher who faked the data in his cancer research. There were allegations of federal grant money being mishandled, also failing to protect patients in some studies. At one point, the National Institutes of Health placed special sanctions on all Duke researchers.
- [18:38]
Unknown Speaker 2
So I gotta be honest, and, you know, people in Durham may not like me saying this, but I think Duke has a lot of work to do to demonstrate that their investigations are complete and that they are doing the right thing by research dollars and for patients.
- [18:54]
Stephen Dubner
Most of the researchers we have been talking about are already well established in their fields. If they feel pressure to get a certain result, it's probably the kind of pressure that comes with further burnishing your high status. But junior researchers face a different kind of pressure. They need published papers to survive. Publish or perish is the old saying. If they can't consistently get papers in a good journal, they probably won't get tenure and they may not even have a career. And those journals are flooded with submissions. So the competition is fierce, and it is global. This is easy to miss if you're in the US since so many of the top universities and journals are here. But academic research is very much a global industry, and it's also huge. Even if you are a complete nerd and you can name 50 journals in your field, you know nothing. Every year, more than 4 million articles are published in somewhere between 25 and 50,000 journals, depending on how you count. And that number is always growing. You have got journals called Aggressive Behavior and Frontiers in Ceramics, Neglected Tropical Diseases and Rangifer, which covers the biology and management of reindeer and caribou. There used to be a Journal of Mundane Behavior, but that one, sadly, is defunct. But no matter how many journals there are, there is never enough space for all the papers. And this has led to some, well, let's call it scholarly innovation. Here again is Ivan Aransky.
- [20:41]
Unknown Speaker 2
People are now really fixated on what are known as paper mills. So if you think about the economics of this Right. It is worthwhile if you are a researcher, to actually pay. In other words, it's an investment in your own future to pay to publish a certain paper. What I'm talking about is literally buying a paper or buying authorship on a paper. So to give you a little bit of a sense of how this might work, you're a researcher who is about to publish a paper. So Dodner et al have got some interesting finding that you've actually written up and a journal has accepted, it's gone through peer review and you're like, great, and that's good for you, but you actually also want to make a little extra money on the side. So you take that paper that essentially it's not a paper yet, it's really still a manuscript. You put it up on a brokerage site, or you put the title up on a brokerage site. You say, I've got a paper where there are four authors right now. It's going into this journal, which is a top tier or mid tier, whatever it is, et cetera. It's in this field. If you would like to be an author, the bidding starts now, or it's $500 or €500 or whatever it is. And then Ivan Maransky comes along and says, I need a paper, I need to get tenure, I need to get promoted. Oh great, let me click on this. Brokerage give you $500 and now all of a sudden you write to the journal. By the way, I have a new author. His name is Ivan Aransky. He's at New York University. He just joined late, but he's been so invaluable to the process.
- [22:17]
Stephen Dubner
Now what do the other co authors have to say about this?
- [22:20]
Unknown Speaker 2
Often they don't know about it, or at least they claim they don't know about it.
- [22:24]
Stephen Dubner
What about fraudulent journals, do those exist? Or fraudulent publications sites that look legit enough to pass muster for somebody's department?
- [22:32]
Unknown Speaker 2
They do. I mean, there are a couple different versions of fraudulent with a lowercase f. Publications. There are publications that are legit in the sense that they can point to doing all the things that you're supposed to do as a journal. You have papers submitted, you do something that looks sort of like peer review. You assign it what's known as a digital object identifier. You do all that publishing stuff and they're not out and out fraudulent in the sense of they don't exist and people are just making it up or they're trying to use a name that's, that isn't really theirs, but they're fraudulent with lowercase F in the sense that they're not doing most of those things. Then there are actual, what we refer to, we and Anna Abelkina, who works with us on this as hijacked journals. We have more than 200 on this list. Now, they were at one point legitimate journals. So it's a real title that some university or funding agency or et cetera, will actually recognize. But what happened was some version of the publisher sort of forgot to renew their domain. I mean, literally something like that. Now there are more nefarious versions of it, but it's that sort of thing where these really bad players are inserting themselves and taking advantage of the vulnerabilities in the system of which there are many to really print money, because then they can get people to pay them to publish in those journals and they even are getting indexed in the places that matter.
- [23:56]
Stephen Dubner
It sounds like there are thousands of people who are willing to pay journals that are quasi real to publish their papers.
- [24:07]
Unknown Speaker 2
At least thousands, yes.
- [24:09]
Stephen Dubner
Who are the kind of authors who would publish in those journals? Are they American? Not American. Are there particular fields or disciplines that happen to be most common?
- [24:18]
Unknown Speaker 2
Well, I think what it tends to correlate with is how direct or intense the publish or perish culture is in that particular area. And generally that varies. Varies more by country or region than anything else. If you look at, for example, the growth in China of number of papers published, what's calculated is the impact of those papers, which relies on things like how often they're cited. You can trace that growth very directly from government mandates. For example, if you publish in certain journals, known as high impact factor, you actually got a cash bonus that was a sort of multiple of the numbers, the impact factor. And that can make a big difference in your life.
- [25:03]
Stephen Dubner
Recent research by John Ioannidis, Thomas Collins and Jeroen Bass has examined what they call extremely productive authors. They left out physicists, since some physics projects are so massive that one paper can have more than 5,000 authors. So leaving aside the physicists, they found that over the course of one year, more than 1200 researchers around the world published the equivalent of one paper every five days. The top countries for these extremely productive authors were China, the US Saudi Arabia, Italy and Germany. When there is so much research being published, you'd also expect that the vast majority of it is never read by more than a handful of people. There's also the problem, as the economics blogger Noah Smith recently pointed out, that too much academic research is just useless, at least to anyone beyond the author. But you shouldn't expect any of this to change. Global scholarly publishing is a $28 billion market. Everyone complains about the very high price of journal subscriptions, but universities and libraries are essentially forced to pay. There is, however, another business model in the research paper industry, and it is growing fast. It's called open access publishing, and here it's most often the authors who pay to get into the journal. If you think that sounds problematic, well, yes. Consider Hindawi, an Egyptian publisher with more than 200 journals, including the Journal of Combustion, the International Journal of Ecology, the Journal of Advanced Transportation. Most of their journals were not at all prestigious, but that doesn't mean they weren't lucrative. A couple years ago, Hindawi was bought by John Wiley and Sons, the huge American academic publisher, for nearly $300 million.
- [27:00]
Unknown Speaker 2
So Hindawi's business model was they're an open access publisher, which usually means you charge authors to publish in your journal. And you charge them, you know, it could be anywhere from hundreds of dollars to even thousands of dollars per paper. And they're publishing, you know, tens of thousands and sometimes even more papers per year. So you can start to do that math. What happened at Hindawi was that somehow paper mills realized that they were vulnerable, so they started targeting them. They've actually started paying some of these editors to accept papers from their paper mill. And long story short, they now have had to retract something like, we're still figuring out the exact numbers when it does settles, but in the thousands.
- [27:45]
Stephen Dubner
In 2023, Hindawi retracted more than 8,000 papers. That was more retractions than there had ever been in a year from all academic publishers combined. Wiley recently announced that they will stop using the Hindawi brand name and they folded the remaining Hindawi journals into their portfolio. But they are not getting out of the pay to publish business.
- [28:10]
Unknown Speaker 2
Publishers earn more from publishing more. It's a volume play. And when you're owned by shareholders who want growth all the time, that is the best way to grow. And these are businesses with very impressive and enviable profit margins of sometimes up to 40%. And these are not on small numbers. The profit itself is in the billions. Also.
- [28:34]
Stephen Dubner
It'S hard to blame publishers for wanting to earn billions from an industry with such bizarre incentives. But if publishers aren't looking out for the integrity of the research, who is? That's coming up after the break. I'm Stephen Dubner and this is Freakonomics Radio. Freakonomics Radio is sponsored by Toyota. Did you know Toyota has an all electric SUV, the BZ4XX. It's a secret worth sharing. Skip gas stations and feel the thrill of electric driving from a brand you can trust. The BZ4X features smart tech that keeps you connected on every drive and its modern design helps you stand out in the right ways. It's all electric and it's all real. The Toyota BZ4X learn more at toyota.com toyota let's go places up your tech.
- [29:29]
Brian Nosek
Game in 2025 with Pluralsight. Pluralsight delivers expert LED technology learning, hands on practice and personalized paths to help you achieve your career goals and land that six figure salary. With over 7,000 on demand courses, you can learn from highly vetted tech experts specializing in cloud, AI, data security and more. Pluralsight works with your busy schedule with bite sized courses that you can fit in anytime, anywhere. Plans are discounted through January 6th. Get started@pluralsight.com what does it mean to be rich?
- [30:02]
Samin Vaziri
Maybe it's less about reaching a magic number and more about discovering the magic in life. At Edward Jones, our dedicated financial advisors are the people you can count on for financial strategies that help support a life you love, because the key to being rich is knowing what counts. Learn more about our comprehensive approach to planning at@edwardjones.com findyourrich Edward Jones Member, SIPC we feel like when we're in cultures that there is no way for any of us to change the culture. It's a culture, my God, how could we change it? But we also recognize that cultures are created by the people that comprise them. And the notion that we collectively can actually do something to shift the research culture I think has spread and that spreading has actually accelerated the change of the research culture for the better.
- [31:06]
Stephen Dubner
That is Brian Nosek. He's a psychology professor at the University of Virginia and he runs the center for Open Science. For more than a decade, his center has been trying to get more transparency in academic research. You might think there would already be transparency in academic research, at least I did. But here's what Nosek said in part one of the series when we were talking about how researchers tend to hoard their data rather than share it.
- [31:35]
Samin Vaziri
Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this my ideas, my data, my designs, my solutions, then I will disadvantage my career.
- [32:03]
Stephen Dubner
I asked Nosek how he thinks this culture can be changed.
- [32:07]
Samin Vaziri
So, for example, we have to make it easy for researchers to be more transparent with their work. If it's really hard to share your data, then adding on that extra work is going to slow down my progress. We have to make it normative. People have to be able to see that others in their community are doing this. They're being more transparent, they're being more rigorous, so that we, instead of saying, oh, that's great ideals, and nobody does it, you say, oh, there's somebody over there that's doing it. Oh, maybe I could do it, too. We have to deal with the incentives. Is it actually relevant for my advancement in my career to be transparent, to be rigorous, to be reproducible? And then we have to address the policy framework. If it's not embedded in how it is that funders decide who to fund, institutions decide who to hire, and journals to decide what to publish, then it's not going to be internally and completely embedded in the system.
- [33:02]
Stephen Dubner
Okay, so that is a lot of change. Here's one problem Nosek and his team are trying to address. Some researchers will cherry pick or otherwise manipulate their data or find ways to goose their results to make come up with a finding that will capture the attention of journal editors. So Nosek's team created a software platform called the Open Science Framework, where researchers can pre register their project and their hypothesis before they start collecting data.
- [33:35]
Samin Vaziri
Yeah, so the idea is you register your designs and you've made that commitment in advance. And then as you're carrying out the research, if things change along the way, which happens all the time, you can update that registration. You could say, here's what's changing. We didn't anticipate that going into this community was gonna be so hard, and here's how we had to adapt. That's fine. You should be able to change. You just have to be transparent about those changes so that the reader can evaluate.
- [34:02]
Stephen Dubner
And then those data are timestamped, I gather.
- [34:06]
Samin Vaziri
Exactly. Yeah. You put your data in your materials. If you did a survey, you add the surveys. If you did behavioral tasks, you can add those. So all of that stuff can be attached then to the registration so that you have a more comprehensive record of what it is you did.
- [34:21]
Stephen Dubner
It sounds like you're basically raising the cost of sloppiness or fraud. Yes.
- [34:26]
Samin Vaziri
It makes fraud more inconvenient, and that's actually a reasonable intervention. I don't think any intervention that we could design could prevent fraud in a way that doesn't stifle actual legitimate research. We just want to make visible all the things that legitimate researchers are doing so that someone that doesn't want to do that extra work has a harder time. And eventually if everything is exposed, then the person who would be motivated to do fraud might say, well, it's just as easy to do the research the real way. So I guess I'll do that.
- [35:02]
Stephen Dubner
The idea of pre registration isn't new. It goes back to at least the late 19th century. But there is a big appetite for it now. The datacolota team came up with their own version. Here is Yuri Simonson from the ESADE Business School in Barcelona. It's a platform that we launched. It's called as Predicted. And it's basically eight questions that people write. Your co author, sign on it, it's timestamped, people, you can show the PDF. And when we launched it, we thought, okay, when do we call this a failure? You know, thinking ahead, when do you shut down the website? All right, if we don't get 100 a year, we're going to call it failure. And we're getting about 140 a day. Now, Brian Nosek says the registered report model can be especially helpful to the journals.
- [35:48]
Samin Vaziri
So in the standard publishing model, I do all of my research, I get my findings, I write it up in a paper and I send it to the journal. In that model, the reward system is about the findings. I need to get those findings to be as positive, novel and tidy as I can so that you, the reviewer, say, okay, okay, you can publish it. That's dysfunctional. And it leads to all of those practices that might lead the claims to be more exaggerated than the evidence. The registered report model says to the journal, you are going to submit, Brian, the methodology that you're thinking about doing and why you're asking that question and the background research supporting that question being important and that methodology being effective methodology. We'll review that. We don't know what the results are. You don't know what the results are, but we're gonna review based on. Do you have an important question?
- [36:38]
Stephen Dubner
So this is almost like before you build a house, you're gonna show us your plan, and we're the building department, we're gonna come and say, yeah, that looks legit. It's not gonna collapse, it's not gonna infringe on your neighbor, and so on. Is that the idea?
- [36:50]
Samin Vaziri
Exactly. And the key part is that the reward me getting that publication is based on you agreeing that I'm asking an important question. And I've designed an effective method to test it. It's no longer about the results. None of us know what the results are.
- [37:05]
Stephen Dubner
And so even if the results are uninteresting, not new, et cetera, we'll know they're legitimate. But there would seem to be a conflict of incentive there, which is that, oh, now do I need to publish this uninteresting, not new result? What do you do about that?
- [37:20]
Samin Vaziri
Yeah, so the commitment that the journal makes is we're going to publish it regardless of outcome, and the authors are making that commitment too. We're going to carry this out as we said we would, and we'll report what happens. Now, an interesting thing happens in the change of the culture here in evaluating research because you said, well, if it's an uninteresting finding, do we still have to publish it? It turns out that when you have to make a decision of whether to publish or not, before knowing that the results are, the orientation that the reviewers bring, that the authors bring, is, do we need to know the answer to this regardless of what happens? Do we need to know the answer?
- [37:58]
Stephen Dubner
Is the question important, in other words?
- [38:01]
Samin Vaziri
Exactly. Is the question important enough that we need evidence regardless of what the evidence is? And it dramatically shifts what ends up being published? So in the early evidence with register reports, more than half of the hypotheses that are proposed end up up not being supported in the final paper. In the standard literature, comparable type of domains, more than 95% of the hypotheses are supported in the paper. You wonder in the standard literature, if we're always right, why do we bother doing the research right? Our hypotheses are always right. And of course it's laughable because we know that's not what's actually happening. We know that all that failed stuff is getting left out and we're not seeing it. And the actual literature is an exaggeration of what the real literature is.
- [38:54]
Stephen Dubner
I think we should say a couple of things here about academia. The best academics are driven by a real scientific impulse. They may know a lot, but they're not afraid to admit how much we still don't know. So they're driven by an urge to investigate and not, not necessarily, at least an urge to produce a result that will increase their own status. But academia is also an extraordinarily status conscious place. I'm not saying there's anything wrong with that. If status is the reward that encourages a certain type of smart, disciplined person to do research for the sake of research, rather than taking their talents to an industry that might pay them much more, that is fantastic. But if the pursuit of status for status's sake leads an academic researcher to cheat, well, yeah, that's bad. I mean, the incentives are part of the problem, but I don't think it's a part of the problem that we have to fix. That again is Yuri Simonson from datacolota. I think the incentives, it's like, boy, do people rob banks because their incentives are there. But it doesn't mean that we should stop rewarding cash. It just we should, you know, make our saves safer because it's good for cash to buy things and it's good for people who publish interesting findings to get recognition. Brian Nosek says that more than 300 journals are now using the registered reports model.
- [40:22]
Samin Vaziri
I think there is broad buy in on the need to change and it has already hit the mainstream of many of the changes that we promote. Sharing data, materials, code, pre registering, research, reporting all outcomes. So we're in the scaling phase for those activities. And what I am optimistic about is that there is this metascience community that is interrogating whether these solutions are actually having the desired impact. And so this is the most exciting part of the movement as I'm looking to the future, is this dialogue between activism and reform. We can do these things, let's make these changes. And metascience and evaluation, is this working? Did you do what you said it was going to do? And et cetera. And I hope that the tightness of that loop will stay tight because that I think will make for a very healthy discipline that is constantly skeptical of itself and constantly looking to do better.
- [41:25]
Stephen Dubner
Is Brian Nosek too optimistic? Maybe 300 journals is a great start, but that represents maybe 1% of all journals. For journals and authors, the existing publishing incentives are very strong.
- [41:43]
Ivan Oransky
So I think journals have really complicated incentives.
- [41:47]
Stephen Dubner
That is Samin Vazier, a psychology professor at the University of Melbourne.
- [41:51]
Ivan Oransky
Of course they want to publish good work to begin with, so there's some incentive to do some quality check and kind of COVID their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.
- [42:04]
Stephen Dubner
And here's a reason to think that Brian Nosek is right to be optimistic about research reform. Some of his fellow reformers, including Samin Vazier, have been promoted into prestigious positions in their field. Vazier spent some time as editor in chief of the journal Social, Psychological and Personality Science.
- [42:26]
Ivan Oransky
So one of the things the editor in chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection. One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate, am I rejecting too many things, et cetera. There was some conversation about whether I was des rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin Des rejected my paper. I found this extremely distasteful and I won't be submitting there again. And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.
- [43:20]
Stephen Dubner
So it was basically like, do you know who I am?
- [43:22]
Ivan Oransky
Yeah, yeah.
- [43:24]
Stephen Dubner
So what happened to you and that journal then?
- [43:26]
Ivan Oransky
It was a tense few months, but in the end I was allowed to continue doing what I was doing.
- [43:33]
Stephen Dubner
Vazier recently took on the editor in chief job at a different journal, Psychological Science. It is one of the premier journals in the field. It's also the journal where Francesca Gino published two of her allegedly fraudulent papers. So I asked Vizier what changes she is hoping to make.
- [43:53]
Ivan Oransky
We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor Editors, the Star Editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions. Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the Star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration. This is something that I think is particularly challenging with short articles like Psychological Science publishes, and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. Authors are competing for those few spots. And so it feels like they have to make a really bold claim. And so it's going to be very difficult to play this like back and forth where Authors are responding to their perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected. But I'm not sure if authors will believe that just because we say that. They're still competing for a very selective number of spots.
- [45:02]
Stephen Dubner
So as a journal editor, how do you think about the upside risk of publishing something new and exciting against the downside risk of being wrong?
- [45:11]
Ivan Oransky
Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there. Sometimes we'll present things with certainty that we shouldn't have. What I would like to be involved in and what I plan to do is to encourage more post publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon and create more incentives for people to do that and do that.
- [45:46]
Stephen Dubner
Well, how would you reward whistleblowers?
- [45:49]
Ivan Oransky
I don't know. Do you have any ideas?
- [45:56]
Stephen Dubner
Right now, the rewards for whistleblowers in academia may seem backwards. Remember the Data Colada whistleblowers were sued by Francesca Gino, one of the people they blew the whistle on. They needed a GoFundMe campaign for their legal defense. So no, the whistleblowers aren't collecting any bounties, nor do they cover themselves in any kind of glory.
- [46:19]
Leif Nelson
Stephen I'm the person that walks into these academic conferences and everyone is like, ah, here comes Debbie Downer.
- [46:25]
Stephen Dubner
That's Leif Nelson, another member of Data Colada. He's a professor of business administration at UC Berkeley. In a recent New Yorker piece by Gideon Lewis Krauss about these fraud scandals, Nelson and his Data Colada partners were described as having a, quote, basic willingness to call bull. So now that you've become part of this group that are collectively, you know, I would think of as the primary whistleblower or police or steward, whatever word we want to use against fraudulent research in the social sciences, what does that feel like? I'm guessing on one level it feels like an accomplishment. On the other hand, it makes me think of a police force where there's the Internal Affairs Bureau where detectives are put to find the Bad apples. And even though everybody's in favor of rooting out the bad apples, everybody kind of hates the IAB guys. And I'm curious what the emotional toll or cost has been to you. Wow, bad question.
- [47:36]
Leif Nelson
No, like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field. Right. They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world. But I don't identify as much with being a police officer or even a detective, though every now and then, people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.
- [48:46]
Stephen Dubner
Even before the lawsuit.
- [48:49]
Leif Nelson
Yeah, the lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything, that's going to be like, well, look, we think something bad happened here, someone is going to be mad at us, and probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved. And it's hard because I know a lot of these people and I know their friends, and I know the friends of the friends. And that carries real stress for, I think, all three of us.
- [49:25]
Stephen Dubner
In the New Yorker piece, there are still people who call you pretty harsh names. You've been compared to the Stasi, for instance.
- [49:33]
Leif Nelson
Yeah, that's real bad. I am not happy with being compared to the Stasi. The optimistic take is that there's less of that than there used to be. When any of the three of us go and visit universities, for example, and we talk to doctoral students, and we talk to assistant professors, and we talk to associate professors, we talk to senior professors, the Students basically all behave as though they don't understand why anyone would ever be against what we're saying. They wouldn't understand the Stasi thing, but they also wouldn't even understand why they almost are at the level of I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to. You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.
- [50:25]
Stephen Dubner
If they were to say that to you directly, why fix what isn't broken? What would you say?
- [50:29]
Leif Nelson
I would say, but it is broken.
- [50:31]
Stephen Dubner
And your evidence for that would be?
- [50:33]
Leif Nelson
Ah, the evidence for that is multifold.
- [50:39]
Stephen Dubner
After the break, multifold we shall I'm Stephen Dubner, this is Freakonomics Radio. We'll be right back. Freakonomics Radio is sponsored by NetSuite. What does the future hold for business? Ask nine experts and you'll get ten answers. Bull market. Bear market rates will rise or fall. Can someone please invent a crystal ball? Until then, over 40,000 businesses have future proofed their business with NetSuite by Oracle. The number one cloud error, bringing accounting, financial management, inventory and HR into one fluid platform. With real time insights and forecasting, you are peering into the future with actionable data. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunity, download the CFO's Guide to AI and Machine Learning at netsuite.com the guide is free to you at netsuite.com freak netsuite.com freak.
- [51:48]
Brian Nosek
Sometimes the most painful part of getting sick is the getting better part. Waiting on hold for an appointment, sitting in crowded waiting rooms, standing in line at the pharmacy that's painful. Amazon One Medical and Amazon Pharmacy remove these painful parts of getting better with things like 247 virtual visits and prescriptions delivered to your door. Thanks to Amazon One Medical and Amazon Pharmacy. Healthcare just got less painful.
- [52:18]
Stephen Dubner
Freakonomics Radio is sponsored by Lenovo. Lenovo is sponsoring Life with Machines, a new video podcast hosted by comedian and tech whiz Baratunde Thurston. Together they will explore how AI can be used for good to power people forward. That's because Lenovo believes their smarter AI is your AI, personalized, easy to scale and delivering outcomes for people and businesses with full stack AI hardware, software and service solutions. Lenovo is transforming diverse industries, organizations and people. Check out life with machines wherever you get your podcasts Can Academic Fraud be eliminated stated Certainly not. The incentives are too strong also to be reductive. Cheaters are going to cheat, and I doubt there is one field of human endeavor, no matter how noble or righteous or honest it claims to be, where some cheating doesn't happen. But can academic fraud at least be greatly reduced? Perhaps. But that would lead likely require some big changes, including a new type of gatekeeper. Samin Vizier, the journal editor we heard from earlier, is one kind of gatekeeper.
- [53:40]
Ivan Oransky
Sometimes, for example, we'll get a submission where the research is really solid but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other. But it's still interesting data. And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.
- [54:13]
Stephen Dubner
And there's another important gatekeeper in academic journals, one that we've barely talked about, the referees who assess journal submissions. Peer review is a bedrock component of what makes academic publishing so credible, at least in theory. But as we've been hearing about every part of this industry, the incentives for peer reviewers are also off. Here again is Ivan Aransky from Retraction Watch.
- [54:41]
Unknown Speaker 2
If you add up the number of papers published every year and then you multiply that times the two or three peer reviewers who are typically supposed to review those papers, and sometimes they go through multiple rounds, it's easily in the tens of millions of peer reviews as a unit. And if each of those takes anywhere from four hours to eight hours of your life as an expert, which you don't really have because you gotta be teaching, you gotta be doing your own research. You come up with a number that cannot possibly be met by qualified people, really, it can't. I mean, the math just doesn't work and none of them are paid. You are sort of expected to do this because somebody will peer review your paper at some other point, which Sort of makes sense until you really pick it apart. Now, peer reviewers, so even the best of them, and by best I mean people who really sit and take the time and probe what's going on in the paper and look at all the data. But you can't always look at the data. In fact, most of the time you can't look at the raw data even if you had time because the authors don't make it available. So peer review, it's become really peer review light, and maybe not even that at the vast majority of journals. So it's no longer surprising that so much gets through the system that shouldn't.
- [55:53]
Leif Nelson
This is a very hot topic, and.
- [55:55]
Stephen Dubner
That again is Leif Nelson from UC Berkeley and datacolota.
- [56:00]
Leif Nelson
Editors, largely in my field, are uncompensated for their job. And reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor and every time I do it, it's basically taxing. The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about, whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis.
- [56:45]
Stephen Dubner
By the way, we have mostly been talking about the production side of academic research this whole time. What about the consumer side? All of us are also looking for the most interesting and useful studies. All of us in industry and government, in the media, especially the media. Here's Ivan Aransky again, we have been.
- [57:09]
Unknown Speaker 2
Conditioned and in fact, because of our own attention economy, we end up covering studies overall. Else when it comes to science and medicine, I like to think that's changing a little bit. I hope it is. But we cover individual studies and we cover the studies that sound the most interesting or that have the biggest effect size and things like that. You know, you wear red, you must be angry. Or if it says that this is definitely a cure for cancer. And journalists love that stuff. They lap it up.
- [57:37]
Stephen Dubner
Like signing a document at the top will make you more likely to be honest on the forum.
- [57:41]
Unknown Speaker 2
Well, you may have heard about that one recently.
- [57:46]
Stephen Dubner
And on that note, I went back to Max Bazerman, one of the co authors of that paper which inspired this series. For Baserman, the experience of getting caught up in fraud accusations was particularly bewildering. Because the accusations were against a collaborator and friend that he fully trusted, Francesca Gino.
- [58:08]
Max Bazerman
So, you know, when we think about Ponzi schemes, it's named after a guy named Ponzi who was an Italian American who preyed on the Italian American community. And if we think about Bernie Madoff, he preyed on lots of people, but particularly many very wealthy Jewish individuals and organizations. One of the interesting things about trust is that it creates so many wonderful opportunities. So in the academic world, the fact that I can trust my colleagues. Colleagues means that we can diffuse the work to the person who can handle it best. So there's lots of enormous benefits from trust. But it's also true that if there's somebody out there who's going to commit a fraud of any type, those of us who are trusting that individual are perhaps in the worst position to notice that something's wrong. And quite honestly, Stephen, you know, I've been working with junior colleagues who are smarter than me and know how to do a variety of tasks better than me for such a long time. I've always trusted them, certainly for junior colleagues, for the most new doctoral students, I may not have trusted their competence because they were still learning, but in terms of using the word trust in an ethical sense, I've never questioned the ethics of my colleagues. So this current episode is really hit me pretty, pretty heavily.
- [59:32]
Stephen Dubner
Can I tell you, Max, that is what upsets me about this scandal, even though I'm not an academic. But I've been writing about and interacting with academics for quite a while now. And the problem is that I maybe gave them overall too much credit. I considered academia one of the last bastions of. I mean, I do sound like a fool now when I say it, but one of the last bastions of honest transformation, transparent empirical behavior, where you're bound by a sort of code that only very rarely would someone think about intentionally violating. I'm curious if you felt that way as well, that you were sort of played or were naive in retrospect.
- [60:13]
Max Bazerman
Undoubtedly, I was naive. You know, not only did I trust my colleagues on the signing first paper, but I think I've trusted my colleagues for decades and hopefully with a good basis for trusting them. I do want to highlight that there's so many benefits of trust. So the world has done a lot better because we trust science. And the fact that there's an occasional scientist who we shouldn't trust should not keep us from gaining the benefit that science creates. And so one of the harms created by the fraudsters is that they give Credibility to the science deniers who are so often keeping us from making progress in society.
- [61:01]
Stephen Dubner
It's worth pointing out that scientific research findings have been refuted and overturned since the beginning of scientific research. That's part of the process. But what's happening at this moment, especially in some fields like social psychology, it can be disheartening. It's not just a replication crisis or a data crisis. It's a believability crisis. Samin Vazier acknowledges this There were a.
- [61:32]
Ivan Oransky
Lot of societal phenomena that we really wanted explanations for. And then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.
- [61:44]
Stephen Dubner
So just how bad is the believability crisis? Danny Kahneman, who died last year, was perhaps the biggest name in academic psychology in a couple generations. So big that he once won a Nobel Prize in economics. His work has been enormously influential in many fields and industries. But in a New York Times article about the Francesca Gino and Dan Ariely scandals, he said, when I see a surprising finding, my default is not to believe it. Twelve years ago, my default was to believe anything that was surprising. Here again is Max Bazerman, who was a colleague and friend of Kahneman's.
- [62:22]
Max Bazerman
I think that my generation fought against the open science movement for far too long, and it's time that we get on the bandwagon and realize that we need some pretty massive reform of how social science is done, not only to improve the quality of social science, but also to make us more credible with the world. So many of us are attracted to social science because we think we can make the world better. And we can't make the world better if the world doesn't believe our results anymore. So I think that we have a fundamental challenge to figure out how do we go about doing that in terms of training? I think that for a long time, if we think about training and research methods and statistics, that was more like the medicine that you have to take as part of becoming a social scientist. And I think we need to realize that it's a much more central and important topic. If we're going to be creating reproducible, credible social science, we need to deal with lots of the issues that the open science movement is telling us about, and we've taken too long to listen to their advice. So if we go from Data Colada talking about p hacking in 2011, there were lots of hints that it was time to start moving, and the field obviously has moved in the direction that Data Khalad and Brian Nosek have moved us. And finally we have Samin Vazir as a new incoming editor of Psych Science, which is sort of a fascinating development as well. So we're moving in the right direction. It's taken us too long to pay attention to the wise advice that the open science movement has outlined for us.
- [64:02]
Unknown Speaker 1
I do think there needs to be a reckoning.
- [64:05]
Stephen Dubner
And that is Joe Simmons, the third member of the Data Colada Collective. He teaches judgment and decision making at the University of Pennsylvania's Wharton Business School.
- [64:16]
Unknown Speaker 1
I think that people need to wake up and realize that the foundation of at least a sizable chunk of our field is built on something that's not true. And if a foundation of your field is not true, what does a good scientist do to break into that field? Like imagine you have a whole literature that is largely false. And imagine that when you publish a paper, you need to acknowledge that literature and that if you contradict that literature, your probability of publishing really goes down. What do you do? So what it does is it winds up weeding out the careful people who are doing true stuff and it winds up rewarding the people who are cutting corners or even worse. So it basically becomes a field that reinforces, rewards bad science and punishes good science and good scientists. This is about an incentive system. And the incentive system is completely broken and we need to get a new one. And the people in power who are reinforcing this incentive system, they need to not be in power anymore. This is illustrating that there is sort of a Roth at the core of some of the stuff that we're doing. And we need to put the right people who have the right values, who care about the details, who understand that the materials and the data, they are the evidence. We need those people to be in charge. Like there can't be this idea that these are one off cases. They're not. They are not one off cases. So it's broken. You have to fix it.
- [65:52]
Stephen Dubner
That again was Joe Simmons. Since we published this series last year, there have been reports of fraud in many fields. Not just the behavioral sciences, but in botany, physics, neuroscience and more. So we went back to Brian Nosek, who runs the center for Open Science.
- [66:09]
Samin Vaziri
There really is accelerating movement in the sense that some of the base principles, principles of. We need to be more transparent. We need to improve data sharing, we need to facilitate the processes of self correction are not just head nods. Yeah, that's an important thing. But have really moved into. Yeah, how are we going to do that? And so I guess that's been the theme of 2024, is how can we help people do it? Well.
- [66:37]
Stephen Dubner
At his center, Nosek is trying out a new plan.
- [66:41]
Samin Vaziri
One of the more exciting things that we've been working on is a new initiative that we're calling Lifecycle Journal. And the basic idea is to reimagine scholarly publishing without the original constraints of paper. A lot of how the peer review process and publishing occurs today was done because of the limits of paper. But in a world where we can actually communicate digitally, there's no reason that we need to wait till the research is done to provide some evaluation. There's no reason to consider it final when it could be easily revised and updated. There's no reason to think of review as a singular one set of activities by three people who judge the entire thing. And so we will have a full marketplace of evaluation services that are each evaluating the research in different ways. It'll happen across the research lifecycle from planning through completion. And researchers will always be able to update and revise when errors or corrections are needed.
- [67:45]
Stephen Dubner
But the need for corrections can move in mysterious ways. Brian Nosek himself and collaborators, including Leif Nelson of datacollada, had to retract a recent article about the benefits of pre registration after other researchers pointed out that their article hadn't properly pre registered all their hypotheses. Nosek was embarrassed.
- [68:09]
Samin Vaziri
My whole life is about trying to promote transparent research practices, greater openness, trying to improve rigor and reproducibility. I am just as vulnerable to error as anybody else. And so one of the real lessons, I think, is that without transparency, these errors will go unexposed. It would have been very hard for the critics to identify that we had screwed this up without being able to access the portions of the materials that we were able to make public. And as people are engaged with critique and pursuing transparency, and transparency is becoming more normal, we might for a while see an ironic effect, which is transparency seems to be associated with poorer research because more errors are identified. And that ought to happen because errors are occurring without transparency, you can't possibly catch them. But what might emerge over time as our verification processes improve, as we have a sense of accountability to our transparency, then the fact that transparency is there may decrease error over time, but not the need to check. And that's the key.
- [69:28]
Stephen Dubner
Still, trust in science in the US has been declining. So we asked Nosik if he is worried that this new transparency, which will likely uncover more errors, might hurt his cause.
- [69:41]
Samin Vaziri
This is a real challenge that we wrestle with and have wrestled with since the origins of the center is how do we promote this culture of critique and self criticism about our field and simultaneously have that be understood as the strength of research rather than its weakness. One of the phrases that I've liked to use in this is that the reason to trust science is because it doesn't trust itself. That part of what makes science great as a social system is its constant self scrutiny and willingness to try to find and expose its errors so that the evidence that comes out at the end is the most robust, reliable, valid evidence as could be. And that continuous process is the best process in the world that we've ever invented for knowledge production. We can do better. I think our mistake in some prior efforts of promoting science is to appeal to authority, saying you should trust science because scientists know what they're doing. I don't think that's the way to gain trust in science because anyone can make that claim. Appeals to authority are very weak arguments. I think our opportunity as a field to address the skepticism of institutions generally and science specifically is to show our work is by being transparent, by allowing the criticism to occur, by in fact encouraging and promoting critical engagement with our evidence. That is the playing field. I'd much rather be on with people who are the so called enemies of science than in competing appeals to authority. Because if they need to wrestle with the evidence and an observer says, wow, one group is totally avoiding the evidence and the other group is actually showing their work, I think people will know who to trust. That's easy to say. It's very hard to do. These are hard problems.
- [71:51]
Stephen Dubner
I agree. These are hard problems, to be fair. Easy problems get solved or they simply evaporate. It's the hard problems that keep us all digging. And we at Freakonomics Radio will keep digging in this new year. Thanks to Brian Nosek for the update. Thanks especially to you for listening. Coming up next time on the show, this sign right here is 12 foot tall. The economics of highway signs. And after that, some 30 million Americans think that they are allergic to the penicillin family of drugs. And the vast majority of them are not. Why does this matter? Nothing kills bacteria better than these drugs. We go inside the bizarro world of allergies that's coming up soon. Until then, take care of yourself. And if you can, someone else too. Freakonomics Radio is produced by Stitcher and Renbud Radio. You can find our entire archive on any podcast app. Also@freakonomics.com where we publish transcripts and show notes this episode was produced by Alina Cullman. Our staff also includes Augusta Chapman, Dalvin Abuaji, Eleanor Osborne, Ellen Frankman, Elsa Hernandez, Gabriel Roth, Greg Rippon, Jasmine Klinger, Jason Gambrell, Jeremy Johnston, John Schnarz, Lyric Bowditch, Morgan Levy, Neal Carruth, Sarah Lilly, Teo Jacobs and Zach Lipinski. Our theme song is Mr. Fortune by the Hitchhikers. Our composer is Luis Guerra. As as always, thanks for listening.
- [73:25]
Samin Vaziri
We are a carrot based organization because we don't have sticks.
- [73:28]
Stephen Dubner
I mean, would you like me to loan you a stick just once in a while?
- [73:31]
Samin Vaziri
Yeah, that would be fun.
- [73:37]
Ivan Oransky
The Freakonomics Radio Network the Hidden side of Everything.
- [73:45]
Samin Vaziri
Stitcher.
- [73:49]
Unknown Speaker 2
Pilots know that weather factors like pop up storms, turbulence and unexpected icing can turn a routine flight into a challenge. But what if you had satellite delivered weather data giving you the full picture of what's around you directly on your Tablet or display? SiriusXM Aviation brings you coast to coast high resolution weather info including radar, winds, icing, TFRs, pireps and more without altitude limitations or line of sight restrictions. Plus it includes features not found on Ads B such as stormtrax, Echo Tops and both cloud to cloud and cloud to ground lightning. SiriusXM's aviation services are compatible with the Foreflight and Garmin Pilot apps. You can also add SiriusXM Entertainment and listen to ad free music plus sports talk, go, comedy news and more. While you fly, fly confidently knowing you have the best information available to make decisions in flight. Visit SiriusXM.com aviation to learn more.
- [74:50]
Brian Nosek
High Five Casino lets you play your favorite slot in live table games like Blackjack with the chance to redeem for real cash prizes. High Five Casino has a giant selection of over 1200 games, including hundreds of exclusive games only found on High Five Casino Casino. It's always free to play and free coins are given out every four hours. Ready to have your own high five moment? Visit high five casino.com that's high the number five casino.com no purchase necessary. Void where prohibited by law. Must be 21 years or older. Terms and conditions apply.
- [75:21]
Stephen Dubner
Do you ever watch TV and think, wow, I'm really good at this? You're right with rewards on sling, watching 30 minutes of TV daily gives you chances to win up to $10,000 in.
- [75:32]
Ivan Oransky
Cash and other monthly prizes.
- [75:33]
Stephen Dubner
Sign up for Sling or stream for free with Sling Freestream to get rewarded for watching TV. Sling lets you do that. Visit sling.com to learn more and get started. No purchase necessary for it were prohibited by law. Visit sling.com for more details.