163. The Data Sleuth Taking on Shoddy Science

People I (Mostly) Admire

Published: Sat Aug 02 2025

Uri Simonsohn is a behavioral science professor who wants to improve standards in his field — so he’s made a sideline of investigating fraudulent academic research. He tells Steve Levitt, who's spent plenty of time rooting out cheaters in other fields, how he does it.

Summary

Podcast Summary: People I (Mostly) Admire – Episode 163. The Data Sleuth Taking on Shoddy Science

Title: The Data Sleuth Taking on Shoddy Science
Host: Steve Levitt
Guest: Yuri Simonson
Release Date: August 2, 2025
Podcast Network: Freakonomics Radio + Stitcher
Episode Focus: Combating fraudulent and misleading research in academia through data analysis and investigative techniques.

1. Introduction to Yuri Simonson and DataColada

Steve Levitt welcomes Yuri Simonson, a behavioral science professor and member of the DataColada team, which includes Joe Simmons and Leif Nelson. DataColada is renowned for its efforts to debunk fraudulent research, call out cheaters, and identify misleading research practices in academia.

Quote [01:19-01:21]:
Levitt: "My guest today, Yuri Simonson, is a."
Quote [01:33-02:10]:
Simonson: "I did some commentary or criticism, I was asked to review a paper for a journal... I went to the original paper... It took me a while, and then I figured out what the problem was."

2. Uncovering Shoddy Research Practices

Yuri recounts his initial foray into identifying flawed research, sparked by a seemingly outrageous study on the impact of initials on life decisions. His meticulous analysis revealed that the study's conclusions were driven by improbable coincidences, such as couples remarrying with the same last name post-divorce.

Quote [02:41-04:56]:
Simonson: "I thought, why would people have the same last name and be more likely to marry?... So I thought, why would people have the same last name and be more likely to marry? Like, how would that happen?... it's such a huge coincidence that even a small share of these people, they can generate an average effect that's sizable."

3. False Positive Psychology and P-Hacking

In 2011, Simonson, along with Simmons and Nelson, published the influential paper "False Positive Psychology." The paper highlighted how common research practices, such as selective reporting and multiple testing, can lead to exaggerated statistical significance, making false hypotheses appear true.

Quote [04:56-05:33]:
Simonson: "We publish a paper that shows how certain practices lead to statistically significant results even when hypotheses are false."

Motivation Behind the Paper:

Quote [05:33-05:36]:
Levitt: "What was the motivation behind writing that paper?"
Quote [05:36-05:55]:
Simonson: "We were going to conferences and we were not believing what we were seeing... There's nothing alarming enough to make us update our priors."

4. The Illusion of Statistical Significance

Simonson humorously discusses a fabricated experiment from their paper where listening to "When I'm 64" by The Beatles supposedly causes time to reverse, illustrating how scientific tone can mask absurd results.

Quote [05:55-07:00]:
Simonson: "You describe an experiment in a very serious tone... 'Will you still need me? Will you still feed me when I'm 64?'... People who listen to that song are almost 1.5 years younger."

5. P-Hacking Explained

P-hacking involves manipulating data or analyses until statistically significant results are achieved. Simonson explains that practices like multiple comparisons and selective reporting inflate the probability of false positives.

Quote [08:18-09:38]:
Simonson: "All you need to do is give yourself enough chances to get lucky... The P value refers to the probability value... If your story's not really true, you'd only get the data that looked like this less than 5% of the time."

6. Case Studies: Francesca Gino and Dan Ariely

Francesca Gino Case:

Simonson details the investigation into Francesca Gino's research at Harvard, where anomalies in data led to accusations of fraud. The team uncovered inconsistencies in survey responses, prompting a $25 million lawsuit against Gino and Harvard.

Quote [28:12-28:33]:
Simonson: "We found a mass of sevens disappearing into ones and twos... The disconnect between qualitative descriptions and quantitative data was a red flag."

Dan Ariely Case:

Similarly, Simonson discusses suspicions of data manipulation in Dan Ariely's research on car insurance, where a uniform distribution of miles driven was impossible, indicating potential fraud.

Quote [40:17-43:04]:
Simonson: "The distribution of miles driven was perfectly uniform... It was a goldmine... the original data showed discrepancies that confirmed our suspicions."

7. Legal and Institutional Challenges

Facing accusations of fraud leads to significant personal and professional strain. Simonson recounts the $25 million lawsuit filed against him, his team, and Harvard by Francesca Gino. Despite initial resistance, support from the academic community through a successful GoFundMe campaign alleviated some pressures.

Quote [37:46-44:02]:
Simonson: "It was hard because defending yourself in the American legal system is very expensive... A GoFundMe project was started, and it raised hundreds of thousands of dollars... It was the only time I've cried for professional reasons."

8. The Role of Academic Institutions

Simonson critiques how academic institutions handle misconduct cases, noting inconsistencies in punitive measures. While Harvard took definitive action against Gino, Duke University handled Dan Ariely's case more discreetly, highlighting a systemic failure to uniformly police academic fraud.

Quote [43:50-44:58]:
Simonson: "Duke conducted its investigation extremely secretively... It's a failure of the institutions to police themselves."

9. Deterrents and Solutions to Academic Fraud

Simonson emphasizes that current deterrents are insufficient, as the repercussions for fraud are often minimal compared to the incentives to cheat. He advocates for proactive measures to prevent fraud rather than merely detecting it post-factum.

Quote [44:02-45:58]:
Simonson: "There's no real punishment. If the worst that can happen is being fired, it's still a win-win for cheaters... We need to prevent fraud by not complicating matters for honest researchers."

Tools Developed:

P-Curve Analysis: Simonson explains the P-curve as a tool to assess the credibility of research findings by analyzing the distribution of p-values across published studies.
- Quote [24:22-27:05]:
  Simonson: "If people are trying multiple things to get to 0.05, they're not going to go all the way... So if you see a bunch of results at 0.04, you should not believe those studies."
Ask Collected Platform: A platform designed to ensure data transparency by requiring researchers to document their data acquisition and analysis processes, making it harder to manipulate data without detection.
- Quote [46:03-47:47]:
  Simonson: "Provide a written record of where the results come from... Journals should require a unique URL with detailed documentation."

10. The Broken Tenure System and Incentive Structures

Simonson argues that the academic tenure system, while fostering creativity and hard work, also creates strong incentives for quantity over quality in research outputs. The lack of penalties for low-quality or fraudulent research exacerbates the problem.

Quote [47:47-49:21]:
Simonson: "The lack of a penalty for low-quality inputs is the broken part... Transparency helps align incentives by making it easier to verify credence in research."

11. Steve Levitt’s Demonstration of Data Manipulation

To illustrate how data can be manipulated to produce desired outcomes, Levitt conducts an experiment where he surveys listeners’ optimism or pessimism about the future climate. Despite multiple data slicing attempts, none of the results yield statistically significant findings, highlighting the ease of creating misleading analyses.

Quote [52:57-53:43]:
Levitt: "I could have tricked you by talking about what we're going to do next... It's just a way to try to have fun with data."

12. Conclusion and Future Directions

Yuri Simonson underscores the necessity for systemic changes in academic research practices, advocating for greater transparency, pre-registration of studies, and open data to enhance research credibility and prevent fraud.

Quote [52:45-53:00]:
Simonson: "We need to prevent fraud by making it harder to commit and easier to detect... The deliverable is a URL with all necessary documentation."

Upcoming Episodes and Additional Resources:

Levitt encourages listeners to explore further episodes on academic fraud, specifically episodes 572 and 573 of Freakonomics Radio, for more in-depth discussions.

Quote [49:42-50:36]:
Levitt: "Check out episodes 572 and 573 of Freakonomics Radio, which you can find in your podcast app."

Key Takeaways:

Academic Fraud is Pervasive: Common research practices can inadvertently or intentionally lead to false positives, undermining the credibility of scientific findings.
Data Transparency is Crucial: Tools like the P-curve and platforms like Ask Collected are essential in promoting honest and reproducible research.
Institutional Failures: Academic institutions often fail to uniformly address and penalize fraudulent research, perpetuating a culture where the quantity of publications is valued over quality.
Systemic Incentives Need Reform: To curb academic fraud, the incentive structures within academia must shift towards rewarding quality, transparency, and integrity in research.

Notable Quotes:

Yuri Simonson [04:56]:
"All you need to do is give yourself enough chances to get lucky."
Simonson [24:22]:
"P-curve just formalizes that. It takes all the P values... tells you you should believe it, you should not believe it, or you need more data to know."
Simonson [47:47]:
"One of the parts of the incentives that is broken is the lack of a penalty for low quality inputs."

This episode sheds light on the critical issue of research integrity in academia, highlighting the essential role data analysis and transparency play in maintaining the credibility of scientific inquiry. Yuri Simonson's work with DataColada exemplifies the ongoing battle against fraudulent research practices, advocating for systemic changes to ensure the reliability and trustworthiness of academic findings.