Austerity: a spreadsheet error?

Summary

---

Podcast Summary: "Austerity: a Spreadsheet Error?"

More or Less with Tim Harford
Date: April 22, 2013
Host: Tim Harford, BBC Radio 4

---

Overview

This episode of More or Less dives into one of the most significant debates in economic policy post-2008: the impact of government debt on economic growth, focusing specifically on the influential 2010 paper by Carmen Reinhart and Ken Rogoff, Growth in a Time of Debt. Host Tim Harford explores how a graduate student’s attempt to replicate the paper revealed errors and methodological issues that could have influenced major austerity policies worldwide. The episode questions the reliability of headline-grabbing economic findings and underscores the importance of transparency and replication in research that shapes public policy.

---

Key Discussion Points & Insights

1. Setting the Stage: The Reinhart and Rogoff Paper

• [00:13] Tim Harford recalls the high-profile presentation by economists Carmen Reinhart and Ken Rogoff at the American Economic Association, introducing their thesis: once public debt exceeds 90% of GDP, economic growth slows sharply.
• Their paper became a touchstone for fiscal hawks advocating austerity during and after the global financial crisis.

"It set this image of a threshold or cliff at 90% public debt to GDP, and beyond that, woe unto the country that passes." — Michael Asch [01:19]

2. Political Impact of the Research

• [02:32] Mike Konzal of the Roosevelt Institute notes the paper’s enormous influence: after meeting with dozens of US senators, Reinhart and Rogoff’s findings amplified concerns about government deficits.
• [06:44] Politicians, including US Congressman Paul Ryan and EU Commissioner Olli Rehn, cited the “90% threshold” to justify austerity measures.

"It really amplified... made perhaps a minor deficit hawk into a raging deficit hawk. It amplified their worries and their concerns." — Mike Konzal [02:32]

3. The Attempted Replication and Discovery of Errors

• [01:33] Michael Asch describes assigning the paper to student Thomas Herndon for replication; Herndon struggled to match the published results.
• [03:30] After failed attempts, Herndon receives the authors’ spreadsheet. Major issues become clear:
  • Spreadsheet Error: Omission of five countries (Australia, Austria, Belgium, Canada, Denmark) from a key average due to a spreadsheet selection oversight. [03:59]
  • Data Omission: Absence of data for some countries (e.g., post-war Canada, Australia, New Zealand) as per Reinhart and Rogoff, which materially alters results.
  • Averaging Method: The methodology gave the same statistical weight to isolated data points (e.g., New Zealand’s single bad year) as it gave to decades of performance in larger economies.

"We had this puzzle that we were unable to replicate the results as Reinhart and Rogoff published them, and that really got under our skin." — Michael Asch [03:20]

"So you average one year for New Zealand at minus eight, with 20 years for Britain at two and a half, and you say, oh, those average out to about minus three." — Michael Asch [04:51]

4. Reassessing the Core Findings

• [05:22] Herndon et al. publish their results: while high debt correlates with lower growth, the dramatic "cliff" at 90% disappears; the effect is more nuanced, with significant exceptions.
• Reinhart and Rogoff respond, acknowledging the spreadsheet error but disputing that it alters their fundamental thesis:

"We do not, however, believe this regrettable slip affects in any significant way the central message of the paper, or that in our subsequent work. Accidents do happen..." — Reinhart & Rogoff statement [06:02]

5. Debating the Real-World Impact and Scientific Process

• [07:08] Mike Konzal criticizes the lack of peer review and transparency given the drastic policy implications: “A lot of people's lives have been impacted by government budgets under austerity.”
• [07:41] Daniel Hamermesh suggests the research didn't directly destroy jobs, but provided intellectual support for policies that influenced real decisions.
• [08:07] Meghan McCardle emphasizes the broader debate continues, warning against letting any single study—even when flawed—overshadow the complexity of fiscal policymaking.

"It provides an intellectual rationalization for things that then affect how people think about the world. And how people think about the world... eventually affects how the world works." — Daniel Hamermesh [07:41]

"The way that this one 90% figure got wrapped into some sort of scientific proof... is sort of unfortunate because it's now obscured what remains a very live debate." — Meghan McCardle [08:07]

---

Memorable Quotes & Moments

• On the “magic number”:
  • "The economists tell us when your debt gets to 90% as a share of your economy, you start slowing down and we stagnate." — Paul Ryan [06:44]
• On statistical scrutiny:
  • "This research has not been peer reviewed. It has not been opened up to public scrutiny..." — Mike Konzal [07:08]
• On the consequences of economic research errors:
  • "Let's not fool ourselves that discovering a spreadsheet error will end the debate over austerity." — Tim Harford [07:56]

---

Timestamps for Key Segments

• [00:13] Introduction and background on Reinhart & Rogoff’s paper
• [01:29] Thomas Herndon starts his replication assignment
• [02:32] Political uptake and influence of the paper
• [03:30] Discovery of spreadsheet error and further methodological critiques
• [05:22] New findings: the revised relationship between debt and growth
• [06:44] Examples of policy based on the original result (Paul Ryan, Olli Rehn)
• [07:08] The call for better peer review and public scrutiny of consequential research
• [08:07] The complexity and ongoing debate over austerity and debt

---

Closing Thoughts

The episode reveals how a single empirical finding—especially when packaged as a clear-cut rule—can rapidly shape global economic policy and public perception. Tim Harford urges caution: in a complex world, the allure of simple statistical thresholds must be balanced with skepticism and robust peer review.

“No single empirical study will be conclusive in a complex world, even if the spreadsheet rows do add up.” — Tim Harford [08:38]

---

Transcript

[BBC Announcer]

Thank you for downloading from the BBC. For details of our complete range of podcasts and our terms of use, go to bbcworldserveys.com podcasts

[Tim Harford]

hello and welcome to More or Less on the BBC World Service. I'm Tim Harford. Let me take you back to January 4, 2010, the Marriott Hotel in Atlanta, the annual meetings of the American Economic association, where the world's leading economists gather to chew the fat. Two respected economics professors, Carmen Reinhart and the former chief economist of the International Monetary Fund, Ken Rogoff, are presenting a research paper called Growth in a Time of Debt. As with all conference papers, it's brief and won't be published in a peer reviewed journal, but it will pack a real punch. Reinhart and Rogoff had been gathering together a large set of historical data about economic growth and government debt, specifically the debt to GDP ratio comparing government debt to the overall size of a country's economy. And they concluded that for countries with a high debt to GDP ratio, growth was dramatically lower.

[Michael Asch]

The paper was of interest from the very start.

[Tim Harford]

Michael Asch, a professor of economics and public policy at the University of Massachusetts

[Michael Asch]

Amherst, it set this image of a threshold or cliff at 90% public debt to GDP and beyond that, woe unto the country that passes.

[Tim Harford]

And Michael Ashe had an assignment for his graduate students.

[Michael Asch]

We asked the students to replicate what they feel is an important paper in the economics literature. And our student, Thomas Herndon, chose Growth in a Time of Debt as his student project.

[Tim Harford]

While Thomas Herndon was getting stuck into his homework, the world was going through an economic crisis and an intense political argument. Most developed countries were in recession and not coincidentally, debt was growing rapidly, sometimes because of deliberate attempts to stimulate the economy, sometimes just because tax revenues were falling and more people were claiming unemployment benefit. Which led to a now familiar let debt increase in the hope of stimulating economic growth and getting out of recession, or cut spending and raise taxes aggressively to get public debt under control. And not surprisingly, Carmen Reinhart and Ken Rogoff's statistical discovery was was in the spotlight. Mike Conchal, a fellow of the Roosevelt Institute, a left of centre think tank, says that the policymakers were listening.

[Mike Konzal]

Reinhart and Rogoff met with 40 senators where they display these results. Many senators were quite impressed with them and it really amplified it made perhaps a minor deficit hawk into a raging deficit hawk. It amplified their worries and their concerns.

[Tim Harford]

But while US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity and A new coalition government in the UK was promising to raise taxes, cut spending and get the deficit under control. All this with the Reinhart and Rogoff result hanging in the air. Thomas Herndon's homework assignment wasn't going well.

[Michael Asch]

He kept coming back to us saying, I've tried A, I've tried B, I've tried C, I can't get their results. Here are the results that I get.

[Tim Harford]

But his professors couldn't see where he was going wrong either.

[Michael Asch]

We had this puzzle that we were unable to replicate the results as Reinhart and Rogoff published them, and that really got under our sk.

[Tim Harford]

So what was the next step to approach Reinhart and Rogoff?

[Michael Asch]

Yes, after attempting, but being unable to replicate the original results, the course ended and we encouraged Thomas to continue his research. He contacted Reinhart and Rogoff and they kindly provided the actual working spreadsheet that they had used to produce the results in the paper.

[Tim Harford]

And you can see where this is going. With the author's numbers and methodology in hand, Thomas Herndon and his professors found some problems with the paper. The first was a basic spreadsheet error.

[Michael Asch]

In an attempt to take an average across 20 countries. Reinhard and Rogoff only included the cells for 15 countries. So five countries were excluded. Australia, Austria, Belgium, Canada and Denmark were simply excluded through this spreadsheet oversight.

[Tim Harford]

Oops. But Thomas and his professors found other issues which had more of an impact on the famous result. The first was the fact that some data was missing for some countries. Reinhard and Rogoff say that at the time they presented their results, good quality data on post war Canada, Australia and New Zealand simply weren't available. Nevertheless, the omission made a substantial difference. But Michael Asch and his colleagues also didn't like the way that Reinhart and Rogoff averaged their data. They say that one bad year for a small country such as New Zealand was blown out of proportion.

[Michael Asch]

New Zealand's single year, 1951, at minus 8% growth, is held up with the same weight as Britain's nearly 20 years in the high public debt category at 2.5% growth, or Greece's nearly 20 years in the high Public Debt category at 2.5% growth. Those are given equal weight. So you average one year for New Zealand at minus eight, with 20 years for Britain at two and a half, and you say, oh, those average out to about minus three. I think that's a mistaken way to examine these data.

[Tim Harford]

Now there's no black and white here because there are downsides to the obvious alternatives too. But still, Reinhart and Rogoff's averaging method was controversial, and it too made a big difference. All these results were published by Thomas Herndon and his professors on 15 April as a draft working paper. They found that high levels of debt are still correlated with lower growth. But the most spectacular results from the Reinhart and Rogoff paper disappear. The correlation is much gentler, and there are lots of exceptions to the rule. Carmen Reinhart and Ken Rogoff weren't available to be interviewed. They've had a busy week. But they did send us a statement.

[Carmen Reinhart or Ken Rogoff (statement)]

Here's an we are grateful to Herndon et al. For the careful attention to our original paper and for pointing out an important correction to figure two of that paper. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper, or that in our subsequent work accidents do happen

[Tim Harford]

and science progresses through the identification of previous mistakes. But was this a particularly expensive mistake? It's certainly true that pro austerity politicians have sometimes leaned on the Reinhart Rogoff results. Here's Paul Ryan, an influential Republican politician in the US presenting his case for cuts in spending.

[Paul Ryan]

The economists tell us when your debt gets to 90% as a share of your economy, you start slowing down and we stagnate.

[Tim Harford]

And Oli ren, the European Union's commissioner for economic and monetary affairs, has mentioned a 90% debt to GDP limit as widely acknowledged, while making the case for belt tightening in the eurozone periphery. Mike Konzal of the Roosevelt Institute is not impressed.

[Mike Konzal]

This research has not been peer reviewed. It has not been opened up to public scrutiny, and for research that is fairly consequential, a lot of people's lives have been impacted by government budgets under austerity. It shows that the level that we're talking about here is not getting the kind of either elite feedback through peer review or, you know, general feedback through public disclosure that should be necessary when it comes to economic policymaking.

[Tim Harford]

Daniel Hammermesh is professor of Economics at Royal Holloway University of London, and he agrees that this paper has had an influence.

[Daniel Hamermesh]

I don't think jobs were destroyed because of this, but it provides an intellectual rationalization for things that then affect how people think about the world. And how people think about the world is, especially politicians, eventually affects how the world works.

[Tim Harford]

But let's not fool ourselves that discovering a spreadsheet error will end the debate over austerity. Here's Meghan McCardle, special correspondent for Newsweek and the Daily Beast.

[Meghan McCardle]

There is other research showing that you can have these slowdowns when you get to high levels of debt. And I think that the way that this one 90% figure got wrapped into some sort of scientific proof of people's prior beliefs about taxation and spending in government debt is sort of unfortunate because it's now obscured what remains a very live debate about how fast we should be cutting, how far we need to get our debt down, and how much all of this matters.

[Tim Harford]

And the next time some fascinating statistical discovery blazes its way into the political debate, bear it in mind. But remember that no single empirical study will be conclusive in a complex world, even if the spreadsheet rows do add up. If there are any numbers you would like explaining or investigating, you can email us. Our address is more or less BBC.co.uk. and you can listen to more editions of this program and sign up for a free download at our website bbcworldservice.com moreorless

[BBC Announcer]

there are dozens of different podcasts now available from the BBC, including news, documentaries, science, business, arts and sports. For details of them all, go to bbcworldservice.com podcasts.