Voodoo polling? Predicting the US election

Summary

More or Less – "Voodoo polling? Predicting the US Election"

BBC Radio 4, Air Date: November 12, 2012
Host: Ruth Alexander

---

Episode Overview

This episode unpacks the world of election prediction in the aftermath of the 2012 US presidential election, focusing particularly on the statistical techniques popularised by Nate Silver and other aggregators. The discussion contrasts these methods with traditional polling, explores friction between pollsters and aggregators, and asks what we can truly expect from data-driven forecasting. In the second part, the show investigates the real impact of Hurricane Sandy on mortality rates in New York—was the catastrophic storm deadlier than a typical day?

---

Key Discussion Points & Insights

1. The Rise of Aggregators: Nate Silver’s Accurate Predictions

• Nate Silver’s Approach:
  • Ruth Alexander explains how Silver, known for his correct 2008 predictions, succeeded again in 2012, accurately predicting the outcome in all 50 states and all Senate races.
  • Silver's methodology involved using all available polls but with increased skepticism, combined using systematic rules to generate dynamic odds updates.
  • Quote:
    “You use all the information, but tend to use it more skeptically. And I also use it in a way that's systematic… it spits out essentially a set of odds that updates each day as new polls come in.”
    — Nate Silver (01:28)

Aggregators vs. Traditional Pollsters

• Criticism from Traditional Media:

  • High-profile critics like Joe Scarborough cast doubt on Silver’s predictions, claiming the race was a "toss up" despite Silver’s 73.6% confidence in Obama's win.
  • Quote:
    “Anybody that thinks that this race is anything but a toss up right now is such an ideologue.”
    — Joe Scarborough (02:02)

• The Bet:

  • Nate Silver bet $2,000 with Scarborough that Obama would win, which he did, further validating Silver’s approach.

---

2. Methodological Battle: Aggregators vs. Pollsters

How Did Aggregators Get it Right?

• Traditional Pollsters Like Gallup:

  • Predicted a much closer race, some even a Romney win in the popular vote.
  • Focused on national surveys and didn’t incorporate state-level nuances in the electoral college.

• Aggregators’ Strategy:

  • Used state-level polling, weighted results from many poll sources, and factored in economic/demographic data.
  • Accurately forecasted state outcomes, which are pivotal given the US electoral system.

Tension Between Traditions

• Frank Newport, Gallup Editor-in-Chief:

  • Notes a “tension” between those who generate original polls and aggregators who analyze their data.
  • Quote:
    “Part of our issue is that if we don't do the polling, these aggregators have nothing to put into their model. They just sit back and take the benefit of our hard work and toil.”
    — Frank Newport (04:04)

• Pollster-Aggregator Symbiosis:

  • Aggregators depend on original pollsters for raw data, but their synthesis yields a sharper picture.

Aggregators as Allies

• Professor Sam Wong (Princeton):

  • Sees aggregators as providing “useful feedback” to pollsters and a “sharper view,” sharpening the craft rather than replacing it.

  • Quote:
    “Aggregators can give useful feedback to pollsters to help shape their craft and so we're a tool for them as well.”
    — Sam Wong (04:37)

  • On media coverage:
    Suggests that outlets use only their own poll data, but some level of aggregation could “really shape their coverage better.” (05:42)

---

3. Hurricane Sandy: Did the Storm Lead to More Deaths?

The Listener’s Question

• Query:
  Did Hurricane Sandy cause more deaths in New York than on a regular day, or even fewer?

Challenges in Calculation

• Immediate Death Stats:

  • Accurately establishing excess mortality after disasters is complex; official counts can take months.

• Expert Insights:

  • Matthew Kahn, UCLA:
    • Points out that changes in behavior during disasters (like staying home) can reduce deaths from causes like car accidents.
    • Quote:
      “It's certainly possible that natural disasters can actually reduce typical urban deaths if households change their behavior…”
      — Matthew Kahn (06:59)
  • On Violent Crime:
    • Staying indoors can also result in fewer violent crimes such as murders.
    • Cancelled elective surgery may also lower death rates from surgical complications.

• Long-Run Effects:

  • Eric Neumeyer, LSE:
    • Stresses that while some deaths (accidental/violent) may decrease, natural-cause deaths—especially stress-induced ones in vulnerable people—might rise.
    • Quote:
      “Roughly 80%, at least, of all fatalities will be natural. In fact, they may well be exacerbated… the additional stress of a natural disaster impact may well trigger what would normally not happen.”
      — Eric Neumeyer (08:16)

---

Notable Quotes & Memorable Moments

• Nate Silver on skepticism (01:28):
  “You use all the information, but tend to use it more skeptically.”

• Joe Scarborough on predictions (02:02):
  “Anybody that thinks that this race is anything but a toss up right now is such an ideologue.”

• Frank Newport on aggregators (04:04):
  “They just sit back and take the benefit of our hard work and toil.”

• Sam Wong on collaboration (04:37):
  “Aggregators can give useful feedback to pollsters to help shape their craft…”

• Matthew Kahn on behavior and risk (06:59):
  “If people hunker down in their homes and change their behavior, they can actually perhaps reduce their risk exposure because a storm is taking place.”

---

Timestamps for Important Segments

• [00:01-01:28]: Introducing the episode and Nate Silver’s statistical method.
• [01:28-02:17]: Silver and Scarborough’s public debate on the reliability of forecasts.
• [03:28-04:29]: Frank Newport (Gallup) on pollster-aggregator relationships.
• [04:37-05:42]: Professor Sam Wong's perspective on aggregators assisting pollsters.
• [06:33-08:55]: Hurricane Sandy’s statistical conundrum—does a natural disaster increase or decrease overall mortality?

---

Summary

This episode of More or Less dissects why aggregators outperformed traditional pollsters in the 2012 US presidential election, the nascent but pointed rivalry between “data geeks” and established pollsters, and how disaster statistics can defy intuition. Through clear expert commentary and memorable on-air moments, Ruth Alexander guides listeners through a nuanced view of how we interpret numbers, be it at the ballot box or during a superstorm.

Transcript

[A]

Thank you for downloading from the BBC. For details of our complete range of podcasts and our terms of use, go to bbcworldserveys.com podcasts hello and welcome to More or Less on the BBC World Service calling number misusers to account everywhere. I'm Ruth Alexander. This week Nate Silver, blogger and statistics geek who correctly predicted the US election results in 2008, proved he wasn't a one trick pony. We asked whether his mistake methods are the way forward for calling elections and Hurricane Sandy. It left a trail of destruction in its wake.

[B]

Good Morning America Breaking news.

[A]

The perfect storm Sandy brings utter devastation overnight. But how did the death toll differ from an ordinary day on the east coast of America? The presidential elections may have had more than one victor this Barack Obama and Nate Silver. Nate is a New York Times blogger who's famous for predicting the 2008 election. Then he correct predicted which way 49 of the 50 states would go. And he got every US Senate race right too. This time he's predicted 50 of the 50 US states correctly. And praise flowed on as diverse outlets as Fox News and the Huffington Post. But this isn't magic, it's maths. Loyal listeners may remember we spoke to Nate just last month. He explained what he did.

[C]

These polls are reported and people tend to treat each of them as being a highly newsworthy event unto itself. My view is that first of all, you use all the information, but tend to use it more skeptically. And I also use it in a way that's systematic. So you design a set of rules for how you process these polls and it spits out essentially a set of odds that updates each day as new polls come in.

[A]

But Nate Silver has his detractors. US Journalist and former Republican congressman Joe Scarborough derided his predictions, saying this on his MSNBC programme, Morning Joe.

[C]

Nate Silver says this is a 73.6% chance that the President's going to win. Anybody that thinks that this race is anything but a toss up right now is such an ideologue.

[A]

In fact, Nate Silver got a lot of stick from Republican commentators who weren't happy about his forecast. But he was so confident he could predict the outcome again, he made a $2,000 bet with Joe Scarborough, who that Obama would win. And of course, Barack Obama did win. And other aggregators also fared pretty well with strikingly accurate predictions. The traditional pollsters, however, their forecasts ranged from overstating Obama's win to predicting that Romney would win by a percentage point or two. One of those was Gallup. Their polls looked at the national picture and predicted Romney would win the popular vote. The aggregators took a different approach. They looked at state polls from across a range of polling companies and and took a weighted average. In the case of Nate Silver, he added, in other data, such as economic and demographic data, which could affect voting. They then made a state by state prediction and worked out how many electoral college votes each candidate would get. From this, they could forecast who the winner would be. I asked Frank Newport, editor in chief of the polling company Gallup, why there was such a discrepancy between their predictions and those of the aggregators like Nate Silver.

[B]

That's a discrepancy, what these aggregators were doing who don't do polling of their own, by the way. They were saying, looking at the electoral College in the us they thought that the number of polls in the swing states showing that Obama was probably going to win was high enough to give him a higher probability of winning. What we were doing was simply saying here's where we think the national popular vote will be, which will be quite close. And in fact the national popular vote was in fact quite close within a couple of points.

[A]

There seems to be a slight tension between you traditional pollsters and this new breed of data geeks, the aggregators. Would you agree?

[B]

I think that there is some tension there and part of it is founded on the fact that they don't poll. It's kind of like the law of the commons. If everybody rationally decides to graze their cow on the town commons, that makes sense. But if everybody does it, there's no grass left and nobody can graze their cow. Part of our issue is that if we don't do the polling, these aggregators have nothing to put into their model. They just sit back and take the benefit of our hard work and toil.

[A]

Nate Silver may be the one being eulogised, but others also use similar methods. One of them is Professor Sam Wong from Princeton University.

[C]

Well, I think there are two possible motivations that a traditional pollster could have you mentioned Gallup since they were reporting Mr. Romney up by five points. I think that perhaps a little self examination might be orderant. In fact, aggregators can give useful feedback to pollsters to help shape their craft and so we're a tool for them as well. The other use of poles is that individual poles can always give very interesting information about subgroups, say for instance, how people over 40 feel or how people under 30 feel. Those are called cross tabs in the polling industry. And I think that those cross tabs are of great interest in understanding what people think. And I think there's lots of opportunity for pollsters to use their craft in ways that, that they've done in the past and can do in the future.

[A]

What is the problem with traditional polls then? Why do you need an aggregator to come along and put all the different poles together?

[C]

All we're doing as aggregators is taking what they do and putting it together and getting a somewhat sharper view. So I would say that it's, I don't think there's very much wrong that pollsters are doing. I just think that there's such a lot of information that they have a new way of sharpening their craft.

[A]

Can you imagine a day where you all work together, perhaps you pay them for their work and there's some sort of partnership?

[C]

I think one problem with the current media approach to interpreting polls is that individual entities tend to use their own polls. USA Today will use its Gallup information, the New York Times and CBS will partner to use their own information from Quinnipiac. And so one thing that they do is they tend to not watch one another. And I think some low level of aggregation on their part would really shape their coverage better.

[A]

Professor Sam Wong Hurricane Sandy, which hit the east coast of the United States last month, left a trail of destruction and killed more than 100 people in the US and and at least 69 people in the Caribbean, more or less. Listener Andrew Thorne has been in touch to ask the following question. I was wondering whether the deaths during

[D]

the recent storm in New York were more or less than the deaths on

[A]

a regular day there.

[D]

I think it would be interesting to see if the unusual weather resulted in a net loss or gain in the death rate.

[A]

Laura Gray's been working out this week's statistical conundrum. So, Laura, what's the answer?

[D]

This is actually much trickier to calculate than it sounds. Finding out mortality rates so soon after an event is difficult as they can sometimes take months to be finalised, checked, and also published. But all the same, I put Andrew's question to Matthew Kahn, professor at the UCLA in the US who's written about the effect natural disasters have on death tolls.

[C]

It's certainly possible that natural disasters can actually reduce typical urban deaths if households change their behavior on those days. For example, driving continues to be a major cause of death in the United and if people hunker down in their homes and change their behavior, they can actually perhaps reduce their risk exposure because a storm is taking place.

[A]

And if people aren't going out, then they're less likely to be exposed to violent crimes. So murders would go down?

[D]

Yes. And there's another change that's likely to have saved lives and that's fewer hospital operations. Now, you might think that this would mean more deaths, but actually, in 2008, we did a piece about a doctor's strike in Israel, or more or less, which showed the opposite.

[A]

What seems to happen is if you don't have elective surgery, you know, the hip replacements and actually some of the much more serious operations, then the risk associated with anaesthetics and surgery disappears. So one thing that happens is your death rate goes down.

[D]

That was Rabbi Julia Neuberger, former head of the UK health charity the King's Fund, speaking in 2008.

[A]

So back to the New York example provided by our listener. Did the storm lead to more or fewer deaths there than on an average day?

[D]

The final word on this goes to Eric Neumeyer, professor of Environment and Development at the lse. He says that while it is conceivable that there could be fewer unexpected deaths, like car accidents and stabbings during a storm because people stay off the roads and in their homes, that's not the full story. Roughly 80%, at least, of all fatalities will be natural. In fact, they may well be exacerbated. Even if you think about someone prone to a heart attack, then the additional stress of a natural disaster impact may well trigger what would normally not happen. So the natural causes of death may actually increase.

[A]

Eric Neumeyer, and thanks to Laura. And you too can email us with your numerical notes and [e-mail] and you can download more editions of the [e-mail] more or less. There are dozens of different podcasts now available from the BBC, including news, documentaries, science, business, arts and sports. For details of them all, go to bbcworldservice.com podcasts.