The Problem with Polling

Summary

---

The Political Scene | The New Yorker

Episode: The Problem with Polling

Date: November 5, 2015

Host: Dorothy Wickenden
Guest: Jill Lepore, professor of American history at Harvard and staff writer at The New Yorker

---

Episode Overview

This episode delves into the evolving role of political polling in American democracy—its origins, its outsized influence today, and the methodological, technological, and philosophical problems that now plague the practice. Dorothy Wickenden and Jill Lepore explore how polling has shaped political strategy, the media’s embrace of “noisy” data, and the urgent need for skepticism and reform.

---

Key Discussion Points and Insights

The Origins and Evolution of Polling

• Modern Polling’s Beginnings

  • George Gallup and the American Institute for Public Opinion revolutionized the measurement of public sentiment in the 1930s.
  • Early polls were seen as democratizing, giving “voice” to Americans outside the political establishment. (02:15)
  • Gallup’s early polls enjoyed a response rate above 90%—a far cry from today's single-digit rates.

• Polling as a Prestigious Endeavor

  • Pollsters once held “quite esteemed” status, akin to progressive reformers.
  • The early promise was that public opinion polling would be “the most important step for democracy since the secret ballot.” — Jill Lepore (02:43)

---

Why Polling is "Teetering on the Edge of Disaster"

• Methodological and Technological Breakdown

  • The shift from landlines to cell phones has dramatically compromised polling accuracy. Over 40% of Americans don’t have landlines; cell phone auto-dialing is restricted by law. (03:55)
  • Demographic differences emerge between those reachable by landline and those who are not, skewing data.
  • Response rates have plummeted from over 90% to “the single digits.” (03:46)
  • People are “sick of answering the phone” and less willing to participate in surveys.

• Political and Philosophical Problems

  • Problems “have been there all along,” not just with technology, but fundamentally political—such as the way polling can interfere with, rather than illuminate, democracy.
  • Even in the 1930s and 1940s, Congress repeatedly investigated the influence of polls. Critics argued that polling could “interfere with the democratic process.” — Jill Lepore (04:58)

---

Polling and the Press: A Problematic Relationship

• Overinterpretation of Noisy Data

  • The press, not just pollsters, perpetuates the problem by “overinterpreting noisy data.” (06:09)
  • Nate Silver (of FiveThirtyEight) points out that polling is often embraced as urgent news, even though the real story is that “there isn't all that much news in the campaign and the media is overinterpreting noisy data.”

• Polling's Lack of Predictive Power

  • Studies confirm polls “do not have predictive power.” They are “immensely entertaining and they're also powerful, are they not?” — Jill Lepore (07:05)
  • Despite this, poll numbers dictate campaign decisions, candidate strategy, donations, and even whether candidates stay in the race. (07:24)
  • “All these polls, we know they're not actually telling us much, but they nevertheless are determining the course of the election.”

---

The Shift from Party Bosses to Political Consultants

• From Party Machines to Political Marketing

  • The 1930s saw a transformation from party-driven politics to a “business model,” anchored in public opinion management and political consulting. (08:11)
  • Early pollsters, advertisers, and newspaper men—such as those at Campaigns Inc.—pioneered the integration of polling and political advertising.
  • Key moment: Eisenhower's 1952 campaign, which ushered in television ads, opponent research, and deep reliance on polling.

• Quote:
  “By the 1930s, [the party machine] reemerges in a completely different form. It's in the form of public opinion management and political consulting. It's just a different machine.” — Jill Lepore (08:11)

---

Polling in the Digital Age

• Obama's 2008 Campaign and the Myth of Digital Democracy
  • Obama’s strategy in 2008 sidelined traditional consultants in favor of social media. (10:20)
  • While new technologies enabled more communication, they didn’t guarantee truly broader participation or deliberative democracy.
  • Studies like “The Myth of Digital Democracy” challenge whether more voices are actually heard, noting a “real consolidation of political authority, not a diffusion.” (10:36)
  • Lepore raises doubt about the quantifiability of public opinion: “There’s a set of assumptions about opinion and attitude that have never really been proven, [but] lie behind the entire endeavor.” (11:40)

---

Media Gimmicks and the Perils of Instant Analysis

• Real-time Sentiment Analysis

  • Lepore critiques technologies like Bing’s real-time “pulse” graph during debates, likening it to a false “veneer of scientific truth.”
  • Quote:
    “It has this eerie veneer of scientific truth, because here is this…it was just about the most preposterous thing I had ever seen.” — Jill Lepore (13:13)
  • She invokes E.B. White: “We should be careful if you take the pulse of the nation, that the nation just hasn't run up a flight of stairs.” (13:35)
  • These gimmicks neglect the need for deliberation, conversation, and authority in politics.

• On Urgency and Volatility

  • The press has a business interest in selling polls as “urgently, instantly necessary.”
    “We blame politicians all the time for the volatility and vehemence of our politics, but I think the press needs to be held a little bit more accountable for the way in which the embrace of volatility is a sales model.” — Jill Lepore (14:29)

---

Advice for Navigating the Age of Polls

• Polling about Behavior vs. Opinion
  • Surveys like the census (measuring behavior/status) remain reliable; opinion polls, less so. (12:20)
  • Critical engagement and skepticism are needed. Not every “pulse” or instant reaction is meaningfully representative.

---

Notable Quotes & Memorable Moments

• “He really presented himself as a social scientist who was committed to the work of hearing the voices of Americans, whose voices were not heard by the political process.”
  — Jill Lepore on Gallup (02:59)

• “Critics said in the 1930s and 1940s, when Congress repeatedly investigated the polls and polling organizations, critics would say this interferes with the democratic process.”
  — Jill Lepore (06:00)

• “All these polls, we know they're not actually telling us much, but they never the less are determining the course of the election.”
  — Jill Lepore (07:40)

• “It's very important for the press to sell these polls as urgently, instantly necessary.”
  — Jill Lepore (14:25)

---

Important Timestamps

• 01:15 — Introduction and Veep satire of pollsters
• 02:15 — Gallup, the invention of modern polling, and early public reaction
• 03:55 — Challenges of polling due to technological change and legal regulation
• 06:09 — Press over-interpretation of polling (“noisy data”)
• 07:24 — Real-world effects of polling on campaign strategy
• 08:11 — Emergence of political consultants and public opinion management
• 09:26 — 1952: TV advertising’s role in modern campaigns
• 10:20 — Obama, social media, and digital democracy
• 12:20 — Reliable vs. unreliable types of polling
• 13:13 — Critique of real-time media “pulses”
• 14:25 — The press’s commercial interest in poll-driven volatility

---

Summary

This episode offers a wide-ranging, deeply informed discussion of how polling has become both more pervasive and more problematic in American politics. Jill Lepore and Dorothy Wickenden probe the shortcomings—technological and philosophical—that undermine poll reliability, highlight how the media’s reliance on polling can distort political narratives, and suggest a return to a more skeptical, deliberative approach to politics and media coverage. The conversation unpacks the historical roots of today’s polling obsession, questions unexamined assumptions about public opinion, and calls for media accountability in an era when the “pulse” of democracy may be little more than a meaningless EKG.

Transcript

[Mint Mobile Advertiser]

As summer draws to a close and the kids go back to school, I know I'm going to want to keep in touch with my kids at a price I can afford. Back to school. Shopping can be a hassle, but your phone plan shouldn't be. That's why I made the switch to Mint Mobile. For a limited time, Mint mobile is offering three months of unlimited premium wireless service for 15 bucks a month. So while other parents are sweating overage charges, I have a little bit more room in my budget for cool back to school threads. Say bye bye to your overpriced wireless plan's jaw dropping monthly bills and unexpected overages, Mint Mobile is here to rescue you. All plans come with high speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Dish overpriced wireless and get three months of premium wireless service from Mint Mobile for 15 bucks a month. This year, skip breaking a sweat and breaking the bank. Get this new customer offer and your three month unlimited wireless plan for just 15 bucks a [e-mail] New Yorker that's that's mintmobile.com New Yorker upfront payment of $45 required, equivalent to $15 a month limited time new customer offer for first three months only. Speeds may slow above 35 gigabytes on unlimited plan. Taxes and fees extra. See Mint Mobile for details.

[Dorothy Wickenden]

This is the Political Scene, a weekly conversation with New Yorker writers and editors about politics. It's Thursday, November 5th. I'm Dorothy Wickenden, executive editor of the New Yorker. Politicians don't get much respect these days, nor do the advisors. Them how to make themselves more popular. Here's Kent, the President's pollster on the HBO show Veep, talking to Ben, the President's chief of staff, and Amy, the Vice President's chief of staff.

[Jill Lepore]

All I do is provide POTUS with the polls. Cold numbers. God, I hate numbers. I mean, there's cultures that don't even have any numbers and they do just fine. Well, Ben, it seems like you're ill suited for your job and you should resign. Ben's breakdown aside, the VP feels we.

[Dorothy Wickenden]

Need a quick military strike here.

[Jill Lepore]

Yeah, precise and surgical, like your lobotomy, Ken.

[Dorothy Wickenden]

Jill Lepore joins me to discuss the role that polling plays in American politics. So, Jill, we weren't always bombarded by poll numbers for the year leading up to presidential elections. How did we get to this point?

[Jill Lepore]

I had forgotten how recently it was that polls were so predominant in our coverage of elections. And the declining status of pollsters, as represented by Kent's unfortunate role in veep, is also a fairly recent vintage. Pollsters used to be quite esteemed. You know, modern political polling began in the 1930s with George Gallup, who founded the American Institute for Public opinion. Gallup was a kind of great cult figure. Much applauded for his civic mindedness. People welcomed the public opinion poll in the 1930s as the most important step for democracy since the secret ballot.

[Dorothy Wickenden]

Why would they think that?

[Jill Lepore]

They would think that for a number of reasons, Mainly because Gallup so insistently told them that Gallup was a great publicist. He really presented himself as a social scientist who was committed to the work of hearing the voices of Americans, whose voices were not heard by the political process and in the 1930s as a kind of great populist. And then social science was on the rise. And maybe in a way that you know how the kind of free path that people who are engaged in startups and who do stem work of cultural cache of the startup in the 1930s, that was social science. One of the things that that meant in the 1930s was that when Gallup's pollsters came to your door and knocked on the door, people answered. So Gallup's response rate, that is the percentage of people that took a soldier survey as a percentage of those who were asked, was well above 90. That is no longer the case.

[Dorothy Wickenden]

You have a piece coming out in the magazine in a week or so where you argue that polling is actually teetering on the edge of disaster. Why is that?

[Jill Lepore]

From a technological vantage and a methodological vantage, there are a lot of problems with polling currently mainly having to do with the transition from landlines to cell phones. More than 40% of Americans no longer have landlines. And the number of polling and market research organizations is so huge, the way the federal government, through the FCC, has tried to regulate that industry is by saying auto random dialing of cell phones is against the law. So given that most Americans no longer have landlines, it's harder and harder for polling organizations to reach Americans. And the Americans who do not have landlines are somewhat different in demographic composition than Americans who do. Also, just people are sick of answering the phone and asking these questions. So the response rate, which started out in the 1930s above 90, is now in the single digits. What I'm really struck by, though, is that the problems with polls have been there all along. And they're not just the technological and methodological problems. They're Fundamentally, political problems.

[Dorothy Wickenden]

And what do you mean by that? And how do pollsters defend themselves?

[Jill Lepore]

To be clear, what pollsters want to do is get the most accurate results and to report information that is reliable. So when Gallup devised the public opinion survey, though he was really interested in finding out what Americans think. He was not that interested in predicting elections, which is actually a pretty dodgy proposition. But Gallup came to believe that it would be useful to use his method to forecast elections because it would prove the reliability of his opinion surveys. And what he was really selling were the results of his opinion surveys to newspapers. As a product, he would report, you know, 40% of Americans believe this, that or the other thing. And how can you prove that you're right? From Gallb's fantasy, You'd believe me if I used the same method to predict who was going to win the next presidential election. But it has all along been the case that there are problems with predicting elections. Critics said in the 1930s and 1940s, when Congress repeatedly investigated the polls and polling organizations, critics would say this interferes with the democratic process.

[Dorothy Wickenden]

It's interesting. The statistician Nate Silver, he had a recent conversation on his website, fivethirtyeight. Com, about the relative strengths of Donald Trump and Ben Carson, and he admitted that the real story is that there isn't all that much news in the campaign and the media is overinterpreting noisy data.

[Jill Lepore]

You know, I think what Nate Silver does is incredibly important and civic minded. There is a barrage of polls out there. It's a sea of polls. And he tries to help people understand the relative merits of this poll or the other poll. But what you're suggesting Silver's remarks point to is an important consideration. And that is that actually the problem with polls hasn't to do with the pollsters. It actually has to do with the press.

[Dorothy Wickenden]

Right. And so we're in a situation right now about exactly a year from the election. Candidates, reporters, pundits, voters and pollsters. Of course, we're all relying on what he calls noisy data. And does it have a predictive power?

[Jill Lepore]

It's been shown not to have a predictive power. To the degree that it's data that has been analyzed, it's been analyzed to a negative result. The fact that they're not predictive is fairly well established. They're immensely entertaining and they're also powerful, are they not?

[Dorothy Wickenden]

I mean, aren't now Republican candidates dropping out because the polls show nobody likes them?

[Jill Lepore]

Absolutely right. So there is a problem, and there's so many Polls. The responsibility is so low. And yet polls are wielding a far greater influence in American politics than ever before. There are candidates who have made decisions based on polls. There are people who donate money based on polls. All these polls, we know they're not actually telling us much, but they never look less, are determining the course of the election.

[Dorothy Wickenden]

This reminds me of a piece you wrote a couple of years ago. It seems part of this process of political consultants replacing party bosses, which is what we had in the mid 20th century. How did advertising and polling come to work together? As I recall, you used the example of Eisenhower's 1952 presidential campaign, which was kind of a turning point because of the advent of TV in many households.

[Jill Lepore]

One of the things that's so interesting about Gallup is the degree to which his work aligns with that of the first political consulting firm, which was called Campaigns Incorporated. And the people who found Campaigns, Inc. Come from the world of advertising. Gallup really comes from the world of newspaper circulation. These people are all bound up in the same set of forces, which is the application of the tools of social science, that is using these quantitative methods to understand consumers. So what happens really beginning in the 1930s is the transformation of the workings of politics from a real party driven system to a business model. And what's interesting about that to me is, you know, when you think about the 19th century American politics and how messy and sort of dirty politics was in the 19th century, and then you think about these heroic progressive reformers, these muckmakers who came in in the 1890s and reformed the political machine and cleaned that system up. They introduced the secret ballot. There are all these, you know, incredibly important reforms to sort of kill the party Mach. By the 1930s, it reemerges in a completely different form. It's in the form of public opinion management and political consulting. It's just a different machine.

[Dorothy Wickenden]

And what about the 50s with the Eisenhower campaign and the advent of TV?

[Jill Lepore]

So Campaigns Inc. Is still on the scene. By the 1950s, they introduced the first television campaign ads. They do opponent research. Most of the elements of modern political consulting, including the reliance on public opinion polls, those are all in place by 1952, when Eisenhower runs the sort of I like Ike TV ads.

[David Remnick]

Right now, we are living through some of the most tumultuous political times our country has ever known. I'm David Remnick, and each week on the New Yorker Radio Hour, I'll try to make sense of what's happening alongside politicians and thinkers like Cory Booker, Nancy Pelosi, Liz Cheney, Tim Waltz, Ketanji Brown, Jackson, Newt Gingrich, Robert F. Kennedy Jr. Charlemagne, tha, God, and so many more. That's all on the New Yorker Radio Hour wherever. Listen to podcasts.

[Dorothy Wickenden]

In 2008, Obama chose a different strategy, sort of shaming Karl Rove and all of the traditional political consultants. And his campaign was largely based on social media. How did that campaign change the way politicians woo voters?

[Jill Lepore]

There are tremendous effects to some of those changes to the reliance on social media. You know, there's this book called the Move on Effect, but there are a lot of careful political scientists who have urged that we be a little bit less gullible about the dramatic transformation that some of these technologies introduce. So there's this kind of interesting book called the Myth of Digital Democracy, which challenges the claims that the Internet is essentially a democratic tool. In the end, if by digital democracy we mean that more voices are heard, more words are spoken, but they're not heard, in fact, there's a real consolidation of political author. I'm really fascinated by those developments. Things are, like, dazzlingly interesting. What if there are these little elves who are crawling the Internet and telling us what people think? That we could find out what people think. But the more you can kind of rush ahead with the technology, from my vantage, the less careful we attend to the deep humanistic questions involving political philosophy that have been asked from the very beginning about the very idea of public opinion, which is, is it a quantifiable thing? Is it something that does, in fact, lend itself to measurement of. There's a set of assumptions about opinion and attitude that have never really been proven, lie behind the entire endeavor. And I think, given the role that polls have come to play, really ought to be asked and asked very, very carefully.

[Dorothy Wickenden]

So how do you, as a writer and an American historian, analyze candidates in the issues? How do you go about sort of sifting through all of that and coming up with your arguments?

[Jill Lepore]

I think surveys that measure status or behavior have been proven to be fairly reliable. So the census is a very useful. Is a very useful tool. It's not a sample survey, but it is a survey. Well, it's surveys that purport to measure opinion, where everybody involved in that work recognizes a fair amount of skepticism required. You know, when you watch the last debate, maybe it was the second Republican debate, I followed this thing that Bing did, this pulse thing. There was a graph like an EKG reading that was going across your screen. If you went to Bing, it was supposed to be like how everybody who was responding to Bing was responding second by second to things that the candidates said. And it has this eerie veneer of scientific truth, because here is this. You know, it's also like the way it was visually represented. It just looks like something seen in, er. You know, it was Gallup who called polling taking the pulse of the nation, and it was E.B. white who said, we should be careful if you take the pulse of the nation, that the nation just hasn't run up a flight of stairs. Really, it was really a visual manifestation of the idea that these tools are actually able to take in real time the pulse of the nation. And it was just about the most preposterous thing I had ever seen. And, you know, I don't need to know that night who won the Republican debate. Making a political decision requires deliberation and it requires conversation and actually requires some notion of authority. So all of our opinions are not, in fact, equal. We haven't all thought about these issues. So the people who are responding second by second on the Bing pulse meter or whatever it is, they're not holding those opinions, they're evanescing those opinions. I guess I'm a great proponent for the idea that there's much less urgency to these matters than the press would make it appear. It's very important for the press to sell these polls as urgently, instantly necessary. We blame politicians all the time for the volatility and vehemence of our politics, but I think the press needs to be held a little bit more accountable for the way in which the embrace of volatility is a sales model.

[Dorothy Wickenden]

Thank you so much, Jill.

[Jill Lepore]

Thank you.

[Dorothy Wickenden]

Jill Laporte is a staff writer and a professor of American history at Harvard. She's the author, most recently, of the Secret History of Wonder Woman. This has been the Political Scene from the New Yorker. This podcast is produced by Jill Dubeuff and Alex barron. For new yorker.com I'm Dorothy Wickenden.

[Katie Drummond]

You can subscribe to this and other New Yorker podcasts in the itunes store. You can find past episodes of the New Yorker out loud, the Political Scene and the New Yorker's fiction and poetry [e-mail] podcast the weekly audio edition of the New Yorker is [e-mail] this podcast is produced by Jill Duboff and Alex Barron. What the hell is going on right now and why is it happening like this? At Wired, we're obsessed with getting to the bottom of those questions on a daily basis, and maybe you are too. I'm Katie Drummond, the global editorial director of Wired, and I'm hosting our new podcast series, the Big Interview. Each week I'll sit down with some of the most interesting, provocative, and influential people who are shaping our right now. Big Interview conversations are fun. I want a shark that that eats the Internet, that turns it all off, unfiltered and unafraid.

[Jill Lepore]

So in a lot of ways, I try to be an antidote to the.

[Katie Drummond]

Unimaginable faucet of reactionary content that you see online. To the best of my ability, every week we're going to offer you the ultimate luxury of our times, meaning and context. True or false? You, Brian Johnson, the man sitting across from me, one day, at some point, as of yet undefined in the future, you will die. False. Tell me more. Listen to the big interview right now, in the same place you find WIRED's Uncanny Valley podcast. Subscribe or follow wherever you get your podcasts.

[Jill Lepore]

From. PRX.