Loading summary
Transcript35 lines
- [00:00]
A
Foreign and welcome to the focus group podcast feed. I'm Sarah Longwell, publisher of the Bulwark, and I am joined here today by Connor Kilgore, one of my moderators and producers of the folks group podcast. And this is not a full focus group podcast episode. We are going to be back officially after Labor Day. So the first Saturday, which I think is 9 6, is when the first episode back from hiatus will be out. And I'm looking forward to getting back to it. Connor, thanks for being here. Why are we, what are we doing here? What's happening? Why are we making this little podcast right now?
- [00:50]
B
So I am also looking forward to getting back into the show. We've just gotten stir crazy around here this summer. We can't stop doing focus groups. But after we did our little reading the comments section take last week, we got many more questions and comments about how we do the focus groups, why we do the focus groups, how we interpret them. And many of them were frequently asked questions. And one thing that I know you like to say around this shop is that just because you're sick of saying something doesn't mean that other people don't still need to hear it. So we wanted to come back to our listeners with some frequently asked questions about our methodology and just how we think about the focus groups and the value we get from them.
- [01:37]
A
Yeah, okay. Love that. Because we do. We get a lot of people asking a lot of different things, usually in the vein of sort of, how do you find people? Like, why do you choose the people that you do? A lot of people want to know why we don't push back on people. And so like, let's talk about those things. Connor, why don't you, why don't you read some of the frequently asked questions and we'll answer them.
- [02:02]
B
So some of these are pulled from the comments section of that reading the comments take we did last week. So it's, that's a little bit meta, you know, reading the comments about the comments. But this is from an episode a few months ago. Someone asked, why do you not fact check your focus group participants when they're making completely false statements? I get it. Americans are dumb. They don't pay attention to things. But come on, you're not dumb. I appreciate them clarifying that we're not dumb, but Sarah, why don't we make it a practice of pushing back on people?
- [02:31]
A
Honestly, it would negate the entire point of doing the focus groups. This is a question that we get a lot. And I understand that it is frustrating for People to, especially when they get to hear the focus group commentary through the podcast. They hear people say things that are wrong that, you know, where they don't have the facts. In the case where they've missed a whole, like, you know, they. They are. They are clearly convinced of something that, you know, the listener knows not to be true. And I understand why people want the satisfaction of hearing a moderator push back and give them the real facts. We don't do it because I am not trying to convince, ever the 10 people in the focus group. I am trying to learn from the 10 people in the focus group so that I can try to build usually campaigns, specifically because I do two kinds of work here. Right. I started doing. I'm sorry for all the preamble, but I'm gonna. I do want to tell you that the reason I started doing the focus groups was not to make a podcast. The reason I started doing the focus groups was because back in 2018, we were trying to figure out a way to primary Donald Trump, and I was trying to figure out if that was possible. And in doing so, I started listening to these voters, and it was like the scales fell from my eyes. Because once you start listening to voters, you realize what kind of information environment they're in. You realize why certain things persist in terms of, you know, just why do they believe one thing, while perhaps I believe or other people believe another thing. And it is figuring out what people think authentically so that you can try to build campaigns to persuade them otherwise. Right. So back then, we were really focused on people who had voted for Trump but really disliked him. Right, Those edge cases. And so I'm not trying to convince those 10 people in the room. I'm trying to learn from them so that I can construct effective campaigns or advocacy material or whatever it is that I'm doing in my sort of of political persuasion life. I just want to know what they're thinking. I always say I want to. I want to listen and learn from 10 people so that I can try to convince 10,000 people. And here's the thing, and Connor, you can verify this. It's not like we never push back with certain facts, or it's not like we never sort of say, like, well, you know, actually this thing is true or this thing just happened. We do do it. It doesn't go well oftentimes with the groups because they also, in the trust that you build during the conversation, they don't want you to have an opinion. And in fact, the creation of an environment in which the moderator is genuinely just interested in what they're saying, creates a kind of safety for them to be honest, which is the number one thing we want from these voters. I want to know why they think the thing. They do not try to correct it and then break the trust that's allowing them to be honest with me. But there have been times for, either because the group got curious or back when I was doing focus groups, or because people Googled me and figured out who I was. And in that, you know, that it does. Like, people get mad at the idea. I think that. I don't know, just that you're. That you have an opinion and are pushing back in an environment where their feeling is, like, I'm just. I'm here to give my opinion, like, authentically.
- [06:11]
B
Yeah. We sometimes tell people that as long as you're telling us what your honest opinion is, then that's the right answer, and there are no wrong answers. And this goes back to a question I think, that we got in the video we did last week where people said, you know, how. How do you know that people are giving you their honest opinion? They're saying that they're voting for Trump because of the economy, but really they are, you know, they didn't want to vote for a black woman for president. And this is part of how we create an environment that's as candid as possible. I know a couple of months ago, I was doing a group of fairly strong Republicans. They'd voted for Trump the last two times, and we were talking about the big, beautiful bill, and I was kind of probing that. Folks were talking about how they wanted the national debt to be handled, and they also wanted to see the 2017 tax cuts extended. Most of our listeners will probably say, well, you can't do both. And I would agree. And so I asked, how do you think about the trade off between continuing these tax cuts and handling the national debt? And people got feisty in saying, well, you can cut taxes and still increase revenue. Some people in this group were really up on kind of the Laffer curve and well versed in, I guess, Reaganomics. But people got a little touchy about it, and you had to kind of quickly move on. But the conclusion we drew from that was, people don't want to deal with that contradiction.
- [07:32]
A
Yeah. And that is like, we are. We are just trying to understand the environment so that we can change it in our. In our other work. And, you know, when I said that sometimes we correct people or like, we do push back, it's not because we want to Correct them. Actually, it's usually in service of, hey, we're having a certain kind of conversation and they've just introduced a fact that's wrong, that if other people in the group think that is true as well, like, we'll throw the conversation off. And so then we might say, well, actually this just happened. But also, like, we're not there to litigate opinions. It's important to understand for us, for the work that we do, that a lot of Republican voters feel strongly that tax cuts lead to revenue growth. And that's a pretty common assumption, actually, in sort of conservative economic circles. And so just because people disagree with it or that it's like a debate that happens doesn't mean that, like, that's the kind of. That's not the kind of thing we would ever come in really, and check. Now what we would do, as Connor said, is ask a probing question that forces them to sort of grapple with the thing that they're saying and to see how they defend it. Because if you want to make an ad that says that the tax cuts are increasing the debt, you want to know how many people are going to dismiss that out of hand because they already believe that actually tax cuts lead to economic growth. Our goal really is like, to learn. And I understand why people can get frustrated hearing things that are wrong. But I just, you cannot do this kind of work. And anybody who does focus groups will tell you you cannot do this work, like combatively or even in a way where you are sort of tipping your hand to the idea that you might disagree with them. I mean, I disagree with constantly. It's like you meet as many people in the world like you're going to disagree with a lot of them. But this is for learning purposes. Okay, sorry, we've, we've exhausted that. Go ahead, Connor.
- [09:40]
B
Sure. So I want to turn to some of the comments from our reading the comments video last week, guy named Benjamin Seltzer wrote, fair warning. My backgr background is in psychological assessment. So I'd like to hear a more in depth discussion of how you structure your focus groups from group selection to question protocols to prevent confirmation bias and other threats to the validity of our conclusions. So I'll let Sarah cook on this, but I would just start by saying that I don't think we are all that confirmed most of the time.
- [10:10]
A
I know. Yeah, if only our biases were confirmed. And in fact, the first groups that I did this, I'm going back to sort of 2018, and at this time I'm still thinking like can the Republican Party be saved? You know, can Larry Hogan primary Donald Trump and win or at least, you know, kind of do a good enough job, get a big enough percentage of the vote that he becomes a viable sort of alternative in the post Trump world? You know, like, it's just these were the kinds of things you're thinking and you go into the focus groups and my assumption is people don't really like Donald Trump. It was kind of an accident of history. There's like a bunch of big Republican field, they split the sort of normie vote. Trump's over there alone in his burn it all down lane. And like this is how it happened. Republican voters don't actually like him. That is not what I heard when I started doing focus groups. Lots of people liked him quite a bit. And you know, it's hard actually listening to people that I see how kind of red pilled they are or how much they live in an information environment that I believe is damaging and poisonous. And so yeah, I, I think we avoid confirmation bias pretty easily. And, and again, this goes back to, let me tell you something, that's a big difference between what we do and basically what everybody else does. If you're doing focus groups, you're being paid to do focus groups. Typically like you're a strategist of some kind or you're a camp, you're a campaign operative and you want to figure out what voters think about, you know, your candidate or about a specific policy issue. Right? You're in a gun advocacy organization. You want to know how people think about gun rights. I think that really can creep into a confirmation bias because you're trying to do is convince some client that like there's a campaign here we can run or people are really on your side if you just ask it this way. Or, or we listen to a lot of people and we think we can get to them on this part of it. That's why people do them, it's not why we do them. We are not being paid by a campaign or anybody else. We do it for our own edification, our own learning for ourselves. And so we are trying to avoid all biases possible. And Connor can talk about sort of how we select people. And look, you're trying to, you're constantly trying to disaggregate certain kinds of people, right? So you want to know like how do college educated voters talk about it versus non college educated voters? How do young people versus older people, men versus women? How do black voters think about it? How do Hispanic voters think about it how do white voters think about it? You know, like you're just trying to get all the different permutations of people. And so, you know, I like, we often. And one thing we don't do is put people who vehemently disagree with each other in the same space because then they fight and you can't learn anything. Actually, the one of the number one things we do is create a space in which like the first thing we say to people is, you're with a like minded group of people, like you all voted for Trump but rate him as doing a very bad job.
- [13:25]
B
And we also say it's start of every group, we're never going to see each other again.
- [13:29]
A
Yeah, right. You want people to feel like the. Now they all sign waivers that allow us to use their comments in the shows and stuff, but they are, they are granted a certain amount of anonymity to say what they think because that is what we want. So we are encouraging them to tell us their view. So I just, yes, I do think we avoid most confirmation bias. But I also will say this quantitative work, like you cannot take one group ever and say, oh, I know what everybody thinks now. Like we do a couple groups a week and we build up a body of understanding about different segments of voters from many, many, many groups. Not just in the moment, but also what we've learned now, doing these for seven years pretty consistently, which I think is unusual. Most people don't do this, most people don'. I don't think there's anybody who's done a larger qualitative survey than us this consistently. And so we're very sensitive more to things that are shifts because we know what a Trump voter who's voted for him three times sounds like. We are very confident that we are. When we, when we hear something that we think is an outlier, like our ears are up or we hear something that's a shift in the moment, our ears are up because we've listened to so many people and we're also. Connor, go ahead, talk about the selection.
- [15:02]
B
Yeah, this is another question that we got. How do you choose participants for the groups? Do you have an intentional composition of age or gender or LGBTQ or socioeconomic status, Race, religion, education that reflects the voting population? I'll start here. And the answer is we have pretty much all of that information when people come into our focus groups. But I think you see, even with high quality opinion polling with several hundred respondents, there are all kinds of flaws in how that can reflect the general population. And when you are dealing with nine or 10 people in a focus group. I would say the precision, especially in terms of people's political views, is the point. So we want to drill down on people who are all we do the Biden to Trump voters all the time, which is a very niche subset of people. And to be frank, we don't really get super precise about do we have the exact right number of Christians or exact right number of white people or black people because there is so much randomness that is involved when you are dealing with such a small sample size.
- [16:10]
A
Yeah. But why don't you talk just a little bit about your screening process because you work with our team here to do what we call double screen. And just to make sure like you talk to people before they enter the group to make sure that they're not, you know, you, because you get people oftentimes who you're like, this is not going to be a productive person in the focus group. Like, they would be too dominating in the group and too outlandish that it would throw the entire thing off. And so we do make sure what we, what we really want to make sure we have are people who can talk in the conversation and that, you know, will kind of play nice ish. In the conversation with others. Right?
- [16:56]
B
Yeah. So we work with a large market research firm whose entire, you know, their entire product is to go out and find people who are interested in various market research surveys. We've talked about this before, but usually those surveys are like taste testing cereal or, you know, different types of cat food. And so like, that's what they're, that's.
- [17:19]
A
What they will bring people in to do focus groups about. It's not typical.
- [17:22]
B
That's right. And usually it's not politics. But they will take our, you know, screener instrument, which is, you know, we want people who, you know, voted for Joe Biden in 2020 and voted for Donald Trump in 2024. And the people taking the survey, by the way, do not know what the answers are to this question that will qualify them for this group. That's very important. And so they'll. Sometimes it takes like several hundred people filling out that screener survey before we can fill a 10 person focus group. And before we get any of these people into our groups, we call them on the phone, we say, you know, hey, you available for this group on Tuesday afternoon? And then we kind of blindly run them through the screener questions again and make sure that actually you voted for Joe Biden in 2020 and you voted for Trump in 2024. And just get a sense of them and make sure that, like Sarah said, they're not going to be super dominant outlier personality in that group.
- [18:18]
A
So what else were people curious about? What else did we need to explain to them?
- [18:22]
B
So these are just some things that I was brainstorming with people in the office about. You know, what are the things that we maybe don't explain as much as we should on this podcast. One thing is why do we choose which groups to do when, you know, why do we spend so much time with these Biden to Trump voters? Why do we spend so much time with just Republicans or just Democrats? We spend so much time with these swing voters and our Biden to Trump is like the main proxy we use for that. Of course, there are non voters as well, and we've done some focus groups there. But this kind of approximates the person who's in the very middle of the electorate, who is not like a hardened partisan, and who comes the closest to deciding elections in this country. And those are the people who, I think we often, you know, people like to complain about the New York Times diner stories, but it's often like a viewpoint that conventional wisdom misses. And so it's one we try to bring as much as possible on this podcast.
- [19:19]
A
You know, this is a good point. So like, we do sort of mock the diner stories because it is, what it feels like is, ah, elites from D.C. and New York and L. A, like going on a. On a little mission to, you know, anthropologically study the people of Northern Pennsylvania.
- [19:40]
B
And he discovers America.
- [19:42]
A
That's right. That's right. You know, people get, I think they get frustrated with me sometimes because they think, oh, you know, Sarah has, has gives these people so much grace and doesn't want to, you know, call them names and stuff. And that is true. I am from a very small town in central Pennsylvania that voted for Trump by 76% or 74%. But it's high, it's high. The district. If I moved back home, my representative would be Scott Perry, the guy who really helped Trump do the election denial and tried to help him steal the election. And so, like, I'm from Trump country. And so it's true when I hear these voters, I don't feel like I'm studying them as much as I feel like I know this person. Like, I know what this is. And I'm thinking, what would it take to persuade that person of something else? What would it take to persuade that person to vote for a Democrat? And I think this is where People should be clear about my intentions, which is I am trying to figure out how to defeat the authoritarian slide in this country, which mainly means defeating Republicans. It used to mean a little bit less of that, but now basically that's what it means because Trump has fully taken over the party and so we live. And if I wrote, if I wrote a book just on the focus groups, it would probably be called Swingers, because that's who we talk to. Like, we are trying to find people who are in a persuasion window and that window moves around. Like in, you know, after 2016, it was, you know, in 2018, we were really talking about voters that felt deeply familiar to me also because these were like the McCain Romney voters in the suburbs. They were college educated and they absolutely didn't want to vote for Donald Trump. But the swinging, your voters have shifted around over the years that we've been doing this. And now it's much more likely to be maybe like black men or young, younger voters, Hispanic voters, white working class voters who voted for Barack Obama but are now firmly maga. Like, you know, just who and, or, or it's has a lot to do with the information environment that they consume. And geographically, you know, more and more like we are, we try to move around geographically because we really think that geography because then who, you know, dictates who you follow on social media, who you're sort of swimming in the same soup with of information, it starts to harden and congeal into sort of geographically specific. Like there's a reason that Perry county, where I'm from, like dominatingly 76% goes for Trump. And so anyway, I just, I don't even know where we were really talking about this other than trying to persuade people and figuring out who is persuadable is the work that we do. It is studying. So like we go and often talk to Democrats and we go and talk to hardcore Trump voters a lot almost to make sure that we have a really good sense of what is the difference between somebody who is absolutely not persuadable versus somebody who is persuadable. And then also like, who are they most persuaded by? Like, we, we know that there is a big chunk of people who like Trump who also really like Bernie Sanders and aoc. Like, that is as, there's like a, there's sort of a, a horseshoe thing at the bottom where people overlap. But then there's like the people up here. And this would be like in Virginia, a bunch of people who voted for Mr. Zippy up vest, Glenn Youngkin. Glenn Youngkin who are probably gonna vote for Abigail Spanberger. And, like, those groups of people are people that we study so that we can figure out, like, who are persuadable voters, where do they live? How can they be persuaded?
- [23:51]
B
Yeah. And I think part of this is we looked at the 2024 election results. We saw so many people who, including in our focus groups, who had maybe never voted for a Republican for president in their life.
- [24:02]
A
Yeah.
- [24:02]
B
I think part of the work we do here is using our imagination about what kinds of people are. Maybe have voted Republican all their lives and maybe, you know, never thought they might vote for a Democrat, but they will find, you know, they'll might find their way there in the future and what kind of person it might take to do that or to win back the people who didn't vote. I know that's a. That's a criticism that we get sometimes, but it's all important.
- [24:31]
A
Yeah. There's tough news in the didn't vote thing. I think there's a lot of people who think, boy, if we just turned out, you know, more of our coalition, we would have done better. And that's true if you know that who you're turning out is actually your coalition. But I think if the higher the turnout, the better it is for Trump or has been. And maybe. I think there were certainly Biden voters who stayed home because they were frustrated with things. You know, one of the things, like, we heard a lot of was people who were just frustrated about they voted for Biden because they wanted their student loans forgiven, and then they weren't and they were mad about it. I don't know. Those. Those are the kinds of things you learn about why someone say some. But we do look at skippers, and we do look at flippers, but it is all in service. Well, actually, I should say, like, it sort of serves two purposes for us. And I think this is where some people don't always understand that. For me, I am trying to do something about Trump and the power that he holds. I want to defeat him, candidates like him, and that means understanding these voters. Like, that's a big part of what I do. That's why I ran Republican voters against Trump. That's why right now, I'm doing Home of the Brave. And these are storytelling projects, because one of the things we've really learned from the focus groups is that the slick ads that you see on Twitter are often much less persuasive than a regular person explaining how Donald Trump's policies are hurting them. Like, that's A deep insight from the focus groups that we've learned over and over and over again, which is why we do a lot of storytelling campaigns. On the other hand, part of what I'm trying to do is I am very aware of the bubbles that people live in. And I think there's a lot of people who, when Trump lost, they were like, but everybody I know voted for Trump. And when Trump won, they were like, but everybody I know voted for Hillary Clinton. And I just think it's important for people to hear what their countrymen sound like and what information they're getting and why they think the things that they do. And it comes from a genuine interest in trying to figure out what we can do to save this country. That's it. That's why we listen to them.
- [26:52]
B
So one more question we got from folks is, how do you go about interpreting one of these groups? If you had 30 seconds to teach someone how to think about qualitative research, I think I know where I would start. But interested to hear how you would address it.
- [27:10]
A
Yeah, I guess I'd say two things off the top of my head. We get the most information in the first 20 minutes of every focus group where we ask the same question every single time. How do you think things are going in the country? And we just, we go around the group and everybody answers this question. And people will say they think things are going poorly, and they will explain why and what you hear from them. And what is always most interesting to me is what do people say unprompted? Right. When they're given a wide open space, what is it the thing that they reach for first as an expression of either what they're happy with or what they're angry with? Because that means that those are the high salience things for them. And then I also listen to what people don't say. Right. So we wait to see. Just recently, this, there was an example of this where we were trying to, you know, we've been doing a lot where we were like, I wonder how Epstein's playing with these sort of disaffected Trump voters, which are people who voted for Trump but think he's doing not a very good job. And Epstein doesn't. Did one person brought it up, but, like, mostly it didn't come up without prompting. Now, when we prompted people, they had opinions on it. But I think about this all the time. Abortion's a great example of this, too, where, you know, for Democrats going into 2024 and going into 2022, obviously the Supreme Court's repeal of Roe was like this enormous issue. But what we found was when you asked people the open ended questioning of how they thought things were going in the country, lots of most people didn't bring it up. Like it came up a little bit, but like, mostly people didn't bring it up. Now when we brought it up, we would. People would get really passionate about it. But we were aware, and I would talk about it a lot sort of in the press and other places, that it wasn't the highest salience thing. And in fact though, and that salience dwindled like it was much more salient for people in 2022 in their local races where it was an issue than it was when it was a presidential issue in 2024. And so we always were like, the thing about abortion is that, you know, unless you can make it high salience for people because when they think about it, they're like, oh yeah, I'm upset about that. But we also understood how complicated in qualitative. Like, I love answers like, well, I'm pro life, but I believe in a woman's right to choose. This is where I was gonna go, oh, was it? I mean, this is like the kind of thing where you're like, say that again. And then, and the first time you hear it, you think, well, that's nuts. And then you hear tons of people say the same thing and you, you learn how to drill down on that and be like, what is that person? What does this mean that so many people say they are pro life, but they believe in a woman's right to choose? That sounds inherently contradictory. And what they mean is, and you learn this as you sort of talk to people is that they are personally pro life, but they do not like these things being rolled back. You especially heard this from women in their 50s and 60s for whom this was the second time around in the debate around this. And like that is the kind of thing that you learn. And you especially learn it if you do lots of these and you ask lots of different people about them. Do you have another example, Connor?
- [30:33]
B
So pro life, but I believe in a woman's right to choose thing is a perfect example of what I was going to say, which I. My favorite thing about doing this stuff is just learning the contradictory nature of people and having that constantly being constantly reminded of that. And sort of my North Star when we're doing many of these groups is oath. It's not to quantify and decide what is the dominant strain of public opinion. It is to take all of the Various schools of thought that come out of these groups and say, oh, this is what it sounds like when someone is pro life but believes in a woman's right to choose. Or this is what it sounds like when someone more mundane example is like, is defending Trump's relationship with Jeffrey Epstein. And we can go on down the list. This is what it sounds like when someone voted for Trump for the first time in 2024. All of those things.
- [31:29]
A
But after taking a pass two previous times, this is what it sounds like for somebody to decide that January 6th was a false flag operation. Or this is what it sounds like for them to decide that it's a what about situation with the Black Lives Matter protests.
- [31:42]
B
Like, this is what it sounds like for a Republican to decide that they don't want to vote for Trump in the 2024 primary. Like, it doesn't mean they hate him. They're just kind of exhausted by all of these, by all the crazy lefties hating him. And that, you know, that's why they're, that's why about four people moved on to Ron DeSantis in 2024.
- [32:02]
A
But it's also how we learned things like, you know, being a politician was a job and so they wanted somebody who kind of looked the part of a politician and seemed like a good, you know, politician. And now they really don't like it when somebody seems like a regular politician. And so you learn to have your antenna up for things like, oh, they're all starting to say that Ron DeSantis seems like a regular politician. They're turning on him. Cuz that is the worst thing you can be right now. Because what they love about Trump is that they think he transcends politics. And so there's just this question about, like, how do we do the analysis is we listen a lot, we ask a lot of the same questions over and over. And so you really do build up an ability to understand how different voters talk about different things. And then sometimes you're like, okay, I'm now, I'm so aware now of how people are going to answer this. We need to, we need to figure out how to swerve. Like, we need to figure out how to ask something differently so that we can try to figure out if there's a different way that people would think about this differently.
- [33:06]
B
Sure. All right, that's all I got here.
- [33:10]
A
But okay, Connor, good job. Way to hang it. Way to hang in there with me while we, while we talked about this.
- [33:17]
B
This is like almost a full length podcast.
- [33:19]
A
It is almost a full length podcast. Well, we hope we gave you focus groupies, a little taste of the show and why we do what we do and how it works so that you're all ready when we relaunch the show in September. You know, we do have some, some races coming up. We got some good governor's races to talk about. We will go through probably some of the Epstein stuff. We've never stopped doing the groups. We just stopped doing the podcast. It's a bear to produce. It's a lot of sound, a lot of different voters. We're pretty careful about how we try to present everything, and we got to find great guests to talk to. But we're excited to get, get rolling again here and excited to be back with all of you. So thanks for sending in those questions. Thanks, Connor.