← The Marketing Architects
The Marketing Architects
Nerd Alert: The Statistical Significance Trap
00:08:49
Welcome to Nerd Alert, a series of special episodes bridging the gap between marketing academia and practitioners. We're breaking down highly involved, complex research into plain language and takeaways any marketer can use. In this episode, Elena...
Loading summary
Transcript20 lines
- [00:00]
A
Nerd alert. Learning is important, right?
- [00:02]
B
Yes, exactly. What a bunch of nerds.
- [00:05]
A
Nerd alerts.
- [00:06]
B
That's right. Marketing Architects. Hello and welcome to the Marketing Architects, a research first podcast dedicated to answering your toughest marketing questions. I'm Laina Jaspar on the marketing team here at Marketing Architects, and I'm joined by my co host, Rob demars, the chief product architect of misfits and machines.
- [00:23]
A
Hello.
- [00:24]
B
Hello. We're back with your weekly Nerd Alert. Every week, I'll take a deep dive into academic marketing research and translate its complex ideas into simple, understandable language for Rob, and of course, for all of you. Are you ready to nerd out, Rob?
- [00:35]
A
I'm feeling like it's a new year and I should not be allowed near a microphone. Elena. So let's do this.
- [00:42]
B
Let's do it. As always, we'll link the research we cover in the episode notes. This week I read a paper titled Statistical Significance and Statistical Moving Beyond Binary. This is by Blakely McShane, Eric Bradlow, John Lynch Jr. And Robert Meyer, published in the Journal of marketing in 2024. But Rob, first, kind of like a gut check for you, when someone says a result is statistically significant, what do you think that actually means in practice? Like, if you're a cmo, you know, hearing that in a meeting, what do you think you're being told? And hey, I spent a long time to be totally transparent trying to teach myself about. So, okay, just putting that out there that I had heard it a lot, but I needed a brush up on what exactly what it means. Yeah, not like what it means broadly, but the details. Right, right.
- [01:28]
A
You know in the Matrix, when Keanu Reeves says, I know kung Fu, that. Oh, man. All right, well, for those of you who have watched the Matrix, which is the rest of you, when Keanu Reeves says, I know kung Fu, that's how statistically significant gets treated in meetings. It sounds like someone unlocked some kind of superpower, but in real life, it's far less dramatic. All it really means is this result is unlikely to be random given the assumptions we made. But that's. It doesn't mean it's important. Right. Doesn't mean it's big. And it definitely doesn't mean you should bet your business on it. My opinion, that's probably more than you wanted to know.
- [02:15]
B
No, that's actually a perfect tee up for this episode. Yeah, that's. That was perfect. That what you just said is exactly what this paper is about. So the authors, they looked at how null hypothesis, significance testing, which is the classic. I know you've seen this equation, Rob. It's like P is less than 0.05 all the time.
- [02:33]
A
I seen that equation.
- [02:35]
B
You know what I mean? Like, that's kind of what people. If you search, if you Google statistical significance, you're going to see that that's really become like sometimes the default marketing research. We've all absorbed this idea that if your P value is below 0.05, then the effect is real and important. And if it's above it, then it quote, unquote, didn't work.
- [02:52]
A
My doctor told me my P value was way down, so I'm taking vitamins for that.
- [02:56]
B
Great. Should maybe double check what your doctor said. Okay. I wanted to break down statistical significance just a little bit more. It's basically a way to check if a result is real or just random luck. So imagine you flip a coin and it lands on heads nine out of 10 times. That could mean your coin special or you just got lucky. And statistical significance helps us figure out which is more likely. So if the result is. This is going to be hard to say this whole time. If the result is statistically significant, it means the chance of it happening randomly is very small. So lower P values mean the result is less likely to just be random luck. Now back to the study. The authors did a mini literature review of 33 recent marketing papers from prominent scholars. They found that all of them used traditional significance testing with the standard P threshold. All of them made reasoning errors like treating significant as proof there's an effect, which is what you were talking about, Rob, and non significant as a proof that there's no effect. Almost none. Follow the practices these authors will recommend, like reporting interval estimates, clearly treating results as continuous, or giving a real rationale for sample size. So a lot of these papers had these challenges. Rob, have you ever seen a result that was statistically significant but then didn't actually matter for the business?
- [04:10]
A
More times than I can count statistical significance, like you said, it just means you cross a very narrow threshold. Right. But it's almost like processed food. It's been refined into something that's like clean and measurable, but stripped of all nutritional value, you end up with results that are technically valid but practically useless. Right.
- [04:32]
B
All right. Now what I like about this paper is it's not just saying, you know, statile significance, don't ever use it again. They're providing some context and what you can do. They're not saying stop using it. They are saying, don't treat it as the stamp of truth. They argue for more of a toolkit approach. So use multiple methods and think of P values, confidence intervals, effect sizes, prior evidence, study design and data quality as all factors to weigh together. So the thing is, it's not that it's bad. But no single number should decide whether something worked. They also say that we should report estimates and intervals, not just yes or no significance, which feels important. Talk about practical importance, not just statistical significance. So is the lift big enough to matter for the business? Is probably a better question to ask and avoid dramatic language like marginally significant or approaching significance, which is basically trying to sneak a weak result through the side door.
- [05:24]
A
Asterisk?
- [05:26]
B
Yes, so. So what does this mean for marketers? First, be suspicious of single number certainty. I'd say this holds true for a lot of different areas of marketing. If someone says it's significant, we know it works. That's a red flag. Ask for what was the actual effect size, what was the confidence interval, and how stable is the result across time channels or segments. Second, push your teams and partners to talk about practical importance. A result could be statistically significant but totally useless in the real world. And third, zoom out from single test to cumulative evidence. The authors make this point. Scientific conclusions should be based on the total pattern of results, not just individual wins or losses. So in marketing terms, don't hang your brand strategy on one test. Look at multiple tests, things like mmm, your brand tracking and your longer term sales response. And finally, demand transparency in methods. I think we mentioned this before. Ask how the sample size was chosen, what assumptions went into your analysis, and did you look at alternative models to tell the same story? That's how you avoid getting seduced by perfect numbers that are actually fragile. So, Rob, I thought this was a fun study because learning more about statistical significance makes me feel smart. But also I think that you could take this takeaway and apply it more broadly. Do you think that marketers rely in general too much on single tests instead of trying to look at bigger patterns?
- [06:41]
A
I know I have certainly been guilty of getting a little overly excited about a single test and trying to extrapolate massive conclusions from it, especially if it's juicy or tells us an interesting story. So there is two sides of it. You can statistically significant yourself into vanilla pudding, or you could potentially hang your hat on one particular test and go, oh, this is amazing. We should build a whole campaign around it. And that danger is obviously overreach. So I think it's a balancing act for sure. I guess I would say both sides of it. One is allow that single test to introduce some energy into where your campaign could go, but then just make sure you've got some data to support it. And is it directionally interesting where you can deal with some probabilistic opportunities that might be showing itself versus getting into the real P dash colon power of two equation you were talking about earlier?
- [07:45]
B
Yeah, that's what it was. Exactly. No, that's good advice. It's a good point, too, about not you could use this. Try to use it for good, too. But that's not good either. Like, if you did a brand campaign and you're like, oh, well, our branded keywords went up this week, but your whole business tanked. Like, you gotta. You could use this for evil too, if you're just focused on one thing for sure. Yeah. All right, time for our Rob GPT. Reading this paper felt like watching a detective at a crime scene this There are clues everywhere, none of them clear, none of them solving the case on their own. But everyone keeps pointing at one tiny clue and saying, that's it, case closed. But real detectives don't work that way, and neither should marketers. A P value is just one clue. It's not the smoking gun. Good decisions come from looking at all the evidence together, not arresting the first number that looks guilty. That's it for this episode of the Marketing Architects. We'd like to thank Taylor de Los Reyes for producing the show. You can connect with us on LinkedIn and and if you like the podcast, please leave us a review. Now go forth and build great marketing Marketing Architects.