Nerd Alert: Why Your Advertising ROI is Wrong

Summary

Podcast Summary: The Marketing Architects

Episode: Nerd Alert: Why Your Advertising ROI is Wrong
Date: July 23, 2026
Hosts: Linda Jasper (B), Rob DeMars (A)

Overview

This episode tackles the complexity—and often futility—of proving the financial return on advertising investment (ROI), even under the best circumstances. Hosts Linda Jasper and Rob DeMars dissect the landmark research paper "The Unfavorable Economics of Measuring the Returns to Advertising" by Randall Lewis and Justin Rao, which reveals just how impossible it is to measure advertising ROI with the precision marketers crave. Throughout the discussion, they blend humor and insight while challenging conventional thinking around marketing accountability.

---

Key Discussion Points & Insights

The Illusion of Flawless ROI Calculations

[00:34 – 01:54]

• Rob humorously describes attributing his good day to various random factors, likening it to marketers attributing success to ads:
  "I'm having a 300% day today... the numbers are flawless. And the numbers are insane. Let's do this." (A, 00:34)
• Linda frames the problem: Marketers are under constant pressure to prove ROI, but most genuinely can’t.
• Rob admits, tongue-in-cheek, he could "prove" ROI with data if asked—while fully knowing it wouldn’t be accurate.

The Research: Advertising ROI is (Almost) Impossible to Measure

[01:55 – 06:51]

• The featured paper is described as a cult classic in measurement circles, with a striking original title: "On the Near Impossibility of Measuring the Returns to Advertising".
• The researchers ran 25 large-scale experiments (half a million to over a million people). Still, the margin of error on ROI often exceeded 100 percentage points:
  "The honest answer was somewhere between this lost us a fortune and this made us a fortune. Which is really useless for deciding did my ad work or not." (B, 05:15)
• To get truly reliable results, each experiment would have to be 9 to 60 times bigger—translating to tens of millions of participants.
• The variability of consumer spending is huge; most people spend nothing, while a few make large purchases—making small ad effects nearly impossible to detect amid the "noise".

The Challenge of Clean Experiments and Targeting Bias

[06:51 – 08:12]

• Most marketers don’t run clean A/B tests; they simply compare exposed vs. unexposed people. But those exposed are often inherently more likely to buy.
• The resulting targeting bias can be 30x larger than real ad impact.
  Linda references a famous report claiming online ads generated a 300% impact—almost certainly inflated by bias.

The Super Bowl Impossibility Theorem

[08:12 – 08:43]

• Measuring the impact of massive ad buys like Super Bowl commercials is especially fraught. Rob emphasizes that Super Bowl ads are about fame and brand-building, not immediate sales spikes:
  "Super bowl is expensive and you are buying fame on the largest stage... this is about fame... you're looking for memorability." (A, 07:05)
• Linda shares the paper’s "Super Bowl Impossibility Theorem":

  To measure Super Bowl ad ROI, a company would need to be big enough to afford it, but small enough for the ad to noticeably move the needle. That situation almost never exists.

Main Takeaways & Practical Wisdom

[08:43 – 09:49]

• Be skeptical of precise ROI numbers.

• Distrust simple exposed vs. not exposed comparisons.

• Measurement over many campaigns and the long run is more reliable than trying to parse single-campaign effects.

• Big, long-term brand effects are more meaningful than chasing immediate, measurable ROI from every campaign.

  Memorable analogy:
  "Imagine trying to hear one person whisper 'thank you' in a packed stadium during a touchdown. The whisper is real... but the roar of the crowd is so loud that proving you heard it... is basically hopeless. The whisper is a single ad’s short-term effect. The stadium is everything else people do with their money." (B, 09:27)

Reflections, Humor, and Theorem Talk

[09:49 – End]

• Rob reflects on the ongoing pursuit of ROI calculation:
  "It is such a battle... proving it is both art and science." (A, 10:19)
• Linda finds comfort knowing that even Google and Amazon researchers admit the difficulty, validating marketers’ struggles.

---

Notable Quotes

• "I am 100% confident that I can prove it. And I'm 100% positive I'd be wrong, but I'm 100%..." (A, 01:29)
• "Most people buy nothing and then somebody buys a couch. Now drop a tiny ad effect into that ocean of noise." (B, 04:13)
• "Be suspicious of precise ROI numbers. Distrust simple exposed versus not exposed comparisons. Targeting bias can swamp the true effect many times over.” (B, 08:43)
• "The Super Bowl Impossibility Theorem..." (B, 08:17)

---

Timestamps for Important Segments

• 00:34 – Humor & setup: Attributing good days to random inputs, opening analogy
• 01:55 – Description of the research paper and its significance
• 03:33 – Discussion on the spirit and abuse of ROI as a metric
• 04:13 – The problem: Measuring tiny effects amid huge consumer spending swings
• 05:15 – Even with massive experiments, ROI margins of error are huge
• 06:51 – Targeting bias explained; why “exposure” groups often mislead
• 07:05 – Super Bowl ad ROI: Fame vs. short-term sales
• 08:17 – The "Super Bowl Impossibility Theorem" detailed
• 08:43 – Study takeaways for marketers
• 09:27 – Stadium whisper analogy
• 10:00 – Reflections and humor on “theorem” and ROI

---

Summary

This episode deconstructs the widespread myth of precise ROI measurement in advertising. Drawing from foundational economic research, Linda and Rob show that—even with big budgets and clean experiments—for most marketers, it’s nearly impossible to untangle the true financial impact of ads from the noise of real-world consumer behavior. They advise marketers to be wary of overconfident ROI claims and to focus on long-term, pattern-based measurement—reminding listeners that, much like hearing a whisper in a stadium, the short-term effects of a single ad are real but almost impossible to isolate and prove.

Transcript

[A]

Nerd Alert. Learning is important, right?

[B]

Yes, exactly. What a bunch of nerds.

[A]

Nerd alert.

[B]

Right? Marketing Architects. Hello and welcome to the Marketing Architects, a research first podcast dedicated to answering your toughest marketing questions. I'm Linda Jasper on the marketing team here at Marketing Architects, and I'm joined by my co host, Rob DeMars, a chief product architect of misfits and machines.

[A]

Hello, hello, hello.

[B]

We're back with your weekly Nerd Alert. Every week, I'll take a deep dive into academic marketing research and translate its complex ideas into simple, understandable language for Rob, and of course, for all of you. Are you ready to nerd out, Rob?

[A]

I am so ready to nerd out. I'm having such a good day. I credit it all to my smoothie, the podcast. I found a great parking spot. And I mean, at the end of the day, I'm attributing 100% of the outcome to all three of those things. Like, I'm having a 300% day today, Elena. I mean, the numbers are flawless. And the numbers are insane. Let's do this.

[B]

Nice. Every marketer's dream. 300% return.

[A]

300% return on my day.

[B]

Yes. That's great. I actually had a day like that. Okay, today we're talking about a paper that sort of argues that most companies have no idea if their advertising actually works. So it's going to get a little intense. But quick question before I jump into the paper in particular, Rob, if I gave you a million dollars to advertise, and then I said, you need to prove to me that it made money, how confident are you that you could actually prove it?

[A]

Oh, I am 100% confident that I can prove it. And I'm 100% positive I'd be wrong, but I'm 100%. I can find the data to prove it.

[B]

Okay, so, yeah, you've got the confidence that you could put together.

[A]

I could put together the numbers. In a way, it'll look like it did. Amazing. But no. Well, can I actually prove that it made money? No.

[B]

Okay.

[A]

But I can make a good case.

[B]

Very honest answer. So the paper we're going to talk about today is titled the Unfavorable Economics of Measuring the Returns to Advertising. I just love that title. It's by economics researchers Randall Lewis and Justin Rowe from Google and Amazon Research, respectively. It came out in 2014, so this is an older paper. However, it's kind of a cult classic in the measurement world, and I'm surprised that we haven't covered it before. And its original title I actually, I told you, I really like that title. The original title is even better. It was on the near impossibility of measuring the returns to advertising. That's an awesome title. Should cut that one. So this is obviously still super relevant because every marketer is under pressure to prove roi. Got to prove to me that these ads are making money. And this paper takes the best case scenario. Huge budgets, millions of customers, perfectly run experiments and shows that even then you usually can't prove it. So before I dig into the experiment, how they set it up, what they found. Rob, what's your feeling in general on ROI as a marketing metric?

[A]

I think the spirit of it is perfect. I think the idea that you're, you're going to spend some money and you're going to get a return on it is important for advertisers to always remember. Right. Like roi. Like, why are we doing this? At the end of the day, we're not in show business. Right. We're in the business of moving product and we should be getting a good return on our marketing. I, I think at the same time it's way overused and way over indexed in terms of its precision and what it actually is. So I think the spirit of it is great and it's a good anchor. We should all remember we're in business and we're, we're about finding a return on it, but I think we abuse the acronym.

[B]

I agree with you. Like the spirit of your marketing should be generating like a positive return for your business. Marketing can grow your business. Totally in on that. But yeah, when it's taken to the extreme where we were believing that only marketing, that you could prove an immediate return on, which some usually isn't even accurate, like, that's when it becomes kind of a talk.

[A]

How are you defining the R in roi? Right. What does that return look like?

[B]

Exactly. So let's talk about what these researchers did. They ran 25 advertising experiments with major US retailers and financial firms. So each of these experiments reached at least half a million people, but most were over a million. Some challenges immediately arose. People's buying is wildly unpredictable. The thing the researchers kept finding is that the swing and how much any one person spends is about 10 times bigger than the average they spend. So to put this in more simple terms, if you think about a store and your average customer spends $7, the typical swing around that is $75. Most people buy nothing and then somebody buys a couch. Now drop a tiny ad effect into that ocean of noise. A generally profitable campaign might lift sales by say, 35 cents a person. So you're trying to spot a 35 cent ripple in waves that are $75 tall. And the results show it in their median experiment. So that had millions of dollars behind it. It reached over a million people, still had a margin of error on ROI of over 100 percentage points, meaning the honest answer was somewhere between this lost us a fortune and this made us a fortune. Which is really useless for deciding did my ad work or not. They went even further than this, though. So to reliably tell a wildly profitable campaign from one that just broke even, the median experiment would need to be about nine times bigger. And to hit the kind of precision that we teach people in business school, we expect in boardrooms, like telling a 10% return from zero, you need it to be over 60 times bigger. We're talking about experiments with tens of millions of people, often way more than brands are investing to see lift. This reminded me of the study. We've covered a lot on the podcast about how important budget is to the outcome of your advertising. Like which people don't talk about that a lot, but like the amount you spend matters so much for the return you're going to get on your advertising. But let's keep going. This actually was worse in most situations because a lot of companies, most companies probably don't always run clean experiments. I think it takes a really sophisticated marketing team to, to run a clean AB test and we have clients who do it really well. And I find it very impressive because it's not, it's not easy. Most people just compare people who saw the ad to people who didn't and they call it a day. The problem is that the people who saw your ad were already more likely to buy in some of these experiments. That's why brands are targeting them. So that comparison can be off by a huge amount. The researchers say the bias can be 30 times bigger than the real effect. So they point to one famous industry report that claimed a 300% impact from online ads, which is almost certainly that type of bias. So if you are running an AB test and you're sort of separating who saw my ad, who didn't, this is why we love like a geo test rather than these digital algorithms picking who sees it and who doesn't, because that can skew it. Rob, I wanted to frame this question in terms of the biggest advertising investment of all, because this one, right, should certainly impact your business.

[A]

Think about super bowl advertising on the Bachelor.

[B]

No, no, maybe, maybe 10 years ago, but they canceled their Last season we didn't even see it. How do you think a brand would ideally measure the impact of something as big as a Super bowl ad? Hmm.

[A]

Super bowl is always an interesting one, right? We debate that a lot. I think we all can agree the super bowl is not a coupon, right? You will see you run an ad. There's a good chance you're gonna blow up your website, especially if you're not prepared. I mean, you've got a lot of eyeballs, a lot of attention, but that does not equal a profitable ad buy. Really. This is about fame. You have much better ways to reach consumers in the near term, much more cost effectively in terms of promotions and other things that you could be doing. Super bowl is expensive and you are buying fame on the largest stage. And you're looking for talk value, you're looking for memorability. You're measuring things six months from now going, do they remember the characters and the distinctive brand assets that you were considering continuing to to anchor in that ad campaign?

[B]

Right. So the traditional sort of roi, immediate return of a Super bowl ad, you'd say it's pretty hard to quantify. You need to look at things much

[A]

greater than it's really hard to quantify. I think it's actually irresponsible if you're looking for short term gains in a Super bowl ad. This is about impressions, about brand building. Use it for that purpose.

[B]

Agreed. The authors would agree with you. So I want to ask that question because they have this theorem in the paper that they call the Super Bowl Impossibility Theorem. And what it is, is they say if you want to learn anything from your super bowl ad, a company has to be big enough to afford it, but small enough that the ad moves their numbers in a detectable way. And for most advertisers, no such situation exists. These brands are either too big to measure it or too small to buy it. So just sort of an interesting commentary on if you're looking for that immediate return from the super bowl, probably not the best idea. Your brand is probably not in a situation to be able to even see it definitively. So the whole point of this paper isn't that advertising doesn't work. It's that measurement on any single campaign in the short term is always going to be hazy. We probably should just be honest about that. So a couple of takeaways from this study. Be suspicious of precise ROI numbers. Distrust simple expose versus not exposed comparisons. Targeting bias can swamp the true effect many times over. Measure across many campaigns over time. Not one at a time because the authors feel that the signal in any single test is too weak. Patterns across the long run are much more trustworthy and lean on bigger longer term evidence and brand effects rather than chasing a perfect short term ROI on each ad, which we would definitely agree with. Now for Rob GPT. Imagine trying to hear one person whisper thank you in a packed stadium during a touchdown. The whisper is real. It genuinely happened. But the roar of the crowd is so loud that proven you heard it with a straight face in court is basically hopeless. The whisper is a single ads short term effect. The stadium is everything else people do with their money. All right, Rob, what do you think of that one?

[A]

I think it's great. And I learned a new word theorem. Use the word theorem in a sentence for a minute. Cause I want to. I want to put that one in my pocket so I sound smarter.

[B]

How about they teach you the Pythagorean theorem?

[A]

The theorem. We developed a theorem. No, I think it's start using that. I'm just going to try to figure out how to use that three times today. My theorem is my theorem for dinner. Is that. No, I. You know, it is such a battle, right? I mean, we all want to be able to figure out the holy grail of calculating roi. And it makes sense that we spend a lot of time talking about it. It's an easy handle. Everybody knows it. But man, proving it is both art and science.

[B]

Kind of nice too, to hear researchers from Amazon and Google saying like, hey, this is impossible to do. That feels kind of good. Like even they're like, this is really not an easy thing.

[A]

So even they're challenged by the theorem.

[B]

That's it for this episode of the Marketing Architects. We'd like to thank Taylor de Los Reyes for producing the show. You can connect with us on LinkedIn and if you like the podcast, please leave us a review. Now go forth and build great marketing. Marketing architect.