← Mobile Dev Memo Podcast
Mobile Dev Memo Podcast
Season 7, Episode 6: Measuring the long-term effects of brand advertising (with Carl Mela and Ross Link)
01:00:07
Loading summary
Transcript63 lines
- [00:02]
A
Mobile game developers no longer need to Pay up to 30% in major app store fees. With Xsolo Webshop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards and analytics. Start today@xsola.com that's x s o-l l a.com or use the link in the episode show notes.
- [00:30]
B
The problem is that distinction needs to be drawn between the competence of the economists and the correctness of their analysis.
- [00:41]
C
Hello and welcome to the Mobile Dev Memo podcast. I'm your host, Eric Suefert. I'm joined by two guests today, Carl Mila and Ross Link. Karl Ross, welcome to the podcast.
- [00:51]
B
Thanks. Good to be here.
- [00:52]
D
Yeah, thanks. I'm delighted to be here. I've listened to the podcast. It's excellent. I'm honored to be asked.
- [00:57]
C
Well, I'm very grateful that you agreed to join me. So, Carl, you were actually the first guest that I've brought on as the result of request. So someone requested that I bring you on and, you know, obviously you're familiar with your work. You're a very prolific academic in the space. And I said, yeah, that's a great idea. Can you introduce us? And he said, I don't know him. I just think he'd be great to be on the podcast. So, fortunately, we do have a number of mutual acquaintances, so. So Garrett Johnson, my friend, connected us. But also you worked with Julian Runch at Duke. He told me that you were the one who brought him out there. Is that correct?
- [01:36]
D
Yeah, yeah. He's fantastic. It's been great interacting with him. I also loved your episode with John lynch and JP Dube. Privacy. Very informative.
- [01:45]
C
You've had some, yeah, fabulous guests.
- [01:47]
D
The question you have to ask yourself, though, is who recommended me? Was it a friend or a foe? Hopefully it was a friend. And someone didn't want to take your ratings.
- [01:55]
C
Well, I didn't interrogate. I imagine just someone who has deep respect for your work. And then you recruited Ross to come along as more of, like, the practitioner's voice. So I'm very, very happy to have you both. Maybe just to kick off the episode, you could both introduce yourselves to the audience and maybe. Carl, we'll start with you.
- [02:13]
D
Yeah, sure. I'm my PhD marketing from Columbia. I've been a Duke professor since 1999. I served as the executive director of MSI and I'm on the board of the Advertising Research Foundation. My early work was on measuring brand equity, which we're discussing today. And then I've shifted more into E commerce and digital advertising and those two streams are coming together in commerce media. I know you've covered that in your podcast, so that's another area of interest. I should also add that I'm from Boston, but I met my wife in Seattle. So I'll let you figure out who I'm rooting for this weekend.
- [02:47]
C
Well, yeah, we'll keep that a secret. And then Ross, how about yourself?
- [02:50]
B
Yeah, sure. So I'm the CEO of a company called Marketing Attribution. We're a measurement firm. We've been around for about six, seven years. I started out, I got my bsnor from Cornell, MBA from University of Chicago, worked at Booz Allen for a few years in their system strategy department, worked at P and G in their media optimization group. And then I had a company called Marketing Analytics. I ran that for 20 years. One of the first marketing mix modeling companies sold that to Nielsen in 2011 and then ran Nielsen's marketing Mix business globally and their emerging MTA multi touch attribution business. Before I started marketing attribution in 2017, I guess I've been around doing eight years now. I guess almost nine.
- [03:24]
C
Maybe just kind of before we get into the questions, what do you feel about the kind of renaissance of media mixed modeling? Because it, you know, it's become. I, you know, and maybe tell me if I'm wrong about this observation too, but it seems like it's become more relevant again, especially with like a lot of the digital first advertisers who historically hadn't used it. What do you make of that? Of the kind of renewed interest?
- [03:43]
B
Well, I sold my first company Marketing analytics to Nielsen in 2011 mainly because I thought marketing mixed modeling was going to die. I thought the way we do measurement was with multiple distribution at the person level. It's definitely the way I would prefer to do it at the person household level. But in the middle of some tests we were doing at Nielsen, it was called Digital Media Consortium. I, I recruited a bunch of companies, Coke, P and G, Pepsi, you know, last company's Clorox and then you know, other companies as well. We had, we had Crux was part of it before Salesforce bought them. But the idea was to test marketing mix modeling multi touch attribution expert, some match panel tests and try and find out what kind of things worked best. And in the middle of the test Google said oh, because the Google was part of it and they were giving us very useful information including person level impressions. They said oh guess what, we can't share that anymore because we're very concerned about data leakage. Well, it turns out that data leakage really means revenue leakage because if you're giving out your personal level impressions, your Google or your Facebook, you're giving out those, you're letting you know companies like us get those personal level impressions. You are releasing your targeting list. That's your targeting list that you just gave out. I gave those impressions, those people because they, I studied them for a while. I know they are in the market for a luxury automobile or something. So that is what they're really concerned about. So that's when they stopped giving out personal level impressions. And that was kind of the beginning. It also was around the 2016 election, Cambridge Analytica and all that stuff. But it really was, the thing I took away from it was this isn't really privacy. This is self interest from the platforms that then kind of merged into privacy. And so that's why I don't believe we're going to get personal level impressions again from Google that we can then merge with Facebook. And so that's why I really believe that was kind of the beginning, the end. And we all are coming back to market mix modeling, which I love. And it's my second best thing after person level stuff. But it does work and I think that that emergence is going to. Resurgence is going to kind of stay.
- [05:32]
C
Yeah, Fascinating. All right, so we, we kind of workshop these questions. I'm really happy with the way the questions came together. So. So what? Just to kind of give the audience a sense of how we'll conduct this. So, so usually when I have two people on, I'll try to have like, I'll identify a main respondent for the question. Of course, the other guest is more than free to, to kind of also chime in, but I'll have kind of the person I've identified as the, as the primary respondent and then, you know, we'll get thoughts from the other person too. So the first question we'll start with Carl, and it is what do most companies get wrong about brand measurement? Both from an analytical but also a conceptual standpoint.
- [06:07]
D
It's a great question. I have several points to make on this dimension. Probably the first one that should be no surprise. Brands like Coke and Apple, they weren't built overnight. It took years and years to build these brands. Why does it take so long to build a brand? It's not just changing somebody's belief about the product attributes, but you actually have to change their preferences. Changing preferences is hard. It takes A lot of work. It takes time, it takes repeated exposure, takes giving people compelling reasons for purchase. You know, discount isn't a compelling reason for purchase as an excuse. And it can't be measured with a small amount of data or an experiment. It's something that takes a very long period of time. I think the second point I want to make on this domain is all too often I see brand strategy follow from what can be measured instead of basing their measurement around their strategy. And this leads to some bad strategies. So the canonical example of this in the 1980s was the release of store level scanner data. Prior to that, sales were measured through biweekly inventory counts in the back room. So they literally had someone go in the back room and figure out how many cans of beans had sold. In fact, if you ever heard from bean counter, that's literally where the term came from. And so if there was a discount for a given week, it would be sort of aggregated away over several weeks. Then scanner online, or excuse me, digital scanner data became immediately available. So companies could easily see the sales lift from a discount and so they started discounting a lot. And advertising is. Advertising effects are hard to measure. We know advertising elasticities are low, but advertising is intended in many instances to build brands. And that's not something that happens on a weekly cadence. And so what happened was companies shifted budgets from advertising to discounting. And so that's when I started thinking about tools to measure long term effects because it's pretty clear that brands were shooting themselves in the foot. But again, these things are hard to measure. More recent incarnation of this phenomena is the advances in measuring direct response advertising. It's very easy to do AB tests, so ADs are measured in terms of direct response. In fact, I think one of your past speakers was talking about one of their main KPIs as being clicks. But clicks and even purchase intent is not brand associations, it's not brand beliefs. I mean, I could drive clicks just by advertising a discount. So basically what's happening is because people can measure direct response so readily and so easily, they've shifted ad budgets to direct response. And I think that is, that's an issue. And that's again why I've worked on some of the long term measurement tools that Ross and I will talk about shortly. A third point, I think advertisers are very tied up for similar reason in targeting, which I think is wonderful, but it underestimates the value of reach. And so let me give you a very specific example. Imagine you have a carbonated beverage brand and you're advertising that carbonated beverage brand. Sort of the best person to reach is the person who's trying a carbonated beverage for the first time on that day. You want them to be activated around your brand when they make that purchase because the customer lifetime value becomes enormous. But you don't know who's just about to start purchase for the first time. I mean, there may be some signals by like age, but not everybody's young when they first drink a carbonated beverage. And so, you know, some of the execs I talked to in that industry say, hey, a really sensible strategy is to advertise to everybody. And I know that if over a lifetime, you know, if someone's roughly 300 days a year, what times 30 years, looking at 9,000 days, 10,000 days, I know that on 9,999 days my ads are wasted and only on one day it's relevant, but it more than offsets the loss. So I think that again, this notion of reach versus direct response, something that's a concern. And one last point in this area, and that's the difference between marketing mixed models and media mixed models. I think just I want to clarify this before we get into it. Media is only a component of brand strategy. And I have a paper, I think it was journal Marketing Research in 2010. It looked across 25 consumer packaged good categories, about 709 brands over five years. And we found that this short term and long term sales elasticity is 1.37 for product and 0.74 for distribution. In contrast, it was 0.13 for advertising and 0.0 for discounting. Now ironically, like if you, if you walk the halls of many research companies, a lot of money is being spent on ad and price measurement, but not nearly as much as being spent on things like product and distribution in terms of developing brands. And again, I think that relates in part to, to this, this measurement. So how does some companies solve this? They have one phase where they use the marketing mix measurement. They use this to figure out the optimal media allocation and then they use a media mix measurement model. So summing up, brands aren't built overnight. Strategy should follow from what can be measured instead. The other way around. Reach is underappreciated and underweighted in a lot of strategies. And we need to think about marketing mix modeling instead of just media mix modeling. Ross, I don't know if you have anything to add on that one.
- [11:30]
B
Well, no, I agree with all that. I can't. I think most people can't do actually a media mix minded model that only has media in it. So I think we always, even if we're only focusing on the media, I believe everybody will put something in there about price and promotion or maybe they'll just take Nielsen's estimates as no syndicated data vendors estimates is gospel. But yeah, you do have to control for those other things. But yeah, we do both types of models and marketing makes miles preferred to us definitely.
- [11:58]
C
So you talked about, you know, this kind of shift to direct response just as a function of like well that's more measurable. And you kind of made a point earlier in your answer about the strategy is essentially as a mistaken approach. The strategy was downstream of, of what was measurable. Right. So you had companies building strategies just on the basis of like what they could measure and then that, that would kind of, that that would kind of result in this general shift towards like well we're going to spend on direct response because we can measure that. And so the measurability of that medium becomes a constraint on our media spend. Is that kind of, are those the same thing? Is, is that a phenomenon? Is that a phenomenon of that, of that mistake?
- [12:38]
D
I think so. And let me give you another example again related to a recent episode of yours. Fundamental question. A lot of companies are asking themselves that AI make ads as well as people. You've had some groundbreaking soon to be seminal studies on that very point where the answer is being sort of longs people don't know it's created by AI. Yes, AI generally does better than people, but again the measures are things like clickability, but that's not brand attitudes and brand associations. So the jury's out. Part of building a brand identity is relentless consistency about what you believe your brand to be like. AI is not tuned to that issue. It's not clear that it can generate executions in a manner over a long period of time that reinforces brand messaging. And so like the jury's out as to whether or not I could use AI over the long term to augment branding. Maybe I can, maybe I can't. But the point is like we don't know because we've been focusing on short term measures.
- [13:39]
C
Right? Well that's, that's the kind of fundamental attention. Did you see that? There was that article that was circulating a few months ago, but it was Mendelez. I might be mispronouncing the name, but they, they have, they make all these different confections. So they have like chocolate. It's a, that's right. Mandalay And I'm forgetting all the relevant details of this article, but, but they had worked with like Deloitte or something to build like a model that could supplement their creative output, but it was trained on their own internal data. Right. So like instead of surrendering his functionality to Facebook or whatever to, to then, you know, we've seen some kind of scandalous results of that. There was a case where Facebook was circulating an ad or they were fighting an ad that they had built on behalf of a brand and they were using like an old woman as a subject of the creative. When the, the brand is a men's T shirt brand. So but anyway, Mondele had built their own model for generating creative which was trained on their own data. So it should be sort of more attuned to their brand sensibilities. Do you think that's a possible solution there? I mean, not everyone can do that, but if you can, is that a possible solution?
- [14:38]
D
I think it's a step in the right direction for sure. Right. We need, we need to think about brand guardrails when we're developing an AI strategy towards creative. Yeah, absolutely.
- [14:46]
C
Got it. So, Raj, well, kind of next question is for you. What are the most common internal political issues that you see arise with marketing measurement within organizations? Who are the less obvious stakeholders for marketing measurement and how do needs sometimes get misaligned with measurement?
- [15:02]
B
Well, those questions are linked up with things Carl does too, quite a bit. But. So I think the biggest political issue in measurement is short term incentives of brand managers. Their incentive system is set up to drive volume for the next one or two years and the owners of the company want the brand to be successful for, you know, decades. So that's, that's a misalignment that causes them to, you know, focus on volume for the next couple of years and not necessarily building long term brand loyalty. That's why I've worked with Carl before on doing long term effects models, is to try and try and change that. So that's the biggest one. That's one that's been around for a long time. Another one that we have is that, you know, we work. I will say that there's less political conflict in what we do now than years ago though. I mean there, there were, there were times when we first started doing this. I mean, I've been doing this since like 91. So at the beginning there were many people that just questioned the whole value of quantitative marketing measurement. So that was the problem. There would be marketing managers who just don't believe it. It's A soft science. You can't really measure this stuff. Truthfully, when I was in business school, that was, I believed you can't measure, you're not going to be able to measure advertising, maybe promotion, but you can do it. So that, that one's kind of gone. But we do have some. So the agencies, the ad agencies, they also, they have an incentive mainly to drive the top line because, you know, same with the brand managers, they're not really responsible for all the cost, for the cost of goods sold. They can't control all that stuff. So you can't necessarily compensate them on profit. So they are mostly incentive to drive the top line. And measurement firms, if you work through it, if somebody tells you I want to optimize profit, profit, it's very clear how to do that in some sort of quantitative optimization. You know, you, you consider all the diminished returns you have all the response curves and you tell them to keep pushing, you know, spending on advertising until the next dollar doesn't drive enough revenue, subtracted cost, cost of goods sold to justify the cost of the media. It just, it's, it's very clear where. How to optimize profit for us as a measurement firm, how to optimize revenue. I'll tell you how to optimize revenues. That's really, it isn't real literally what people really want to do. But they'll say that I just, I'm focused on revenue. Because you focus on revenue, you spend a bazillion do oh, you got and then borrow more and spend that too. So what you really need is you need some sort of weighting between the two and then judgment comes in. So as a measurement firm, it's much more clear how to optimize profit. We tend to focus on that. And so therefore the agencies generally want the client to spend more money than we tell them to. So that's a conflict that we run in on that one. Another one is kind of near and dear to my heart is like, who does the measurement? Okay, we're a measurement firm and then everybody's got, all the different people that do measurement will have their own incentives and biases and so on. You know, we measurement firms, their typical biases, not me. We don't do this. We focus on straight shooting. Just tell people the truth. That's how you stick around and keep doing this for 30 years. But there are a lot of firms that if you tell your incentive system is to tell the client the highest roi, to ring out the highest ROI you can get from that data, give that to the client because the people that are paying you generally that's what they want to show. It's either brand management, they want to show their management or it's the agency or it's the platform. Pretty much almost everybody that's our stakeholder, except maybe the CFO wants us to show a sometimes somewhat arguably unreasonably high roi. So that's the measurement firms platforms, they also want to show a high roi. And they'll do that using a couple methods that tend to show high roi. I don't think they're all, you know, being evil about it. They're using methods that some people tell them is good. But some of these methods do tend to give you a higher roi. One is match panels. If you just, just look at a very simple here's people who are exposed, here's people that were not exposed. A reasonable person would think well that's a good way to do it. Problem is with all the high sophisticated targeting mechanisms that there are today, people that were exposed were not exposed for some random reason, they were exposed if they're likely to buy. So then you compare people exposed, people are not exposed. You know, oh wow. Here I looked at a whole bunch of people that saw a loan ad recently because I mean they watched a loan television program or something and because of that I hit them with an ad. Well, geez, that's why, you know, that's why they took out more loans. They were already more likely. That is a big problem. And that is in most platforms measurement system, it's in there. It's not in some of the best ones like say Google's for example, I think they do a very good job with something called ghost ads. It's a complicated way. It's in an auction. Again, in their defense, I don't think everybody's evil out there. But auction. Trying to do a randomized control trial in an auction system is quite complicated. I think Carl knows about that really is. And it's like when do you substitute in the control product before the auction or after the auction? You have to do it after the auction. And so Google does that most people do it before because it's so complicated. So that's. And then, and then. So it's between the match panels, the sort of biased auctions, measurement platforms tend to give biased estimates. So that's another thing about who's doing the measurement and what is their incentive system. And then you asked about less obvious stakeholders. We have a lot of stakeholders like at the beginning, you know, I've been Doing this for a long time. So things have changed. Beginning it was not everybody, the CFO didn't really care when we first doing this stuff. But now everybody cares and pretty much everybody supports quantitative marketing measurement and they support marketing mixed modeling too. For whatever reason either because they understand the thing I talked about about giving away your data, it's giving away your targeting list. And for whatever reason they don't trust the platform. So marketing mixed modeling is very accepted. So our stakeholders include key one is the insights of the analytics team, them the brand management, going up to the CMO, media buyers inside the company finance and the CFO. And we do work with the C suite CFO, CMO. I know the CEO at some of our biggest like $10 billion PL companies are aware of the stuff that we do trade promotion management because we do marketing mix modeling. I've been doing that for a long time. We try to, we try to model everything, price and promotion and have it all be reportable. So therefore we work with trade promotion people at some of our clients revenue management on pricing. We do pricing models along with it and the ad agency, the platforms and the platforms themselves also they pay us directly. We do a fair amount of studies for all the major, not all of them but like you know, maybe five, six of the biggest platforms to to do best practice studies for them for the benefit of their client using like you know, credit money or something like that. But then they have their, there's another stakeholder and finally our favorite stakeholder procurement. Just love them. But with this many stakeholders, if there's any less obvious ones, I don't want to know about them basically what I'd say about that. And I think he had one more question which was about alignment, about misalignment or something like that. Yeah, you can get misaligned when you're doing, when you're a measurement firm like us, marketing mixed modeling. When you work going from one client then you go another another brand. It's kind of, it's kind of the same thing. Okay, guess what? Today it's the same thing as yesterday. We get all your marketing. We put it in the best model we can do with the most granular data we can do and we report on the ROI of it and recommend what you do next year. So it's kind of the same thing every time. But it is important still at the beginning to ask the client what is their needs, what do they care about? Oh, we're moving into a new sales channel. Oh, we have new product introduction. Oh, we're Concerned about this competitor, things like that. You do want to know what, what is your, besides the normal stuff, what is it? And then as long as you do that, then generally your measurement is aligned with the needs.
- [21:58]
C
Carl, did you have anything to add?
- [21:59]
D
Just three, some reinforcements. First, I want to give a shout out to Garrett Johnson because Ross mentioned ghost ads. I know Garrett's been a guest on your show and I think that's a really nice thing that every marketer should be familiar with that's doing digital advertising. I want to underscore his point about the challenges of an independent analyst giving an honest answer to firms who have a vested interest in spend. Because the number of employees that you have and your power in the company is correlated with your annual budget or spend. And so if your budget gets cut, your power gets cut. So there's a tremendous incentive for a lot of these companies not to want to listen. In fact, I co chaired the American Marketing Association School of Market Research for a couple of years and we always did a top 10 list where we surveyed the attendees on what they saw as the biggest problem in market research. Number one, and not by a small amount every year, was their managers telling the design studies to show that whatever initiative they had worked. And that's a real danger. Finally, I also want to echo this point about brand managers. Unless you have a long term measurement system that we're going to get into shortly in place, I tell my students who are brand managers, I say the best thing you can do is ruin your brand. Just cut ads, discount the hell out of it, move a ton of cases. Yeah, so what if people now think it's a cheap brand you've harvested, leave that mess to the next person. Because if you go in there and you invest in the brand and it's brand equity and do a lot of brand building, that's a cost. It takes time to build brands. So you get the cost, but you don't immediately get the benefit or you're competing against the other brand manager. Look, that's a really big problem, the one year tenure of brand managers. Unless you have a system in place to deal with this, and again, we'll give some examples later on how to manage this.
- [23:56]
C
Yeah, I feel like marketing, I mean, especially when you, when you go to big firms, like, I mean, imagine the kind of firms that Ross works with. Like the marketing dynamic is probably so rife for issues of principal agent problems because it's like to your point, Carl, it's cool to spend a lot of money. It's really fun to be invited to super bowl, you know, suites and get Taylor Swift concerts and to have your TV ad being talked about by your family and friends. It's really cool. It's a really nice thing. And so, you know, whereas, you know, shareholders maybe would have a different opinion if they were more attuned to the, the kind of underlying performance. Right. And so like there is, I guess there's a little bit of an alignment just naturally, inherently in that. But like, then it's a question of like, who wants the bad news. Right? Because if the measurement's going to give you bad news, the external, you know, unbiased measurement company that you're working with is going to give you bad news and the marketing team is the one paying them. Then you might run into misalignment just naturally, right? So that I've seen a lot of cases where the CFO steps in and says, I don't trust that I'm going to be the one interfacing with the measurement company. I'm going to pick the measurement company, I'm going to audit their performance and I'm going to be the one that decides, you know, whether you're spending the appropriate amount of money. But just one, one quick question I want to follow up. Ross. So you talked about agencies and kind of managers kind of having the same, like a little bit of the same sort of bias to just driving that top line revenue. Because with agencies I think it's pretty easy to understand because that's how they get paid. Right. And then with marketers it's, you know, it's more just that what I just talked about is cool to spend a lot of money. Could you maybe just expand on that, the idea with the agencies a little bit more? Because I think that's something that seems obvious, but it kind of sneaks in more nuanced ways than maybe some people understand.
- [25:47]
B
Well, I mean, the way that's coming to my world, even in the last few weeks with two different platforms, actually maybe the last couple months with two different platforms is that we did a best practice studies for them and then identified, you know, different types of ads that have the highest ROI or, you know, yeah, highest roi. There's different measures of roi. Whether or not you include margin in or cost of goods sold or not depends. Sometimes that's confidential. But you know, we focus on roi. And then the, well, this was, this was actually the platform. But, but the agency does the same thing, which is that they want to drive revenue and therefore they even will question whether they should be focusing on ROI or not. I've seen, I've seen that. They say, well why, why don't you tell us which ads were most effective and we'll do those. Effective? There's, there's, it's kind of getting in the weeds here. But effectiveness in our world is generally defined as the incremental revenue per impression or thousand impressions regardless of how much those impressions cost. Then you add in the cost of the impressions and then it's incremental revenue divided by media spend and then you might, might multiply the revenue by the margin to get the actual incremental profit. But, but the agencies and the platforms are saying are asking us, why don't you focus on effectiveness and forget about, you know, not roi? Because you're going to stop, you're going to stop spending way too soon if you focus on roi, but you don't really maximize roi. We do these market mix models, then we put them in an optimization like a nonlinear optimization, big ones that we've worked with professor at Stanford Madeline Adele on an S curve optimization algorithm specialized for S curves. But what I'm saying is that you don't actually maximize ROI after you're done with your marketing smile. You maximize profit or revenue. So anyway, I guess what I'm saying to you would ask him to get a little bit more into the details by the agency. It ends up, up, you know, having to convince them that you even do need to look at costs. You know, you can't just maximize the fact that forget about the costs. Somebody gives you a billion dollars of. Spence. Yeah, I mean our clients do spend, someone do spend a billion dollars on advertising globally, you have a billion dollars to spend. You don't just, you can't forget about costs. What you have to do is you have to maximize the income. If all you focus on revenue, fine. Focus on incrementality. Forget about margin. You still have to get the maximum incremental revenue per dollar. So you get the most revenue for your billion dollars. But anyway, that's, it's, it's, it gets in the weeds about that kind of stuff with the agency.
- [27:57]
D
Yeah, CMOs love ROI because they can walk to the CFO and say, look, these numbers are eye popping. Even if the marginal profit is negligible.
- [28:05]
C
Yeah, let's get real.
- [28:08]
A
Half the time you're not sure if your campaigns work. You just kind of hope they do. Attribution numbers look good. But what if those conversions would have happened anyway? Incrementl helps you find out for real. No tracking people, no guessing games. Just clear answers on which campaigns actually move the needle. If you want to stop guessing and start knowing, go to Incremental. Because good marketing isn't about getting clicks, it's about getting results. Check them out@ Incremental.com that's I n c r m n t a l dot com or click the link in the show description.
- [28:45]
C
Carl so you wrote kind of a canonical piece. If brands are built over years, why are they managed over quarters? And in that piece you argue that a disproportionate focus on short term data supports a constant cycle of price discounts that may end up eroding brand health and pricing power by increasing consumers price elasticity, which could be economically destructive in the long term. You proposed two measurements of brand performance that firms should incorporate into their strategy. Quantity and price premium. Can you explain what those are? And how would you suggest that firms adapt these metrics in their broader measurement portfolio?
- [29:18]
D
So let me start with the last question because it's easiest. How should firms adapt these metrics? To be blunt, call Ross or someone like Ross. The reality is the concept's actually very easy. But you know, when you talk to someone with 20 years of experience, they know the benchmarks, they know the errors, they know the norms. There's a lot of trial and error over the years and so much money is spent on advertising. You really need to be with firm that does things correctly. And to Ross's earlier point that you could trust, they'll tell you if something isn't working. In terms of the example and what this all about, let me give a very specific anecdote that got me thinking about this because when I was an mba, I remember learning about, you know, how do I evaluate the profitability of a discount. So I was taught you just look at the lift. If the lift is really big, tout way sort of loss in margin, then the promotion looks great. Go ahead and do the promotion. But let's think this through. See I sell a lot more at smaller margins, so overall my profits are up. But let's suppose I am discounting. What happens if I do it next month? Next month consumers have learned to lie and wait for a discount. So the baseline sales, where the sales and things are off promotion are very low and the incremental response is even bigger. So if I go through and I do the things I was taught when I was UCLA MBA on evaluating profitability promotion, the incremental lift is even higher and so the promotion looks more profitable. Next thing you know you're in a death spiral and so what you really need to do is take a look at these baseline sales and these promotional lifts. And the promotional lifts are basically price elasticities. Higher price elastity, higher promotional lift. So the idea is, let's go run a marketing mix model. And in the marketing mix model gives you two very, very useful pieces of information. It gives me the baseline sales. The baseline sales is typically something like an intercept in the model. It tells me if the marketing, all the marketing tactics and the price are the same, which brand sells more. The idea of brand selling more is a measure of brand power. And then I could look at elasticities again, how sales change with price. And the lower elasticity, the higher you can raise your price and therefore the higher margins you have, you think of luxury goods. So if a brand does really well, hopefully you see high baselines and low price elasticity. So you could track these over time. So in one paper I wrote, I remember doing these buy plots where we looked at the path of brands over five, six years where we just plowed these two things and we saw real movements. And interestingly enough, as we'll talk shortly, those movements can be tracked back to your marketing strategies. So one of the things again, I found repeated in my research and I was very relieved when others found the same thing. Ross has more cases than I do. Not surprisingly congruent with the example I gave you a moment ago is that a lot of discounting tends to show up in subsequent periods as greater lift and lower baselines. Now, advertising tends to lead to higher baselines. So linking this back to the product manager example, if one of my students took my advice and trashed their brand, what would you see by the time that person's review is being written up, you would see baseline sales go down and price last needs to go up. And so even though they've sold a lot of cases, the manager is going to say, well, you trashed my brand. And this has created a very powerful incentive to get companies back on track to managing brand health over the long term. Ross, do you have anything you want to add on this? Because I know you've done so many more of these studies than I have.
- [32:57]
B
Well, I mean, this is a classic paper. One of our clients was in it. Steve Gary from Clorox. No, it's a classic paper. I mean, Carl's. The kind of long term effects, honestly. And so no, we, I don't think it was this paper that the algorithm is that about the modeling the base volume and the priceless. The only thing it came out of there I Think it was a subsequent paper, but no, that is classic. And then we built our algorithm on another paper of Carl's later on.
- [33:20]
C
Well, so what I loved about that. So in the paper you talk about, you give it the example of La Coste, right? And so like it's, it's interesting because I've always known Lacoste is like that kind of premium tennis brand. I, I didn't know that, that they were, it was acquired. So the story of it is it was this, this French brand that was, is known as kind of like, I guess a luxury kind of, you know, tennis apparel brand. And it got acquired and the acquired just cut the price and tried to, you know, just increase the distribution. And then as a result, you know, it became worth a lot less because they just the okay, well they were selling more but at far lower prices. And then the original owners bought it back and kind of returned it to the sort of luxury status it had. But then you also talked about Nike and, and so with Nike, with the footlocker distribution, Nike had said, hey look, here's the, these are the terms. If you want to sell our shoes, you have to position them as like kind of high quality goods. And so the paper was from 2007, but Nike had the more recent experience, I guess in the last like two years of, they kind of shifted into D2C and then they started relying more on just like you know, direct response advertising and then pushing, you know, promotions to, to make that more effective. And that I think diluted the brand, right, made a lot more accessible. And then, and then, you know, they ran into trouble with that. And I think they brought in a new cmo. I think they brought a new CEO if I remember correctly. But, but they had to kind of revert strategies there. Just maybe if you could just kind of expand upon those. I just, I just thought those two examples were really interesting, especially the Nike one with the more of having kind of taken the opposite approach as what they did as what you depicted in the paper.
- [34:48]
D
So Nike's challenges have, have evolved somewhat. But let me just back up briefly and, and why did I bring up those specific examples? There's always some conflict with channel. In the case of Nike in the paper, you know, the channel wanted very liberal terms on credit from Nike and Nike didn't want to give liberal terms of credit. So these conflicts are all about how do I allocate the value I've created in the brands I've built. So I create the value, then how do I allocate and to Me, this is all a function of who owns the customer. So if the customer is loyal to the retailer or is the customer loyal to the brand? And in fact one of the very reasons we see Shopify you gained so much market share in E Comm relative to Amazon is because Shopify sits on the back end, Amazon sits on the front. Amazon controls the customer experience doesn't give a company a chance to do any branding or brand building unlike if I use the backend tools at Shopify. So Nike was extremely successful in owning the customer. Fast forward to now. It's a different world. The barriers to entry of DTC are pretty low and so you have all sorts of fragmentation and ankle biting. So you know, Nike's a really good brand, it has a great reputation, but maybe it isn't the single best brand for bike shoes, you know. And so the fact that I've got more DTC and ankle biters means Bike has to fight a little bit harder in thinking about how to own the customer and how that affects their advertising. I think that also the poster child recently for all of this is the acquisition of Kraft Heights by private equity. Ross, I don't know if you've been part of the measurement in that space, but they went big on controlling costs and part of that was cutting advertising. And you know, it's been, it's been a really rough time for Kraft Heights. So you can't cost cut your way to excellence, to be honest. You have to invest in build brands. Hope that answers the question, Eric. Long winded.
- [36:54]
C
No, no, no, no, that's great. And I guess, you know, I feel like private equity would be especially bad at that at operating a brand centric company because the whole goal of private equity is to buy an undervalued asset, kind of integrate some efficiencies and then flip it in a few years. Well, if you look at ad spend and even if you've got these kind of provable results of. Look, when we support these brand initiatives, they drive real value. Yeah, fine. They drive real long term value. We're not going to be the owners in the long term. Right?
- [37:30]
D
Yeah. By the way, that's one reason an analyst, some analysts I don't know if you've worked with anywhere else. Some analysts have started adopting these measures too as they evaluate companies.
- [37:37]
B
Yeah, I did work with some people at Vanderbilt that were helping out Wall street firms evaluate mergers and acquisitions and using cross price elasticity. See what happened. If these companies came together, would they be able to raise prices more? Have we done that?
- [37:50]
C
Russ? I wanted to switch gears into measurement. What are the differences in approaches to marketing measurement that you see across verticals for successful companies? So how does effective marketing measurement differ? For instance, E commerce retailers versus app developers versus large CPG brands, Or does it.
- [38:06]
B
Okay, okay. Well, we haven't worked with every single vertical, but we have worked. I mean between my first company Marketing analytics and then when I was at Nielsen and then now here at Market Attribution, we have covered a lot of verticals, including gaming, like you mentioned, Eric. Oh, we've done that. There's not many verticals that we haven't done. I think some B2B things we haven't done probably because that's a little bit, that's kind of different. That's more like CRM, which is not really our specialty, but I would say. So how they differ? Well, first of all, the KPI. So we'll model sales in some verticals. We'll model downloads for some loan applications sometimes also we may model different portions of the funnel of the purchase funnel. So we may. So for a bank we might, you know, model loan application, you know, loan applications and then loans given out or something like that. There's different levels of the whole funnel. We can measure the data differs. E Commerce, retail, they generally have better data and they have sales at the consumer level. If you, if you have, if you know who your consumers are, you can do a lot, a lot better stuff with, with modeling and. Yeah, because you know, you know, sales by consumer and you don't always. Not necessarily like E Commerce. Oh yeah, they would know everybody they sell to. They don't necessarily know a lot about them. They may sell through some sort of affiliate and then not know everything about the who it went to. But so the data differs and then the methodology differs. So if you do have consumer level information, like randomized controlled trials are the gold standard in measurement of pretty much everything in medicine and in marketing and all kinds of stuff. They are, they are the gold standard and you can make sure you don't have any biases with them. But it's a lot easier to do if you do sell directly to consumers. So that's easier to do if you're E Commerce or you're a retailer. We've done some RCT randomized controlled trials that were, I thought were super interesting. This was in CPG we had a 10 million household panel basically that we could send ads to and see how they behave. So 10 million households, something like 30 million people, you know, 100 million devices or something. It was it was a rather large experimental lab that we had. It was super awesome. We couldn't measure as many campaigns at the same time as we can do with marketing mix. We can only do like maybe six at a time. But it was so expensive because we had to pay an identity management firm. If you're, if you're a retailer or you're E Commerce, you might not have to do that. So that's more practical in those verticals. B2B, like I say, generally doesn't have enough volume. You can't just sell like one or two giant, you know, industrial machines a year and do some kind of model on that. That's kind of hard to do. You need some kind of volume. So B2B is kind of. And then long term effects, you know, this podcast is kind of focused on long term effects that also your ability to do that differs by vertical quite a bit. And well, even, even just any kind of measurement for let's say for automobiles that has to. You can't. Automobiles have such a large long purchase cycle that it's really difficult to model sales with a marketing mix model for cars. So we generally mod something like an attitude like purchase intent or some, some kind of thing like that. There are some products that have a very long purchase cycle, like mattresses that we are able to model. I don't know if it's because they tend to have really strong holidays. So like, you know, there's going to be a few periods where the people are going to buy a bunch of mattresses. President's Day, like right now, Labor Day, those are big seasons for that. So you can, you can measure, you can see the effect of your advertising in some sort of short period of time. So I'd say that, that, that those are differences across, across there and then as far as like you know, really successful, you know, some of the successful marketers who we work with, I think an E commerce company that we're working with is like really, I think they're one of our more successful clients. And I'll say here's some things about them that I think help well to quantify what they, what how good they're doing. They went from about 250 million sales to about 550 million over the time that we've worked with them. More than double sales. Their CMO is a super awesome guy, cracked us all up and he is always pushing us to push the measurement envelope forward. He had us doing our first share of search models that were popularized by a guy named Les Binet. I think he's in the from the UK but share of search it's an interesting thing. It's, it's a potential long term effects model, although I think it has some pros and cons for that. They also pushed us to do our first affiliate marketing. That's an important thing for E commerce where you sell through affiliates, you give them like a rather potentially large commission, like 20%, 30% or 2% depending on what kind of affiliate it is. So, but so modeling that and like how much extra values would I have sold without that affiliate referring it to me, you know, I prefer not to give them 20%. And then working tightly with their media guy, we turn things around really fast for them. I just love the way these guys are pushing everything. So for them after each of their major holidays we will get the model turned around in five weeks. So five weeks after the holiday ends and the Sunday of the holiday or whatever, the Monday of the holiday, five weeks after that we're done. We've collected all their data, working with their media guide, we've run the model, we've given the results, put it in an optimization, recommended what they do and we're done and we're ready for the next holiday. So I think guys like that, that really push, try new things, push the envelope always, you know, I think those guys get good results. It's good to be good at measurement.
- [43:01]
A
If you're swimming in dashboards but still arguing about what actually drove installs.
- [43:05]
C
This is for you.
- [43:06]
A
Branch is an AI powered MMP built for growth marketers who care about signal quality and outcomes, not just reports. You can quickly answer questions like how is my TikTok spend really performing? Or which partners are driving next Net new users and even launch campaigns that move users from offline to app without breaking attribution. Branches AI proactively surfaces what's working, flags issues early and takes care of the busy work like link creation and tagging so you can move faster and spend smarter. Learn more at Branch IO. That's Branch IO.
- [43:42]
C
So next question is for you both. Maybe I'll let Carl start and then Ross you can follow up. But what are the pros and cons of the various methods of measuring long term marketing effects?
- [43:52]
D
So let me outline the ways to do it and then the pros and cons. And I think Ross will probably take a dive into sort of some applications. But three basic approaches are using surveys to measure brand attitudes, brand associations, sales based measures like baseline drivers as we discussed earlier. And then you know, some people use stock returns tobid's Q specifically Tobin's Q is a measure of market value relative to assets. The idea is the market value is higher than the assets. Some of that is related to brain spraying. Surveys are easy to measure, but they're often a weak link to purchase. So for example, purchase intent R squares on purchases are not high and you have serious response bias because you know, who has time to fill in surveys. I don't know about you, but I haven't done many surveys in a long time. Sales again, this is the marketing mix type things that Ross is talking about. Easy. It's data that are typically part of the, you know, the standard tech stack, martech stack of a company. It's behavioral, it's field based and so it's a really nice tool to get a read on how a brand is trending. The downside of course is it's difficult to link directly to profits, which is sort of the key financial KPI. I mean it can be done. The stock price, it does go into firm value, which is again what you need to raise capital and sort of like it's the end game. But it's, it's a bit of a challenge like spend to stock price for a couple reasons. First, stock price isn't reported the brand, so I can't figure stuff going out on the brand level. The second thing you know is costs aren't many. Costs aren't under the marketer's control like cost of goods and then you know what is the result you're looking for. Ironically, if you have a null effect, then you're at optimal spend because if you're spending too little, you should spend more, raise profits. If you're spending too much, you spend less to raise profit. So if I'm looking at a direct link between spend and profits, the only way you know you're doing it properly is a null result. You know, of course null results have all sorts of problems, not the least of which being you can't really test the null result because of power issues. So those are sort of three and the pros and cons. Ros.
- [46:03]
B
Yeah, no, Carl outlined them. I think there were attempts at measuring long term effects that go way back. I came across a paper just in the last few days when I was talking to my team about getting updates from them about what they thought about long term effects models that worked or didn't work. They told me about one way back from 1978. It was called Assessing the Long Term Value of Advertising. I don't know if you've ever heard of that one, Carl from some guy, Naram Dala from J. Walter Thompson from 1978. They told me about this one. So it's been going back. I don't really think that one's super promising. So with the different methods we've tried brand awareness, we tried using that, that's like a survey method. It had a nice, we worked with Miller Brown together on that. I'm now part of Kantar. We got a base awareness that was served as a long term effect and it was big and so that was good, that was good about it. But it is survey based. It's, it's a relatively small sample. People also get confused with awareness data. People get confused about brands that they've seen. So you'll ask them, you know, are you aware of this brand or something? Or you where this brand's advertising? And they will say data, they will say they've seen ads that weren't even running at that time. But. And then what we find out it is, it's another brand that's kind of similar to it that they got confused about. So things like that make it survey data. As Carl said, it's like there's biases in it, selection biases and other things and there's people just don't remember stuff. Exactly. And so that's one kind of that one. We've had other survey data. There's a thing called brand asset valuator that, that's interesting data. It goes back a long time. It's got a lot of brands in it. We use that data, you know, it's perception data. Thought it got decent results. Senior management wasn't convinced that, that a survey data is going to let us double all our ROIs. I mean the general rule of thumb is that what you measure in a market mix model is like the effect on sales in the first year maybe and then you should double it to get the total effect. Something like that. And, and so the, the 2x multiplier, whatever we got out of that study, the senior management didn't take much action. As my, as my recollection we've also tried other behavioral metrics like, like share of search. That was the one that was popularized by Les Binet. His first stuff was on using durables. I think it was phones and something else and it looked really cool and I do think that it's a cool model to use. But it's not clear to us that branded, what we're talking about is branded organic search. You're searching specifically for some sort of keyword that has the brand's name in it. You would like to your advertising to drive people to search for your brand and learn about it, stuff like that. So that's why branded organic search is a good thing to drive. But anyway, it's not clear that a good indication of long term brand loyalty or anything in things like CPG or soap or you know, toothpaste or something like that. But I, I believe it in durables. Nielsen used a state based model for a while that was based on new and repeat buyers. That sounds quite promising to me. If you can track somebody like if you're a phone company, a telecom company and you track and you've got somebody that's a customer for several years, I think that that that could be promising if you watch somebody over a few years. I haven't personally done that one with a phone company but there's been several attempts at using that like tracking buyers over time, see how many are new, how many repeat and trying to have a model try and predict why. How do I get new customers that then stick around for a long time? Those are very promising. Another one was done by ncs, that's Nielsen. That's an alliance Nielsen had with a company called Catalina Solutions that now doesn't do this anymore. But a woman named Leslie Wood, she's done a lot of cool stuff. She did a household panel long term effects model that had new and repeat buyers. Also the problem with that is in CPG and you have like a panel there's a lot of churn in these panels. So like after a year, so many of the panelists have turned over that she could really not measure long term effect for longer than one year. And that's not really what we call long term effects. We generally call them long term effects after that. So we've used a lot of them and I will say that none of them are perfect and it's a hard thing to measure the effect on your sales. Like you know again these, some of these things I thought at first when I heard them is like I don't think it can be done trying to measure your effect on your sales two years out or three years out out but you know you can get decent estimates and, and the best one, the best model that we've done and like I say none of them are perfect but the best one in terms of like getting intuitive results consistently that are accepted by the client and that the client comes back and buys from us. Again I like to repurchase as much as the next guy is Carl's two part model. That one that I guess he did outline the kind of basics of it in an HBR article. But then later on he developed some methods to try and estimate it. So we do two parts of it. We estimate the impact of your advertising on base core sales like on a quarterly. We do quarterly data so we don't, we wash out the spikes and we just want to know total volume, especially core volume that you're driving and like you know, in the long term. So we model that and then we model your base elasticity and see what happens. How does your media, how do your promotions affect those things? The core volume? Do you keep having giant promotional spikes that drive down the volume in between the promotions that you're selling at full price? Is that what you're doing and how are you affecting pricelessly? Nobody wants a high price elasticity that helps you cut price. People want a low price elasticity so you can raise it. So raise price. So that's a really good model. We've run that model for 16 years. When I first started working with Carl, it was a long time ago, I guess it was 2010. We, I brought it into Nielsen when Nielsen acquired us. They're still using it to this day. To my, to my knowledge, no other long term effects model that we've ever used. We've used a bunch of them. I listed some. We've used more than that too. I think none of them lasted with more than two years with us anyway. Maybe with somebody else. But both us, we went away. So this is, is the only long term effects model that I've had a long term relationship with. So I thank Carl for that. And then I think I'll say that one thing I'll say about Carl's model is not all verticals support the price elasticity part of it. Sometimes we'll be in a vertical, you just can't get a good pricelessly. So we just do the core base volume part of it and that's good enough. And that that is the key is that we're modeling sales. So there's a lot of long term effects models. If you're modeling some kind of attitude or something else that that management doesn't really believe that that's connected to sales model, sales model, base core sales. Everybody can agree that that's a good thing to do. And I think that's part of why it works really well.
- [51:54]
D
Yeah, I remember that. I remember Ross coming down to Duke 2010. Remember the time we were spending on the whiteboard. Oh, you had some great Ideas. Ross mentioned Les Binet. For those who are not familiar with him, you probably should look him up. He's an icon in the advertising industry. He's a stalwart on protecting branding in the advertising mix. And I know he's done a lot of work in the space of brand management, baseline sales.
- [52:18]
B
I'm just saying Les Binet is. The way he talks. When he did this thing about share of search, it was during COVID and I remember watching it just thinking, oh, this is so relaxing and nice. He's just a great speaker. Nice guy.
- [52:30]
D
Yeah.
- [52:30]
C
So Russ, that was, that was a great segue to the next question. I know we're just at time, but maybe I'll try to squeeze this last question in. So Carl, you know, we're kind of talking in the context of like, like a lot of, you know, brands that are sort of established, they've got long histories. Maybe you know, something like Coca Cola going back more than a century. What tools does a CMO have for like a nascent brand? So imagine like a brand, a D2C brand that's just starting up, right? What, what tools does a CMO have with a nascent brand to make the case for investing in this, to investing in brand advertising, to, to build these kind of like long term brand effects. How would they, how would they make the case to like the CEO or like their investors that this is something that should be invested in?
- [53:07]
D
So I had one thought on that and Ross had one thought on that. I'll share my thought on that and jump to Ross. But analogs are really important here. The reality is you can't make up data. Remember it takes five years or more of data to really truly understand long term effect. You could think of this in terms of data variation. How many long term cycles of spend can you really have? Over a very short quarter you can't. So if this stuff's being measured at quarterly variation baselines, I mean to have any statistical power, you need enough quarters really to link long term strategy to quarterly baselines. But there are analogous brands and channels. So one of the really interesting discussions that you've probably fielded on your show or you may soon is with ads woven into generative AI, what's their performance going to be relative to search engine marketing? Right? And then you start thinking, well, you know, are these analogous? You know, and they should be in any way. It's like the SEO tools are same. I mean, I still have to have my website be discoverable. All those tools we did to, to get higher SEO and Better SEM tend to apply in this market. So it might be reasonable to say like as an initial guess I can expect an analog. The same would be true for new social channels with existing social channels adjusted for demographic and those sorts of things. But I want to caution, you know, you can't think of a brand new channel as brand. You can't prove a brand new channel's brand building until you have enough data to prove that is brand building. You might have good faith, a good reason analogs might be part of that. And Ross, like I think you have some experience again with Lotus's old measures and rules around this.
- [54:50]
B
Yeah, I mean that, that, that the original 2x thing came from. I meant to mention this during earlier discussion but There was that 1978 paper that I only heard about until recently. But there's a classic paper that was done in 95 by Len Lodishjit Abraham, Bruce Richardson, a couple other people from IRI and Wharton. It was a summary of 55 in market experimental estimates of long term effect of TV advertising. That was experiments like using behavior, what they call behavior scan markets. Those were IRI now called Sukana but way back when they had small cities, Peoria or something, little tiny cities that were kind of isolated from other media markets. And then they would, you know, run some ads in one city and then do something else that different or within the same city. I think that's split cable. That's right. Within the same city they had some people saw one ad, some people saw another ad or didn't see an ad and they did that and then watched for like three, two or two years out. They watched for two years out I think. So that was a very interesting. That's where they got the 2x multiplier. First thing I said the long. The total effect from an ad is generally twice the measure that you get in the short term. That rule of thumb, all kinds of long term effects models, we all go back and make sure we're somewhere near two otherwise we're probably doing it wrong. But yes, I agree with Carl that, that, that for a brand new brand you probably should just use the 2x thing. There have been some attempts to do like Nielsen after we brought Carl's model to Nielsen, some other guy that, that, that came in with my team to Nielsen, he took that model after I left and then tried to make it so it could work. Because like with Carl's model we generally say we have to be at the client for three years because we get there and then we have two Years of back data. And then once you've been there for three years, we have five years of data that then we can use them. Carl's model works pretty years of data. And so that's what we'll say. But you know, that's like kind of hard to sell, especially if it's a new client coming in. So I can understand why Nielsen tried hard to make it try and work for a new brand. And what they did was they took ad stock and just kind of extended ad stock way out. That's been done by some ad stock is for anybody. Probably most people listen to this, probably know what that is. But ad stock is you take GRPs or impressions and then you kind of do exponential smoothing. I'm just kind of smooth amount of the future. And then you use that as your variable. And then through that you get an effect going on in the future. You try different decay rates.
- [56:55]
D
Rates.
- [56:56]
B
It's pretty much the same thing as exponential smoothing, except it has some kind of set. Different people, different companies have different kinds of proprietary or maybe they admit it different kind of saturation estimates. They lay on top of the, on top of the smoothing thing. But trying to get a new brand and just doing ad stock to go out in the future, it doesn't work too well because it makes all the. You spread it out so far in the future it just looks like a pancake. And it's highly correlated with the intercept itself. It doesn't work good. The one, one thing that we did do that was that for. For a new brand. Carl and I did find this one out just a couple days ago was we. Well, actually I knew about it, but I didn't really think how that is pretty good. It was for in pharma or actually we're doing pharma or OTC over in Europe. And over there they have a mixture of the things that we buy as OTC over there. Some of it's pharma and some of it's otc. But they have a measure that we could do getting over there from some market research firms called average weekly recommendations. So we advertise to the healthcare professional and then we get data on how much they're recommending that product to their patients. And so like I say, even some OTC things like nicotine replacement therapy, that is you go to a doctor over there to get a recommendation. And so we'll model AWRs and then we'll model the effect of AWRs on sales. So we get kind of two kind of long things. Effective marketing on AWRs that has a big lag in it. The effect of AWR is on sales. That has another lag. We get out about a year maybe in terms of the effect. But you know, like I said, it's about the cutoff of where you might call it long term. So bottom line, I agree with Charles with Carl that benchmarks are probably the best thing. There's a couple other things that are decent.
- [58:24]
D
Yeah, Ross, I like your example. That's actually called the surrogate variable technique by a bunch of academics. The problem, your application is fantastic. The problem is the assumption in these models is that this intermediate measure, the surrogate, fully mediates the effect of advertising on future outcomes, which just is insensible in most applications. But not yours. It's actually a really good application, Carl.
- [58:48]
C
Russ, this was a fantastic episode. I learned a lot. I imagine everyone listening did too. How can people follow you? How can they consume your content? How can they find you on the Internet?
- [58:58]
B
I'm@www.marketingattribution.com.
- [59:01]
D
yeah, and thanks for having us. It's great to inject long term back in this again. I love that it's top of people's minds. In terms of my case, I like to say that if you're going to have a kit, think about how to name them and that people can search them and they're the only real result that shows up. Like, to my knowledge, there's maybe five Carl Milas in the entire world. So to find me is pretty easy in a, in a Google search. Just, you know, enter my name and it'll pop. And if brands are built over years, why they manage up a quarter is just do a simple search on that will pop up at hbr. I think that is a really good starting place for a lot of people if they want to take a read into what's going on here. So thanks again.
- [59:41]
B
Yeah, thank you, Eric.
- [59:42]
C
No, you're, you're just, just pointing out to people that that is a HBR paywall gated article. But the PDF can be found through Google search. Well, so they can, they can find it that way. I really appreciate both your time. Thank you very much and enjoy your weekend.
- [59:56]
D
Thank you.
- [59:56]
B
Next slide.