
What is incrementality testing? “I’ve been doing more interviews with journalists lately and realize I need a better answer to this question,” says Haus Head of Strategy Olivia Kory.
Loading summary
Olivia Corey
Welcome to Ad Exchanger Talks, the podcast devoted to examining the issues and trends in advertising and marketing technology that matter most to you.
Alison Schiff
This episode is sponsored by the trade desk Edge Academy, the online learning platform designed to help marketers stay ahead in the evolving world of digital advertising. Check out their certifications, expanded course curriculum, and more, all led by the people who are shaping the world of ad tech. Enroll today for free at edgeacademy.thetradedesk.com.
James Hersher
Welcome to AD Exchanger Talks, our weekly podcast with interesting people who bring perspectives from across the worlds of data driven advertising. I'm James Hersher, Ad Exchanger senior editor. Filling in for Allison Schiff. Joining me today is Olivia Corey, head of strategy at House, an ad measurement and incrementality testing startup. She's held stints at practically every link in the online ad supply chain, Starcom, Tubemogul, Netflix, Quibi, and Sonos before House. Olivia, welcome to the show.
Olivia Corey
Thank you, James. It's good to be here.
James Hersher
So, before we get into what incrementality testing is and other timely topics, let's start with the Olivia Corey story a little bit. How did you end up where you are and. Yeah, where exactly are you too?
Olivia Corey
Yeah, I'm actually in suburban Detroit right now. Was in LA for a very long time. Uh, and then the pandemic hit. I was at a startup, Quibi, failed streaming startup of 2020. So it was like a nice career break. Um, and we decided to move home where I grew up to Michigan. So have been here for a few years and just braving the coal.
James Hersher
So this isn't. This isn't House headquarters too. This was. This was your place?
Olivia Corey
No. Yeah, we're fully remote. Large group in San Francisco, but we're fully remote. And yeah, here mostly for personal reasons. We have a couple children now and it's a great place to raise a family.
James Hersher
Right on. Well, yeah, let's get into maybe, you know, still sticking with the Olivia story. So how did you, you know, what, what was it that attracted you to eventually House? But, you know, how do you end up in this category? This sort of part of our world?
Olivia Corey
Yep. I was actually introduced. I was very lucky to be introduced to incrementality measurement. Pretty early in my career. I worked at a media agency called Starcom, part of, part of Publicis. Fun fact. I actually met my husband there. We sat next to each other working on the Allstate business. But Allstate was actually. Was actually pioneering this stuff pretty early. This would have been like 2012 or 2013, but they were running these kind of controlled holdout experiments to better understand where they should be investing. And so I think that experience helped me land my first kind of brand side role at Netflix. And I mean, Netflix was just a playground for learning, was surrounded by brilliant people. And I like to say that Netflix really understood the contribution of their marketing. They understood pretty clearly what each marketing activity was driving to the business. In a world where there's tons of organic demand, tons of noise. I mean, this was 2016. The company was just exploding in terms of growth. And there's no secret here, the reason they were able to do it is because they just staffed that operation with a lot of amazing data scientists and causal economists and in product managers. And so, you know, I would say like it was a, it was a very sophisticated operation, but also very expensive. And that's what, you know, ended up leading us to house into actually building this company is we, we thought, you know, what if we could enable access to that level of experimentation that really can transform a business, but do it as SaaS for less than the cost of a headcount and fast forward a few years and that's what we've been able to do.
James Hersher
I'm curious about the Netflix thing because I do remember, I mean, Netflix did have this reputation for having such smart data science and this was sort of behind its own marketing. And this was the days of a million dollar prize for recommendations that are better than Netflix or something like that. And I remember a couple women from Netflix at a programmatic I.O. in San Francisco, someone like dropped out. We had a last minute like hole in the agenda and they just like jumped on stage day of to do a presentation that they. Oh, like they had a presentation in their back pocket. They'd done like the week before on something like, you know, sort of ghost ads and testing. So it was like the best presentation of the show. But you know, is that like, okay, if you are Netflix, a really sophisticated marketer and sort of like data science group, like how, you know, does that, how much of like a leg up is that now that like, because this was before the days of like Netflix being in ad tech, like, you know, taking on that side of things, like does that, you know, it's like, oh, are you starting over? Are you still doing like baby steps? Even if you are like a sophisticated marketer, you know, you're Netflix. Like, you know, does that necessarily mean you could like have a sophisticated ad tech operation on the other side?
Olivia Corey
Yeah, it's a great question. So, so you're curious if, like, if all of the advancements that they've made in terms of the product itself and in marketing, like if that carries over to their, their advertising operation now that they sell ads.
James Hersher
Yeah, yeah. Like there might be lots of retailers who have a great marketing team but like their retail media business feels very like toddler ish or something like that. Like, you know, so sometimes I feel like there's an interesting split.
Olivia Corey
I think about this all the time and this is all speculation because I obviously haven't been there in a bit, but I was very surprised that they, when they started doing ads, I was very surprised they went with Microsoft as the partner. Cause it was just so, it was, it was, it was very much against like everything I had experienced there, which was like homegrown technology and significant build versus kind of buy mentality. And I have heard, you know, since then, over the past year or so they are bringing it in house and they are going to I think build the engine that we all expect from them in terms of sophistication. My guess again, just speculating is that they wanted to get that ads product up really fast and so they had to go buy a third party. So yeah, I mean they should be the best in the world at this. I know that they have the talent and like what I loved about marketing there is that Reid, the CEO, he wanted the marketing to work like the product where it was very algorithmic in the same way. Like no two people see the same Netflix in terms of the service itself. He wanted to take those principles to the marketing and he really wanted to focus on experimentation and building a culture of experimentation. So I would expect over time that that would also kind of infiltrate over into the ads group as they have more time to build this thing. I would imagine it was just too rushed and they had to just get an MVP out. But I'd imagine it's moving in that direction.
James Hersher
MVP being minimum viable product. We gotta cover all our three letter acronyms here. Yeah, that makes sense. And I do think that's you know, just with like Walmart and Trade Desk or something there is almost as like everybody's like assuming, even if it isn't like official that like at some the plan is to like squeeze the vendor out and for this to be a sort of in house thing. But yeah, it's always interesting to see how those, how those play out. Let's, let's get to house itself and you know, some very like ad tech in the weeds perhaps. But okay, let's start with what is incrementality testing. Like how do you sort of define that in this, you know, people know like multi touch attribution and attribution in general. Okay, like what is incrementality testing?
Olivia Corey
Yeah, I've been, I've been doing more interviews with journalists lately and I realized I need a better answer to this question of like the. Explain it to me. Like I'm a, you know, I'm a, I'm a six year old. So you can tell me, James, what you think of this one. But what I like in incrementality too is the establishment of causation. Everything in this world of marketing measurement historically has been built on correlation. And I like to give the cappuccino analogy of you walk into a coffee shop and at the front of the register there is a sign for a cappuccino and then you order a cappuccino. Did that sign actually make you order the cappuccino? Or does it just so happen to be the case that that sign is sitting in front of everybody who orders a cappuccino? And that's what we're trying to tease apart with these concepts of incrementality is we have a counterfactual. We're able to understand what would have happened anyway in the absence of an ad or some sort of intervention, such that we know that any given test that we run is truly incremental to the business. And so what I like to compare against is like a randomized control trial in healthcare where if you're rolling out a new drug, you have some group of users who get the placebo, you have some group of users who get the drug, and then you look at the difference between those two groups to really understand the efficacy of that drug. That's what we're doing in terms of incrementality testing. And I've been, I've been hearing a lot of confusion in the industry in terms of what incrementality is and what it is not. And when I really kind of boil it down like to its core, it's the idea that we have the counterfactual to understand what would have happened anyway. And that's how we're really able to establish causation.
James Hersher
Do you feel like, can the platforms themselves, which are also have been rolling out, sort of. Oh, like, you know, clearly they, oh, our advertisers are using the word incrementality. Okay, like here's an incrementality product. Are those like, could a Google incrementality product work? Or does it have to be Someone sitting on top of all of those because I imagine there's like, you know, everyone's, everyone's saying they contribute quite a bit.
Olivia Corey
Yeah, what's, what's fascinating about, about where we are right now in kind of modern marketing is that I actually think incrementality is a bigger problem than ever because these systems Google and Meta are leading here are like really good at finding people who are going to purchase. Like they have loads of engineers and machine learning people whose job it is to like build models and build out, build algorithms that are going to deliver an ad to somebody who's going to purchase, like deliver that ad to the right person at the right time. And thus it becomes even more important because these systems are getting better and better at getting kind of in the way or like in the path of somebody who is just about to purchase. And so you can run these. There are a few types of incrementality tests that you can run and just to kind of to level set on the different types of experiments. There are user level experiments and there are geo experiments. Meta and Google can actually run these user level experiments where based on their systems they can create a holdout group of users and say we're not going to deliver ads to this group. And this is called conversion lift. And Meta has a version of it in Google as well. The issue with user level studies is not so much that you can't trust the platforms. I do think they're doing great work here, but a lot of folks really don't want to trust these platforms to grade their own homework, if you will. But the larger issue is that with signal loss and everything post iOS 14 and ATT, it's very hard for them to allocate users into a treatment and a control and like follow them through to purchase. It's just like hard to track that these days at a user level. And so that's what moves us into the world of geotesting and just to kind of close it out on the platform studies, they each have their own methodology, so they're all doing it in a slightly different way. And thus it makes it a little bit hard to kind of compare those reads that you get from Meta to a Google or a snap. And so that leads us to geotesting and why we focus there is you get the same kind of standardized methodology across every single channel and platform and then you also don't need to worry about any user level data or pii. We just totally bypass that by just looking at aggregated sales in regions that got A treatment that got meta, for example, versus the regions that didn't. And so it's a lot more durable in terms of privacy as well. And then there's like this kind of third type of incrementality that I see thrown around, but it's a lot less precise, a lot more directional of just like what I call a natural experiment where you just turn something off for some period of time and you look at how the business changes. So we don't have that, you know, that holdout group, if you will, but you can kind of run a natural experiment to like see what's happening after you take an action. So those are the three types and why we focus on geotesting. And we think it has some really great properties.
James Hersher
Yeah, it's tough. It's really feels like a wild and it really has gotten the bar of sophistication, I feel like has increased so greatly from the days when it was like, oh, I could have an interesting sort of product and put together a compelling video and like Facebook might just like run from there. Like once I generate like a hundred sales, Facebook is just being like, all right, that's, that's a very useful seed. And like, here's just a, just a river of, of lookalikes here coming down. Like, you know, it just feels like it's, it's gotten so much tougher and I don't know if like the social platforms can, can even like get you there anymore. Like, I don't, you know, is it just that like the signal loss isn't there, it's just, you know, iOS all these changes much more difficult? Or also is it just that like Facebook is like, you can you even be like a social based brand that's viable and you're just, you know, make your living on Facebook and Instagram?
Olivia Corey
I think it's a few things. I think part of it is macro. I think, you know, when, when things aren't going well, you blame the platforms for not working as well. But it might just be the case that people aren't buying right now and thus like, you know, the CPAs or the CAC and Meta is going up and you're like, oh, what's wrong with Meta? In reality, it's just that demand is softening. So I think that might be part of it. For some of these brands, especially these brands who really peaked during COVID it's really just hard to comp that year over year demand. The second thing is signal loss and these iOS changes. Daniel McCarthy, who's an academic, wrote A very insightful paper about the effects of ATT Apple's changes on E Comm brands and they ran some pretty thorough analysis showing that it has had an impact and it has had a disproportionate impact on smaller businesses in the E Comm world. So signal loss is definitely contributing. And then the third thing is these platforms and this is why, you know, I feel so strongly that you should be incrementality testing is they're getting better at targeting users who are in market to convert. And so that might be a problem and just something that you need to check and really validate for yourself in terms of what you're getting and making sure that meta is not just claiming credit for these sales but that they're actually driving sales and influencing sales that wouldn't have happened otherwise. And a system like Advantage Shopping or Google's Performance Max are very heavily suspect. And so that's where as those systems or as those platforms roll out these new products that are more black box and heavier on ML and machine learning, you do need to make sure that you are getting incremental outcomes and not just the low hanging fruit.
James Hersher
I do think that that is a sort of a dynamic once you get into this sort of data driven online advertising space like it is Google, Amazon, Meta but also like Walmart, Target, like they don't my feel like I don't think that those like once you get into the platform place like you don't believe in organic conversions in my opinion like Walmart or Google, like Amazon, like honestly like in there, like they're just not built like they don't. No, no, no conversion is organic. Like as every con, any organic conversion is actually just an opportunity for them to improve their attribution in a sense. Like right. Like they want to, they want to convert 100%. So I think, yeah, I just think it is like a, it's a rough incentive system. Just like structurally the incentives are very bad. I kind of feel like for, for advertisers being diligent about what is really being contributed.
Olivia Corey
Yeah, like meaning like all these retail platforms now are just running ads to try and like claim credit for that.
James Hersher
Right. Any like anyone who is going to show up, go to the store and was absolutely going to buy, you know, you're like Campbell's Soup. Like that's just from like a Walmart or Target perspective. It's like all right, we should have just had an ad in front of this person. And I feel this way about like the search products too. Like not just performance Max and Advantage Shopping, like the social stuff, but like, you know, brands that are handing over their search budgets or something to brands. And like, this retailer might know, okay, so this person buys dish soap every five weeks, and three or four weeks in, they just start, like, they can just, they serve you an ad and like, we know this person's probably going to buy this and just start to claim credit for what are organic sales.
Olivia Corey
Yeah, I like to say that the closer this ad exposure is to the point of purchase, the more skeptical you should be. Meaning, like, if you're running these Amazon search ads and someone is searching for this brand on Amazon and you get an ad, like, you should really be running more rigorous testing to make sure you need to be spending. And this is the beauty of geotesting is we can suss that out. Like, if they just shut it off some of these retail partners or Amazon in part of the country, we can see what would have happened anyway. And we can kind of better explain some of the metrics and what you might expect organically had you not been spending.
James Hersher
This is a good time. It's actually a late time, but we got to cut to ad break and we will be back with some other assorted topics I'm eager to talk about jumping into what we're thinking about for 2025. Stick with us.
Alison Schiff
I'm Alison Schiff, managing editor of Ad Exchanger, and today I have Stephanie Patterick with me, GM of Editorial and Editor in chief at the trade desk, where she leads the content strategy and execution. And she also has more than 20 years of experience in media and journalism. Stephanie, welcome.
Olivia Corey
Alison.
Stephanie Patterick
I admire your work. So happy to be here.
Alison Schiff
Thank you very much. So what shifts or forces are driving the evolution of the ad tech industry? This is such a big question.
Stephanie Patterick
It is a big one. And I think we're in a remarkable moment in time because we're really witnessing the maturation of the ad tech industry. And by that I mean, you know, just in my career as a, as an advertising journalist, I've seen programmatic go from this niche part of the ecosystem that transacted primarily in remnant inventory to a vital player that's supporting premium content. And I think that three forces are sparking this. It comes down to consumers, regulators, and innovators. And first, I think it's important to understand that more people are spending time on the open Internet than in walled gardens. So they're spending more time streaming shows, listening to podcasts, reading news. And second, the Department of Justice has really held Google's feet to the fire this year and that has exposed the danger of black boxes. So everyone in the supply chain is being scrutinized and there's a pressure to add value, not just extract it. And all of this really represents a massive opportunity for innovation and for ad tech to keep evolving with quality content, supply path efficiency, and really transparency for all parties at the fore.
Alison Schiff
What is one piece of advice that you'd give to someone looking to take their career to the next level?
Stephanie Patterick
I love this question so much. Get outside of your own expertise to understand the big picture. I think one of the biggest mistakes that people make in their careers is to silo themselves within one area of expertise. And I want to be clear. Specialization is valuable, but the more that you can get curious about what your stakeholders know and the deeper you understand how the entire media ecosystem works, I think the more brilliant you'll be at what you do. So don't be afraid to play the role of a reporter and ask the subject matter experts in your organization what they know. I think you'll be surprised at how generous people are with their knowledge.
Alison Schiff
Advice for your career, advice for life. And how can the trade desk Edge Academy help a marketer with their career?
Stephanie Patterick
We talk about transparency a lot in our industry, and one place where it's vital is education. At Edge, we have a strong belief that the more everybody understands programmatic, the healthier the whole ecosystem becomes. And we really believe that this knowledge is an advantage for anybody who's looking to grow their career or their business. And this goes for the cmo, the planner and the trader. Edge has served the advertising community for 11 years and it's evolving to stay ahead of the curve. So we've got new classes on timely topics like omnichannel Advertising, Connected tv, as well as foundational certifications like trading essentials and marketing Foundations. So it's just a fun challenge to meet learners where they're at and help them navigate this space that changes, as you well know, literally every day.
Alison Schiff
Stephanie, thank you for the insights and for the advice.
Stephanie Patterick
It's been so fun to be here. Thanks for having me.
James Hersher
Welcome back. I want to start segment two with a topic I want to make sure not to miss. It is sort of an easy one perhaps to forget about, but I think is a really interesting, interesting right now. And it also gets back to, you know, that question of like can, can Facebook can now meta like can they get you there on their own like Google, Amazon even. And that's, you know, the growth of Applovin as in e Commerce and D2C trying to get into this space of not just like mobile gaming advertising but you know, hey, we're a viable sort of retail and E commerce advertising channel. This is in a closed beta sort of period. I think right now like Applovin has really not taken the, the training wheels off but it seems like just on the sort of social channels, a lot of like excitement. We, we may even have already reached the point where it's like there's some pushback to the excitement. But yeah, where are you? Are you bullish? Bearish? Somewhere in between on Applovin?
Olivia Corey
Yeah, I, I've been fascinated by the Applovin story. We are recording this in early December, mid December and we've already experienced a bit of a rollercoaster where I think folks who have been in the mobile world know Applovin very well. When I was working at Quibi as a mobile app, if you're looking for app installs, you have likely tested or bought Applovin. So them kind of coming out of nowhere into the ecom world happened over the summer. In August, September, they started this pilot and our brands who we work with at Haus, we work with a lot of e comm brands were intrigued because Applovin was offering some free credits to get started. So they were like, why not let's get started? And they launched it with I don't think they had much in terms of expectations and it ended up performing really well on their attribution models and what they were looking at day to day where it appeared that Applovin was hanging in there alongside some of their key channels. But the next question of course is is this incremental or is Applovin kind of again this is the challenge with all these systems. Are they just picking off low hanging fruit that maybe Meta or another channel is driving?
James Hersher
It feels like it's tough for Applovin though like with these like Google, you know, the opportunity to just like poach the low hanging fruit as you say, is so clear like the, you know, the Google search. But Applovin, like people aren't, you know, if they serve an ad in like a mobile game, like that's not. That doesn't feel like such a direct like obvious conversion. Like you know, they're not just like less of sort of the cynical like snatch of a conversion of the attribution as opposed to like generating something, you know, if it's Applovin serving the app serving the ad, it feels like it's a conversion that's happening a little later.
Olivia Corey
Down the road, yeah, you're absolutely right. Is that like this doesn't seem like an environment like search where it's like easy to just kind of like pick off the low hanging fruit. These are mobile games. I don't think Candy Crush is in their network but games like Solitaire and Bingo where you're playing your game, you are waiting to go to the next level and you get served in interstitial that you have to, it's kind of a forced view. So I agree with you. You wouldn't expect it to do great on a last click attribution model just based on the environment and the inventory. But alas, like they saw pretty strong signals and so then came the questions around, all right, can we get a house test on this? Because we want to really validate what we're seeing. And so we started testing in November and we've at the, at this point of recording we've completed six experiments and they've been generally positive. The spend levels are on the lower end and the experiments that we're running now are at a step up. So when we saw initial success we know a lot of these brands are increasing spend and retesting and we have some bigger brands getting into the mix. So that open question is like can they maintain this performance at scale is like open question number one and then open question number two and this may have been what you're referring to in terms of some of the pushback to the hype is are they able to go find new customers? And this is what these E comm brands want like in terms of their model and the economics of their business is like they need to be driving revenue amongst new customers and not repeat customers. And that's, I'd say an open question with what we've seen with some of our experiments is that it is incremental but for some of these brands it's bringing in repeat purchasers who have purchased these brands before. And so that's open question number two. And in a system like Applovin can't as easily just exclude new customers from a campaign as like a Meta or a Google. So this might be a technical limitation that they have and it might be a challenge that they're going to be up against. But I've been, you know, we've run a lot of experiments and it seems like Meta and Google take the lion's share of any E comm brand spend. Like you know, between Meta and Google usually a brand is spending 80% of their budget there and then you have these secondary partners like TikTok, like Pinterest and Apple.
James Hersher
Yeah, I think partially is that people are just excited to see, you know, a new name. Oh, that's nice. Like someone else. Like, even if they're sort of on the margins. And I do think, like getting that, making that transition to like, you know, the way like P Max is just like an open product. Anyone? Hey, like, everyone jump in here. Whereas I do think, like Applovin, like, as long as it is in this closed beta, the numbers are so like, they just control those levers. And even, even meta, like, okay, like, even at meta scale, Advantage plus kind of like came out of a weird, like, hybrid beta over the course of 18 months, it felt like. And there was this long period where like, brands are like, I've kind of like signed up and I'm just like waiting, like, hoping I'm getting that email that I'm part of advantage shopping now. And I feel like that that was meta just being like, oh, like, you know, if we let all our fashion brands in, like, that ruins everybody's like, conversion. Like, you know, they can kind of, as long as it's in a beta program, they control. They control the demand, they understand their supply. Like, Applovin could be like, okay, we can generate great returns for this candy brand, but we can't let all our candy brands in because then no one's going to have good returns.
Olivia Corey
That's a theory right now is that there are only a few hundred brands in here and so there's a lot less competition than Meta where there are millions of advertisers in the auction. So that's a theory. But it's just exciting to see a new channel emerge and to see performance this strong. It's not every day.
James Hersher
Do you feel like there are going to be getting speculative 2025, are there new names beyond Applovin that are sort of in contention you feel like could make this switch?
Olivia Corey
I mean, one of the themes coming out of Black Friday Cyber Monday that I'm seeing is like, a lot of these brands really want to diversify. And they're not just saying it like they. They've been having issues on meta. A lot of folks kind of talking about this publicly since February. Maybe some weird, like, system quirks and bugs that have impacted meta performance. And so there is this theme I'm noticing around diversification. It's going to be a priority in 2025. I'm bullish on YouTube. I think brands just really like, if they're measuring YouTube correctly in terms of moving off of user level attribution and just like into the kind of in, in your own data. In terms of sales, what is it driving? It can, it can like really hang in there as a performance partner. So I've always been bullish on YouTube. I think it's underutilized. TikTok Shops is the other big one. A lot of testing into TikTok shops right now.
James Hersher
Do you feel like the ban is sort of like, I don't know, everyone could kind of not believe it. It's like it's sort of. But is that a real thing? Okay, if that happens, is this doing. Speaking of app love and getting in the mix, do we sort of lose one from the mix? TikTok's an important tier too.
Olivia Corey
Yep. Yep. So much news right now between Applovin and like the Meta health and wellness restrictions. You almost like forget that we might lose TikTok. So yeah, that it's, that's the reality is I think that's, that that's very much a possibility.
James Hersher
Yeah. And I do want to get into the, these changes at Meta too. Health, wellness, beauty, like definitely a big one. I think sort of you know, generally like sensitive categories. They're, they're expanding and you know, I think that yeah like also because that, that category is such like a powerhouse social commerce. Like those are, those are like the big, I think it's probably like related but those are the big tick tock shop sellers it seems like. But yeah, like what, you know, so what are people gonna, gonna do about that? Like these seem like big changes. They're, they're making health and beauty wellness brands. Not being able to advertise using the conversion API essentially not base anything on conversions.
Olivia Corey
Yeah, it's been, it's been challenging to talk about this one because we have so little information and so a lot of this is just based on conversations that you're having with, with different brands who are hearing different things from their reps. I think the, I think the impact will be severe for the brands who are impacted, but I don't think the impact is as widespread as we might think. For example, supplements like you know, wellness kind of vitamin supplements that promote, you know, energy and focus and gut health, like those are going to be fine. I don't like, I think those we're in the clear and where we're really going to see impact are these companies that might be, might be passing like what might be perceived to be sensitive data around like doctors and prescribing medications. And so if you're a brand that has doctors prescribing medication. That is where it gets really touchy. And for those folks, like, yeah, they're not going to be able to optimize to conversion. And we know, we know that Meta even says, as a best practice, that these systems are really good at doing what you tell it to do. So if you can no longer, like, point them to a conversion event, we should expect some impact. And I imagine that these brands are kind of already getting started on diversifying their channel mix. Although then you've got to wonder if the other platforms are going to follow suit, if this is like, regulatory.
James Hersher
Yeah, I mean, the difference between how, like, Instagram approaches this and TikTok approaches this is crazy to me. Like, I think Tick Tock is getting a little closer. But I remember a year ago doing sort of like a deep dive on the Tick Tock platform and someone was showing me, you know, someone was showing all the things you can't do, all the things that Instagram wasn't allowing as, like, this was like a supplement and vitamin brand. Someone else who did like face creams and they couldn't. Oh, like, they sort of felt like, oh, like Feel Good Friday. Like, and the Instagram was like, no, like, even though that's not. Like, you might not think that that's like cynical behavior. It's like, oh, like Feel Good Friday. But that's like targeting emotion. Whereas on TikTok, like, you could go, you could like go target like wrinkle cream to people who are like, like, hashtag, like feelingso old. Like, you could target, like supplements for vitamin D to people who are like, journaling for depression category. It's like, oh, my God, like, I cannot believe that. But so, yeah, it does make me think, like, okay, like, I could see why Instagram is like, okay, we really need to put up some guardrails. Like, it's hard to, you know, even if you don't think you're doing something cynical, like, if you are connecting a conversion to like, this media and the behavior on social media, you're going to end up creating these audience segments that are like, people who feel really bad about themselves. Like, people who are like, down people who have just like lost gambling and like, you know, it's just. You're going. It's almost unavoidable.
Olivia Corey
Totally, totally. And that's why I wonder, like, meta, Meta. Being conservative here feels like it's just the start of more to come for other, these other platforms and channels. So I, I don't think this is going to be just like isolated or limited to meta. And so then you just, you have to figure out how to optimize your campaigns in the absence of conversion events. And I think we should probably expect some efficiency hits but you know, brands are getting really creative. We're seeing some like very clever hacks and we'll get through it the way we did the Apple changes, but it is pretty severe in terms of the impact.
James Hersher
Well, I guess hopefully it means there's an opportunity there for everybody else. But there is demand.
Olivia Corey
There's a silver lining in that. Maybe if you are forced to, you know, broaden out a little bit and you, you can't do this conversion optimization, maybe that's more incremental, like buying the old fashioned way of like reach frequency, probably less efficient but like maybe it's more incremental and maybe they're able to introduce the brand to more, to more audiences.
James Hersher
This is just the long road home to yeah, upfront buys and yeah, great. Well, I think that's a good place to end it. Maybe not the most optimistic take for 2025, but you know, get your, get your testing game together should be a fun one and more news to cover. So Olivia, thanks for joining.
Olivia Corey
Thank you so much, James.
Alison Schiff
This episode is sponsored by the Trade Desk Edge Academy. Whether you're a seasoned programmatic pro or are just starting out in digital advertising, the Trade Desk Edge Academy's expert LED courses and certifications are here to help you stay relevant, ready and ahead of the curve. Enroll today for free at edgeacademy.thetradedesk.com.
AdExchanger Podcast Summary: "For Incrementality Testing, It’s One Step At A Time"
Release Date: December 17, 2024
Host: AdExchanger Talks with James Hersher
Guest: Olivia Corey, Head of Strategy at House
The episode opens with James Hersher, AdExchanger's senior editor, introducing Olivia Corey, the Head of Strategy at House—a startup specializing in ad measurement and incrementality testing. Olivia brings a wealth of experience from various roles across the online ad supply chain, including positions at Starcom, Tubemogul, Netflix, Quibi, and Sonos. Her diverse background sets the stage for an in-depth discussion on incrementality testing and its evolving role in the ad tech landscape.
Notable Quote:
"I've been very lucky to be introduced to incrementality measurement pretty early in my career."
—Olivia Corey [02:44]
Olivia Corey delves into the concept of incrementality testing, differentiating it from traditional attribution models. She emphasizes that unlike correlation-based attribution, which merely identifies associations, incrementality testing seeks to establish causation by determining what would have occurred in the absence of a specific marketing intervention.
Key Points:
Notable Quote:
"At its core, incrementality is the idea that we have the counterfactual to understand what would have happened anyway."
—Olivia Corey [10:57]
The conversation shifts to the complexities of relying on major platforms like Google, Meta, and Snap for incrementality testing. Olivia highlights several challenges:
Notable Quote:
"With platforms like Meta and Google, they are getting better at targeting users who are in market to convert, making it even more important to validate incrementality independently."
—Olivia Corey [14:46]
James brings up Netflix’s reputation for sophisticated data science and marketing operations, questioning whether their foray into advertising compromises their established standards.
Insights:
Notable Quote:
"We thought, what if we could enable access to that level of experimentation... but do it as SaaS for less than the cost of a headcount."
—Olivia Corey [04:08]
Olivia discusses the current state and future of incrementality testing, touching upon:
Notable Quote:
"Geotesting allows us to bypass user-level data issues by looking at aggregated sales in regions with and without treatment."
—Olivia Corey [09:01]
A significant portion of the discussion centers on Applovin's transition from mobile gaming advertising to e-commerce. Olivia shares insights from early experiments conducted by House with Applovin:
Notable Quote:
"In our experiments, Applovin is bringing in repeat purchasers who have engaged with the brands before, raising questions about true incrementality."
—Olivia Corey [27:20]
The episode also explores recent regulatory shifts, particularly Meta’s new restrictions on health, wellness, and beauty advertising. Olivia explains how these changes affect advertisers:
Notable Quote:
"Brands are getting really creative with campaign optimization in the absence of conversion events, often facing efficiency hits but potentially reaching more diverse audiences."
—Olivia Corey [38:29]
Post-ad break, Stephanie Patterick, GM of Editorial at The Trade Desk, joins the conversation. She discusses the maturation of the ad tech industry, driven by consumer behavior, regulatory scrutiny, and innovation.
Key Points:
Notable Quote:
"More people are spending time on the open Internet than in walled gardens, presenting a massive opportunity for ad tech to innovate with quality content and transparency."
—Stephanie Patterick [21:25]
The episode concludes with a forward-looking discussion on the state of ad technology and the importance of incrementality testing. Both Olivia and Stephanie emphasize the necessity for continuous experimentation, diversification of ad channels, and robust measurement techniques to navigate the complex and evolving landscape of digital advertising.
Final Takeaways:
Notable Quote:
"As brands diversify and regulatory landscapes shift, having a solid incrementality testing framework ensures that marketing efforts are truly driving incremental business value."
—Olivia Corey [39:02]
For those interested in enhancing their knowledge of programmatic advertising and incrementality testing, The Trade Desk Edge Academy offers a range of expert-led courses and certifications. Enroll for free at edgeacademy.thetradedesk.com to stay ahead in the ever-evolving world of digital advertising.
This summary aims to encapsulate the key discussions and insights from the podcast episode, providing a comprehensive overview for listeners and non-listeners alike.