← Marketecture: Get Smart. Fast.
Marketecture: Get Smart. Fast.
Meta Advantage Plus and Incrementality: Findings From 640 Tests
00:26:34
Loading summary
Transcript45 lines
- [00:02]
A
By now you've probably heard about Market we put on two sold out events jam packed with the most insightful advertising content around. Speakers included Eric Seoufer, James Borrow from Universal Ads, Mark Grether from PayPal, Olivia Corey from Houzz, and of course me and the Market Extra crew. Well, this spring we're coming back bigger and better with a two day that's shaping up to be a must attend event March 10th and 11th in New York. We're putting on the new Tentpole event in collaboration with Adweek and TV Red and you absolutely need to be there. Early bird tickets are 25% off and qualified brands and agencies can be comped. Go to marketlive.com right now. That's marketecturelive.com to get the Early Bird discount.
- [00:49]
This podcast is brought to you by audiohook, the leading independent audio dsp. Audio Hook has direct publisher integrations into all major podcast and streaming radio platforms, providing 40% more inventory than what can be accessed in omnichannel DSPs. What's more, audiobook has full transcripts on more than 90% of all podcast inventory, enabling advanced contextual targeting and brand suitability. Audio Hook is so confident that in addition to CPM buys, they offer the industry's only pay for performance option where brands can scale audio and podcasting with peace of mind knowing they are only paying for outcomes. Visit audiohook.com to learn more. That's audiohook.com.
- [01:40]
B
Hi, thank you for having me. Jeremy Arie, honored and humbled to be on stage today. I appreciate you giving me the floor. We're talking about AI and outcomes today, which is something we've been digging into a lot at Houz. I'm going to spend the next 20 minutes or so walking you through a research report led by my amazing colleague at House, Tyler Horner, where we looked at hundreds of incrementality tests to understand Meta's new automated AI product, Advantage Plus. And just to kind of set this scene, I wanted to take a little trip down memory lane and reflect on what Meta ads and campaign management looked like back when I was buying them. So humor me for a moment. The year here is 2015. I was an account manager at Tube Mogul. My best friend was Jeremy's dog Charlie, this little Westie here. And like he was the dog that made me want a dog. I don't know if any of you guys have that in your life, but yeah, that's Jeremy's dog keeping me company through some late nights. And my boss was actually Matt Kiska, who's also in the crowd. But in digital ad buying at this time, like everything was manual. The name of the game is kind of test as many audiences and add permutations as you possibly can, prune those bottom performers and repeat. So it was pretty painful. Very automated or sorry, very manual, not at all automated.
- [03:07]
And then we fast forward 10 years where we are now, where we're finding ourselves kind of like nearing the logical conclusion of this evolution where Meta and Google are really taking the wheel. They're saying, hey, give us a budget and we'll do the rest. Not even give us a budget and creative anymore. It's just like give us a budget, we'll do the creative. Whether we like that or not remains to be seen. And so with this, I would say this new campaign type Meta Advantage plus is really the poster child of AI media buying. And with this, Meta is still giving advertisers a little bit of control at this point to kind of like take back some of those manual controls. But it sets up this research question, is it worth continuing to resist or is it really time to just fully embrace these AI advertising tools? @ the end of all of this automation, our brands actually better off. Like that's what we wanted to really answer when we embarked on this research. So we looked at this over 18 months. We looked across all of our house customers in 640 incrementality tests to really answer this question. As I was, as I was walking up on stage, I was remembering a moment kind of in the middle of this evolution. It was probably around 2018 or 2019. Just a funny anecdote because I was just talking about Netflix where I worked for a period of time as we were transitioning our buying at Netflix from manual really specific niche kind of interests over to like no targeting, broad based buying. Just send Meta that sign up event and let them do the rest. I was working on US digital marketing at Netflix and I was working on a show called the Ranch which starred Ashton Kutcher. And I remember having to explain to him that we're not targeting his show, we're not setting any targeting parameters and it's like this middle America sitcom. And he was just really upset and I was remembering that and I had to explain it to him. And now it's just common knowledge that Meta is better at this than we are. But I remember in the moment it was quite controversial. Just a funny anecdote talking about these incrementality tests. Why do we focus on incrementality? Eric set it up in the intro. But I'm going to tell a story that's loosely based on reality to really hammer this point home, which is, let's say I'm at Jeremy's house. He's a few years ahead of me in his career. He has a sick home theater setup. He's got the sound bar and this is talking about Sonos. I used to work there. So he has this amazing home theater set up. He's got the subwoof the surround sound and I love it, it sounds great, it looks great. I decide I'm going to buy these speakers. I go home, I do my research. I realize if you do want the sub and the surround, which you should, unless you live in New York, maybe you don't need it, I don't know. But if you want that. This is a multi thousand dollar purchase. So I decided I'm going to wait till they go on promo, maybe buy in a couple months when I get my bonus check. And in that time Meta is just hammering me with ads. But I was already going to buy. I didn't need the ads. Those are the conversions on the top. That is an ad platform reported conversion that Meta will take credit for. But it's not an incremental conversion. And we as marketers really should be striving to drive incremental outcomes. This is the age of outcomes. We want to influence outcomes that would not have happened anyway. And so that's what we're looking for. Here on the bottom is somebody who's still in consideration and that ad is actually getting them over the edge. Again, Meta doesn't know the difference. And every time Meta is going to spend on those users on the top because they're cheaper, they're lower in the funnel, they're easier to acquire. So we as marketers really need to check these systems and make sure of what we're getting. How do we do incrementality testing at house? We run Geolift holdout tests and we think Geolift has some great properties. So number one, it's a standardized methodology across all channels. You can test social in the same way you test search in the same way you test offline like television. So we like the cross channel comparability. We also like that in the post iOS 14 era we don't need to rely on any user level data. Geolift just requires aggregate kind of sales by day, by zip and then you're off to the races. So. So it's really easy from a privacy standpoint. And what we do is we say in Order to kind of get that counterfactual though, what would have happened? Anyway, we turn off marketing in some percentage of the country. We use statistical methods to make sure the region selection is balanced and that the analysis is sound. And then out the other end, after a few weeks we start populating results. And what's Also cool about GeoLift is we look not just at.com sales but, but we look at Amazon and retail as well. That's really huge, especially you know, as users just continue to buy based on their preferences. I think it's naive to think that they're going to buy where you tell them to buy, they're going to buy where they want to buy. So just a few, just a few details on the data before I get into it and I share some of the findings. Like I mentioned, 640 incrementality tests run over 18 months. Average test duration here is about 3 is about a month and then some of the language advantage plus this is Meta's AI product. We'll dig into it. Manual campaigns is what someone lovingly referred to as boomer buying. But this is the old way of buying where you add a lot of interest and you do this, you, you say, basically I think I can outsmart the machine. Incrementality factor is the ratio between the platform, the platform reported conversions and the conversions that we're reporting in our incrementality tests. Let's say Meta is reporting 10 conversions for the test period. We see five of them are incremental. That's a 0.5 incrementality factor of 50%.
- [08:54]
Iroas incremental return on ad spend. This is taking ROAS a step further and only including that incremental revenue that would not have happened anyway. And then a ptw. This is a House term post treatment observation window. We know that AD effects are not immediate. So what we do is after a test ends, we end the test, we revert back. But we follow the behavior of these markets for some period of time to understand latency and any lagging impact. So I'll refer to that observation window as the PTW or the post treatment window. Are you ready to see some data? Okay, three, three slides of findings. The next slide is not so rosy for Meta, so I felt like I needed to include this.
- [09:37]
In general, when you zoom out 10,000 foot view, Meta is working. The impact is undeniable. Meta represents 77 of the 100 highest lift experiments we've ever run at house. Now, lift is mostly a function of spend. So what this really means is that 77 of the top 100 tests just means these advertisers are spending more on Meta than any other channel. This one going from left to right here, brands are highly dependent on Metta. I think this is a fascinating stat. Metta on average is driving almost 20% lift to the brand's primary KPI, whether that's new customers or revenue. What this means is that if they turned off Metta, their business level revenue would come down by 20% overnight or over two or three weeks. Whatever the test period is, that's a really big deal. And so I felt like I needed to contextualize this. When we talk about asc just is saying that there's a reason these customers or these brands are spending so much on Meta. It really does work. Impact is more immediate than what we see with YouTube and with CTV. So 96% of the meta studies we ran detected significant lift by the midpoint. And the average lift in that post treatment observation window was only 26%. In contrast to a YouTuber, we actually see a lot more of that lift happening in the post treatment window. So you know, meta, I'd say hit hard, hits hard quickly.
- [11:00]
If you are only looking at click based attribution in the meta platform, you're going to see that it's actually under reporting the impact, which is pretty interesting because we all think meta tends to take too much credit. If you're only looking at click based attribution, they are actually under reporting by about 15% on average. Now if you're running a click plus view attribution model, we tend to see meta over reports so very dependent on the attribution window that your agency or your team is setting. And then omnichannel impact going back to Amazon and to retail. For brands doing at least 25% of their business outside of DTC, we actually see that 32% of that impact goes to Omni Channel to Amazon into retail. And what this means is if you're not looking at retail at Amazon impact, you're probably under reporting or underestimating the effect of these ads by the tune to 30%.
- [11:54]
All right, so now for the not so rosy findings. This is where things got really interesting for us. And I will say when we went into this research, I was certain that Advantage plus was going to outperform. I was ready to come in and say everyone, relinquish control, give it up, they win. And we didn't see that. Um, so this is our deep dive into the the AI buying product advantage plus is only winning 42% of the time in head to head tests. So we're seeing a 12% lower IROAs at an 18% lower spend. So even with Advantage plus spending less, it's still less efficient. This really surprised me. It was confusing for me. I had to like, I went through the seven stages as a marketer.
- [12:43]
And so I think it's close. Also, it's not like this is not a huge margin here, but you want it to be better. You want it to be better and you want to feel good about that less lagged effect with Advantage Plus. So what this means is where we were seeing about 17% of that lift coming in that post treatment window for a. We're seeing a lot more lift coming in the post treatment window for manual. So it's hitting harder, quicker, which makes sense. Then Advantage plus is actually over reporting incrementality by about 12 percentage points on average relative to manual. So this is, I think I have a few theories for why this is happening. It's the headline of this slide. Is the algorithm actually too good? Is what I've been wondering. Is it so good at finding purchases that it's actually targeting these people who are already going to buy? It's like circling the bottom of the drain. That's one theory for why this is happening. Then the other theory is that we looked over 18 months. It's possible that with the Andromeda updates things have gotten much better. And so if we cut the data based on more recent studies, maybe this tells a different story, which is something that we're going to look into.
- [13:58]
Pausing here for a quick reminder. When we published this research back in July, we tried to capture all the nuance here and we even put in the research report. This is not an indictment of asc. Then everybody started grabbing the headline of like ASC doesn't work. I knew it. Don't use it. And so this is a screenshot of a LinkedIn post saying like, according to House, you should stop using asc. That's not what this means at all. This means that for 42% of the brands in the study, Advantage plus actually did outperform Manual. So it's close to. And the learning here. And the correct takeaway is that you need to be testing this for yourself. So what do we do about this with the a learning, knowing and wondering if this is just kind of capturing users who are already going to purchase or it's really good at identifying intent. What do you do about it as marketers? And so that leads to what I Think Eric Seifert coined as signal engineering. I'm certain he's going to talk about this in his presentation today of what if we point Meta toward a different optimization event? What if we say don't optimize for purchases, actually go help us prospect and find new audiences. This is the any marketer D2C you talk to right now is saying my rolling reach on Meta is stagnant. I'm not introducing the brand to new audiences. It's not a prospecting tool anymore. And so this is their answer to that. They're trying to kind of hack the system with what we're calling signal engineering, where it's saying instead of pointing Meta and these systems toward a purchase, what if we point it to something shallower in the funnel like an add to cart or a site visit or a PDP view. And so that's a way in which folks are trying to hack the system and prospect. We're seeing a huge increase here, 121% in testing frequency. We're actually seeing pretty good results here. So just 14% lower Iroas than purchase optimization, albeit much lower spend though we are starting to see the spend here picked up pick up quite a bit. So 85% lower daily spend. Really hard to compare these apples to apples, but encouraging signals that this might be a way to kind of get Meta out of that bubble and introduce your brand to new customers. And then we're also seeing with mid funnel optimization, stronger omnichannel effects, so better impact on Amazon and retail and also more lagging impact in terms of long term effectiveness. So this is encouraging. We're seeing a lot more of this and I think it's only going to increase as the system gets better at identifying intent. So rounding this out, this is my last slide. What to take away from all of this. Number one, Meta consistently delivers, but brands really need to be testing this trade off between automation and incrementality. Number two, let's learn to think of Advantage plus as the intent identifier. Really, really good at doing the thing that you tell it to do. And you need to remember that number three, we need to balance out these Advantage plus campaigns with mid and upper funnel strategies as well. And then lastly, I mentioned this a couple slides ago, but do not take this as gospel. You really need to test for your own business. Every brand is a little bit different, you know, based on how long you've been spending on Meta, based on your consideration cycle, the price point of your brand. So please test for your own business.
- [17:27]
And yeah, with That I actually want to open it up for questions, if anybody has any. I think we have like you know, five or so minutes and happy to take any questions from the.
- [17:38]
C
Audience. Hey Olivia, I was going to ask from the companies that showed that Advantage plus worked particularly well, were there any patterns you saw amongst those companies? I know the spend was about just over a million a month, but.
- [17:50]
A
Any other patterns in terms of how well established their brand was or other.
- [17:54]
C
Sales channels or other media channels they're running.
- [17:56]
B
On? Yeah, the one I think this makes intuitive sense, but smaller brands are having more success with this and I, I do wonder if it's like if you try to beat Meta and you try to outsmart Meta, you better be really good at media buying, you know, Whereas these smaller brands who are newer to Meta, there's no way they're going to be able to outsmart the system. So we did see Advantage plus looking better for smaller brands. And one other interesting finding, I don't know what to do with this is like if you're hedging, if you're not sure, so you're doing 50% manual and 50% advantage plus that actually didn't work as well. So you kind of want to like go all in on one strategy and just commit to it because we saw the. And maybe there's some duplication happening and bidding against yourself, but using both campaign types alongside one another didn't seem to work very.
- [18:45]
A
Well.
- [18:47]
Hi Olivia, I was kind of trying to interpret the results as maybe longer purchase cycles might be better off with Manual because you did indicate that the like the look back window or whatever you're calling it is less impacted by their AI. Is that a good kind of.
- [19:04]
B
Assumption? I think so. I think that's a really good theory that the Advantage plus works really well at identifying intent quickly. So if you're a longer consideration product then it's not going to work as well. That makes sense to me. Who knows? I wish would be really cool if they peeled back the curtain a little bit on the algorithm and what is actually a really good signal for.
- [19:30]
What are the intent signals that most correlate to buying or to purchase. I think that that would be really.
- [19:37]
C
Cool. Olivia, I have a quick one. You said that they're over reporting incrementality. I know Meta has their own incrementality attribution product. Anything more you can say about that product and how that.
- [19:51]
B
Works? Yes, I'm really glad you asked this question. So incremental attribution is the really, I think a hot topic Right now they just rolled this out in the general release recently and what they're saying here is you can select incrementality as the optimization setting. And we looked into this as part of this research. The results are very mixed right now. And my theory on that, on why it may not like again in theory this sounds great, right? Like of course I want to drive incremental outcomes and this model should be delivering on that. My theory is like their legacy optimization product has been running forever. This is a new thing, it's going to take some time to work. They just probably have a little bit less data. Like if you think about what's feeding that model, it's conversion lift studies. You need a very high volume of incrementality studies to train that algorithm and maybe it's just too new, but you've got to hope and you have to assume that it's going to get better over time and it will get to parity. So the incremental optimization product should be a good antidote to this. If you think, you know, with signal engineering, if you're pointing it towards incremental users, it should be working.
- [20:58]
C
Better. So two questions. The first one is you said Andromeda made a difference. The look back was 18 months. So in the closer to the later half of the like the six months path, did Andromeda make a big difference?
- [21:15]
Their whole like Andromeda lattice, all their AI stuff that they mentioned keep mentioned on learning.
- [21:19]
B
Costs. Yeah. So I wish I had the answer to that question. We actually haven't cut the data in that way yet. But that's a theory that is a current kind of hypothesis for why Advantage plus didn't win out in this study is maybe we look back over 18 months. Maybe if we look at the last six months, maybe this tells a better story. I don't have that data but it's something that we're going to look.
- [21:39]
C
At. All right, so, so then that's the follow up. Right. So from my understanding is the new systems focus mostly around how creatives is served and the efficiency. So did you guys, when you do these incremental like look back windows, did you account for the CTRs within the platform themselves? Like for the creatives or it's just mostly all the posts like go to the website, we see the conversion, all the pixel.
- [22:01]
B
Data. Yeah. So your question is are there any trends in terms of the creative that was running and any sort of correlation with.
- [22:08]
C
Incrementality? Incrementality is focused on the whole.
- [22:13]
A
Chain.
- [22:13]
C
Right. You look at the Whole chain but you rarely. I don't understand how does like the click through rates on the platform, on the creatives will affect the incrementality on the back end. So did you guys see any of those pattern or just mostly measure the back end stuff like on the.
- [22:30]
B
Website? Yeah, that's a really good question. We didn't look at that at all. I think a lot of these D2C brands, Nick probably knows you can look at CTR, you can look at CPMs, but at the end of the day all they care about is the cpa. So that is something that we could look at in terms of the ctr. Is it correlated with better incremental returns? We didn't actually look at that because then what do you do with that? Do you end up like do you optimize for CTR and does that confuse things or perhaps muddy the picture? So definitely an interesting question and something we should look.
- [23:04]
C
At. Based on Ari's question, I had a question on did the results that you guys were saw, did any of the results, were they impacted by looking at your immediate attribution windows or the lag effects or combination of.
- [23:20]
B
Both? So question question here is did, did the time window impact results? Okay, this is a great, I love this question because when you look at YouTube, we were just recently released a YouTube report earlier in the year and it got a lot of traction and we saw like 70% of the effect of YouTube was happening in the post treatment window. Here is like the opposite. Like there's like only 20 or 30% of the effects are happening in the post treatment window. Which signals to me that meta is more immediate. It's, it's much down, much more like down funnel in terms of capturing that.
- [23:56]
C
Intent. Was it the same for the 42% versus like the.
- [24:00]
B
58? So advantage plus worked quicker, less lagged effects than.
- [24:05]
C
Manual. Oh, interesting. Okay, my second question was.
- [24:11]
You know, in the, the signal engineering hypothesis that you guys have, would that be running that model in lieu of a low funnel model or both.
- [24:22]
B
Simultaneously? Yep, that's a great question. Right now brands are carving out like 20% of their budget for the signal engineering mid funnel optimization. And then as it works they're moving more and more budget over from purchase optimize. But it's definitely in a long side right now they're not replacing purchase optimization with mid funnel entirely. So they're just taking that like that last tranche of the purchase spend. Basically if you think like your last dollars in are the least efficient, they're taking that out putting it in mid funnel and those first dollars in mid funnel are a lot more efficient than those last dollars in.
- [24:56]
D
Purchase. I liked this kind of signal engineering topic as well. I think we have often found that the meta algorithm finds its solution very quickly and I think that's part of kind of appeasing the person who's putting their money in there. Like you show them results really early on, you see how Meta quickly kind of finds the right creative and I think I'm wondering whether you see differences in this kind of holdout area versus the non holdout area in kind of this focus on the people in the checkout lane. Right. I mean I think part of the reason could be that.
- [25:32]
Meta is very good at finding the kind of people that you know are in this kind of your first example where they, where they will buy the product in any case. And do you find in this kind of mid funnel, upper funnel, do you see more evidence of these people taking these steps that you would identify with kind of an early consideration.
- [25:49]
B
Customer? Yeah, you know, I want to be running longer tests. Like that is something that's a personal goal for 2026 is like that. It's very hard to answer that question without running a longer experiment and actually having evidence of that playing out. But you have to assume that for the mid funnel, for the signal engineering that there is more effect here happening later on that we don't see in a three, four or six week test even. And so this is something I just.
- [26:19]
Something I wish of marketers is that we had a little more patience. There's so many interesting insights that you could unlock with a six month experiment, but nobody wants to run six month tests. I try all the time and I.
- [26:31]
A
Fail.
- [26:31]
B
So. All right, I'll leave you with that. Thank you so much everybody. Appreciate.
- [26:35]
A
It.
- [26:49]
Thank you for listening to the marketecture podcast. New episodes come out every Friday and an insightful vendor interview is published each Monday. You can subscribe to our library of hundreds of executive interviews at marketecture tv. You can also sign up for free for our weekly newsletter with my original strategic insights on the week's news at News Market tv. And if you're feeling social, we operate a vibrant Slack community that you can apply to join@adtechgod.com.