B (8:54)

Iroas incremental return on ad spend. This is taking ROAS a step further and only including that incremental revenue that would not have happened anyway. And then a ptw. This is a House term post treatment observation window. We know that AD effects are not immediate. So what we do is after a test ends, we end the test, we revert back. But we follow the behavior of these markets for some period of time to understand latency and any lagging impact. So I'll refer to that observation window as the PTW or the post treatment window. Are you ready to see some data? Okay, three, three slides of findings. The next slide is not so rosy for Meta, so I felt like I needed to include this.

B (9:36)

In general, when you zoom out 10,000 foot view, Meta is working. The impact is undeniable. Meta represents 77 of the 100 highest lift experiments we've ever run at house. Now, lift is mostly a function of spend. So what this really means is that 77 of the top 100 tests just means these advertisers are spending more on Meta than any other channel. This one going from left to right here, brands are highly dependent on Metta. I think this is a fascinating stat. Metta on average is driving almost 20% lift to the brand's primary KPI, whether that's new customers or revenue. What this means is that if they turned off Metta, their business level revenue would come down by 20% overnight or over two or three weeks. Whatever the test period is, that's a really big deal. And so I felt like I needed to contextualize this. When we talk about asc just is saying that there's a reason these customers or these brands are spending so much on Meta. It really does work. Impact is more immediate than what we see with YouTube and with CTV. So 96% of the meta studies we ran detected significant lift by the midpoint. And the average lift in that post treatment observation window was only 26%. In contrast to a YouTuber, we actually see a lot more of that lift happening in the post treatment window. So you know, meta, I'd say hit hard, hits hard quickly.

B (11:00)

If you are only looking at click based attribution in the meta platform, you're going to see that it's actually under reporting the impact, which is pretty interesting because we all think meta tends to take too much credit. If you're only looking at click based attribution, they are actually under reporting by about 15% on average. Now if you're running a click plus view attribution model, we tend to see meta over reports so very dependent on the attribution window that your agency or your team is setting. And then omnichannel impact going back to Amazon and to retail. For brands doing at least 25% of their business outside of DTC, we actually see that 32% of that impact goes to Omni Channel to Amazon into retail. And what this means is if you're not looking at retail at Amazon impact, you're probably under reporting or underestimating the effect of these ads by the tune to 30%.

B (11:54)

All right, so now for the not so rosy findings. This is where things got really interesting for us. And I will say when we went into this research, I was certain that Advantage plus was going to outperform. I was ready to come in and say everyone, relinquish control, give it up, they win. And we didn't see that. Um, so this is our deep dive into the the AI buying product advantage plus is only winning 42% of the time in head to head tests. So we're seeing a 12% lower IROAs at an 18% lower spend. So even with Advantage plus spending less, it's still less efficient. This really surprised me. It was confusing for me. I had to like, I went through the seven stages as a marketer.

B (12:42)

And so I think it's close. Also, it's not like this is not a huge margin here, but you want it to be better. You want it to be better and you want to feel good about that less lagged effect with Advantage Plus. So what this means is where we were seeing about 17% of that lift coming in that post treatment window for a. We're seeing a lot more lift coming in the post treatment window for manual. So it's hitting harder, quicker, which makes sense. Then Advantage plus is actually over reporting incrementality by about 12 percentage points on average relative to manual. So this is, I think I have a few theories for why this is happening. It's the headline of this slide. Is the algorithm actually too good? Is what I've been wondering. Is it so good at finding purchases that it's actually targeting these people who are already going to buy? It's like circling the bottom of the drain. That's one theory for why this is happening. Then the other theory is that we looked over 18 months. It's possible that with the Andromeda updates things have gotten much better. And so if we cut the data based on more recent studies, maybe this tells a different story, which is something that we're going to look into.

B (13:58)

Pausing here for a quick reminder. When we published this research back in July, we tried to capture all the nuance here and we even put in the research report. This is not an indictment of asc. Then everybody started grabbing the headline of like ASC doesn't work. I knew it. Don't use it. And so this is a screenshot of a LinkedIn post saying like, according to House, you should stop using asc. That's not what this means at all. This means that for 42% of the brands in the study, Advantage plus actually did outperform Manual. So it's close to. And the learning here. And the correct takeaway is that you need to be testing this for yourself. So what do we do about this with the a learning, knowing and wondering if this is just kind of capturing users who are already going to purchase or it's really good at identifying intent. What do you do about it as marketers? And so that leads to what I Think Eric Seifert coined as signal engineering. I'm certain he's going to talk about this in his presentation today of what if we point Meta toward a different optimization event? What if we say don't optimize for purchases, actually go help us prospect and find new audiences. This is the any marketer D2C you talk to right now is saying my rolling reach on Meta is stagnant. I'm not introducing the brand to new audiences. It's not a prospecting tool anymore. And so this is their answer to that. They're trying to kind of hack the system with what we're calling signal engineering, where it's saying instead of pointing Meta and these systems toward a purchase, what if we point it to something shallower in the funnel like an add to cart or a site visit or a PDP view. And so that's a way in which folks are trying to hack the system and prospect. We're seeing a huge increase here, 121% in testing frequency. We're actually seeing pretty good results here. So just 14% lower Iroas than purchase optimization, albeit much lower spend though we are starting to see the spend here picked up pick up quite a bit. So 85% lower daily spend. Really hard to compare these apples to apples, but encouraging signals that this might be a way to kind of get Meta out of that bubble and introduce your brand to new customers. And then we're also seeing with mid funnel optimization, stronger omnichannel effects, so better impact on Amazon and retail and also more lagging impact in terms of long term effectiveness. So this is encouraging. We're seeing a lot more of this and I think it's only going to increase as the system gets better at identifying intent. So rounding this out, this is my last slide. What to take away from all of this. Number one, Meta consistently delivers, but brands really need to be testing this trade off between automation and incrementality. Number two, let's learn to think of Advantage plus as the intent identifier. Really, really good at doing the thing that you tell it to do. And you need to remember that number three, we need to balance out these Advantage plus campaigns with mid and upper funnel strategies as well. And then lastly, I mentioned this a couple slides ago, but do not take this as gospel. You really need to test for your own business. Every brand is a little bit different, you know, based on how long you've been spending on Meta, based on your consideration cycle, the price point of your brand. So please test for your own business.

B (17:27)

And yeah, with That I actually want to open it up for questions, if anybody has any. I think we have like you know, five or so minutes and happy to take any questions from the.

C (17:37)

Audience. Hey Olivia, I was going to ask from the companies that showed that Advantage plus worked particularly well, were there any patterns you saw amongst those companies? I know the spend was about just over a million a month, but.

A (17:49)

Any other patterns in terms of how well established their brand was or other.

C (17:53)

Sales channels or other media channels they're running.

B (17:55)

On? Yeah, the one I think this makes intuitive sense, but smaller brands are having more success with this and I, I do wonder if it's like if you try to beat Meta and you try to outsmart Meta, you better be really good at media buying, you know, Whereas these smaller brands who are newer to Meta, there's no way they're going to be able to outsmart the system. So we did see Advantage plus looking better for smaller brands. And one other interesting finding, I don't know what to do with this is like if you're hedging, if you're not sure, so you're doing 50% manual and 50% advantage plus that actually didn't work as well. So you kind of want to like go all in on one strategy and just commit to it because we saw the. And maybe there's some duplication happening and bidding against yourself, but using both campaign types alongside one another didn't seem to work very.

A (18:44)

Well.

A (18:47)

Hi Olivia, I was kind of trying to interpret the results as maybe longer purchase cycles might be better off with Manual because you did indicate that the like the look back window or whatever you're calling it is less impacted by their AI. Is that a good kind of.

B (19:04)

Assumption? I think so. I think that's a really good theory that the Advantage plus works really well at identifying intent quickly. So if you're a longer consideration product then it's not going to work as well. That makes sense to me. Who knows? I wish would be really cool if they peeled back the curtain a little bit on the algorithm and what is actually a really good signal for.

B (19:29)

What are the intent signals that most correlate to buying or to purchase. I think that that would be really.

C (19:37)

Cool. Olivia, I have a quick one. You said that they're over reporting incrementality. I know Meta has their own incrementality attribution product. Anything more you can say about that product and how that.

B (19:51)

Works? Yes, I'm really glad you asked this question. So incremental attribution is the really, I think a hot topic Right now they just rolled this out in the general release recently and what they're saying here is you can select incrementality as the optimization setting. And we looked into this as part of this research. The results are very mixed right now. And my theory on that, on why it may not like again in theory this sounds great, right? Like of course I want to drive incremental outcomes and this model should be delivering on that. My theory is like their legacy optimization product has been running forever. This is a new thing, it's going to take some time to work. They just probably have a little bit less data. Like if you think about what's feeding that model, it's conversion lift studies. You need a very high volume of incrementality studies to train that algorithm and maybe it's just too new, but you've got to hope and you have to assume that it's going to get better over time and it will get to parity. So the incremental optimization product should be a good antidote to this. If you think, you know, with signal engineering, if you're pointing it towards incremental users, it should be working.

C (20:58)

Better. So two questions. The first one is you said Andromeda made a difference. The look back was 18 months. So in the closer to the later half of the like the six months path, did Andromeda make a big difference?

C (21:14)

Their whole like Andromeda lattice, all their AI stuff that they mentioned keep mentioned on learning.

B (21:19)

Costs. Yeah. So I wish I had the answer to that question. We actually haven't cut the data in that way yet. But that's a theory that is a current kind of hypothesis for why Advantage plus didn't win out in this study is maybe we look back over 18 months. Maybe if we look at the last six months, maybe this tells a better story. I don't have that data but it's something that we're going to look.

C (21:38)

At. All right, so, so then that's the follow up. Right. So from my understanding is the new systems focus mostly around how creatives is served and the efficiency. So did you guys, when you do these incremental like look back windows, did you account for the CTRs within the platform themselves? Like for the creatives or it's just mostly all the posts like go to the website, we see the conversion, all the pixel.

B (22:00)

Data. Yeah. So your question is are there any trends in terms of the creative that was running and any sort of correlation with.

C (22:08)

Incrementality? Incrementality is focused on the whole.

A (22:12)

Chain.

C (22:13)

Right. You look at the Whole chain but you rarely. I don't understand how does like the click through rates on the platform, on the creatives will affect the incrementality on the back end. So did you guys see any of those pattern or just mostly measure the back end stuff like on the.

B (22:29)

Website? Yeah, that's a really good question. We didn't look at that at all. I think a lot of these D2C brands, Nick probably knows you can look at CTR, you can look at CPMs, but at the end of the day all they care about is the cpa. So that is something that we could look at in terms of the ctr. Is it correlated with better incremental returns? We didn't actually look at that because then what do you do with that? Do you end up like do you optimize for CTR and does that confuse things or perhaps muddy the picture? So definitely an interesting question and something we should look.

C (23:04)

At. Based on Ari's question, I had a question on did the results that you guys were saw, did any of the results, were they impacted by looking at your immediate attribution windows or the lag effects or combination of.

B (23:20)

Both? So question question here is did, did the time window impact results? Okay, this is a great, I love this question because when you look at YouTube, we were just recently released a YouTube report earlier in the year and it got a lot of traction and we saw like 70% of the effect of YouTube was happening in the post treatment window. Here is like the opposite. Like there's like only 20 or 30% of the effects are happening in the post treatment window. Which signals to me that meta is more immediate. It's, it's much down, much more like down funnel in terms of capturing that.

C (23:56)

Intent. Was it the same for the 42% versus like the.

B (24:00)

58? So advantage plus worked quicker, less lagged effects than.

C (24:05)

Manual. Oh, interesting. Okay, my second question was.

C (24:11)

You know, in the, the signal engineering hypothesis that you guys have, would that be running that model in lieu of a low funnel model or both.

B (24:21)

Simultaneously? Yep, that's a great question. Right now brands are carving out like 20% of their budget for the signal engineering mid funnel optimization. And then as it works they're moving more and more budget over from purchase optimize. But it's definitely in a long side right now they're not replacing purchase optimization with mid funnel entirely. So they're just taking that like that last tranche of the purchase spend. Basically if you think like your last dollars in are the least efficient, they're taking that out putting it in mid funnel and those first dollars in mid funnel are a lot more efficient than those last dollars in.

D (24:55)

Purchase. I liked this kind of signal engineering topic as well. I think we have often found that the meta algorithm finds its solution very quickly and I think that's part of kind of appeasing the person who's putting their money in there. Like you show them results really early on, you see how Meta quickly kind of finds the right creative and I think I'm wondering whether you see differences in this kind of holdout area versus the non holdout area in kind of this focus on the people in the checkout lane. Right. I mean I think part of the reason could be that.

D (25:32)

Meta is very good at finding the kind of people that you know are in this kind of your first example where they, where they will buy the product in any case. And do you find in this kind of mid funnel, upper funnel, do you see more evidence of these people taking these steps that you would identify with kind of an early consideration.

B (25:48)

Customer? Yeah, you know, I want to be running longer tests. Like that is something that's a personal goal for 2026 is like that. It's very hard to answer that question without running a longer experiment and actually having evidence of that playing out. But you have to assume that for the mid funnel, for the signal engineering that there is more effect here happening later on that we don't see in a three, four or six week test even. And so this is something I just.

B (26:18)

Something I wish of marketers is that we had a little more patience. There's so many interesting insights that you could unlock with a six month experiment, but nobody wants to run six month tests. I try all the time and I.

A (26:30)

Fail.

B (26:31)

So. All right, I'll leave you with that. Thank you so much everybody. Appreciate.

A (26:35)

It.

A (26:49)

Thank you for listening to the marketecture podcast. New episodes come out every Friday and an insightful vendor interview is published each Monday. You can subscribe to our library of hundreds of executive interviews at marketecture tv. You can also sign up for free for our weekly newsletter with my original strategic insights on the week's news at News Market tv. And if you're feeling social, we operate a vibrant Slack community that you can apply to join@adtechgod.com.