Cody (3:29)

Yeah, we're investing more in video based channels than we ever have before. And our media mix, our creative mix looks so different year to date this year than it did a year ago the year before that. We still put most of our budget in the Meta. But even within Meta we're investing in new content campaigns that aren't going to drive that conversion right away. So for us, as we spend more into stuff like that, as we spend more into YouTube, linear TV, CTV, like there's no other way to measure these channels other than an incrementality test, at least for the first like layer of defense. We would not be able to rationalize the view content stuff we're doing in Meta, for example, if we didn't have these incrementality tests because those are serving to a much higher percentage of new people. Our consideration period is 6 months, 12 months longer than that at times. And if we were just looking at like the one day click or 30 day click benchmarks that we use our MTA tools for, that would never show up. We've tried to do it before that always get turned off. We run an incrementality test and now we are able to validate, hey, this view content stuff is driving incremental orders. It's driving incremental orders at a profitable cost per incremental order. And taking it one step further, now we are able to create a relationship between the incremental return on ads that we see in our incrementality testing tool, whether that's house, whether that's meta's user level incrementality testing and create a relationship to that data with what we're seeing in North Beam. So now we've validated the concept, we validated the tactic and now we can actually optimize it on a day to day using a very, very, very different one day click benchmark in North Beam, which is the tool we use versus our purchase conversion campaign. So like that's a great example of where we've started with this outer layer of incrementality testing to validate the concept as a whole. And then we're taking it. I mean Eric talked about like the layers of measurement. Like the outer layer here was the incrementality test and now we've been able to connect that and create a relationship between that and what we optimize our day to day off of and optimize that every single day. So things like that, like that's not going to show up in these methods of attribution. That or you know, click based, impression based. It's really impossible to validate that any other way 100%.

Connor (5:40)

There's a stat. Well, they were talking about Gen Z. There's two things here. I got it written down. Gen Z spending 25% a day consuming content, crazy stat. I don't know if that includes sleeping time, but my screen time's not too far off from that frankly, so I shouldn't be talking. But the Gen Z is less likely to click ads, right? So like just immediately that fact alone breaks down all sort of deterministic first party MTA data where it's like you couldn't possibly measure the effectiveness of an ad on an 18 year old versus a 65 year old in the same way like that 18 year old is likely just not being influenced by the same content. Clicking ads and converting in the same way, whereas what we're trying to do is like, making sure we're generating value from both of those people. And that requires different forms of measurement for those listening. There's a lot of people here who will have just seen operators with Yoni, but they talked about. I do give Meta a ton of credit for really explicitly trying to align their business platform with this idea of concurrentality. And again, that's a brand's true objective and I think has always been. And we're finally finding the words and nomenclature to talk about it. So anyway, just. I think that's been like a big. A big trend of this whole thing. Cody, I've got a question for you here. Yesterday they asked who is measuring incrementality? And there weren't a lot of hands raised. I think we are still. I think marketing operators and listeners are on the relative forefront of talking about this in the case of E commerce. So for people who are walking away today and saying, this is what I want to be measuring, this is how I want to be building my business and setting goals, what would you suggest? How do they start?

Connor Rowland (7:16)

Yeah, I mean, I think it's really cool, like, just to go back a second, and Sean said this as well, is like, Meta is the only platform that's actually encouraging everybody to go and testing for mentality. And to me, that shows a ton of confidence in what they're doing and building. It's kind of like somebody saying, like, hey, audit me, irs. Like, I'm confident in what I have here, and, like, I want you to bring in a third party to actually validate it. And I don't see the other channels doing that, you know, at all. In fact, it's actually something where we've gotten tremendous pushback from some other channels. Won't mention any of them here, but if I have to, I will after this. No, but I think it's. I think it's really cool. I also think it's interesting that, you know, two years ago was the measurement 360, and there's a lot more. Mmm. And incrementality is definitely the buzzword here. And my guess and take on that is, again, it just. Just simplifies it. And really, at the end of the day, what we are all after is incrementality. And it's easy to get so complicated with all this measurement stuff. And you have mta, you have, you know, in platform, you have incrementality, you have geolifting mentality, you have user level. And I think just really stripping it down to, like, this is the most important. This is the North Star again. You can have multiple, but the majority of businesses are doing that. I think it's really cool. So I do give Meta a lot of credit for that.

Connor (8:28)

Yeah, you know, I'll jump on because I'll call out the platform, I'll try to curry fake with my meta reps.

Connor Rowland (8:33)

We probably have a few. It's too long.

Connor (8:35)

Well, no, no, I was going to say Google because yeah, meta's saying, hey, come grade our test. We know that we're driving outcomes that align with your business goals. Google's not doing that. What Google's doing is they're moving everyone to ach int taking away our credit card points. So it's like there isn't a more stark contrast I think between the two. But you know, maybe I'm biased. Sitting here at the Meta Performance Summit, my question was what do you suggest to a brand just getting into this form of measurement?

Connor Rowland (9:03)

You know, if you're just getting into it and don't have budget or even like need for a tool, it's, it's amazing that meta has the, the user level. Anyone at any budget size can run a, you know, a holdout and a conversion loss study, which I think is so cool. So I highly recommend that. And then I think as you grow and as you scale and you want to invest in this more, I definitely think bringing on a third party incrementality tool can be really helpful. One of the reasons why, and maybe people like this as much, but you're able to compare apples to apples. One of the things for incrementality testing is not just what, what is this? You know, I roas but how does this compare to the rest of my channels and the rest of my strategies to understand like what's most effectively driving, you know, new conversions and you should really be using a consistent way to measure that across channels. And so that's where having like a, a completely independent, unified 33rd party provider, you know, we all use house here. Beloved sponsor is a great way to, you know, be like, I'm going to measure this the same exact way on meta and on Google and TV and then I can really compare very evenly 100%.

Connor (10:09)

Conor, anything you would add?

Cody (10:10)

Yeah, I think, I think meta having their own native tool that, you know, none of these other ad channels have, it really just unlocks marketers to learn more in the same amount of time. So now instead of maybe having to do like a, a tactic or a channel level test in house, like to test something within Meta you can actually handle that inside of Meta with their user level holdout test and then go use the incrementality testing tool like Houzz to measure something that you just natively cannot do in a YouTube or, or a linear TV or a CTV. So that's how we're starting to think about it is like we've unlocked new content and it's becoming a really big part of our media mix. There's another 10 tactics and optimization types that we want to test for. Let's go handle that stuff with Meta's native tool and let's go continue to validate some of these other channel tactics that do not have the natively built in testing that Meta has. So that's what we'll go in. Like for example, we validated YouTube in a major way last year. We're not going to roll into ACL'd up YouTube test. There's no other way to run a holdout on that channel except for using a GEO mentality. Hold on test to it does. YouTube does not offer that functionality. So that's the things we're going to use House for. We're going to go in deep on the video view stuff, the reach stuff, all the different campaign objectives with Meta's native tool and we're just going to learn more quicker by using both. So I really don't look at it as a one or the other. I think it allows you just to learn more faster.

Connor Rowland (11:34)

Yeah, we've test running with both of them right now. Even I've test running on NETA with both of them and I think just like the more you can build a culture of experimentation and I remember our first test that we ever ran with House, we got an IROIS back and it was like what do I do with this? It was just a different language almost compared to ROAS and cpa and obviously it's higher because not all of it is incremental. And I think where it gets really fun is when you start getting multiple experiments on your belt and your team kind of starts understanding which of the channels are performing in different ways and then you can kind of just build off of them. But I think just building that culture internally and it starts probably with the growth team and the marketing team, but then obviously you probably need like operational buy in and finance teams to buy into that as well.

Connor (12:13)

Totally. And I do want to talk more about operationalizing experimentation like you just mentioned. Eric talked about that a bit in the keynote yesterday. To go back quickly, I'd like to talk about what I would suggest for a brand, maybe not even thinking about measuring incrementality, but just trying to optimize towards incrementality. We at Ridge had done a quarter billion dollars in revenue before we ever ran a lift test. And to go back to my point around like incrementality being a sort of conceptual thing before, it needs to be necessarily like a number that you're measuring is like Kristen, our rep is here and I remember speaking with her and she's like, have you ever done conversion lift test? I'm like, no. She's what are you talking about? I described to her and I think this works for a long time, especially for smaller brands is like if you are driving impact in the business, you should be able to tell like if spend goes up, revenue should go up. And I'll see a lot of brands say, and we won't get into dunking on Google too much, but brands will. You know, if you're, let's say you're doing $10 in revenue, you're spending $3. Every day you begin spending $5, you'll want to see an increase in top line revenue. And you can at least begin to tease out that like that true impact of those additional dollars. And what you don't want to be doing is just blending down those results. You don't want to be going from a 3x Mar to a 2x Mer without increasing top line revenue. So you want to be thinking about that marginal return. And for some people it's just observing it. Like one thing we still do is we still look year over year, what's the delta in revenue? What's delta and spend and we call it like a delta. It's like a delta mer essentially. And that is in a way trying to ensure that we are driving like an incremental out. What we don't want to be doing is spending $200,000 more and generating less revenue. So if I were just beginning to think about it, I'd just be doing more like relative comparisons like that and trying to tease out that's the.

Cody (14:02)

We did the same thing before we ever ran a conversion left test too. It was all just like period over period. Here's what we did to our ad spend now. What is revenue doing? What is our post purchase survey data doing? It's really just this mindset. And like before we ever do anything in our marketing stat, before we ever think about designing the test, whatever it is, we are first designing how we're going to measure this thing before we ever were using meta's native tooling before ever using House. That's exactly what we do. I always point to this Amazon test we ran where we we cut back spend 40, 50% and we saw revenue over the next 30 days drop 5%. Like that is fundamentally not maybe the most scientifically controlled incrementality test like you would get with like a meta conversion lift or a test. But that is still a New York measuring incrementality there. And what we learned is that additional Amazon spent was not very implemental over that 30 day window and any brand of any size can do that. You do not need to spend whatever these tools cost to do that.

Connor (15:02)

Yeah, yeah, Post purchase survey is another good one. We were talking about linear TV which is extremely hard to measure via geolift to let just get super expensive as soon as you need to start targeting DMAs or something. So we use post purchase survey and like what that looks like for us and we're going to spend a couple million on TV this year is like we're just. You say it's a form of measuring incrementality, It's a form of trying to measure incrementality. Right. And my favorite quote was from, I think it was the VP of Growth at Wayfair. Someone had previously said incrementality is an old star. And what he'd said was it's more like a constellation of stars. And that like totally resonated because like you've got a couple different stars that you're trying to put together. Because I'd go even further and say like we are at the point where we are just beginning to understand the galaxy. We've got like crude telescopes, we're in like the ancient Greek era of astrology. And it's like we don't quite know what's happening, but we are putting in the tools. We have the mindset of we know where we want to go and then we're just trying our best to try to where possible, measure that type of outcome.

Cody (15:57)

Yeah, and I think, I mean that I don't know if this is a good transition into like the, the tactical ways we go about triangulation because I think like when iOS 14 happened, whenever that happened, that's when all these MTA tools became like the gold standard for measuring performance. And I still think that I know at least the hackspot, they are a very critical part of our measurement stack. But step one is measure it through an incrementality test. I'll go back to the View content example. We validated the tactic as a Whole through the incrementality test that we ran with House, we're still using MTA to scale that and optimize that on a day to day. We validated it. We created a relationship between what that incrementality return is with what we're seeing in platform and now in the day to day as it pertains the measure or optimizing the audiences, the actual ads that we're running. We're not running mentality test over and over and over again on that. We are using our MTA data to actually optimize the ads and the audience that we're serving to. So it's like that, that incrementality test becomes that kind of outermost, you know, first line of defense as it comes to measurement. Then you can optimize it using, you know, your MTA tool, your last click tool, your in platform data, whatever it is. And that's like this kind of stepwise approach to how we go from the furthermost like layer of measuring to the innermost day to day. And that's like what that triangulation kind of looks like in practice, I would say. And I think that's triangulation has become a buzzword a little bit and I think it is important to talk about like the operations of it. And that's at least how we go about it at Hedgepod.

Connor (17:27)

Yeah, yeah, yeah. And we're super similar in the way I've described it, which it was, I forget to actually describe. Oh, there's someone from resident hump Jared, friend of the pod, spends billions of dollars a year and what he was describing was like you have this top down form of measurement that's like a geolift holdout or something where it's like, hey, we don't know exactly which ads drove which conversions. Eric, keynote speaker yesterday, discussed this as like probabilistic forms of measurement. What you get is we did a holdout on Meta for the month of May. In that period you can get a pretty good sense of what your absolute incremental return was, especially if you're doing the study as scientifically as possible. But you just get it for that snapshot of time. So what you're saying is you're taking that data point which you can feel confident in, you're forming a relationship to your first party MTA data to your in platform metadata so that in June you no longer have that geo lift read, all you have is that MTA data that day to day user level data and it helps you make the decisions that you need to, in order to optimize between campaigns or ad sets or channels or whatever else.

Cody (18:29)

Yeah, yeah. And if, if you were to look just at our MTA data or in platform data and saw what we were spending on view content and you compared that to our purchase conversion, you'd be like, what are these guys doing? These are, these are terrible media buyers, they're terrible marketers. Like, it looks bad, but that's because it's a very different person we're going after in that purchase conversion versus View content. So yeah, that's exactly right. And like knowing where that backs out to over time and we will retest it to make sure it's not different at different moments in time, but. Yeah, that's exactly right.

Connor Rowland (19:01)

I just, I just think our industry, like is getting more sophisticated and it kind of has to because it's, you know, challenging times. Like ads don't perform the way that they used to prevent create iOS aim in part to that. And then also there's economic challenges, there's more competition. And so I just think the same way that a lot of us had bunch of growth to profit and there's obviously a lot of value augmentation stuff we'll talk about. I just think also the way we measure success, yes, there still potentially is a value of MTA tools and I think everybody will have their own way to do it, whether you use an mmm, whether you use mta. But I just think how success gets ventured hasn't changed. And with all of these privacy issues and changes, plus brands getting larger and more sophisticated, plus Gen Z changing their media consumption outlets, it's like the way that we even view success is changing. It's probably not even fair to say, oh, it's not working because mentality really is working. And I think, at least for me in my career, that was a big thing that I had to get my head around because I was so used to click data. And there's probably a lot of finance teams that are the same way. And I've talked to a lot of these brands like that. We're like, hey, we think that this is working for us, but we can't really justify it because we're getting a worse last clip ros or worse in platform ros. But we think it's doing something. And I've talked to marketers who don't have the tools and I really think mentality is what gives you the tools to measure that and be able to very objectively say. And I think the one part where maybe, like I caution what you guys pretend to disagree about is like I think like post purchase survey trian is good and helpful but none of it is causal. And the one thing that is really helpful about instrumentality is it is causal. So you're, you're changing one variation variable at a time and yeah, there are certain limitations but I do think it allows you to have a more objective conversation especially with a finance leader or somebody like that to be able to say like this is for a fact, for a high degree of probability what's happening when I make these change versus I think there's so much noise and the whole ecosystem that sometimes when you're changing one variable you get such a clear insight that allows us to make decisions. So I personally am in favor and kind of react rejuvenated body to focus on incrementality not the whole measurement. 360 and again they, they still have a time and place but I think if you have to pick one, do you guys agree? If you have to pick one and then MTA incrementality you go incrementality or not.

Connor (21:16)

Yeah, I struggle with this one a little bit because again I think, I think there's a difference between incrementality as a concept and overall business objective versus a form of measurement. Like I actually I think we use geo lift and incrementality synonymously which isn't quite true because I'd say MMM, MTA, GeoLift are all tools in achieving better incremental outcomes. If you had to choose one then a geolift holdup that's identifying the true impact and lift of a given tactic or new channel is totally what you want. The clarity is fantastic but there are just trade offs especially over periods of time. My point earlier around like glib studies are great but they're for a snapshot of time at a given scale at a given consumer environment, it's like largely inapplicable six months later. So you need something else to be able to make decisions. And that's why we're in the crude telescope phase of incrementality isn't.

Cody (22:17)

If I could only choose one outside tool I would choose an incrementality testing tool and then use the channel level data to optimize platform data. If I had to choose one.

Connor Rowland (22:29)

All right, let's move it on a little bit. I think one of the things that we shouldn't overlook that's really cool is I think meta is tying the optimization to the measurement where I think previously it wasn't and it was like great. All right, if Everybody's using mortality test and studies, but the system is still going towards a one day click or a seven day click. Doesn't exactly align with that. And so how do you guys feel about incremental attribution? And like do you think that that's on the right path? And there's a lot of value of saying, okay, we hear you, we know you care about incremental conversions, like why pay for non incremental ones? And we actually have a way to optimize those to ensure you're getting as many of those as possible.

Cody (23:07)

That was, that was like the biggest. Maybe call it like a breath of fresh air. For me this week is, I mean I think we've probably been thinking a lot about incrementality for the last year and a half, two years, but for the longest time like every ad channel, still optimizing just for flat out conversion. So just to hear from the meta team how much they're investing and building their advertising technology around optimizing for incremental conversions like that. I mean that's the goal. That's like the ideal state, right? Is our measurement and what the tools are actually optimizing for are 100% the same thing. And for most tools it's still not that. So it's cool to see meta like hearing what marketers care about and building their technology around that. I mean I think that is so critical.

Connor (23:53)

I love the vision. I'm super excited for the future where I get to wake up, accept the auto generated opportunities for recommendations, optimize for incremental attribution. Just print money. I'm super stoked. AI generated creative. Love it. I wish for that future we'll see how there's going to be some navigation between here and there.

Connor Rowland (24:14)

You won't really have to do that much marketing at that point yourself. You could probably just be CEO of Bridge @ that point.

Connor (24:20)

Anything else you guys want to hit on the incrementality from? Otherwise I'd like to. It's 126. I'd love to do Q and A if people have questions at maybe 15 minutes left.

Connor Rowland (24:30)

Can I give Meta some feedback? I know Yoni didn't ask me this but Andy sitting here, Sarah sitting here. I think this might be a spice one but we've done a few predictions episodes where like Connor and I tweeted and gone over some predictions and we're doing pretty well. I think you had one where you said Stephen A. Smith is going to be like a, like a mainstream, mainstream figure. I think Ross stilling pretty well. What else do we have? I feel like we talked about incrementality being one of the big buzzwords, so that's going pretty well. We did not talk about Trump and Elon having a massive breakup. We probably should have.

Connor (24:58)

That is breaking news.

Connor Rowland (24:59)

Yeah, breaking news. But one feedback that I have, or my prediction is I think meta will come out and say that we are having a hard time reaching new people and we hear that and we know that's happening. I haven't heard that for sure, but I know Yoni talked about some of the tests where you're able a bit differently. I would imagine that that's one of the next direction where there's value, there's profit right now, there's incrementality. And I think that's going to be one of the biggest focuses. Most brands that I talk to that seems like that's the biggest challenge is just we're trying creative diversity, we're trying all these different things, but it just seems like we're still showing our ads to the same people. And maybe it's a challenge of asc, maybe there's exclusions. I fundamentally believe that there are issues with exclusions and they just don't work the same after all these privacy issues. So I think that is probably what's going to be next on the roadmap and something that I think is a really big challenge today that my guess is meta will come out in the future and say, we know that this is a challenge, we hear you, and we have solutions that we think are going to help with this.

Connor (26:01)

Yeah. And I think. Well, would you say if incremental attribution does what it's branded to do, you're inherently reaching new people or just does it matter to you that you're reaching new people or do you just want to drive incremental outcomes? And do you think reaching new people is like the first step in that?

Connor Rowland (26:18)

It's a great question. You know, you're optimizing for the purchase event, which is really any purchase, and there are exclusions, but I just don't think that they work. And I think there's a lot of spillage in there. So even though they might be, hey, I want to reach only incremental conversions, I think it's really nuanced, but I think people lump together incremental and acquisition often and most brands, majority of their media spent is new customer acquisition. So I think that there have to be better tools for brands that truly only care about that as much as possible to Actually get that. So I think like for us what we find meta is very incremental. Like we've ran hold out tests for like 47% of our. It's a 47% lift. So like 47% of our new customer are of even our orders are generated by meta. Not always as profitable as we like so we got some work to do there. But more importantly our spend is going to more returning customers than we would like.

Connor (27:11)

Yeah, you and I fundamentally disagree about this. We haven't had a good fight about it.

Connor Rowland (27:15)

Well we just have different businesses that we have like 50, 60% of our business will be repeat this year and you guys are probably much, much different. So that's probably my guess.

Connor (27:28)

Totally. Yeah.

Cody (27:29)

Do you do much? Because I would argue that those tools actually like already exist in the form of just everything outside of not purchase conversion. And like that's been like I, I have you can tell. But like the most exciting thing I've had happen on our ad account in the last really probably since me and London started at hexcloud was this like we'd never had a way to measure it before and when we validated that like just the juices really got turning and also a much higher net new visit rate. So I think those tools do exist and I think they've gotten better over time and now we just have a way to measure it. But you're saying more so like within the purchase conversion at auction is what you're talking about.

Connor Rowland (28:08)

Yeah, and I think there's value in both. That was another prediction this year that like those campaigns like full funnel campaigns are going to be like just like de facto best practice strategy. So yeah, I definitely agree with that and I think especially for brands that are at our size but also your AOV and consideration like it makes total sense and then if you're only going to be able to measure that within that. But I do think there's probably some value in getting more new reach but keeping the purchase then.

Cody (28:35)

Right.

Connor Rowland (28:36)

So I think, I think that's really both. I did notice like two years ago they did talk a lot more about that at the summit of like a full funnel strategy and I didn't hear much about it today.

Cody (28:43)

Why do you guys think maybe the, maybe the optimizing towards incrementality is the full funnel strategy. Right. Like that is baked into that optimization setting. It's going to help you reach, you know, people in the funnel that would not have converted if they didn't see that ad. It's gonna help you Reach new people. So maybe that, maybe it's like it was being talked about, just not with that, that vocabulary. It's like incrementality equals more full funnel approach.

Connor (29:08)

Hopefully that would be my take too. And it goes back to like, it's something we've always been trying to do and just recently we've really like defined the term for it. And actually at this point meta is trying to optimize directly towards it versus before they were trying to convince you of it, convince you that it was happening. And frankly, do you can make the argument that for many brands that was required to drive the outcomes that you needed?

Connor Rowland (29:30)

My theory is brands are doing it more and more and so it wouldn't be as incremental for Meta to focus on that with their marketing. And so they're focusing on trying to drive to the new products. That's right.

Connor (29:41)

Yeah, yeah, yeah, that's a great point. The incremental changes in ad accounts, it's like they've already hit the saturation. They're deep on the saturation curve of the adoption of reach. And now you got to talk about.

Connor Rowland (29:50)

Something very nary, Jerry. Marginal return on that. I next plan.

Connor (29:55)

You guys want to talk about creative?

Cody (29:57)

Yeah.

Connor (29:57)

Okay, cool. So, because my takeaway is we talked basically half the day about aligning optimizations with measurement and this idea of incrementality. I think it covered that pretty, pretty comprehensively. The back half of the day was about creative diversity and AI, just high volume, diverse content. Connor McNeague, you talked about how you guys are developing creative today. Is AI playing a role in that? Like, Jason kind of spoiled it earlier that you guys are actually producing less creative. So give us a rundown of the hexag processes.

Cody (30:31)

Another reason I was so excited about what I heard yesterday and today is I said this in our group chat, but I felt like kind of the TLDR is the input.

Connor Rowland (30:44)

Hold on. That's a very online term for Connor. We joke because he's very offline. Usually we have to teach him.

Connor (30:51)

He's like doing yoga and hiking in Denver.

Cody (30:53)

They know the Internet trends before I do, and I learned from them on the podcast. The tldr though, was the same. I don't think many of the inputs that were recommended yesterday were too different from the previous three years. It's like give the system a lot of creative, a lot of different creative, and by doing so you're going to go reach more people because different people are going to resonate with different narratives, with different ad types, all the things and the big change was we just made our advertising technology way, way, way better through AI enhancements. So now we're still recommending the same inputs. It's just inputs are going to go and work way better for you and I think that's really empowering because I know like all three of us have been investing heavily into creatives over the last however long, you know, I'd say is in terms of how we're using AI. First off, I'm really excited to test out meta's like gen generative creative tools. It's mainly in like the pre production process right now. Like we're using it a lot for writing, for briefing, for scripting, stuff like that. We're going to start testing it more. We talked about this but I don't think the ad account ready stuff is quite there yet. So that's kind of where we're at with our process. We're very excited to test all of the video editing tools, all the static design tools, even some of the landing page development AI generative tools I think are all going to be really exciting. Again I don't think we're there yet but we're definitely playing around and testing it. And yeah, we're just focused on, on bigger, higher quality swings. And I think Yona, you hit on that, you're like 30 still a lot and like we've scaled back our volume but that was because we got to a point where we're launching like you know, 7, 8, sometimes 10 AD sets per week. Often with you know, anywhere between three to you know, 10 ads in that ad set. And then I'm coming back and I'm looking, it's like we got you know, no more than like you know, 8, 10 conversions in a single ad set in a seven day window. And at ROV that's like, you know, it's kind of, you know, it's a good thing to have our aov but it also means just less purchase volume through the machine learning. So that was our big thing. I was like, I think we're splitting and like segmenting our ad account way too much. So I'd say we're still producing a lot of creative but now instead of producing eight new ad sets per week, a bunch of variants under each, we're producing three, four and we're focused on like bigger thematic swings, we're focused on putting more spend through less ad sets and just getting learnings quicker, getting more skill quicker. But also like we're sitting on this library of content that we've built and produced in the last three years. And a lot of that content, like, we ran it for a week, two weeks, and we didn't like how it looked in the MTA compared to our control metrics, and we turned it off. So I think we have literally thousands of ads that we're talking about, like, what's the best way to operational wise, like a relaunch or retest strategy. We have literally thousands of ads that we've launched and ran for 10 days and turned off. So part of it's like, bigger thematic swings for the new stuff we're making. We're shooting at Nancy Silverton's house today, making ad reviews, TV commercials, all sorts of things. Like, massive swings that we've never done. When the part of it's like, let's go look at all the ads to be launched and turned off in a week or two weeks in the last three years and just like relaunch it. I mean, in theory, based on what I heard yesterday about, like, Andromeda and all these. These improvements in the tech, like, those ads should have a much better chance of performing now. So I think that's also very empowering for brands is like, go revisit the ads you've turned off. Like, those maybe were not a winner at that point in time, but they very much could be again. And we're thinking a lot about that.

Connor (34:28)

I think that's a really interesting point because we've also gone through periods of time where we've decided we've been overproducing creative. And basically that's because we haven't been happy with. You launch a bunch of ads, how well ad spend would get distributed across them. So then you end up launching more ad sets, you have more constraints, you end up building larger backlog, it's more costlier to test. And then you haven't tested all your creative and it was really expensive to do it.

Cody (34:52)

More comp. Like you're more competition. Like, you got 10 ad sets running, all the same targeting. Like. Like, who's to say that they didn't perform well just because ad set A was competing with ad set B? Like a lot of variables, 100%.

Connor (35:04)

So we're getting point where if Andromeda does what it says it does and can reference far more creative and do better ad matching, then we can produce even more creative, launch it in larger ad sets, be more cost efficient with the way we're testing it and be like, more trusting of those outcomes.

Cody (35:19)

Yeah, More simplified ad accounts. Like, like Matt was talking about letting an ad Sit there for four weeks and then all of a sudden like it gets picked up and scaled. Like that's very empowering for, for us and our teams because it's a lot less involved than, you know, having 10 campaigns and a hundred ad sets and all that.

Connor Rowland (35:35)

Yeah, that was, that was, that was what I tweeted yesterday. I'm surprised you didn't have any document. You put one of yours in there usually.

Connor (35:40)

Yeah, sorry.

Connor Rowland (35:41)

No, that's good. I said. Yeah. Based on all the AI model updates, you know, to metal system, I fail to see how creative testing in an ABO and then scale them to a, you know, asc. This best practice in the future. Because I agree with that just based on everything that they're rolling out and saying. And if Meta is really going to get that good at predicting which ads are going to perform and who to serve what. It doesn't make sense to me to force spend to certain ads. And I get that new ads don't have the same learning set as other ones. Maybe you have to put them in a new ASC with all new creative, but still not try to force spend to it. Especially I think at the scale that's going to be required. The testing structure is probably an important consideration that I think has to change. And that's something that maybe my prediction for this year, next year is like, I do think how we test creative will probably have to change as the systems evolve totally.

Connor (36:32)

And it's convenient that Ads Manager is working in a way that it can utilize more creative and the cost of creative going down to a significant level.

Connor Rowland (36:41)

Have you guys seen that though, like what Yoni was saying earlier of like, like having an ad that doesn't really do much and like four weeks later turns off because like it's got me rethinking it. I haven't seen it, but we usually test in a ABO and we'll shut things off after a week if there's no timeline. But like maybe we're finding a local maxima and like shooting ourselves in the foot by doing that.

Connor (37:00)

Yeah, we definitely err on the side of there's probably too many constraints and.

Cody (37:04)

Manual edits happening in the ad account. We'll see it in, you know, the way that we do creative testing. It's still very segmented, but we see that in our scale campaigns. Our approach with scale campaigns is like, just drop it all in there, right? Let Meta decide and like we might turn something off if it's getting a ton of spend and not performing well, but we will Absolutely. See, like an ad that Maybe was the seventh higher spender in the scale campaign for the last 30 days. All of a sudden, like, shoots off, and two weeks later, it's like the number one spender.

Connor Rowland (37:32)

I'm thinking one of my takeaways. Let me know what you think. I'm sorry. Let me know what you think of this. Is doing like a. Almost like an intermediate campaign between, like, a scale campaign and a touching campaign. But like, anything that, like, doesn't get spent or you like, like putting that in like a zombie ASC and just, like, leaving it there and seeing if it gets spent. Sarah's saying potentially, I don't know now it doesn't want to. All right, if you spend but you.

Connor (37:51)

Like it, you'll spend on it.

Connor Rowland (37:52)

Well, but you spend to it. But no, but you don't force spend to. It's still on the market.

Cody (37:56)

That is brand marketing. You heard it here for zombie can talk. Get it going. Someone tweet that?

Connor Rowland (38:01)

No, but I think to just see if it then gets spent Again, not forcing it, but seeing if the machine picks up on it maybe two weeks later. Yeah, totally. What's the heart?

Cody (38:11)

And I think the overall approach of more ads and less ad sets and let the best ad tech in the world do what it does best is why the general guidance that nodes brand should be trending towards.

Connor (38:23)

So to pivot a little bit and then we have it like five or six more minutes. And then if people have questions, I'd love to do Q and A. One thing that stuck out to me from the creative presentation yesterday was this. Well, it felt like two things were happening. One, with AI generated ads, it'll be easy for the barbershop at the Foreigner to launch ads on that and them to be pretty cool. And that's awesome. The other thing that was brought up was, like, cultural resonance, and those feel like almost completely different ideas. I personally find it hard to believe that Meta will be AI generating culturally resonant ads for rich. And that might be a good distinction of, like, advertiser scale. You brought this up earlier, Connor, where like, you guys are spending more time on culturally resonant creative. You're doing the shoot today with the OG Housewife.

Cody (39:13)

You told me. Oh, with Bethany Frankel.

Connor (39:15)

Bethany, yeah. All right, cool. So we've got that coming up that is like, Meta is never going to AI generate that ad for you. And I think that's a really interesting sort of distinction to make where it's like, yeah, what is the most valuable for brands to be Doing from a creative perspective is trying to nail that. And then it almost feels like maybe once you've got culturally resonant content, interesting thematic, creative that actually AI can just play a role in making it more diverse from like a placement perspective or like how does that video become a GIF or static ads or whatever else. So it was something that I hadn't quite thought through until the last couple days, but stuck out to me. What do you guys think?

Cody (39:52)

I think that first off I love that because I think it impact at the end of the day, digital marketing almost swung so far that people that were marketers were hanging their hat on being super technical, which is important. But at the end of the day that's not the core of marketing. Marketing's messaging and resonating with emotion. So I think this is swinging back in the direct of like you know, Meta first Econ, first brands really becoming like marketing or to not just like, oh, we're really good technical ad buyers and like we know how to operate klaviyo, whatever that, whatever the platform is that you're technical with. So yeah, I 100% think it's, you know, the, the job of the brand now is to produce the, you know, the hero asset, the single, you know, 45 second video that tells that narrative really strong. And now instead of spending that additional 10, 15, 20 hours that brands are spending, like all right, well now I need, need four new visual hooks and five new text hooks. And what if we had this graphic treatment? Like I think that hopefully becomes less of a, of a, called a burden but just less of a requirement maybe. And now you're just pumping in that initial Nancy Silver 10 or that initial Bethany Frankel video ad. And to your point, like now, now it's Meta's generative creative that's doing the text overlay, the variations, the different cups and then. And again I think it's empowering because it lets these creative strategy teams just focus on taking really big swings like create 5 to 10amazing net new videos in the next three months and then let Meta turn those 10 videos into 110 versions of those videos. So I think it hopefully means less manual doing of that stuff and just let the ad tech do that.

Connor Rowland (41:35)

Yeah, there's a quote, I've seen her on Twitter a few times about AI and I think it's like I want AI to, I don't want AI to do the creative so I can do my laundry. Like I want AI to do the laundry so I can focus on the creative. And I think like that's how I see it where it's like. I think there's almost a term that should be different than generative AI because we're not really hoping to generate a lot of. Net move assets. Iterative, was that one of you guys said earlier? I think like that's a great term. How I think about it, like we'll use AI, like AI voiceovers on a lot of our stuff. It's very hard to tell. And again like that's just like we have existing footage, how can we patch it up? Or maybe we have a creator when you just need to add in this one line. I think same thing with what kind of understand like you know, we need all the formats, right? Like that's best practice to go server. And all the formats are like I think like John was saying yesterday, it's like you don't want server static ad in, you know, a real, real placement. Like you want it to have size correctly, you want it to have music, you want to have some animation, stuff like that. And I think like how. I totally agree with you, but just taking those time consuming things and resizing grid outs and testing new hooks on them. I feel like AI should be able to do that, you know, very easily. But I don't think it's just going to be where you upload a product catalog feed and it's done. And if all of the it's no longer a lever to do targeting and buying things like that because that's done by AI, then I think really like what levers do you have? And I don't think there will be a world where everybody can just put their feet up there and how to create them because everything is going to look the same and be very similar. So I still think there's that really human element to creative and storytelling that is going to be just even more of a focus on the future.

Connor (43:11)

The job security, you're saying?

Connor Rowland (43:14)

Yeah, hopefully. Hopefully.

Connor (43:15)

Fingers crossed. The interviewer actually hit a point that I really liked. And you guys have heard me say this before, but it was with. During the Eric keynote at the end. But he said AI helping you reach the local maximum, right? You're introducing a new idea, a new concept. You maybe have to do that in fewer assets than you've had in the past because via iterative AI, which I said earlier, so I'll take credit for coining the term, it can get you to that, that, that local maximum. Make sure you're maximizing the value of it across placements and surface and things like that. And then, yeah, to your point, the. The culturally resonant stuff seems like it, but at least for some amount of time will have to come via human intervention.

Cody (43:56)

So, yeah, hopefully some amount of time.

Connor (43:58)

For some amount of time.

Connor Rowland (44:00)

I do think there's like a funny juxtaposition between, like, AI, AI, AI. And I think it's clearly going there. But also like, Meta is very clearly leaning into partnership ads and finding success with that. And it seems like brands should be focusing more on partnership ads just based on their push into it and communication. And one of the reasons those things work so well is because they are very authentic and they're real and people are engaging with creators that they might know and follow. And so it's just very different. And I think it goes to show you AI is definitely going to be the future and you will be able to create all these assets. But. But people are probably going to respond to people for a very long period of time.

Connor (44:36)

100%. Do we have any questions? We've got a mic ready for those listening he talked about, do you break out reels in order to serve in those placements? And the same sort of deal with partnership ads, sometimes they don't get delivery within, like the BAE campaign, which you guys want to take up.

Cody (44:51)

Yeah, so we do not. We do mainly automatic placements. Maybe there's a use case for not doing it, but I'd say most of the time just our evergreen operations is automatic. We are uploading 9x16s every time we upload a new creative. So when it does get served in the reels, it's going to be very native. You know, it's not going to be like a one by one with. With added, just like whatever color scale is in the creative. So that's how we approach that. Again, it's probably worth testing. I've seen brands break out reels and it works really well. We don't break out the partnership stuff on its own, but we will sometimes have like multiple scale campaigns because we'll notice that we sometimes have a tough time, like bringing in, let's say we have a creative testing ad that's in its own campaign. We launch it into our like, primary scale campaign. We see that sometimes that will not spend because it's, you know, you said this too. It's like competing with a bunch of other ads that have tons of historical data. So we'll create like a, like a new scale campaign at times where it's like, all right, let's take all the winners from the last three months. And give them their own world to operator operate in. I don't know, I mean you could maybe say that as to segment it but we just want to get as much done behind these these creatives that we've gotten good signal on. So but that's not unique to partnerships like that could be. You might have five partnerships ads with like five static ads with five like high five brand reels. Like it's not, we're not segmenting out partnerships as much as it's just based on hey this is a winning period from the last three months and we don't have high confidence that it can go and compete with like our hero Gordon whitelisted ad that's been running for two and a half years.

Connor (46:31)

One thing I would add, we have not gotten partnership or white listing ads to work. It was one of our red boxes on the opportunities for so we're working on it. I'm bought into the future there. I think we have to do it. I wrote down for what it's worth, we talk about thematic diversity in creative and partnership ads just seem like a really easy way to get to thematic diversity. So we're going to double down there. But what we are doing right now to go back to the idea of measuring incrementality, I think there's a small chance I say they don't work via our first party data. Just like doesn't seem to. It doesn't outperform it within meta ads platform. It doesn't outperform in our first party MTA data, nothing like that. I think there's a chance that it has a higher incremental impact and we should be looking at it with a higher multiple. So we're in the midst of doing a geolift holdout for whitelisting ads specifically to see if it doesn't have a similar impact to something like more top of funnel optimized ad that you'd love.

Connor Rowland (47:24)

To update us on one of the test of the week that we do once a month.

Connor (47:27)

Yeah. And if anybody else wants an update unsubscribe from marketing operators. Are there any other questions?

Connor Rowland (47:32)

Right here.

Connor (47:33)

Amazing.

Sarah Hunt (47:33)

What's up? Holy Client partner on the Shelters team. Thank you. This is awesome. In regards to creative AI, what creative AI platforms are you guys using or testing? I'm meeting with Pencil AI tomorrow and I'm curious. I didn't see any guys names on that website so have you tested them? What other ones are you guys using to kind of deploy and iterate on.

Connor Rowland (47:54)

Creative that's out there. Oh God. Sorry. Icon is really cool. That's like a newer one. I would say most of this stuff is like not there yet when you're talking about. I feel like we all want our teams to be using it because it's probably going to be very soon. I think it depends on your brand standards and things like that. And I think again, it's more there from an iterative component. I think icon is cool. We're using it. We use that for our voiceovers and you can't even tell. We can use it for some editing and just playing around with some static generation, but that's cool. I mean, obviously even just ChatGPT is doing a lot of really wild stuff. What are you guys using?

Cody (48:27)

Yeah, I mean, we're not, we're not there yet. Well, we have like assets coming like from the, the platform that's making the image and going in the ad account. We will do like, like I said, we're doing a lot of brief writing and then we'll use Chat GBT to like get a, like a low fire, like a wireframe. But that's part of the brief, right? Like that's not, not to go in the ad account account. But if it's like we have a bunch of, you know, we have a brief with a bunch of words getting direction and then maybe at the bottom of it there's like something we generated in Chat GBT and that's part of the brief, but then that's all going to a designer to actually make it ad account ready. So. But yeah, just, just Chat GPT as now.

Connor (49:04)

Yeah, I, I, I would agree with that. Like we use ChatGPT to generate a bulky old bifold on fire that had worked for a bit, that was pretty cool, but then like ingested and warped. In photoshop we use VO3. We have a hook right now of somebody chainsawing an old piece of luggage and like the clothes spraying out. That's pretty cool. In terms of being very experimental, we've got those AI generated maybe ads running right now. I don't know where that was created, but we got those going.

Connor Rowland (49:30)

Yeah, it seems like right now it's like pre production and post production, but not as much production.

Cody (49:37)

What are some of the web engineering AI tools that. Do you guys have any top of mind that you're like, this one's the closest to us being like, hey, here's a Figma file and it does all the engineering and like that page is ready to go. We're not there yet, but are there any of those that you think are front runners.

Connor (49:57)

I think you've played around with them more.

Connor Rowland (49:59)

I play around with a few of them. I haven't seen an amazing figma to code. I also think Shopify is a little different because as we've written in Liquid there's Shop dev and Instant is the owner of Shop Dev and so they're doing a pool. But one thing I'll do, we have a new custom bundle coming out in a month or two and my team and I had some ideas for how to do it and it was in my head and I couldn't find any other examples from other brands that have a bundle that's built this way. So I built it yesterday morning in Lovable which is one of them. It's like full and lovable just in 10 minutes. I even did a voice memo of this is what I want and I sent that to our designer, to our US designer as well just to be able to show the steps I wanted them to flow. I know that you guys said you're doing it but I feel like that part is really helpful so far but not quite production or anything I've seen.

Cody (50:43)

So it's really the pre production process that I think all three of us are getting the most value out of it right now.

Kristen (50:49)

You guys have heard a lot of different content this week at Summit from various speakers and from meta. What's maybe one thing for the listeners who weren't able to join today that you're going to be going back and doing differently testing new product or changing up your ad account. What's something tangible that they can take away?

Cody (51:06)

I mean the the one we immediately stacked our new E buyer was the incremental conversions. We haven't we've played around with some like a lot of the value stuff we've done but we have not not test that optimize them for that. I think that's like the no brainer like duplicate your best scale campaign and just switch that attribution setting. I think that's the most actionable and that we're. I would hope it's already live after the message we sent yesterday morning.

Connor (51:33)

You'll be happy to hear this. That question came from Kristen, our Meta rep. Two things. One thematic diversity and how can we as efficiently as possible get to thematic diversity and that's why when we're planning go to market for a new product it's like let's just get partnership ads as a line item so we can how do we build the playbook to do that more efficiently? That's One, and then two is the budget liquidity in the ad account. So we're, we're hitting the points of more ads and fewer ad sets, allowing meta to optimize across ad sets or load it all into an ase, let it do it that way. Those are two things I think are super actionable. We're like kind of beating around the bush of both of those. I think just we have to commit. I think that's the future.

Connor Rowland (52:12)

So that's when for us continued just like obsessive focus on our experimentation roadmap. So incrementality is measuring all of it obviously testing incremental fusion and then the whole Value suite to really be able to try to orient towards what we care about the most is like the biggest like strategic thing. And then just doubling down on creative and partnership ads are working for us. Seeing the focus on that, we're going to double down on that and then can always get a lot better at creative diversity.

Kristen (52:41)

And Cody, for the value suite, which is the one you're most excited about?

Connor Rowland (52:45)

Definitely.

Connor (52:47)

Five. All right, we got one more. One more.

Sarah Hunt (52:52)

So I mentioned a lot about mentality. Like we've been hitting the pavement on this for quite some time. What was the wake up call for you that like Last Click was just not it and how did you like start to change the gears in your mind to start be more focused on like incremental outcomes, adopting on house, doing commercial tests and like Cody touched on it a little bit. But I'm curious if you can go back and talk maybe like two, three years ago, like what was Cody thinking about that and how did you change to move away from Last Click?

Connor Rowland (53:18)

For us it was, I mean it was probably a plateau. So all good things and innovation comes from, you know, necessity. But I think it was a plateau where we felt like we were growing as quickly and just knew we needed something different. And it was combination again needing to reach new people. So combination of wanting to do these strategies like upper funnel reach strategies and like again like Connor mentioned before, like it just looks terrible if you're looking at it attribution. And then same thing with like YouTube where that was performing pretty well for us. We thought it was working and we were spending to a point where if it wasn't incremental our blended numbers would look worse. But I didn't really have the confidence to scale more into it at those attributions so just needed more confidence to be able to fight and go get budget for it. So that is probably what led to it for Us?

Cody (54:00)

Yeah. I think for us is we spent a lot of time scaling into video first channels and like for the longest, it took me a while to kind of like get out of the MTA tool and actually just be like, wait, hold on, let me think about how I like, like, like I watch YouTube, how do I consume YouTube ads? And like I don't click these things. And I'm like looking at the YouTube ad content we had. I'm. I'm looking at just like where like food content was trending on YouTube. I'm like, the quality is impressions. Like I just, there's something off here. Like I'm looking at like the NTA reported data and then you go like one day click from to one day view and it's like a massive jump. And like one day view is not our like discovery to consideration period. So it was some of those kind of realizations where I'm like, I'm just, we're never gonna be able to measure these channels effectively without some sort of income and topic testing tool. So that was a big one. We started just seeing our net new reach go like down and down and down in some of our channels and the, and like the platform click look really good. So I'm like, all right, well I don't know whether or not we need to be having this high a frequency with our app ads. And then you see other scenarios where you are really amazing like click based roas or CAC or whatever it is, but the net new visit rate is super low. It's like, okay, well again, do we really need to be showing these people these ads or they've converted anyway. So there are certain channels that we've actually like have the opposite where it's super, super low. Net invisible rates super high roas on a click basis, you run a holdout and some of the worst like cost rank product orders we've seen. So, so really a combination of all those things. We've just like stacked these use cases and it's like, I'm sure I'll have another five, six months route.

Kristen (55:40)

Okay, last question. My name is Sarah Hunt, I'm a client partner here at Meta and for you guys, if you were talking to the meta engineers right now and they could build any product for you, what do you want them to build?

Connor Rowland (55:54)

Cody? I know I might have noticed, but I'll let you guys go first.

Connor (55:59)

If I could have the Meta engineers build any product. Well my like tongue in cheek answers, I'd solve like a really big problem like hunger.

Cody (56:10)

Hey, buy more Hexbi what do you think we solve my hunger. I would like to, I would like to figure out like a way to. And maybe this isn't. Maybe it's already built, maybe it's incremental conversion. But I would love to see what. How can we improve the rolling reach on our purchase conversion campaigns. Like that's been been just going down and down and down which is like a function of our spend and we've unlocked a little bit with non purchase conversion. But to your point earlier is like how can we get net new reach up to 40, 50, 60% again on purchase conversion campaigns without also compromising. We are a high price point product. We need to show many ads to one person to get them to convert. So what's that sweet spot between. And we're not 100%, you know, first impression of the brand, but we're also not, you know, 8 or 9 or 10%. So like what's that sweet spot? Maybe it is the incremental conversions though.

Connor Rowland (57:05)

Yeah, I think, I think by the way you're a much better person than I am. Yesterday morning somebody from fair was showing some like the just non ad related research that you guys are doing about like helping. It's almost like the darling thing with sensory and stuff like that. And all I'm thinking about is like how does this help my ads convert better than me? Yeah, I think something like that. I think something that would be really cool is almost like a full funnel campaign in one and have different objectives in there and meta like automatically allocates different amounts of spend to a reach or to a view content or to like prospecting within one campaign and I think that would be really cool. So just kind of same along the lines of how do we make sure we're reaching new people and yeah, if you could get exclusions to work again, that would be amazing.

Cody (57:53)

Yeah, yeah, yeah.

Connor (57:54)

All right, cool.

Cody (57:54)

That's it.

Connor (57:54)

We'll call it a wrap. Thanks everyone for listening.