Turning Incrementality Tests Into Action That Makes You Money

Summary

---

Podcast Summary: Ecommerce Playbook – “Turning Incrementality Tests Into Action That Makes You Money”

Date: February 26, 2026
Hosts: Taylor Holiday & Richard Gaffin (Common Thread Collective)
Guests:

• Olivia (Chief Strategy Officer, Haus)
• George (CMO, Cozy Earth)

---

Episode Overview

This episode takes a deep dive into one of the most complex and pressing challenges for modern DTC (Direct-to-Consumer) brands: how to not just conduct incrementality and holdout tests on marketing spend, but how to take the results and operationalize them into clear, profitable actions that drive business outcomes. Hosts Taylor Holiday and Richard Gaffin are joined by Olivia from Haus (a measurement company specializing in experimentation) and George from Cozy Earth (a leading home goods and apparel brand) to candidly discuss the messy realities behind incrementality testing, how it upends commonly held beliefs about marketing performance, and what it really takes to make such test results actionable in day-to-day marketing operations.

---

Key Discussion Points & Insights

1. The Incrementality Testing “Journey” (10:08–13:28)

• Initial Shock, Then Disillusionment: Many brands start with enthusiasm about incrementality testing, but are often shocked by lower-than-expected incremental ROAS (Return on Ad Spend). This is especially true for brands used to reporting much rosier numbers through in-platform metrics.
• Olivia: “We’re sort of through the first chapter of, like, 'okay, we need this.' And now it’s, 'How do I use this as an operating system and really operationalize this?'” (05:43)
• Testing Is a Methodology, Not a One-Off: Establishing testing as a recurring methodology—rather than a series of isolated events—is critical as brands mature.

2. When to Start Incrementality Testing (21:00–24:41)

• Brands shouldn’t wait until they’re “big enough;” issues may arise when organic and repeat demand increases, complicating what used to be clear attribution paths.
• Taylor: “Incrementality is a step to take when you’ve built distrust in your present measurement system… change happens if and only if dissatisfaction is large enough.” (21:34)
• Misattribution of all new customer revenue to Meta (Facebook/Instagram) remains a common, flawed assumption for many DTC founders.

3. Understanding Holdout and Incrementality Tests (19:08–20:59)

• Olivia: “We fundamentally believe at Haus that incrementality requires a holdout. You need a counterfactual to understand what would have happened anyway... Otherwise, you can’t attribute causation.” (19:08)
• There’s a stark difference between “turn-off” natural experiments and true randomized controlled holdouts.

4. Operationalizing Test Results – Core Challenges (13:28–38:43)

• The Replication Crisis: “One month it’s this, and the next month it’s that. And there’s no clear reason why the result changes so much. It can be frustrating.” – George (14:27)
• Testing Frequency vs. Value: Running more tests doesn't necessarily create more value—only actioning on test results matters.
• Taylor: “Are you running a test, or is testing a part of the way you’re going to do measurement? Those are very different places.” (13:17)
• No Universal “North Star”: The best efficiency metric for marketing should be directly tied to organizational goals—whether that's free cash flow, contribution margin, or TTM EBITDA. Only then does incremental contribution margin become an actionable metric for channel investment. (26:57–29:34)

5. Applying Incrementality Factors (“IFs”) in Budgeting (34:05–38:43)

• Cozy Earth approach: Apply an incrementality “factor” (the ratio between incrementality-tested and in-platform ROAS) to in-platform performance to assess the true impact, but results can be highly volatile.
• Olivia: “If you get a read that’s, you know, 1/3...and then six months later you get 0.8...that’s just too blunt of an instrument to actually be useful in decision making.” (36:58)

6. Actionable Frameworks for Brands (44:03–88:45)

The Three Haus Playbooks (44:09):

• Efficiency: Cut costs.
• Redistribution: Move budget from underperformers to best performers at a flat budget.
• Scale Up: Invest incremental spend in new channels to drive growth.

“As long as you’re moving budget from low performers to higher performers, you will get better. It still might not be where you should be.” – Olivia (44:09)

Redistributing Budget: Channel vs. True Return

• The real threshold is not “channel-vs-channel,” but “channel-vs-potential scale of profitable return.” (47:03–48:05)
• Cutting unsuccessful media is necessary—there’s no reason to deploy capital that doesn't generate profit, even for the sake of “growth”.

Frequency and Method of Redistributing Budgets

• Channel-level factors: Most leading brands assign incrementality factors by channel, then use either MMM (Marketing Mix Modeling) or other frameworks for reallocation.
• Reallocation cadence: Monthly is common for channel-level redistribution, but day-to-day tactical adjustments are often made in response to campaign performance and overall contribution margin.

---

Memorable Quotes & Moments

• Taylor: "How does your organization decide what’s true? What is the standard for truth inside your company?" (24:41)
• George: "For us as a brand, it’s really difficult to figure out what creates the outcome. Still figuring it out. We don’t have it down to a science yet, but we’re working on it." (14:27)
• Olivia: "There are a bunch of brands who will fill those testing slots. And it’s testing for the sake of testing...it’s not how we drive value for our customers." (13:28)
• Taylor: “If you are deploying money and it’s generating negative money...you’re accelerating...you're killing yourself faster, deploying the capital than if you stopped.” (50:14)
• On Model Trust: "I actually have more confidence in Olivia than I did the prior team." – George (41:29)
• Re: Long-Term Effects: "The idea that you’re generating new customer revenue where your incremental return nine months later is growing, like, hundreds of percent, is deceptive to the point of criminal behavior." – Taylor (60:39)
• On Brand Owner Mindset: “If my head of marketing came to me and was like, ‘Hey, Taylor, we're going to invest your dollars and you’re going to get them back never, but it’s going to grow the top line,’ I’d be like, no.” (54:38)

---

Practical Recommendations & Tactics

When Facing Poor Incrementality Results:

1. Creative Testing:
   "Most often when I see step-change improvements in performance, it’s a creative swing." – Olivia (82:31)
2. Expand Distribution:
   Moving into new channels like Amazon or retail can surface hidden incremental returns often missed when only measuring D2C.
3. Account Structure:
   Audit retargeting levels (especially in enterprise accounts), optimize click vs. view-based segments, and assess signal engineering (sometimes focusing on shallower conversion events can help).
4. Cost Controls:
   Channel-wide cost controls (e.g., cost caps in Meta) may help drive efficiency, but need close calibration and testing.
5. Narrow the Mix:
   Excessive channel complexity (“channel diversity is the killer of clarity”) can make actionable learnings impossible. Focus investment on big ideas in your best-performing channels.

---

Defining “Success” with Incrementality Testing

• It's a Journey, Not a Switch: Implementing and actioning incrementality results is an ongoing process of progressive refinement—a continual effort to be less wrong over time, not a one-and-done event.
• Progress is Error Band Reduction: The real sign of progress is not always increasing incremental ROAS, but a tightening between measured incremental outcomes and your expectations—i.e., less volatility and fewer huge surprises.
• Measurement ≠ Fixing: “Measuring things doesn’t make them good. It just assesses what is.” (82:00) The real work is in taking hard results and building business strategy around them—and sometimes, that means acknowledging tough truths and changing behaviors.

---

Timestamps for Major Segments

• Setting the problem & bios: 01:16–07:11
• The journey of incrementality: 10:08–13:28
• Types & setups of tests: 19:08–20:59
• When should brands start testing?: 21:00–24:41
• How to apply incrementality findings: 34:05–38:43
• Action plans & frameworks: 44:03–88:45
• Optimizing creative/distribution/structure: 82:13–83:13
• Big picture takeaways: 88:45–end

---

Final Takeaway

Incrementality testing exposes uncomfortable truths for DTC marketers, but embracing it as an ongoing operating system is the only path to informed and sustainable growth. The journey is never done—actionability relies not just on measurement, but on organizational clarity, brutal honesty with results, and the discipline to channel dollars toward what’s truly profitable, not just what fits convenient narratives.

---

Memorable Analogy:
Taylor: "If you want to understand IROAS, think about a golf handicap and how every new score doesn’t suddenly become your handicap." (89:45)

---

For brands struggling to move from measurement to action: Accept that measurement is imperfect, resist the urge to chase false certainties, and focus on building an actionable truth over time—one that survives not just in models, but in P&Ls and cash flows.

---

Transcript

[A]

I flew here to beat Taylor in golf and he bailed.

[B]

So by the way, your hand, you're a good metaphor. Your handicap is basically what this is. By the way. If you think about an incrementality factor in an IRAs, that's what your handicap is.

[A]

I didn't play to my handicap.

[B]

You should think about that though.

[A]

Last time we played.

[B]

Take that away for use of you that are golfers. If you want to understand IRAs, think about a handicap and how every new score doesn't suddenly become your handicap. Exactly.

[A]

Oh yeah, that's a great analogy. Good job.

[C]

That's a really good one.

[B]

Thank you.

[A]

This episode of the E Commerce Playbook is brought to you by Applovin. If like a lot of brands, you're looking for ways to profitably diversify your ad spend, you need to check out Axon by Applovin. Axon connects brands to 1 billion plus potential new customers across top ranked mobile games. Axon ads are full screen videos followed by multiple interactive screens getting a minimum of 10 seconds of undivided attention and a median watch time of 35 seconds. 70% of brands see higher LTV from Axon customers compared to other platforms. It's super easy to launch and brands have been able to scale to 100k per day in spend within days. Axon is offering a 5k ad credit. When you spend 5k, just go to Axon AI CTC to set up your first campaign.

[B]

All right, welcome to a very, very special episode of the E Commerce Playbook podcast. We're in person today. This is the office. We're in a little bit of an odd setting between two ferns here because we had some power outage issues because of the big rainstorm we got in California. It rained for 12 minutes and we nearly lost the power grid. But I'm really excited to have two of, I would say, like my close Internet friends. You guys are my close Internet friends. George. You've, you've manifested into my real life a little further. Yeah. Olivia and I just met for the first time in real life, but we hang out a lot in the DMs.

[A]

Today is your first time meeting.

[B]

The first time we've ever met in person.

[A]

Wow.

[C]

I didn't know that you were adversaries for a while.

[B]

No. You know, I was going to start with all the positives and you just

[A]

went negative to everyone on the Internet, so.

[C]

No, I think, I think this, this might be a death match.

[B]

No.

[A]

Okay. That's what I'm saying.

[B]

I'm going to set this up differently. Don't listen to them. They're actually here to help me and to help you. What I navigate, what I think is one of the most challenging things for operators and marketers to deal with right now, which is that we've all been inundated with the idea, and many of us accepted that incrementality and holdout testing is like a critical component to running a modern marketing playbook. But I think that there is a gap in the market in information about how to actually operationalize these results and put them into practice. And I know we're experiencing this. I feel a level of confusion myself that's been challenging. So I've brought two of the smartest people I know who are dealing with this all the time. Olivia, obviously, at House, what is your exact title?

[C]

I'm the Chief Strategy Officer.

[B]

Chief Strategy Officer. That's a C level, folks. You heard it there. That's a C from Olivia. And she's probably run more tests than anybody in the world. And then George is the CMO at Cozy Earth. Okay. CMO at Cozy Earth. And amazing. Do you guys call yourselves. What was like, the specific category you would say Cozy Earth is?

[A]

Is it a home goods?

[B]

Home goods, but also apparel.

[A]

Apparel. Bath. I mean, home goods is bath, bedding. And then we have apparel.

[B]

Amazing business, who's also participated themselves as an individual brand in a lot of testing. And so. And then at ctc, we run our own set of testing and holdout and Mmm. And all sorts of different things that we have to then participate in deploying for our customers. And so building this into practice from setting up the test, where to test, when you get a test, what do you do? How do you actually turn this into action? Is something we wrestle with George directly a lot with. And then Olivia and I are always kind of bouncing ideas back and forth because I think it's challenging. And so I think we're here to do that. But maybe for those of people who don't know you guys, just give us 35 seconds of your bio and what you do at your corresponding companies. Olivia, go first.

[C]

Sure. But before we start, though, George, you. You pulled our ltv, right? Cozy Earth.

[B]

Yeah.

[A]

Oh, yeah.

[C]

Can you put on the record here

[A]

$1,400 from Olivia in the last year.

[B]

What about me?

[A]

I think it was $100, and most of the orders are free because he texts me asking for free things.

[B]

Okay, well, also, but here's I I1 up you. My mom works for co. That's true.

[C]

What?

[A]

That's true.

[C]

Does work how did this. How did this not come up?

[B]

Because I needed to save some of it for the podcast.

[A]

Yeah.

[B]

So my mother works at the retail store here in Corona del Mar for Cozy Earth. So we're on. We're. We're just on the. The take side of the Cozy Earth relationship. You're a giver.

[C]

All right.

[A]

Granted, I didn't look at your wife's profile. Yeah, she's the real spender.

[C]

The pajamas are a great gift. I'll do that plug right now. But if you have a gift to get, go for the pajamas. So what do I do? 30 seconds on Haus. So I actually was one of the first customers of Haus at Sonos. I was leading growth. We had a last click attribution model for D2C pointing us in one direction. We had an mmm, pointing us in another direction. And sort of pitched experimentation is the way to, like, really start to quantify what marketing was delivering to the business. And it was just going so well in such a short amount of time that I made the move over. I can't believe I am a vendor now, but have been doing this now for almost four years. And. And this is. I'm excited we're having this conversation, too. I think this is really important because the first couple years of House was just convincing people that incrementality is important. And I think we're. We're sort of through that now. We're sort of through the first chapter of, like, okay, we need this. And now it's. How do I use this as an operating system and really operationalize this. This practice in a way where it's driving business outcomes? So that is what I do. I do a little bit of everything, but mostly spending time with our customers and making sure that they're getting value out of. Out of this. This practice. I say it's. It's. It's more than a product.

[B]

It's.

[C]

It's the whole thing that, you know, the. Both the product and the. The kind of strategic layer that comes with it.

[A]

Cool. Oh, you got something.

[B]

Well, I was just saying Olivia is out here for a customer success event with a lot of her customers, and I enjoy watching her do her job. So much so that I'm constantly just, like, trying to convince her to come work for me. Sorry, Zach. And she just keeps saying, no, I'm gonna say that. So you're. You're in good hands. But because I love watching her care about the work that she does, and she cares a lot about it, and so that's why I, like, I always tell people arguing with them on the Internet is a compliment. It's the highest compliment I could give. It's because I means I really value your opinion, and so I. I want to run up against it to learn how, like, you got to where you are. And Olivia is one of those people for me. So excited to have her here in person. And she's. She's important, y'. All. She's, like. She's hobnobbing around with the CEOs of all of the most important platforms in the world. So she won't say it, but she's. She's in the know and connected.

[C]

No comment.

[B]

Cool.

[C]

No comment.

[A]

Yeah. Olivia's awesome. I've been at Cozy Earth for eight years. I started when we had, like, three or four employees. I was doing customer service, and then we started leaning into digital marketing a little more. I. I owned that. And the company's grown, and I. I got to stay in my seat. I haven't done a bad enough job to get fired yet. So anyway, that's. That's my background, obviously. Relatively involved in our. All of our holdout test and our relationship with house and experience. Excited to. Excited to chat today.

[B]

And so I'm gonna do the thing where I compliment both of them after they. Because they're gonna undersell themselves. But George is if you're somebody who's like, maybe you're in your first job at a marketing company and you want to understand the playbook to. How do I someday continue to grow in this organization to the point that you could be the CMO from. Where'd you start? Customer service rep.

[A]

Customer service rep.

[B]

Right. From customer service rep to cmo, like, George is the person. And. And here's. Here's the key. Genuine curiosity and humility and a constant desire to learn. George is a CMO who asks questions in every conversation that he's in and asserts no presumed understanding or knowledge that would be greater than anybody else. He's around. He's constantly trying to learn, and that's why he keeps growing, is because he keeps trying to find ways to do that. And so it's why I enjoy talking to him. Cause I just talk most of the time. You know, it's easy.

[A]

That's a great relationship. Come out here. He yells at me, tells me what I need to do better, and I love it. That's why we. That's why I pay him.

[B]

Yes.

[C]

George, I'm so glad you're here.

[A]

Thank you, Olivia. I'm glad to be here.

[B]

Would you have done it with if you, if he wasn't.

[C]

Come on. But I am so glad you're here.

[A]

Of course I'm thankful. Yeah, I've, I've been looking forward to this. It's fun. We'll get this on film. We have these conversations all the time. Hopefully it's helpful to somebody. We'll see.

[B]

Yeah. So maybe we set the stage a little bit with some of the journey that we've been on. Because this isn't about cozy Earth today in the specific ways, but I think your journey is illustrative of what actually a lot of people will experience if they go down this road. So whether they're just starting their measurement journey or they've been on it for what, two and a half years now or however long you guys have been in it, there's a, there's a process here that I watch people go through all the time where there's a conversion moment where suddenly they decide incrementality matters and then there's this like enthusiastic testing process and then that first test comes back and in some cases it could be inconclusive, it could be. In some cases it could be great, in some cases it could be bad. But then begins this like process that sort of like the trough to disillusionment. It's sort of like Gartner's hype cycle that we go through every time with every customer as it relates to this technology or this process where they realize that there is no one single answer that solves all the business problems and it becomes almost discouraging at first. So Olivia, I'd be curious if like you've walked a lot of people through this, like how do you think about setting up how people should relate to the idea of a measurement journey? If they're just getting started or maybe they've had one or two results.

[A]

Brands like Ridge, Ashley Furniture and Wayfair are unlocking incremental revenue with Axon by Applovin. Axon ads are full screen videos that run in top ranked mobile games with a median watch time of 35 seconds. That's longer than a standard TV commercial. You can get up and running on Axon in under an hour. Using the creatives you already have. DTC brands have scaled a six figure daily spend within days of launching and you'll get a 5k ad credit when you spend 5k. Go to Axon AI CTC to get started.

[C]

Yeah, so this is a new metric like we are establishing baselines on for a lot of our customers. They've never actually run an incrementality test before. And I think those are actually the hardest customers to, to, to educate in the early days because a lot of our, a lot of our customers, bigger brands who have data, science teams who have done this before kind of come in knowing what to expect in terms of. All right, these results are not going to look like you're in platform roas, like they are potentially going to be lower. We, it's, it's not uncommon for us to see cost per incremental acquisition in the multi multiple thousands of dollars. And so for brands coming in who have never gotten these baseline reads in terms of incrementality, it can be shocking. And to your. There is definitely that, that journey that we go on with them. I'd say though, like the hardest moment is not after that first test because I, I still think there's a lot of excitement around. All right, well this read wasn't where I thought it, it would be, but let's go test some other channels and those are going to be a lot better and we can then, you know, reallocate budget from low performer to high performer. I think the, the hard part is when all of the reads come back and then you have them all in hand across your key channels and they're all under one in terms of incremental roas. And in that again, we're used to seeing this like when we got started with a large entertainment brand, you all know it, their CPI was $4,000. And by the end, you know, through a year or two of working together, we got it down under a thousand. And that, and that's why you do it. And to me it's like hard to, I think it's, why would you expect incrementality to be good out of the gate? You know, so that's the, what we try to preach is like this is a journey, this is a How do you improve upon this number? Not the report card that you take back. But it's tough because I, I, I, you know, we'll, we'll get into it. But sometimes our, our results do show that, that customers are, are overspending in marketing. And I think that's, it's an unfortunate reality that most brands actually don't want to, to face in terms of, you know, a lot of, a lot of these brands are trying to grow and they're trying to invest back into the business. So that's something we're wrestling with every day. In terms of know as partners, what is the best way to guide them.

[B]

I think I'm gonna, I often play this role of like poking at Olivia over certain positions. And one of the ones that you would be easy is that like okay, you are a business who makes money. As I continue to persist in testing so the idea that this is a journey or the idea it's a thing that I always need to do feels very self serving as an idea. And I think you and I have gone back and forth on like whether the application of the benchmarks is sufficient or how you, how much you have to test. Everything is a phrase that gets used a lot. But I think that it is important to understand that this is not that we are taking in every one of these cases a moment in time snapshot that reflects what was true for a period of time that is subject to lots of reasons why it might change. And so the idea that a single answer would be sufficient for all time is just not true in any way, shape or form. And so I think that there are, there is certainly a reality that brands need to begin to embra. Embrace that. I'm not running a test, I'm making testing a part of my methodology.

[C]

Yeah.

[B]

And I think that disposition change is an approach to entering into this is really important. Is, are you running a test or is testing a part of the way you're going to do measurement? Yep, those are very different places. I think.

[C]

Yeah, we should have done this up top of like, let's just like lay out all of our incentives and make it super clear. And you said like we, we're a measurement company, we want customers, we want these brands to be running more tests. I actually haven't seen like a great correlation there. There are a bunch of brands who come in, you get two testing slots a month. There are brands who will fill those testing slots. And it's testing for the sake of testing. That's actually not how we drive value for our customers. Like I don't think there's a, a very high correlation between number of tests run and value. It's the, are you actioning on the results? Are you doing anything with them? And so that is like, that is, you know, where and how we're successful is like are you actually using these results to make business decisions? And that's our metric. That's, that's what I, that is I would say our number one determinant of ROI and value.

[B]

So George, how did, tell us a little bit about your measurement journey? When did you start? How do you, how did you think about the role of incrementality and how has that evolved as you've been going through it.

[A]

Yeah. So to Olivia's point earlier, everyone comes in with a perception of what the return is on Meta. And I was having a conversation actually yesterday with the brand doing about $15 million and the in platform is like a 3.5 and he's assuming that all of his new customer rep and when you start looking at the math, you're like, there's no way it's driving a three, it's probably driving closer to a one. He's never ran a holdout test, but he's operating under the assumption that it is driving a 3X ROI. So I think there is this moment when a new brand comes in, they're operating and so were we before we started testing that the in platform number is reflecting some version of reality. Whereas in, when we got our first test result back, we were like very, very different than what we were seeing in Meta. So and then from then on, like we've almost always had not cool, we've almost always had a conversion lift study on or a house study on. And so we've been able to see what Taylor mentioned, which is a very wide range of results where one month it's this and the next month it's, you know, it's this. And there's no, there's no clear reason why the result changes so much month over month and our effort is always to increase that number and you know, two and a half years in, it's. It, it can be frustrating because I feel like we are running out of levers to actually change the outcome and it's really difficult, at least for us as a brand to figure out what creates the outcome. So that, that, that's kind of our journey. And again, still figuring it out. We don't have it down to a science yet, but we're working on it.

[B]

So I want to lay out some of the things that I watch that's common from what you just described to what I see as like the E Commerce brand measurement journey is that when you start out earlier stage businesses, let's say you're in the seven figure range, Meta ROAS becomes a really strong indication of your actual revenue realization. There's not other channels in the mix. You don't have existing organic demand, you're not even really, you don't really have existing customer revenue. So it is a very trustworthy signal of the impact of your media. So your organization builds orientation around improving that measurement. It works its way into all your dashboards, it becomes present in lots of ways and Then you begin this process of two things occur as you mature. You expand distribution of the media. You add in Google now there's some TikTok maybe there's a little app love and we're going to try out YouTube. That introduces measurement complexity. Then you also get more of your revenue coming out of your existing customer base through email and sms. As a channel grows as a percentage of revenue that has an effect on the actual incrementality of your media and channels. Meta's he's going to pixel all those purchases. Those people have probably seen ads when they see an email purchase. And that's sort of phase two is like okay, existing customer revenue growth and slight channel introduces. Now ROAS degrades a little bit in quality. And so generally we start to try and to move into MTA is sort of a first step where now we're trying to assign some sort of multi touch value to these different interactions. We're trying to sort out between the channels. Maybe an increased focus is where AMER became a metric that I think Andrew really probably deserves credit for really driving this forward a lot. Acquisition marketing efficiency ratio so a focus on new customer revenue begins to emerge as really important. Then a third thing happens in my experience which is distribution expands. Now you're on Amazon, now you're in retail. Now you have online retail competitors so your actual product distribution expands. And then you start introducing top of funnel media. Now we're into TV or podcast or whatever. And now any sort of click based measure basically sort of collapses on itself in terms of its efficacy. And so you move as an organization as you mature, you have these signals that were really reliable that it's hard to tell the exact moment that they stop being reliable. And so every organization has to go through this process of sort of recreating the measurement framework for themselves. And I feel like now what got introduced and you, you guys deserve a lot of credit for popularizing this is that wait a second, we need to sort of ignore all of this click or view or attempts at any sort of mta. We need to actually try and create a causal relationship between the spend in any location and the ethic, the efficacy on the actual business metric you care about. And so that's where I think holdout testing has really emerged as popularizing. Maybe you could sort of define these terms a little bit for people because I think sometimes hold out, scale up, hold it like there's different ways that they get defined but maybe you could give us the most common incrementality tests and what they're attempting to do.

[C]

Yeah. So the, a lot of people say what they do is incrementality. I think we kind of fundamentally believe at House that incrementality requires a holdout. You need a counterfactual to understand what would have happened anyway. This is classic scientific method, randomized control trials in healthcare. Like, you cannot, you cannot establish causation without the counterfactual of what would have happened anyway. And so if you are, if you are spending up and you don't have a holdout, that's interesting. But you don't know what your business would have done if you hadn't spent up. And so there are different ways to answer different types of questions. The what is the baseline incrementality of this channel? Is a different test design than how much should I spend on this channel? And both, I would say, require different cell structures and test designs. But fundamentally, at the core, like what we believe is that, is that that holdout group, you need to turn off your marketing or whatever you're testing in, in, you know, across some group of users on the conversion left side or across some regions in order to actually understand the, the counterfactual. So, yeah, that's, I think a lot of people also kind of conflate these, like, natural experiments with we turned this thing off and then we observed, which is a great fallback solution. Like, I think we still recommend these tests, we call them time tests. But it's really hard because there's so much happening in the business at the same time. If you turn something on right ahead of Black Friday, Cyber Monday, it is impossible to actually tease apart cause and effect and disentangle which of those channels you, you lit up are actually driving your business.

[A]

I mean, we were doing that before where we would just scale things up or turn things off. And it was a terrible, it was a terrible system. We had no reads, we had no idea we would make assumptions, then we'd action on those assumptions and then they'd be wrong. So a couple of questions though. You mentioned, like, for new brands.

[B]

Yep.

[A]

There's this point in time where they need to start introducing incrementality testing. I have conversations with brands that are, you know, five to $15 million, maybe half of their revenue is coming from repeat. They have no idea how much of the demand is organic versus repeat versus meta. They just assume all new customer revenue is coming from meta because that's the only paid channel that's turned. At what point should they start testing incrementality? Because I actually think it's earlier. I think there's a lot of brands that have a decent amount of organic being driven and they're attributing all of that to meta.

[B]

So, yeah, we probably both. I'd be curious your answer here too. I think that incrementality is a step to take when you've built distrust in your present measurement system to generate the effect that you want on your business. So I think about all these as tools to some end. And the end is you want to understand the levers of growth so that you can deploy capital and generate a return. If your present measurement system you feel confident internally that you can understand based on the whatever information you're using, where to deploy the dollars and get a return, then great. When we, if I analyze all my gong transcripts, which is something we do a lot like our sales calls, we plug them in and we ask, what is the number one reason people come to ctc? It's not because their business stops growing, it's because their business has stopped growing and they don't know why.

[A]

Yeah.

[B]

And so it's when uncertainty clouds your measurement system and you don't know what to do, or you have a lack of confidence, or there's fighting between marketing and finance, or there's disagreement organizationally around how you should deploy the next dollar and what the levers of growth are, you have a problem and you need to reevaluate the measurement system, which likely at that point will begin the process of including testing into that thing. That's, that's my experience.

[A]

I would say that a lot of them have a false sense of confidence in their system. Like in my conversations with them, they're like, no, no, this all has to be coming from matter. Because that's like. But when you actually dig into the business and you look at the revenue driven from email and SMS and you look at the revenue driven for. From organic or like some other channel, I would say there's a lot of small brands that would say, oh no, I'm confident, I trust this. But they don't, they don't even know

[B]

this goes back to what. When does change happen, though? Change happens if and only if dissatisfaction is large enough. And in those cases, we, we interact with those people all the time and you can't actually change their mind. Yeah, that's the problem is that someone has to be willing to examine the present thing through a critical lens in order to get to a new adoption of a new idea. And I just think that a lot of times, if it's working for them then like they're going to persist in that system. And that's my experience. It's hard.

[C]

Yeah. Well, I, I continue to be surprised. I always ask this question of like when, when, when I'm having an intro call like how much of your business do you think is driven by paid? And so many people answer that question based on MTA attributed data and I'm like no, no, no, throw that away. If you turned off paid tomorrow, how much Right. Would your business come exactly. And so many can't answer that question now. I think the, the moment where you need holdout testing is where you can't do that anymore. Like if, if you can turn some, turn off your, your Meta and you can immediately see the impact to the business, I don't think you need holdout testing. But. Yeah, but in there are moments I'd say like when organic demand starts to take off, when you're doing partnerships, when you're selling in more channels. I typically see it around like low to mid eight figures is like when you're, when you're diversifying your channel mix enough and like when you have enough organic going that you can actually confidently answer that question by just shutting it off.

[A]

Yeah.

[C]

So, but yeah it's, it's so many people still answer that question based on like how many sales came in through an attributed click.

[A]

Yeah.

[C]

And, and that's just fundamentally flawed.

[A]

Or they just say everything that's new is meta. I'm serious. It's like very, it's, it's concerning when you talk to people because there's just this perception that meta is driving X percent of the business and then they try to grow the business by scaling into Meta and the ROI is not what they anticipated.

[B]

Well, so what you're getting at there though is like is not just. It may be both presently true that meta is the primary driver and that when they scale the curve of degradation is actually something they're like acutely unaware of. So I think there's lots of ways to use the testing to answer different questions. I don't actually think that you can really test. There's certainly from a data standpoint, there's too early. But I really think that having an early foundation for measurement as part of an experimentation of how your organization determines truth, like this is an idea I care a lot about which is like how does your organization decide what's true? Like what is the standard for truth inside of your company? And I think that it really interrogating the core KPIs that drive the behavior of the organization and why they are true. Like what needed to exist to validate that expectation is something that leaders should be like highly like should highly scrutinize. Like why is this the thing that orients all our behavior around? And usually it's because you've designed the financial incentives that way. This like almost always comes back to the financial incentives why which is why it's so important that you as a CEO in particular design the incentives around causal levers. Like you want your people actioning things that create real value. And so you should interrogate that a lot. So I we're not afraid to do it early I is what I'd say,

[A]

I guess is there what is meta's threshold? You might know this Olivia, for a CLS study, is there a small threshold?

[C]

I think it's universally free for all.

[A]

Why would everyone not do a cliff that's available if you have talked to your meta rep about doing the study? Because that's going to give you some sort of a read on the incrementality of the channel. That that was the first thing we did and then that opened the door to more testing and you know, helped guide us down that path.

[C]

But yep, I agree. Taylor, you, you, you asked this question on on Twitter. I'm very curious what your answer is. What do you think should be the North Star metrics for a marketing team? And I'm not, I'm intentionally not seeing a growth team for the person who spends the the advertising budget. What should they be gold on as their North Star KPI?

[B]

So I believe that the whole organization should or your orient and flow out of the company a financial objective for how shareholders intend to realize value. So stay with me for a second because I'm not going to give you a simple answer here. The number one job for the shareholders is to decide how they're going to create liquidity. If they're going to do that through the distribution of capital, then free cash flow is the governing metric that the organization has to orient itself around. Which leads to a very different media buying strategy than one that's trying to produce a P and L level profit. These are different things. If you're trying to create a sale event someday and you're trying to maximize TTM EBITDA over some period of time, then you should try to orient your behavior and flow down through the PNL as the view which is likely going to lead to things like contribution margin. Then you're going to try and figure out where you generate incremental contribution margin and then you're going to orient around that. So I actually think that the crisis that I see is a crisis of leadership at the shareholder level to define the organizational objective that the marketing team is trying to accomplish. That is the number and, and fighting or disagreement at that root level about how the company intends to make money. I'll give you another example that where this comes in conflict is in the distribution strategy. Because like all of the future growth flows out of some corporate strategy that includes retail, it includes wholesale, includes international expansion, it includes.com and includes Amazon. And media should be used to support business growth, not channel growth. Like and oftentimes they get isolated and siloed into these things. So the first answer is do you have clear alignment with on the fin corporate financial goal? Do you have clear business strategy for where what channel that growth is going to come from? And then is the media team laddering up to the effect on those channels and if so they can be really really effective and then we can decide the right input metric for them to focus on. In most cases what I think it should be is incremental positive contribution margin on all points of distribution. So if you're, and if you, if, if, if retail is a little bit more complicated because it's not as simple to say that driving retail sale through equals incremental dollars for the company because there's a sales rhythm related to POS and store accounts and things like that. So the simplest way is to say we call it your digital enterprise, your online business and say everywhere you're selling online that you have control over hold out. So like usually this is Amazon, if you run Walmart.com or any like you know, self serve marketplace where you can get geo level revenue data, all your media should be held to an incremental impact on all of that revenue all the time. That's what I think. And, and you should, your efficiency goal should ladder up to whatever the organizational obligation is to margin and growth and work from there.

[C]

What, what about the scenario where your brand for whatever reason can't be running an always on all up holdout test? Are are, are they doing, are you

[B]

running always on so not, not always on all up. But I, we, we would advocate for always on channel level.

[C]

Okay.

[B]

Especially we, we think about measurement roadmap relative to what we would call as like the variable business impact. So when we think about designing your testing roadmap we would say where's the most if you take the, the media spend by channel times the variability of our sample size of test results that represents the bounds of possible incrementality in that challenge channel multiplied by, by the dollars. Right. And so what I want to do is as fast as possible, I want to narrow the error bars on your most valuable channel and I want to work through it sequentially like that to try to make the most immediate impact on the place where you're spending the most money. So most of the time that begins with a meta holdout. It represents 66% of share of wallet. I think across our portfolio I think North Beams is pretty similar. And so for most brands that's the most important question to answer and then we can sequence through it from there. But all up I am a fan of as well. It just, it's a little harder for me to determine the right corresponding action afterwards.

[C]

Yep. Okay, so you're just, you're, you're running channel level tests. You start with the most, the, the highest spend channels and then you get your factor, you recalibrate based on the factor versus the benchmark that you started with. And then basically in a month where you're not testing, let's say you test Google the next month. So meta's off in terms of, of incrementality tests. You're not testing meta, you just use the factor from the last month. And, and you think like ahead of growth could go to their boss and say like this was my incremental contribution margin in this month based on kind of assumptions on incrementality from prior month's tests. Like I'm trying to figure out how

[B]

you, how you measure fast forwarding to the end of what we're trying to solve for here. Yeah, that's right.

[C]

That, so that. Okay, so you, you, you believe the North Star metric for a head of growth is should be based on the results of channel level holdout tests.

[B]

The world I'm going to advocate for in the future is something called IMR incremental.

[C]

Are you rolling that out right now?

[B]

I've done a thread on this before. We're working towards a broader implementation our own world. But what I'd love to see. So IMR would stand for incremental marginal return. So I think that the errors that we make in reporting return is we report revenue and we report non incremental revenue. So I don't want to report either of those things. I want how many marginal dollars did I generate divided by the investment. And that would present always, it would normalize everything to 0% is neutral. And then it would read like a portfolio, like I would my Robinhood account. You'd see plus 20%, plus 15%. And you could compare every channel to each other in a normalized view of return. That to me is way more intuitive. Like if I look at it and I'm getting a 15% return on my money, then I don't have to do this calculation of like, what's my gross margin, what's the like? Because even if I say my Iroas is 2.2 to 1, that doesn't actually tell me anything. I don't actually know if that's good or bad for each of your businesses. But if I was reporting incremental marginal return on a percentage basis, then I could look at it and I could also create a threshold like that is consistent with return on invested capital. So if I'm a CFO and I'm like, our hurdle rate as an organization to deploy capital is an annual return on investment of 22%. Well, then that's your IMR threshold. Like, otherwise, don't deploy the capital right now. Now. So you. So, so to me, that would be the ideal state of how we would get to reporting. Unfortunately, right now I'd say the first step of educating people is just to move to IROS is to move from Ross to IROS is step one. My goal for step two is IROS to imr. So that would be my.

[A]

Interesting.

[B]

Yeah.

[A]

Always learn from what do you.

[B]

What do you. So. Well, what do you do? What give me how. What is. What is true about the measurement complexity? Because I think you represent what is true in a lot of organization, which is. There's a lot of information.

[A]

Yeah. So with Meta, we've been able to have a test pretty much always on for the last two years through the conversion of studies and because the test result varies so much month to month, and we worked with Taylor on this and basically what we decided, what a year and a half ago is we would weight the results of the I, you know, the incrementality studies and we would apply that to our in platform. So if we're getting a 1x incremental return and the in platform is showing a 3x, we know that we have to. We have to. I mean, I guess if that's within

[B]

you, you guys call them ifs, right. Incrementality factors.

[A]

Incrementality factors, yeah. So it's a multiplier on the end platform and we operate within that constraint and try to hit the target. And then we know like, you know, you're assuming a correlation between the in platform number and the incremental return, which hasn't always been true for us, which is a debate and something, you know, that has caused me pause. But I don't think there's a better way to do it currently. So we are operating that way for the channels. This is where it gets a little harder though. So Olivia mentioned they have two cells and we're running, call it 10 channels. So we might only be able to do a Google channel or, or maybe you're identifying a single tactic within Google. So you might only be able to do a non brand search Google test once every six months. And so that means you get two tests per year and the result varies. You know, like Taylor said at the beginning of the pod, it represents a period of time and as we've seen with Meta, the result varies pretty dramatically based on the period of time that the test is deployed. So that's where I think I have a harder time operationalizing it. Because the tests are so infrequent and the results vary so widely like it, it makes it very difficult to have any confidence in the impl increment. What did you call them? Incrementality factor. Eif incrementality factor for those channels in particular.

[B]

So this, you, you brought up two really good points that I want you to address because I, that I, I think they're worth like really wrestling with. One is the replication crisis, which is like tests don't come back the same all the time. What do you do in light of that? And then the second is the disassociation between optimization and attribution, which is this idea that like we're meta's still optimizing off of their data. Not the incrementality factor or those things. Is the platform result actually connected? Does the factor play out every day? Is that real? So replication crisis and disassociation of optimization and attribution. What say you?

[C]

I actually, I think that this can be really challenging because the, like the. If, if the. Let's, let's start from the beginning. It is most definitely a good thing that you now know 1/3 of your platform reported ROAS is incremental. Right? Like you have to. I think we all need to acknowledge that that is way better than the starting point of assuming that that is all incremental. Now your challenge is if you get a read that's you know, 1, 1/3.33 if and then six months later you get a 0.8 point. I think this is a problem. I agree. I think this is, this has been like again, we're very early, we're use a baseball analogy, like second or third inning of the whole incrementality movement. I think this has been really challenging of like, that is. That's just too blunt of an instrument to actually be useful in decision making.

[A]

You lose the real time application.

[C]

Yep. Yeah. So if you, you know, then, then you get confused of like, all right, it was 0.3, now it's a point 8. Do I just split the difference? Like, we need to level this up. And this is why I actually really respect the way you are challenging this position of there has to be a better way to do this in terms of operationalizing it versus just using the most recent result. So I agree. I think that that's been, it's been a struggle. We've just sort of, We've, we've had to, We've had to experience as we've gone because we're one of the, this is like the first time we're all building an OS around incrementality. And I think that can be frustrating. But I do, I, I do believe this is better than using that 0.3, even if it ends up being a 0.7, is better than all of it is.

[B]

Okay, I actually, I, I'm actually gonna push a little bit here to say that I actually think it's better than even that representation. Here's what I would suggest is that, Let me ask you, do you believe that it's actually true that your media is more or less incremental at different points in time?

[C]

Yes.

[B]

Yes. So. So I, there's this phrase, I think, with data analysis that I like, which is data reveals truth. Okay. So in those snapshots, what we found is that at different periods of time, your media was more and less incremental. Okay, so we could look at that and we could be frustrated. We could say, why are the tests different? Or we could say we are observing nature. That's what science is. This is an observation of your media. It's not. The test isn't creating the outcome. Right. We're observing what is occurring. So what it allows us the opportunity to do. And this is where like measures of central tendency as a mathematical principle are like, really powerful. So if I have a, like a distribution of data, a bunch of different points on a scatter plot, the idea that the median at any given time would represent the clo. If I, if I had to play the game be least wrong in my guess, that point represents the least likely to be wrong given the present sample distribution. Now, what I like is that as long as the, the ends of the sample like the, the, the. The low and the high aren't constantly expanding as long as the results continue to fall within some continuous range. Over time, that median becomes even more powerful because what it begins to represent is this idea that like, that is actually a best guess at any given time, which I think with measurement we have to acknowledge that's what we're always doing. It's always our best guess. We're just trying to be less wrong. And to your point, the Facebook Ross was wildly wrong. So like we already got way better by just eliminating that data point. And even though it's still frustrating that 0.3 and 0.8 are far from each other, 0.6 is still way better than 3.2. And so like this is this idea, I call it progressive truth. We're trying to be more right over time with. While accepting will never be definitively right.

[C]

Now, the. Here's the problem. What I think is great about Cozy Earth is you were running meta lift studies before you came in. I think you were even running holdout tests with another partner. And so we kind of came in with the shared understanding of, all right, these results are not wildly far from what you've seen in the past, which, which I think made you trust them more quickly. Yeah, but that's a, this can be an issue of like, if you have never run a meta conversion lift, then there is, there is a long period of just trying to get customers to believe the result. And so I think that was a great starting point and foundation. I think it's great that directionally, I don't know if Taylor fully agrees, but directionally the household out results have lined up to meta conversion lift. I think you think that's great when multiple sources kind of.

[B]

Yeah.

[C]

Circle around the same data point. So I think that's been great. Now the question is, is there variance because incrementality can change over time or because these are just really noisy test results? And that's also a reality that we need to, we need to, I think, wrestle with is you can run bad tests. And so that's. I, this is why, like, I, like, I really do believe that the science matters. And if we, if we are, if we know we're running clean tests, then that makes it easier to buy into this reality that incrementality is changing over time. But it also just could be pure noise. Like, I've seen other vendors that will run these lift tests on like $20,000 of monthly spend.

[A]

Yeah.

[C]

And against like a very large baseline in terms of that business level revenue, like there's, it's very hard to, to actually land on a point estimate there that you can trust. So just to say this is why it actually matters to run good tests of like, we've run good enough tests that we feel good. We, we. And then we've, we've validated them through different kind of methodologies that we do know incrementality is changing over time and I think that's important.

[A]

Yeah. And I will plug House real quick. She didn't pay me to say this, but we were with a different vendor beforehand.

[B]

Okay. The vendor disparagement. I, we are.

[A]

Look, I'm not saying any names. All I'm saying is it's been night and day better. Their operation is much cleaner. I'm not, you know, super familiar with the testing methodology, but I do have more confidence in Olivia than I did the prior team. And so yeah, brand is.

[B]

Brand is trust.

[A]

A good partner is, is definitely helpful in this. And the good. We trusted the results. There was even debate when I first started talking to you about whether brands should trust the CLS study because it's meta grading their own homework. And I'm like, well, they gave themselves an F, so. So I think we can probably trust the result.

[B]

Yeah, like meta CL studies are often very poor in. Sometimes are. Because the other thing is, is that one of the things I think is really important is bringing in the other channel revenue. So a CLS study is not going to include your Amazon revenue. So in many ways it's always going to underappreciate its effect. So I think that it actually, it's a totally reasonable way to do it as a, as a starting point. And yes, you can make bad tests, but I think regardless, like, I, I don't think you think the fact that there are different test results for Cozy Earth are result of your poor test design.

[C]

No, no, absolutely.

[B]

So my point is that like it's, it can happen with good test design,

[C]

but that's, that is because there we can. With good test design, we know that incrementality is actually changing over time.

[B]

Yeah, I think, Yeah, I think incrementality

[C]

test design, you don't know is it, is it the poor test design or is incrementality changing over time? You don't have a lot of confidence.

[B]

Yeah, I mean I, Yeah, it's, it's hard to. Tests also come with their own confidence intervals and expectations of ways to analyze how strong the signal was on that individual test that like is a spectrum it's not a binary.

[C]

Yep.

[B]

And so that in and of itself is a piece of this that you talked about something I believe too, which is that there should be a model that weights each test result relative to the aggregate set of historical tests to your database of tests. We do this with our database of tests that give you some present best guess. But, but now I want to talk about like, okay, so let's say we've done that. Let's say we've run all the tests we did good test design. We've got these different outcomes and we've picked a point now median whatever or modeled result relative to the confidence of each test and seasonality and all the things that we might use as inputs. Let's say we have a number. Then the question becomes, okay, what do you actually suggest people do with this number? So you go first because I don't know what the House position is on how to operationalize an if.

[C]

Pause. We talked about replication. Was there another big, well, optimization.

[B]

Well, I think this is what.

[A]

This is basically that question.

[C]

All right, so what do we do with the if? Yeah, well, okay, so what do we do with the if? The. The way when, when a customer signs up with House, we show them three kind of playbooks they can in or tracks they can sort of run with House. It is the efficiency track where I want to cut costs, I want to get more efficient. It is the redistribution framework where we, our budgets are flat redistribute funds, you know, across channels in order to, to grow at a flat budget. And then there's the scale up budget of we want to add new channels and we want to increase spend to grow. A lot of, most customers don't want to get more efficient, even if that is the, the you know, technically correct thing to do. Based on our test results, I'd say most of our, most of our brands are in the redistribution track where it's okay, we're a healthy business, we can invest back into the business to drive growth, we can spend some money. We have this budget we want to reallocate across channels in order to, and this is what we work with these brands on is like as long as you're moving budget from the low performers to higher performers, you will get better. It still might not be where you should be. And this is Taylor, I think this is a thread you're going to pull on of like you're still, you'll, you're still spending inefficiently, but you're spending more efficiently than you were before. You started actioning on these results. And so that's where a lot of our customers are at right now. And, and I, I think there's probably still a lot of work to do. But they're moving budget from relative underperformers into, into stronger performers. Even if your results come back and they're not profitable.

[A]

Can I, can I dig into that? Because we've, we've had problems with this where it's like, okay, we run a test for Google non brands in March, and then July comes around and we run a test on YouTube. And you know, maybe the difference is minimal. You know, it's not a huge difference, but there is a little bit of, you know, the performance is a little bit better on YouTube, but you just tested two different periods of time. How can I actually confidently move budget from non brand to YouTube when we just said that there's massive variance in the test test results month over month? That's where I have a hard time is like, how. Because we're, we're acknowledging the fact. And again, this is all live. So I apologize, we didn't discuss this beforehand. Colby specifically was like, I'm done running test unless I'm pinning channels head to head. Colby's art.

[B]

So, so this is, this is. I think that your experience of this is because you work with budget managers, not owners. They have budgets that they have to spend. So of course they're just trying to allocate the given budget that the organization gave them at the most efficient level possible because they need their budget next year and their job is to allocate budget. If you're the owner, I want you to, like, answer the question as if it's your money.

[C]

Yep.

[B]

Because then the threshold isn't channel versus channel. The threshold is channel versus return. And this is my big thing. Is that it? And you said it. They're not performing as well as they should. I wouldn't call it should. I would call it profitable. Like was, could you deploy the dollars in a way that made you money back? And it's not channel versus channel at all. It's channel versus potential scale of profitable return from the channel. I don't care what any of the channels do. I'm not taking budget from here and putting it here. It's channel versus profit. If the channel can generate profit, you get more dollars. I don't care what happens anywhere else, because you can't do the comparison that you just described. It's not actually possible. It's not a real solution. And so instead, what you're just Doing

[A]

is wait, why can't you, why can't you go head to head in a channel during a given period of time?

[B]

You just described why.

[A]

I can. If I say, hey, you know what, I'm deciding between non brand and YouTube and I'm going to run that test March through April. Then at the end of that read, it's the same period of time I can confidently say, okay, non brand outperformed YouTube over that period.

[B]

Okay, then what do you do with the meta budget?

[A]

Hold steady, hold the plan. I mean what is.

[B]

You have two cells. This was her point earlier. You like. But my question is if they're both profitable, they're not. Okay. So if one is and one's not, why. Why is the. Why is the relationship to each other the measure versus the relationship is to the return on investment that you want as a business.

[A]

Yeah, it shouldn't be, but in a lot of cases it is. It's.

[B]

You are. So the question is why is why do organizations behave that way?

[C]

Are you, are you asking why, why we do channel comparisons? Why not just say I tested YouTube in July, forget about that non brand test result from four months ago. I'm going to either dial up or dial down YouTube based on this result and my, my goals in that moment.

[B]

Yeah,

[C]

because I think a lot of the results might suggest that you should. That many brands are over investing in marketing and that. But this is. We. You talked about this up top of like what is the founder's goal for the business? It's not always clear in my experience of like they're trying to build in some optionality and they really want to show growth and they're willing to invest a little less efficiently in order to continue showing high growth. I don't, I don't think a lot of founders come in with a clear plan of what they want to do with the business.

[B]

I agree. That's why I said when you asked me the question of the problem. The problem is lack of clarity at the shareholder level because how am I

[A]

as a media buyer or lack of alternative investment. Like, and I understand you're going to say okay, distribute the, distribute to the shareholders.

[B]

That's your job as a CEO. If you don't have an investment thesis for the cash and you're just sitting shareholder money in treasury at like what like a 4% yield? Why would I.

[A]

But okay, but we. Okay, so really the job should be you're supposed to, you need to grow the business and you, you need to grow bottom line, you need to do both. You have to hold both things at the same time and say, okay, if, and we've talked about this extensively, if all of our channels are not hitting the target that we've set, we need to figure out a plan B. It can't just be okay. Stop spending money and shrink until you're dead. Right. It has to be okay. What's plan B? And how else can you deploy funds in a way that's different than what you've tested that hasn't proven profitable?

[B]

If there was a hypothetical business that had tested a bunch of channels and was finding that none of them were profitable, they are actually killing themselves faster deploying the capital than if they stopped. That's the problem. You, you, the, like the, the, the frame is false. Like if you are deploying money and it's generating negative money.

[A]

No.

[B]

You are accelerating.

[A]

If there's no, if there's no testing being.

[B]

No, but, but if you do the test and you find out that it's bad.

[A]

No, no, no. I'm saying, okay, you figure out that none of the channels that you've tested so far.

[B]

Yes.

[A]

Have been profitable or to the target that you've set. And then you redeploy into new channels and you keep testing and you keep testing and then your overall efficiency doesn't improve because you're in a constant state of testing. Right. I'm not. I, I guess what I'm saying is you get all the reads back. The channels aren't performing the way you'd like. You either, you either stockpile the cash, which is kind of what you're saying, or you say, all right, we're going to redeploy into X channel or X Initiative. And I would say that's currently what we're doing. It just takes time. Those things are not as easy as

[B]

spending running money through meta. I agree. That is the problem. It's too easy to be bad.

[C]

And we sort of glossed over this. But you're assuming, Taylor, that this brand has already gone on that journey and they don't have any levers to improve incrementality, which, which we don't always see. Like, often there is a lot of low hanging fruit in terms of account structure. Creative. There are like levers you can pull

[B]

because this, this is. Okay, fair. I, I, give me them though.

[C]

So let's say, all right, if you've tested, if you've tested this channel three times, you've made changes and it's still not working. Yes. Like we need to move budget out, but we often see there is There are some opportunities to fix it. So. Okay, let's keep going now. So they've tried to improve it and it's not getting back matter.

[A]

I actually think that conversation is probably more valuable to the audience. What are the actual. What are the things when they get back that first CLS study and it's not what they had hoped. What things should they turn to to increase that score? Before we move back to the problem of. I would say most of your audience probably has things they can do to improve the performance before they revert to, okay, don't invest in Matt anymore. Let's, let's try.

[B]

I think the first question though that we're like, I don't think we've resolved is what's the standard? Like, what's the expectation to deploy capital? I think that, that because everything falls against that constraint, that constraint breeds the creativity to the solution against that problem. When you capitulate the standard, then there is no reason to be creative in solving the problem.

[A]

Well, how do you, how do you suggest they set the standard?

[B]

They define it against the profitable contribution of that dollar against whatever the hurdle rate for investment of capital is in the organization relative to all the options of allocation of capital. If you can go launch a store and you can make 40% return, then the, the hurdle rate for the dollars is actually that 40% return. Now the, the problem is, is that like I agree to your most point. Most business owners don't consider the alternative investments of capital. They don't. And so they just only know. Or they're slow.

[A]

Or they're slow. Like the alternative investment.

[B]

So is a meta return that pays you back 70 cents on the dollar.

[A]

Right.

[B]

That's like really freaking slow.

[A]

True, true.

[B]

And it actually causes a hole today. Like that's really big.

[A]

Yeah.

[B]

So I, I think the idea that that's fast is just fake. Fake. Now what it does is it creates cash. It creates a balance sheet solution that's different than a P and L solution. So this goes back to like, is the goal to create cash? Well then yeah, my gross margins, turning that back into cash at a lower return might be worthwhile.

[A]

Also there's subscription businesses who, you know, like Target.

[B]

That's a realization of value timeline question that I would be in support of.

[C]

I'm going to pause real quick because I forgot to mention this, but I just want to acknowledge how I see. I work. We work with so many agencies and so many brands. How rare it is for an agency. This is the only compliment Taylor's going to get. Probably Ever. How like this conversation is not being had out in the open right now because agencies are mostly billing on percent of spend and they have no incentive to tell or suggest to a brand that they, they slow down. So this is just like, that's why I feel like this conversation doesn't, isn't, isn't being had is because there are not a lot of agencies who are willing to have that conversation. So it's, I just think it's an interesting role reversal here where like Taylor is the agency owner is saying like no, you, you should not spend if the incrementality results are coming back sub one because we never hear that.

[B]

I just like I'm, I'm a business owner. It's my money. Like if my head of marketing Grant sitting right behind us came to me and was like hey Taylor, we're going to invest your dollars and you're going to get them back never. But it's going to grow the top line and you're going to buy 70 cents for a dollar. I would be like no Grant, I don't want to do that. Like I can't. I, I'm just so, so, so often I don't actually believe that there's somebody in the organization that actually believes that's what is happening and has said yes, we want that. Yeah, I just, I, I don't, I think it's a lack of clarity or belief around the measurement system usually.

[A]

Okay, yeah.

[C]

There's one, one thing we haven't talked about which is long term effects and, and so often these tests, you guys cozier has been I think really fun to work with because you, you're, you've been running longer tests which I just think are inherently more interesting. But most of these tests are three, four weeks if you. And then we run this post treatment window where we observe the behavior of the markets after the test and we'll see more, more kind of come in through, through that window. Like if you believe that these effects compound over time then, then we are understating results and we have talked about this a lot at house. I'm actually curious how you all feel about this of like should we model long term effects into our results? And there are some people again imagine a larger org who wants to look good who love that idea. And then there are people like Connor McDonald who's like hell no. Like I come to houzz for the cold hard truth. Do not model my data. And I want it to be no black box transparent. I can recreate these results. And so we've been torn on. Like we, we kind of know these are both average efficiencies, not marginal efficiencies. So that's one thing is like if you just pull down that last 20% of spend, you could be in the realm of efficiency here. So that's one thing. And then the other thing is these are short term effects, not long term effects.

[B]

Yeah.

[C]

And the only way we can, we can actually like provide an evidence backed answer to long term effects is running a very long test. In the absence of that we'd have to model it. And so I'm curious like if that's, if, if you, if, if like should we, should we be modeling this because we know that there is some effect that we can't capture through an experiment.

[A]

Well, I have a lot of thoughts. So when we first ran our CLS studies with Meta, they came in and tried to discredit the results with this argument that there's some sort of long tail effect of the spin that they aren't capturing.

[C]

Oh no.

[A]

Okay, well stupid Taylor's shoulder keeps hitting him in the shoulder.

[B]

Shoulder. It's a big shoulder.

[A]

Not really. I wouldn't consider it big is tearing holes in a shirt though. So yeah, basically META came in, we got the first read, we weren't happy with the result. We're going to pull budget back and they come in and they say hey wait, there's actual effects that you can't see in the study. You should keep spending. And I'm like okay, maybe, right, maybe logically you think about it. Okay, great. The problem is we start running tests with you. There is a post treatment window. So for example, we ran a test in Q4 of last year, the post I think the actual treatment window was six weeks and the post treatment window was six weeks into holiday. So if you think about that timeframe, if you're spending your marking dollars in October, a lot of brands assume that is generating revenue on Black Friday on those peak sales days. The change in return from the time we killed treatment to the post treatment window was almost zero for new customer. And so for a while was there

[C]

was latency for existing for existing almost

[B]

100 which is also odd.

[A]

But anyway, so like the new customer it shows in Anmar who's sitting over here. We've talked about it a lot. I, I love the theory. Until you start looking at the post treatment windows because there is. Why would the impact all of a sudden after the post treatment window increase? Yep, it wouldn't, you know. So like if there is no impact in the post treatment window, there's probably no long tail effect that's at least significant. And that's where for at least our brand, I don't even care to see it. I, I'm with Connor. I would prefer to just. It appears that there is no real long term effect at least on new customer acquisition.

[B]

And like first principles, like so. Because my thing about this is that I am all for whoever has a hypothesis about the long term value and we'll put their name on it and create a number to publish it and defend it and we can, we can all act accordingly. No one will do that. And when you do do it, the post treatment windows which I commend you on for actually attempting to try and do it, it's. And you get that result. Well then you go, well what do I do now? But the first principles, for me, the idea that I scroll through an Instagram and I had an ad impression and nine months from now I can't remember what I had for lunch 20 minutes ago. Like the idea that there's suddenly now I do think that where this becomes more real and where it becomes more true is when your distribution broadens to where you encounter the product unexpectedly in the wild in retail. So where all of a sudden I'm walking the aisles of Target now the recall connection. Because buried in my mind somewhere is that ad. Oh, I, I can do that. And so that effect of ad stock, which comes out of a study that was started in I think the 70s is when the initial ad stock theory was developed. It was all retail distribution. And so the idea that you would connect the dots later in your brain is very different to me than the idea that you would suddenly go, oh you know what, I'm the website and buy now as a result of the ad I saw nine months ago. So I think that's really important to understand how ad stock effects and memory recall play out when the product gets re encountered differently. And so if you're proctor and gamble, I think this is probably real important and can show up in lots of different ways in NCS studies and retail and different things. But the idea that you are generating new customer revenue where your incremental return nine months later is growing like hundreds of percent, I think is like deceptive to the point of criminal behavior. Like it's like insane almost.

[A]

Quick thought on that. Oh, go ahead, Olivia.

[C]

Well, I, I think one, I think it's great that you guys did a test to answer this question like that first. I would, I commend you on that. They actually ran, I think it was a seven week plus. Seven. Seven week.

[B]

14 week total.

[C]

14 week test. That's awesome. Now we have an answer to that question. And if you're, if you're not seeing much come in that. I think that that's, that's important. I guess. My, my question is how much, how much distribution do you guys have in Amazon and wholesale?

[A]

I mean, we have a little bit on Amazon Varies. Maybe 5% of our total.

[B]

It's a big problem. It's one of the. It would unlock a lot more, probably less.

[A]

So yeah, we need to fix our distribution.

[C]

But discovered you guys in retail like a little boutique in Michigan. Yep.

[A]

No way. Well, good job, wholesale team. Anyway, one of the. What were we saying?

[B]

Talking about every call.

[C]

Criminal distribution.

[A]

Oh, I was. You were describing like seeing it on the shelf. My thinking on Meta right now is meta is the equivalent of seeing it on the shelf. So you create the ad stock through other channels. Right? So meta.

[C]

Yeah.

[B]

So they see it on podcasts and

[A]

then meta is the freaking target shelf. That's. And that is why Meta, at least for us at this stage, for our product category, it's less incremental because it is more mid funnel. It is, it is that you are relying on some sort of brand awareness to actually drive the purchase.

[B]

So you're saying they heard about on the podcast and they see it on Meta.

[A]

Yeah.

[B]

Oh, I see. I see. So you're saying that there's some initial implant that happens on a different channel.

[A]

Yeah, yeah.

[B]

It's the 47 leg stool.

[A]

Yeah.

[B]

That I, that is sort of my issue with it. But, but let's go back to like, I want to go back to. Because I feel like we didn't get to. I have a factor. What do I do? And so like, I want to just get to like. Because this is the decision I actually feel on the hook for. And so I feel pressure. Like, okay, we ran the test. George is looking at me, clients are looking at me going, okay, tomorrow you have to allocate my dollars. And the dashboard said something. Scale up, scale down. Good, bad. What. What do I do?

[C]

Okay, well you got. You guys do something too. So let me, let me explain what we do. If you are only using our experiments product, then we line up all the incrementality factors and we look at the, you know, incremental ROAS by channel or CPI by channel. And then we redistribute budget based on cpi. It's like, it's just like looking at a CPA in North Beam, instead of looking at North Beam, you're looking at, at house on a CPIA now, every day when. Well, how often are you, how often are you doing budget reallocations? Like do you, should you be touching it and moving it every day? Like, I think Matt and Google would say that, that that is hurting the system. I think every, every eight. And I don't, I don't buy media but like I feel like every agency owner has a POV and like how much you should be moving in and dialing up and down these knobs. Right?

[A]

Yeah.

[B]

So weekly. So what's the answer Right now you're giving your perspective.

[C]

Yeah, so I think so. So, but that, that you cannot use that chart to answer how much budget should I move from channel A to B? You can we kind of say like move 20%. You know, we're kind of loose about it. That is what MMM is for. MMM is actually drawing that curve and telling you exactly how much money to move to basically have your marginal incremental efficiency be neutral across, across all these platforms. And so the MMM answers the question of all right, exactly how much do I move? But without that you're kind of again, you're playing loose with like, all right, how much budget? Maybe I do 10 or 20% shift and then I retest. I don't. But I don't know that. And, and this is, we're learning into this as well because experimentation has never been like this. Always on OS of like how, like how much, how often are you making daily allocation decisions? Are you making weekly allocation decisions? Like how often?

[A]

And there's a lot of debate on how frequently you should be. I know companies that are doing it on the hour, I know companies that are doing it every day. I think, I know people that really believe in cost caps. But for us, yeah, I think for digital right now it's a two day. But for channel level distribution, that's a monthly decision. And I'll shout out Colby McDermott, he's great. He, he basically has a monthly meeting with all of our acquisition channels and he allocates budget by the most recent or, you know, and we're, we're discussing changing, changing this methodology, but by the most recent incrementality read. So if Meta got a 1.2 and podcast is a 1.5 and that's not an incrementality read, that's a direct code. You know, we measured that. We can't measure that through incrementality. He has basically a process to reallocate budget and Then as far as during the month goes, there is the incrementality factor applied to the in platform reported number. And then you try to hold just that standard so your budget moves up or down real time based on okay, if we need a 3x in platform to get our 1x incrementality or Iroas, then you know, we can't have performance drop below 3 is how we're doing it.

[C]

Taylor, how do you guys. How, how would you say that? How do you operationalize factors?

[B]

So there's a similar to you in the sense that ide every channel gets a factor. If we haven't run a test, we use our benchmark factors, then we'll use an MMM to allocate the proposed budget against those channels. We'll break that into a daily expectation of every channel. Because the reason daily matters to me is because you run sales, you create moments you run. There's reasons why days have different available opportunity for you as a business. The market availability on Meta as an example is a function of usage. There are wild variations in available inventory by day. Like the idea that you would spend the same every day is so flawed I can't even begin to tell you how erroneous this idea that you should spend the same amount of money every day is. So if you need to make some allocation, this is why part of why Costco controls are so good is because they actually allow you to allocate based on the available market, not your presumption of it. I. I'll save that diatribe for another time. But so, so you start there. But here's the key. Okay. And then you have your factors in the channel, but then you have to play the game on the field. It's just like you go into the game with a game plan and then guess what happens on day one. You sent an email, you ran your ads and you made some amount of money and you either made the amount of money you thought you were going to make or you missed and the email either worked or it didn't. The the ads either were effective or they weren't and the money's in your bank account or it's not. And then in light of that reality, you have to course correct, you have to change your behavior because if you don't, then you are just going to be subject to the limitations of the plan as it is. And there you're going to be wild error bars no matter what model you use, no matter what factor you're trying to do all the time. So you have to Constantly be using your best daily discretion against the present reality of what's happening to try and map towards it. I'll give you an example of the way that incrementality factors can fail. When you guys present an incrementality study result, do you break out new versus returning customers? Are the factors the same on new and returning customers?

[C]

No.

[B]

Okay, so if I have a campaign and in the period that it was tested in, it was 50, 50 split between new and returning and suddenly it's 80 returning in 20. Is the factor the same? Of course not. So same thing. You'll do factors on click versus View. Like there's lots of different ways to present it, but if the underlying application of that factor changes, then you have to be conscious of that and that will happen all the time. So you have to constantly be mining for why the reality, which is the revenue, the contribution margin for that day is disassociating from the factor in some way. And you have to try and account for that as much as you possibly can to try and move the thing that actually matters, which I would say is contribution margin on a monthly basis. Like you have to try and move that number as much as you can.

[C]

Are you doing that manually? Is that like people digging into the ad accounts and into the data or do you have a system for that?

[B]

There's a, there's a. We have a lot of things that contribute to it now. So the planning process is part AI, part of data science team, part into part qualitative human planning overlaid against the marketing calendar, email, send, schedule, all the things that go into building the plan and then on a day in, day out basis, it's a combination of AI analysis of the performance, human analysis of the performance, that, that ultimately the, the action still today is in the hands of a person. I'm hopeful that maybe that won't be the case in the future, but for now it is.

[C]

So is the factor.

[B]

Then you can drop it in Grant.

[C]

Is it not static? Is it dynamic every day? Like are. Are your teams adjusting the factor based on what they're seeing? In, in these cases, the factor is

[B]

not allowed to change your behavior relative to the factor.

[C]

What you do with it?

[A]

Yes.

[C]

What you do with the budget. Okay, that's interesting.

[B]

So I. You only update the, the factor in the face of new evidence.

[C]

Yep.

[B]

Let me ask you something. Go ahead.

[A]

So let's say you run any. So this is very common. You run an email campaign.

[B]

Yes.

[A]

The meta in platform number is now a five.

[B]

Yeah.

[A]

Usually it's a Three. So are you assuming that that day you had a 2 1/2 x incremental ROAS or are you assuming that I'm excluding?

[B]

It sounds like you have bad exclusions.

[A]

No, sounds like that's every single meta account.

[B]

That's not true. If I have really good exclusions to new customer acquisition on a quick basis.

[A]

You don't have really good exclusions.

[B]

You need three things. You need Klaviyo, audience exclusive and pixel exclusion and your shop and your Shopify audience excuse. And there will be leakage. Yes, of course. But you can get it. No, it'll be sub 10% maybe.

[A]

No, it will not. Yes, Meta has an official stance on this. I just met with them. There is not 10.

[B]

What do they think it is?

[A]

30 to 40 to 40%.

[C]

They said that?

[A]

Yes. And if that's. I probably shouldn't say that.

[B]

If that's true, then what is the value of a CLS study?

[A]

What if that's they're working on it

[B]

telling me the user match rates are that bad then this whole thing collapses on itself. What is the value of optimization? How can you actually attribute a purchase back to the end?

[A]

Don't be mad at me.

[C]

I'm just happy. I mean that's great for our business, right?

[B]

That's if.

[C]

If the so so Met is only able to match 60 to 70%.

[B]

Yeah, now we're getting clipped. Don't, don't, don't.

[A]

Do I. Look, I don't know for sure there's.

[B]

There's.

[A]

That's not Olivia going to be in

[B]

a disruptive meeting and have to explain.

[A]

But I can tell you right now

[C]

they know that Taylor doesn't run your ad account so he's able to just totally trash him.

[A]

Yeah, I don't care. He basically. I know they. They are admitting that the exclusions product does not work and they are working on revamp. That was a big talking point. There is a reason there are a lot of companies spending on waste not and blot out and all these companies that improve exclusions and match.

[B]

Okay, but if they do then. Then you use all those people.

[A]

Yeah. Yeah.

[B]

So you're. And so what are you telling me your match rate then yours. I don't know the tools that you use.

[A]

Sure. Ours is probably sub 10%.

[B]

Okay, so then. So then when you send an email it should not affect new customer revenue on a click basis and certainly not on a new customer revenue on a one day click.

[A]

Well why does. I mean sure in theory it shouldn't but I can pull up we could go through all of your clients accounts and you're going to see when they send an email that the in platform roas spikes and that's just the truth.

[B]

No.

[A]

Yep. No, we're going to do it after this call and I'll, I'll make my first tweet about that result.

[B]

I wish it were true that we perfectly execute our ideology everywhere. We don't. We have gaps. You can find gaps. There's no doubt about it. But the, the, the point is that if you are executing that then the effect should be minimal. Now what I'll say is that I think a harder question is is the incrementality during a sale moment the same as a non sale moment? Those kinds of questions I think are worth interrogating and I'm actually I give our, our strategists real leniency if they want to make a case for why they human qualitatively adjusted a factor on a day and changed their behavior based on some insight like that I care most that behavior and belief are aligned in most organizations. They're wildly disparate. If you can defend why you change the factor on Tuesday and Wednesday during the sale to a lower number because of you were going to send a bunch of emails you were worried about over reporting you wanted to be conservative then I will support you in that position. Like there are novel edge cases all the time and I think that's one of them. So I, I just think though that what you're doing is, is what I watch every organization do which is that they undermine the information all the time to where no scenario is actually actionable because you can poke a hole in almost every decision.

[A]

The question I'm asking is how do you handle that? And you're saying just don't mess with it.

[B]

No, I'm saying have a, have an opinion and express it.

[C]

But pre commit.

[B]

Yes.

[C]

Before that. Before it comes up on how you're

[A]

going to be clear we don't change our factor if we send an email on the end platform then why did

[B]

you just do that?

[A]

Because I wanted you to admit that be but that that is something that should be because people might think about that. People might say like well when I send a sale it goes to a 5x how should I factor what Olivia

[B]

said predefined the measurement and believe within it and then adjust it at a different period of time. I watch but, but this is what you did is what I watch happen in every organizations all the time which is somebody says something, somebody undermines it Somebody says something, somebody undermined. Yes. Well, I agree. Right. Like when you, not if you, you, you assert that they are revealing truth and you are building a population of results about what happened at various moments in time that then if you assume that they're supposed to be the same then they will undermine each other.

[A]

Well the problem.

[B]

But they're never going to be the

[A]

same us as we assume. So every month we get, if we get a bad result, we implement a plan to fix the result. We're always trying to improve the incrementality of the channel and then maybe the, maybe the performance improves for the next three months and we, we, we assume that the performance improved because of X change. For example, we added waste not. I have no affiliation with them. I don't, I don't even know if I like software. In September of last year and had our back to back highest incrementality scores ever in a CLS study and I assumed that that was maybe in, you know, causing the high incrementality scores get to January not as good. So I, I think that's where for me, I expected to start this journey and start stacking wins and see the score improve gradually over time as our strategy shifts. And that just hasn't been the case.

[B]

Here's, here's one of the reasons why and this is where I'm going to ask Olivia to give us a preview of what they're trying to become in the world. But I think the reason why is because your optimization is not connected to your measurement and so you aren't actually meta. It's the delivery of your dollars is not actually in trying to affect incremental attribution. Like, and I'm hopeful, like to me if you're on this journey, I don't know how you couldn't be also committed to using the ad product. Now I get that Meta is saying like oh, there's some errors, it's got some stuff to work out and they undermine their own ad product as it relates to incremental optimization. But to me the end state here has to be, it has to be that we are optimizing for incremental effects. Otherwise why would we assume that trying to optimize for something else will create incrementality? Because it's not going to. Which is why one of my predictions in the future is that House is going to build an automated media buyer and ultimately like what I would think of as like a dsp, like a demand side buying platform that actually does optimize for their incremental measurement system. So win product.

[C]

I'm going to go back a second. One of the unintended benefits from doing this as a SaaS product is pre committing to how you're going to do the analysis before you do it. Like we have found so many instances of if you have an in house data science team they are engineering the result that the marketing stakeholder wants. We have like this is, I'm not just giving you know like a few examples. Like this is just prevalent inside of these organizations is the data science team is tweaking the results in a way to land on what, what these stakeholders want. And so that's really important. That's why I like that we're doing this hands free with the, with the like ROI piece. Like you, you, you wanted to be stacking wins. I think there are two questions. Is it, are you actioning the tests? Like because that's again that's where the ROI comes in. And this is Taylor, I like what you said of like you kind of like you can't continue to throw in these you know, hypotheses or reasons why not to believe. Like you actually have to commit to this and make decisions on it. And then question two of like stacking is. I think this is, we talk about. Taylor and I talk about this all the time. It's really hard to actually and this is what's so ironic about a measurement company is like it's hard to actually know like maybe your business, it would have been down more if you didn't, if you didn't actually test like maybe even if it's, if it's, it's really hard to actually tie these changes that you're making to like business level outcomes given all the noise in a very large business.

[A]

Well to your point earlier about counterfactual there. Exactly. There is no counterfactual. Even though you're running the holdout, there's no counterfactual to business before you made the change there.

[C]

There is no follow. Follow up test of like made this change.

[A]

Yeah.

[C]

Ran this test now run the test

[A]

versus the old and we could actually do that.

[B]

Yeah. What do you mean?

[A]

Yes, there is not. We haven't all haven't done it. We haven't done it. Trying to think of how we would structure that.

[B]

Well we ran an all up test where we held out all media. To me I think of that as like a portfolio test. I think you redesign the portfolio and you test it again.

[A]

Yeah. And you just continue to. But that's the point. So this is this is the whole problem. If I, if I really. This has got me thinking about the past. I think about what's happened. Right. We have a poor result. We change the whole account from seven day click one day view to only seven day click. We get two months of strong roi. We're feeling optimistic, we think perform and then down. You know, it's like we constantly make changes. We get false signal that it's working. We reinvest in the channel because I don't think it actually has anything to do with the change we made. I think it has everything to do with the demand for the category at the time and that, that ultimately is where I'm landing for our business is it's much more, at least Meta's performance is much more dependent on market demand for our product categories than it is any media buying strategy that we're going to implement.

[B]

I think that's, that's true in a very mature category very late on the customer adoption curve. I think there are just countless businesses even in your very closely aligned to your category.

[C]

Yeah.

[B]

That prove counter to that where they're growing right in your face.

[A]

Not, not they would not say Meta is their main strategy.

[C]

I mean George, you asked me earlier like what are the follow on tests that people run to improve their result? Like have you done like have you really, have you invested in like creative testing? Like obviously I know that's the biggest lever. Like you feel like you've swung through all the way on that.

[A]

Yeah. And he can disagree but no, I

[B]

think, I think you guys have done.

[A]

We, we have invested and that was Meta's thing. We actually, you know, we made some hard decisions last year. We let go of a lot of our creative team and rebuilt a growth creative team whose only focus with Meta was Meta ads. And Meta and some other partners were like this is the, this is the lever. This is going to improve your performance and it hasn't really.

[C]

So I've. We saw a 50% improvement in CPI at Netflix through a creative template test. We changed our template that we ran the shows through 50% win. Other than that we were never able to see anything more than like a 10 to 15% win on accounts.

[A]

I think for us we were already at a point where the creative like I think there are businesses where like let's say you have an ad account that's just running static ads and like really bad video create. I think there's opportunities for creative diversity to really help them. I think for us we just got to a point where there's A diminishing return on the creative diversity. And maybe you know, our meta reps, I just said they were in town, we met with them and I said hey, you guys made us make hard decisions last year and it hasn't really paid off. Explain why. And they, they said, and I said are we not doing a good enough job at creative diversity? And they're like no, that's not it. You guys are doing a good job.

[B]

So what did they say it is?

[A]

Well, they said the January test, just to let everyone know, the January test for poor performance was because of an ad placement.

[C]

Have you seen it in fan audience network?

[A]

No, it's not the audience network thing, it's the on reels. So it's like an ad placement. It's a piece of junk placement. It's an ad that pops up on a reel that's unrelated to your product.

[B]

Yeah.

[C]

And it, yeah, platform changes.

[A]

It performed really well in Q4 for us because we're running promo. And then when promo turned off, that placement turned, you know, started performing really poorly. And then spend was still going there because of the strong performance in Q4. And they're claiming that's why January's result has been less than ideal.

[B]

And these are all just things like again, so your incrementality read in that month was true. And so it goes on the plot of things that might happen right there are there, there, there is a built in dispersion of potential future outcomes laid out by your historical ones. And that is one of the metabugs can happen in the future. And so the way that you consider for that is that you allow it to inform the expectation of the future and then you'll have periods where you over underperform it. But, but I, what I would like to see is that you want to see the monthly relationship to your measure tightening over time. So like if, if we started in month one and you know I'm going to make up numbers right now, platform said three and we got back a one, that's a 300% delta to performance. If we get to the end of this and we're working off of a 0.7 and the result was 0.4. Well, we have really narrowed the error bands through which we're making decisions. That to me is a sign of a really good testing program. You're actually getting the truth now. Fixing it is sort of a different thing than measurement. Like these are, it's really important to us measure. Measuring things doesn't make them good. Yeah, it just assess what is. But it's really powerful.

[A]

Right.

[B]

And this is where they get to. Olivia gets to sit in the seat of just like we will tell you the truth. And that is really important. It's a really important part of the process. But it doesn't lead to fixing it and that becomes where the real work happens and where it becomes really hard. So I empathize with that.

[A]

I feel like we've given a good answer on that on how to fix it if they get a score back that's less than. I mean we've hit on creative diversity. Is there, are there been other things that you've seen that you feel like most brands should try as an effort in an effort to increase the score?

[B]

I think there's a lot of things I could give you a lot of a lot of things here.

[C]

There are some levers I think creative is, is the biggest lever. Like that's most often when I see like step change, improvements in performance. It's a creative swing.

[A]

Yeah.

[C]

Expanding distribution, like you know that's, that's huge.

[A]

Meaning. So if you're talking to a 10 million dollar Ecom business, meaning go live on Amazon, meaning TikTok shop, you know, what do you mean by distribution?

[C]

Like expanding into, into Amazon and retail.

[B]

Yes.

[A]

And do you measure lift in retail?

[C]

Yeah.

[A]

You do?

[C]

Yeah. You, you know you have to get the sales by the by retailer. Like Target's easy, Walmart's easy. Yeah. Because that, that, I mean that's something we talk about all the time is if you are doing business in Amazon, Amazon and wholesale you are definitely understating your results. If you're only measuring on D2C.

[A]

Yeah.

[C]

So that's a, you know that's a very common reason why these brands think they're incrementality results are low is that they're not pulling in all their sales channels.

[A]

Yeah.

[C]

So yeah that can make a difference. Account structures. Interesting. Like I think that there are accounts. This is, this is an enterprise problem like 50, 60, 70% retargeting. I am not exaggerating.

[B]

No doubt about a lot of low

[C]

hanging fruit on like an account that really hasn't been, hasn't been optimized in terms of, of best practices. And so account structure we see click versus view based optimization has helped the signal engineering has been mixed in terms of. All right, I need to point meta toward a shallower metric to help it go prospect new customers. It's not a home run every time but like we have seen a lot like Ridge talks about it, the alternative objectives.

[A]

Yeah.

[C]

Getting them like out of this cycle of just like over delivering to this very small bubble pocket of potential customers. So we've seen people get out of it. We have a lot of success stories around not just creative but account structure. But you know I think you, it seems like you guys have pulled a lot of those levers already which is a different, that's different from where a

[A]

lot of these we've pulled levers. I will say like the top of funnel objectives is something we could lean into a little heavier. The creative diversity I feel like we've pulled the lever you know over a 12 month period and the exclusions yet to be seen. But if the, if the results rebound post January which they're starting to after turning that placement off, there's still an argument to be made that fixing the exclusions help.

[B]

Well I actually, I actually think that the data says something different that the data says that you should spend more on existing customers than you are.

[A]

Sure.

[B]

And this, this is actually I think a big, this is a contrarian opinion that I know everybody I think brands, big businesses with large customer files are massively underspending on existing customers and that they get, they would get more incremental return by in terms of IROAs. In terms not in terms of incrementality factors but in terms of if you want to improve your IROs that you would actually get it more from existing customers than from new customers.

[A]

Well all of our test results show that. What I'm saying is you need clarity on, on how many dollars actually went

[B]

through the other thing. So for you, the other thing I'll say so you say it's creative. I would say product is number one. It's like to go into a category where there are tailwinds like you just described like you there if you're in a category that's growing I think you're going to do better than a category that's stagnant and mature and highly competitive which is where you are and all the profit gets competed away down the two is Amazon. If you guys were on Amazon, your meta irs, you're on Amazon as broadly distributed and you introduce Amazon revenue into the measurement your IRS will be higher because you are having an effect there and the fact that you're not measuring it is suffocating your view of your own performance. So that's so productive would be one second would be that and then account structure like I I think is I, I think that if you wanted to improve your IROs or IMR you should introduce cost controls on all the dollars,

[C]

like you say sale out cost controls. Is that what you just.

[B]

On every dollar.

[C]

How about partnership?

[A]

Like if cost controls did what you think they do, there would be clear indicators that it's more incremental. No, no, because it would be. Because you'd be spending less money.

[B]

No.

[A]

Yes you would.

[B]

That's, that's not what you. No, it's, it's incremental return on ad spend. Not incremental factor. It's not the IROS factor that's higher. This is the confusion that people have when they say it's real. A channel's really incremental. That means it has a really high incrementality factor relative to what the platform reports. Like what I'm talking about is the incremental return on ad spend, the efficiency of the dollars.

[A]

And have you seen that? Have you seen more efficient dollars? I agree with that.

[B]

That's a choice.

[C]

I'd be lying if I said we've run a lot of these tests. We've run a few head to head cost cap, lowest cost tests and what we could do is we could like label all of the accounts that we've tested in meta and we, we could, we could probably do that in another follow on report but with the head to head test, we've seen, we've seen not, not much difference.

[B]

So what you have to do in the test though, you'd have to see what the cap was set at relative to what the expected target was on the other one. Because if they're the same then it's like, well that's not going to be.

[C]

We got to answer this question though, for sure.

[A]

I mean that's a.

[C]

Guys, what about partnership ads? That's where I've seen a lot of wins.

[A]

We run a lot of partnership ads. I mean we're doing most of the best practices. There's not like a lot of like, oh well, we, you know, we're part of the disruptors group. It's really just a spend level decision. That's what it comes down to. Do we want to pullback spend divest investment in Meta or. The other point is, and we haven't

[B]

talked about this isn't just Meta either, it's every.

[A]

Yeah. The other thing is, is where we have a relatively diverse mix which creates a high baseline.

[B]

Yeah, yeah. That would be my third thing to do to fix it would be wildly narrow the media mix. Shout out to you Connor and Cody and Connor for bringing this up on your podcast the other day. Channel diversity is the absolute killer of clarity. Nobody has any idea what's happening when you get into 10 channels. Nobody has any idea. And so complexity breeds inefficiency.

[C]

Y the mantra I like to to preach is like either metag, Google big ideas and nothing in between because it will like you will get more creative if you're focused on big ideas and not just trying to, trying to like light up a new channel with the same creative and thinking that's going to create a step change improvement. So for sure I like that mantra of like if you're going to go outside of those two channels, make it a big swing instead of just repurposing what you were doing in another channel.

[A]

Yeah, 100%.

[B]

Well I hope this was helpful in some way shape or form. I think if anything what I hope you can learn to sit in is to accept the difficulty of if you want to become a measurement based company and you care about this idea of progressive truth about building being less wrong over time as an organization that it's not going to be a binary, it's not going to be one test and you solved it all that you're on a journey and that building structural decision making inside of your company through complexity is really important. And these are really smart people that are here to help. And we're as you can hear, we haven't figured it out for Cozy Earth. We're working together, we're working through the

[A]

problem, made some progress. It is not, it is not super easy but I will say the clarity is very helpful.

[B]

Yep. And you know, as much as we're competing day to day, we get to work with House a lot, including in the case of Cozy Earth. And I'm an Olivia Stan and very appreciative for you coming to hang.

[C]

Oh thanks Taylor.

[A]

Thanks for having us, George.

[C]

Thanks. George flew here.

[A]

Yeah.

[B]

Just to be on the podcast.

[A]

I flew here to beat Taylor in golf and he bailed.

[B]

So by the way you're hit, you're a good metaphor. Your handicap is basically what this is by the way. If you think about an incrementality factor in an iros, that's what your handicap is.

[A]

I didn't play to my handicap.

[B]

You should think about that though. Last time we played take that away for use of you that are golfers. If you want to understand IROs, think about a handicap and how every new score doesn't suddenly become your handicap. Exactly.

[A]

Oh yeah, that's a great analogy. Good job.

[C]

Really good one.

[B]

Thank you.

[A]

Peace.

[C]

All.