Podcast Summary: Ecommerce Playbook – “Turning Incrementality Tests Into Action That Makes You Money”

Date: February 26, 2026
Hosts: Taylor Holiday & Richard Gaffin (Common Thread Collective)
Guests:

• Olivia (Chief Strategy Officer, Haus)
• George (CMO, Cozy Earth)

Episode Overview

This episode takes a deep dive into one of the most complex and pressing challenges for modern DTC (Direct-to-Consumer) brands: how to not just conduct incrementality and holdout tests on marketing spend, but how to take the results and operationalize them into clear, profitable actions that drive business outcomes. Hosts Taylor Holiday and Richard Gaffin are joined by Olivia from Haus (a measurement company specializing in experimentation) and George from Cozy Earth (a leading home goods and apparel brand) to candidly discuss the messy realities behind incrementality testing, how it upends commonly held beliefs about marketing performance, and what it really takes to make such test results actionable in day-to-day marketing operations.

Key Discussion Points & Insights

1. The Incrementality Testing “Journey” (10:08–13:28)

• Initial Shock, Then Disillusionment: Many brands start with enthusiasm about incrementality testing, but are often shocked by lower-than-expected incremental ROAS (Return on Ad Spend). This is especially true for brands used to reporting much rosier numbers through in-platform metrics.
• Olivia: “We’re sort of through the first chapter of, like, 'okay, we need this.' And now it’s, 'How do I use this as an operating system and really operationalize this?'” (05:43)
• Testing Is a Methodology, Not a One-Off: Establishing testing as a recurring methodology—rather than a series of isolated events—is critical as brands mature.

2. When to Start Incrementality Testing (21:00–24:41)

• Brands shouldn’t wait until they’re “big enough;” issues may arise when organic and repeat demand increases, complicating what used to be clear attribution paths.
• Taylor: “Incrementality is a step to take when you’ve built distrust in your present measurement system… change happens if and only if dissatisfaction is large enough.” (21:34)
• Misattribution of all new customer revenue to Meta (Facebook/Instagram) remains a common, flawed assumption for many DTC founders.

3. Understanding Holdout and Incrementality Tests (19:08–20:59)

• Olivia: “We fundamentally believe at Haus that incrementality requires a holdout. You need a counterfactual to understand what would have happened anyway... Otherwise, you can’t attribute causation.” (19:08)
• There’s a stark difference between “turn-off” natural experiments and true randomized controlled holdouts.

4. Operationalizing Test Results – Core Challenges (13:28–38:43)

• The Replication Crisis: “One month it’s this, and the next month it’s that. And there’s no clear reason why the result changes so much. It can be frustrating.” – George (14:27)
• Testing Frequency vs. Value: Running more tests doesn't necessarily create more value—only actioning on test results matters.
• Taylor: “Are you running a test, or is testing a part of the way you’re going to do measurement? Those are very different places.” (13:17)
• No Universal “North Star”: The best efficiency metric for marketing should be directly tied to organizational goals—whether that's free cash flow, contribution margin, or TTM EBITDA. Only then does incremental contribution margin become an actionable metric for channel investment. (26:57–29:34)

5. Applying Incrementality Factors (“IFs”) in Budgeting (34:05–38:43)

• Cozy Earth approach: Apply an incrementality “factor” (the ratio between incrementality-tested and in-platform ROAS) to in-platform performance to assess the true impact, but results can be highly volatile.
• Olivia: “If you get a read that’s, you know, 1/3...and then six months later you get 0.8...that’s just too blunt of an instrument to actually be useful in decision making.” (36:58)

6. Actionable Frameworks for Brands (44:03–88:45)

The Three Haus Playbooks (44:09):

• Efficiency: Cut costs.
• Redistribution: Move budget from underperformers to best performers at a flat budget.
• Scale Up: Invest incremental spend in new channels to drive growth.

“As long as you’re moving budget from low performers to higher performers, you will get better. It still might not be where you should be.” – Olivia (44:09)

Redistributing Budget: Channel vs. True Return

• The real threshold is not “channel-vs-channel,” but “channel-vs-potential scale of profitable return.” (47:03–48:05)
• Cutting unsuccessful media is necessary—there’s no reason to deploy capital that doesn't generate profit, even for the sake of “growth”.

Frequency and Method of Redistributing Budgets

• Channel-level factors: Most leading brands assign incrementality factors by channel, then use either MMM (Marketing Mix Modeling) or other frameworks for reallocation.
• Reallocation cadence: Monthly is common for channel-level redistribution, but day-to-day tactical adjustments are often made in response to campaign performance and overall contribution margin.

Memorable Quotes & Moments

• Taylor: "How does your organization decide what’s true? What is the standard for truth inside your company?" (24:41)
• George: "For us as a brand, it’s really difficult to figure out what creates the outcome. Still figuring it out. We don’t have it down to a science yet, but we’re working on it." (14:27)
• Olivia: "There are a bunch of brands who will fill those testing slots. And it’s testing for the sake of testing...it’s not how we drive value for our customers." (13:28)
• Taylor: “If you are deploying money and it’s generating negative money...you’re accelerating...you're killing yourself faster, deploying the capital than if you stopped.” (50:14)
• On Model Trust: "I actually have more confidence in Olivia than I did the prior team." – George (41:29)
• Re: Long-Term Effects: "The idea that you’re generating new customer revenue where your incremental return nine months later is growing, like, hundreds of percent, is deceptive to the point of criminal behavior." – Taylor (60:39)
• On Brand Owner Mindset: “If my head of marketing came to me and was like, ‘Hey, Taylor, we're going to invest your dollars and you’re going to get them back never, but it’s going to grow the top line,’ I’d be like, no.” (54:38)

Practical Recommendations & Tactics

When Facing Poor Incrementality Results:

1. Creative Testing:
   "Most often when I see step-change improvements in performance, it’s a creative swing." – Olivia (82:31)
2. Expand Distribution:
   Moving into new channels like Amazon or retail can surface hidden incremental returns often missed when only measuring D2C.
3. Account Structure:
   Audit retargeting levels (especially in enterprise accounts), optimize click vs. view-based segments, and assess signal engineering (sometimes focusing on shallower conversion events can help).
4. Cost Controls:
   Channel-wide cost controls (e.g., cost caps in Meta) may help drive efficiency, but need close calibration and testing.
5. Narrow the Mix:
   Excessive channel complexity (“channel diversity is the killer of clarity”) can make actionable learnings impossible. Focus investment on big ideas in your best-performing channels.

Defining “Success” with Incrementality Testing

• It's a Journey, Not a Switch: Implementing and actioning incrementality results is an ongoing process of progressive refinement—a continual effort to be less wrong over time, not a one-and-done event.
• Progress is Error Band Reduction: The real sign of progress is not always increasing incremental ROAS, but a tightening between measured incremental outcomes and your expectations—i.e., less volatility and fewer huge surprises.
• Measurement ≠ Fixing: “Measuring things doesn’t make them good. It just assesses what is.” (82:00) The real work is in taking hard results and building business strategy around them—and sometimes, that means acknowledging tough truths and changing behaviors.

Timestamps for Major Segments

• Setting the problem & bios: 01:16–07:11
• The journey of incrementality: 10:08–13:28
• Types & setups of tests: 19:08–20:59
• When should brands start testing?: 21:00–24:41
• How to apply incrementality findings: 34:05–38:43
• Action plans & frameworks: 44:03–88:45
• Optimizing creative/distribution/structure: 82:13–83:13
• Big picture takeaways: 88:45–end

Final Takeaway

Incrementality testing exposes uncomfortable truths for DTC marketers, but embracing it as an ongoing operating system is the only path to informed and sustainable growth. The journey is never done—actionability relies not just on measurement, but on organizational clarity, brutal honesty with results, and the discipline to channel dollars toward what’s truly profitable, not just what fits convenient narratives.

Memorable Analogy:
Taylor: "If you want to understand IROAS, think about a golf handicap and how every new score doesn’t suddenly become your handicap." (89:45)

For brands struggling to move from measurement to action: Accept that measurement is imperfect, resist the urge to chase false certainties, and focus on building an actionable truth over time—one that survives not just in models, but in P&Ls and cash flows.