Podcast Summary: "Incrementality Playbook Every Ecommerce Marketer Needs"

Marketing Operators
Hosts: Connor Rolain, Connor MacDonald, Cody Plofker
Date: March 17, 2026

Episode Overview

This episode dives deep into the concept of incrementality in ecommerce marketing—how to operationalize it across an entire organization, implement a culture of continuous testing, and use a rigorous, hypothesis-led approach to measurement and media planning. The hosts unpack the practical realities of running incremental tests, moving beyond geo-holdouts to a holistic framework for growth, and emphasize building a culture where marketing investments are measured and optimized for their true additive effect on business outcomes.

Key Discussion Points & Insights

What is Incrementality and Why Does It Matter?

• Definition (00:42, A): Incrementality represents "additional" outcomes that would not have occurred without a specific tactic.
• The Real Goal: Marketing’s purpose isn’t just to get results, but to prove those outcomes wouldn’t have happened otherwise. It’s about distinguishing true growth from cannibalized or misattributed revenue.
• Quoting notable research: “Teams that ran a certain number of experiments per year had like 17% lower CAC... just an always-on program where you’re constantly testing things.” (00:49, C)

Mindset Shift: From Platform Metrics to Truth-Seeking

• Cultural Change (06:03, C): “You’re almost trying to be truth-seeking, and you’re trying to have more questions than answers and kind of using incrementality to guide that in everything you do.”
• Shift away from reporting vanity metrics to focusing on what’s truly moving the business.

How to Operationalize Incrementality Across the Org

1. Hierarchy of Metrics (06:03, C):

   • Know which metrics are directional vs. causal (e.g., MER, MTA, MMM)—use true causal data for decision-making.

2. Constant Experimentation:

   • Example: Hexclad team runs frequent tests across lines of business and channels (08:38, A).
   • “We ran 58 tests last year... The reason we’re able to do that is we basically run four separate lines of business right now. Between wallets, rings, travel, tech.” (00:10, B)

3. KPIs Should Reflect Incrementality (07:14, A):

   • Case Example: Hexclad compared Shopify vs TikTok Shop demographics to verify if their TikTok Shop channel was reaching a new, incremental audience.
   • “11% of our audience that buys through Shopify is 18 to 34 years old. But on TikTok Shop, that percent is 32%... So that’s like an incrementality-first approach.” (07:54, A)

Applying the Incremental Framework in Retention & New Channels

• Retention Testing (10:01, A):
  • Email/SMS programs often overstate ROI; need to implement user-level holdouts and measure true incremental impact.
  • “Retention’s another tricky one because ... just because you sent an additional email or someone clicked an additional email still doesn’t mean it was necessarily an incremental outcome.” (12:20, B)
• Collaboration with Tech Partners:
  • Example: Working with Post Pilot to extend test period for direct mail or SMS, ensuring measurement of true incremental impact (15:33, A).
• Expectation From Partners (17:34, C):
  • “Any partner knows if you’re going to be asking me for money, prove it with incrementality. This is what we care about—very clear and firm.”

Structure & Sequence of Testing (+ Notable Quotes)

1. Baseline—Channel Holdouts:

   • Test spend vs. no-spend to establish baseline incrementality.
   • “Step one is just like channel level holdout, right? Hey, no spend versus spend. Is this channel incremental?” (58:01, A)

2. Scaling Up & Segment Testing:

   • Run scaled-up tests (multiple spend levels) and test new objectives or creative strategies.
   • “Now we’re starting to string together ... we tested view content last year for a long time, got a really good readout... now we’re leaning into other non-purchase conversion objectives.” (20:49, A)

3. Iterative, Hypothesis-Led Adjustments:

   • Sequence tests back-to-back, analyze outputs, fine-tune strategies.
   • “What I love about it is it’s so objective, causal, and simple. You get like two metrics, right?... But then it’s like, all right, you get a bad test, like why? Then you have to dive in, make decisions, follow up, test again.” (17:34, C)
   • “Meta performance hasn’t been what we wanted. First I need to get a baseline, right? A lot of things have changed... so we ran another test and made a bunch of changes...” (22:17, C)

Embracing Directionality Over Perfection

• Not Every Test Needs a True Holdout (32:46, B):
  • It’s often more practical to optimize for directionally improved outcomes, not academic perfection.
  • “What matters to us is just improving the incremental ROAS. We’re not trying to get in a peer-reviewed journal.” (32:46, B)
  • “If your KPI is so far from your target right now, you’ll know if this worked or not, even without the perfect data.” (32:47, C)

Optimization Examples, Channel Mix, and Team Enablement

• Brand Video Integration:

  • Ran true holdout tests adding brand video to purchase-optimized campaigns, saw 40% lift in incremental outcomes (55:11, B).
  • “We saw a 40% lift. So it was 40% more incremental in the cell that had the brand videos, which is a massive win.” (55:11, B)

• Testing at Scale (50:38, B):

  • Ability to run more than one test a week is credited to having multiple lines of business and infrastructure for concurrent testing.
  • “We ran 58 tests last year, which is like more than one a week... Ridge is four separate lines of business... so we can basically run concurrent tests.” (50:38, B)

• Culture Building:

  • Encourage team to make the case for incrementality.
  • “Part of building that culture is just like the Socratic questioning of: ‘Prove this is incremental.’ And over time the team learns what is and isn’t a data point that proves incrementality.” (61:46, A)

Notable Quotes & Memorable Moments

• “What matters to us is just improving the incremental ROAS. We’re not trying to get in a peer-reviewed journal.” (32:46, B)
• On Experimentation Culture:
  “Teams that ran a certain number of experiments per year had like 17% lower CAC... just an always-on program where you’re just constantly testing things.” (00:49, C)
• On Brand Video Test:
  “On the MTA data, we saw a 40% lift... 40% more incremental in the cell that had the brand videos.” (55:11, B)
• Incrementality in Retention:
  “You should be setting it up internally so that you can observe what is it doing in addition to what would have otherwise already happened.” (11:28, B)
• Defining ‘Incremental’:
  “That’s all incremental really means, right? It’s additional. You would not be getting that outcome... if you were not doing that tactic.” (00:42, A / 09:44, A)

Timestamps for Key Segments

• Introduction to Incrementality & Experiment Culture: 00:00–01:07
• Case Study—Testing TikTok Shop vs Shopify Audiences: 07:14–09:44
• Retention and User-Level Holdouts: 10:01–13:50
• Testing Structure & Iterative Process: 17:34–26:39
• How to Sequence Tests & Analyzing Results: 20:49–28:05
• Precision vs Directionality in Measurement: 31:53–35:05
• Brand Video Incrementality Test: 54:24–55:27
• Team Operationalization & Test Volume: 50:38–52:44
• Summarizing the Playbook: 58:01–61:46

The Incrementality Playbook in 3 Steps

1. Channel-Level Holdouts:

   • “Hey, no spend vs spend. Is this channel incremental?”

2. Scale-Up Tests:

   • “How much can we spend and have it still be incremental? Use scaled spend tests.”

3. Optimize the Channel:

   • “Test channel-specific strategies: brand videos vs not, view content vs purchase conversion, etc. Stay in this optimization loop as the core long-term tactic.”

Final Thoughts

• Build Directional Conviction: Perfect data is rare; prioritize actionable direction over academic rigor.
• Cultivate a Culture of Experimentation: Reinforce a truth-seeking, hypothesis-driven environment—make “prove it’s incremental” the expectation.
• Leverage Tech & Vendor Relationships: Push partners to support real incrementality measurement.
• Incrementality Beyond Revenue: Use the incremental framework for everything—retention, brand awareness, even retail pilots.

Recommendation:
Implement these incremental testing principles not just as a measurement tactic, but as an organization-wide cultural shift. Let incremental thinking guide metric selection, creative strategy, and every marketing investment decision.

For more, listen to the full episode for tactical detail and additional case examples.