Podcast Summary: "Incrementality Playbook Every Ecommerce Marketer Needs"
Marketing Operators
Hosts: Connor Rolain, Connor MacDonald, Cody Plofker
Date: March 17, 2026
Episode Overview
This episode dives deep into the concept of incrementality in ecommerce marketing—how to operationalize it across an entire organization, implement a culture of continuous testing, and use a rigorous, hypothesis-led approach to measurement and media planning. The hosts unpack the practical realities of running incremental tests, moving beyond geo-holdouts to a holistic framework for growth, and emphasize building a culture where marketing investments are measured and optimized for their true additive effect on business outcomes.
Key Discussion Points & Insights
What is Incrementality and Why Does It Matter?
- Definition (00:42, A): Incrementality represents "additional" outcomes that would not have occurred without a specific tactic.
- The Real Goal: Marketing’s purpose isn’t just to get results, but to prove those outcomes wouldn’t have happened otherwise. It’s about distinguishing true growth from cannibalized or misattributed revenue.
- Quoting notable research: “Teams that ran a certain number of experiments per year had like 17% lower CAC... just an always-on program where you’re constantly testing things.” (00:49, C)
Mindset Shift: From Platform Metrics to Truth-Seeking
- Cultural Change (06:03, C): “You’re almost trying to be truth-seeking, and you’re trying to have more questions than answers and kind of using incrementality to guide that in everything you do.”
- Shift away from reporting vanity metrics to focusing on what’s truly moving the business.
How to Operationalize Incrementality Across the Org
-
Hierarchy of Metrics (06:03, C):
- Know which metrics are directional vs. causal (e.g., MER, MTA, MMM)—use true causal data for decision-making.
-
Constant Experimentation:
- Example: Hexclad team runs frequent tests across lines of business and channels (08:38, A).
- “We ran 58 tests last year... The reason we’re able to do that is we basically run four separate lines of business right now. Between wallets, rings, travel, tech.” (00:10, B)
-
KPIs Should Reflect Incrementality (07:14, A):
- Case Example: Hexclad compared Shopify vs TikTok Shop demographics to verify if their TikTok Shop channel was reaching a new, incremental audience.
- “11% of our audience that buys through Shopify is 18 to 34 years old. But on TikTok Shop, that percent is 32%... So that’s like an incrementality-first approach.” (07:54, A)
Applying the Incremental Framework in Retention & New Channels
- Retention Testing (10:01, A):
- Email/SMS programs often overstate ROI; need to implement user-level holdouts and measure true incremental impact.
- “Retention’s another tricky one because ... just because you sent an additional email or someone clicked an additional email still doesn’t mean it was necessarily an incremental outcome.” (12:20, B)
- Collaboration with Tech Partners:
- Example: Working with Post Pilot to extend test period for direct mail or SMS, ensuring measurement of true incremental impact (15:33, A).
- Expectation From Partners (17:34, C):
- “Any partner knows if you’re going to be asking me for money, prove it with incrementality. This is what we care about—very clear and firm.”
Structure & Sequence of Testing (+ Notable Quotes)
-
Baseline—Channel Holdouts:
- Test spend vs. no-spend to establish baseline incrementality.
- “Step one is just like channel level holdout, right? Hey, no spend versus spend. Is this channel incremental?” (58:01, A)
-
Scaling Up & Segment Testing:
- Run scaled-up tests (multiple spend levels) and test new objectives or creative strategies.
- “Now we’re starting to string together ... we tested view content last year for a long time, got a really good readout... now we’re leaning into other non-purchase conversion objectives.” (20:49, A)
-
Iterative, Hypothesis-Led Adjustments:
- Sequence tests back-to-back, analyze outputs, fine-tune strategies.
- “What I love about it is it’s so objective, causal, and simple. You get like two metrics, right?... But then it’s like, all right, you get a bad test, like why? Then you have to dive in, make decisions, follow up, test again.” (17:34, C)
- “Meta performance hasn’t been what we wanted. First I need to get a baseline, right? A lot of things have changed... so we ran another test and made a bunch of changes...” (22:17, C)
Embracing Directionality Over Perfection
- Not Every Test Needs a True Holdout (32:46, B):
- It’s often more practical to optimize for directionally improved outcomes, not academic perfection.
- “What matters to us is just improving the incremental ROAS. We’re not trying to get in a peer-reviewed journal.” (32:46, B)
- “If your KPI is so far from your target right now, you’ll know if this worked or not, even without the perfect data.” (32:47, C)
Optimization Examples, Channel Mix, and Team Enablement
-
Brand Video Integration:
- Ran true holdout tests adding brand video to purchase-optimized campaigns, saw 40% lift in incremental outcomes (55:11, B).
- “We saw a 40% lift. So it was 40% more incremental in the cell that had the brand videos, which is a massive win.” (55:11, B)
-
Testing at Scale (50:38, B):
- Ability to run more than one test a week is credited to having multiple lines of business and infrastructure for concurrent testing.
- “We ran 58 tests last year, which is like more than one a week... Ridge is four separate lines of business... so we can basically run concurrent tests.” (50:38, B)
-
Culture Building:
- Encourage team to make the case for incrementality.
- “Part of building that culture is just like the Socratic questioning of: ‘Prove this is incremental.’ And over time the team learns what is and isn’t a data point that proves incrementality.” (61:46, A)
Notable Quotes & Memorable Moments
- “What matters to us is just improving the incremental ROAS. We’re not trying to get in a peer-reviewed journal.” (32:46, B)
- On Experimentation Culture:
“Teams that ran a certain number of experiments per year had like 17% lower CAC... just an always-on program where you’re just constantly testing things.” (00:49, C) - On Brand Video Test:
“On the MTA data, we saw a 40% lift... 40% more incremental in the cell that had the brand videos.” (55:11, B) - Incrementality in Retention:
“You should be setting it up internally so that you can observe what is it doing in addition to what would have otherwise already happened.” (11:28, B) - Defining ‘Incremental’:
“That’s all incremental really means, right? It’s additional. You would not be getting that outcome... if you were not doing that tactic.” (00:42, A / 09:44, A)
Timestamps for Key Segments
- Introduction to Incrementality & Experiment Culture: 00:00–01:07
- Case Study—Testing TikTok Shop vs Shopify Audiences: 07:14–09:44
- Retention and User-Level Holdouts: 10:01–13:50
- Testing Structure & Iterative Process: 17:34–26:39
- How to Sequence Tests & Analyzing Results: 20:49–28:05
- Precision vs Directionality in Measurement: 31:53–35:05
- Brand Video Incrementality Test: 54:24–55:27
- Team Operationalization & Test Volume: 50:38–52:44
- Summarizing the Playbook: 58:01–61:46
The Incrementality Playbook in 3 Steps
-
Channel-Level Holdouts:
- “Hey, no spend vs spend. Is this channel incremental?”
-
Scale-Up Tests:
- “How much can we spend and have it still be incremental? Use scaled spend tests.”
-
Optimize the Channel:
- “Test channel-specific strategies: brand videos vs not, view content vs purchase conversion, etc. Stay in this optimization loop as the core long-term tactic.”
Final Thoughts
- Build Directional Conviction: Perfect data is rare; prioritize actionable direction over academic rigor.
- Cultivate a Culture of Experimentation: Reinforce a truth-seeking, hypothesis-driven environment—make “prove it’s incremental” the expectation.
- Leverage Tech & Vendor Relationships: Push partners to support real incrementality measurement.
- Incrementality Beyond Revenue: Use the incremental framework for everything—retention, brand awareness, even retail pilots.
Recommendation:
Implement these incremental testing principles not just as a measurement tactic, but as an organization-wide cultural shift. Let incremental thinking guide metric selection, creative strategy, and every marketing investment decision.
For more, listen to the full episode for tactical detail and additional case examples.
