Limited Supply: S9 E9 – Incrementality, Data, the Funnel, Oh My! (with Olivia Kory of Haus)
Host: Nik Sharma
Guest: Olivia Kory, Haus
Release Date: September 18, 2024

Introduction to Incrementality Testing

In this episode, host Nik Sharma delves deep into the nuances of incrementality testing in the Direct-to-Consumer (DTC) marketing landscape, joined by Olivia Kory from Haus. The conversation primarily revolves around the shift from traditional attribution models to more sophisticated incrementality testing to maximize growth and marketing efficiency.

Olivia Kory opens the discussion by likening incrementality testing to randomized control trials in healthcare:

“Think of randomized control trials in healthcare… that's incrementality testing. We have a counterfactual to understand what would have happened anyway in the absence of this intervention.”
[00:00]

Understanding Incrementality vs. Attribution Models

Olivia provides a comprehensive explanation of incrementality testing compared to Multi-Touch Attribution (MTA) and Marketing Mix Modeling (MMM).

• Incrementality Testing: Establishes a causal relationship by comparing a treatment group with a control group to determine the true impact of a marketing intervention.

• MTA and MMM: Primarily correlational, analyzing patterns and relationships without establishing causation.

Olivia emphasizes the importance of creating a counterfactual scenario to measure what would have occurred without the marketing effort:

“MTA and MMM, they are more correlational, but they’re not causal. With incrementality, we’re looking for what’s actually driving the business.”
[07:20]

Olivia Kory’s Journey in Paid Media Marketing

Olivia shares her extensive background in paid media marketing, highlighting her experiences at:

• Ad Agency in Chicago: Pioneered incrementality testing with Allstate around 2012.
• Netflix (2015-2016): Ran global incrementality and NGO experiments, significantly reducing acquisition costs by up to 90%.
• Quibi (2020): Witnessed the challenges of making media incremental in a low organic demand environment.
• Sonos: Transitioned to Haus to enhance marketing strategies beyond last-click attribution.

The Role of Haus in Modern Marketing

Nik introduces Haus as a pivotal platform for conducting incrementality testing, enabling brands to:

• Configure regional test and control experiments.
• Measure incrementality across both online and offline channels.
• Optimize marketing budgets by identifying points of diminishing returns.

Olivia supports this by detailing how Haus's geotesting methodology offers consistency and precision:

“With geotesting, you’re running the exact same methodology standardized across every channel you test… it’s resilient to all the privacy changes.”
[13:07]

Types of Incrementality Tests

Olivia breaks down the three primary types of incrementality tests:

1. Geographical (GEO) Experiments:

   • Advantages: Privacy-compliant, standardized across channels, and resilient to data privacy changes.
   • Challenges: Historically noisy data, requiring advanced methodologies like synthetic control to enhance precision.

   “All three cells are statistically indistinguishable from one another. So, the behavior of the regions in cell one are exactly the same as cell two and cell three.”
   [43:48]

2. User-Level Studies:

   • Conducted by platforms like Facebook and Google through their conversion lift tools.
   • Limitations: Primarily effective for brands heavily invested in these platforms and struggle when expanding to multiple channels.

   “Once you start expanding outside of those two channels… that’s when you go up funnel, you start spending in more channels… you really need geotesting.”
   [17:53]

3. Natural Experiments (On/Off Testing):

   • Involves turning a marketing tactic on or off and observing the resultant impact.
   • Drawbacks: Less precise due to the absence of a true counterfactual.

   “If you want to just do that burst method where you go out with a bunch of different types of branded content… we can run a time series test on off and see what’s happening.”
   [41:17]

Practical Applications and Insights from Experiments

Olivia shares real-world applications and learnings from her tenure at various companies:

• Netflix: Discovered that channels like brand search and affiliate contributed minimal incremental lift despite appearing strong in last-click attribution.

• Sonos: Identified that while certain channels didn't drive direct incremental revenue, they shifted customers from retail to DTC, enhancing overall profit margins.

  “Even though on last click attribution, it looks like your best channel, incrementality tests showed no impact.”
  [09:17]

• Q4 Case Study: Analyzed immediate versus delayed lift across various campaigns, finding that delayed lift often exceeded immediate lift, especially for higher price-point products.

  “Delayed lift exceeded immediate lift in 73% of the experiments we ran in Q4.”
  [26:03]

• Video Campaigns: Discovered that video campaigns like YouTube and OTT drove significantly more delayed lift compared to search-based campaigns.

  “Video campaigns drove 286% more delayed lift than search video view campaigns.”
  [26:54]

Diminishing Returns and Marginal Efficiency

A key highlight of the episode is the exploration of diminishing returns in marketing spend. Olivia explains how understanding the marginal cost per incremental acquisition (CPIA) can guide optimal budget allocation.

“Diminishing marginal returns help you understand where you are on that curve before you start increasing the cost, before you are no longer growing at the clip that is efficient for you.”
[41:49]

She illustrates this with an experiment design involving varying spend levels across different regions to identify the optimal spending point:

“We ran low, medium, and high spend pressure in different cells and found that between medium and high, we were fully maxed out.”
[42:00]

The Future of Brand Marketing

Olivia posits that the future of brand marketing lies in integrating incrementality insights to drive sales effectively. She challenges the traditional notion that brand marketing must rely solely on big agencies and expansive campaigns, advocating for leveraging platforms like TikTok, Meta, and YouTube for scalable brand building.

“People are really only buying brand marketing insofar as it drives sales… the future is going to be observing maybe shallower KPIs like site visits in real time, but running longer experiments with that post-treatment window to understand ultimately how those leading indicators are driving sales.”
[30:25]

Integrating Brand and Performance Marketing

Nik and Olivia discuss the symbiotic relationship between brand and performance marketing. Olivia underscores the necessity of building brand equity to support scalable performance marketing efforts, referencing lessons learned from Quibi and other ventures.

“At Quibi, we could have the best performance marketing engine in the world, but if there’s no brand and there’s no brand equity, you’re going to hit that wall in terms of rising costs.”
[33:28]

Innovative Approaches to Organic and Influencer Marketing

The conversation also touches upon testing organic social strategies and influencer content. Olivia shares insights on handling organic bursts and seeding creator content, emphasizing the need for flexible testing methodologies when dealing with non-traditional marketing channels.

“If you want to run a burst where you post a bunch of UGC content over a few weeks, we can observe and run a time series test on off and see what’s happening.”
[39:55]

Conclusion and Final Thoughts

As the episode winds down, Olivia reiterates the importance of delayed lift and encourages listeners to adopt incrementality testing to uncover hidden efficiencies and opportunities within their marketing strategies.

“Make sure you’re looking at delayed lift.”
[44:21]

Nik wraps up by highlighting the critical takeaways about moving beyond traditional attribution models to embrace a more scientifically rigorous approach to marketing measurement.

Key Takeaways

• Incrementality Testing: Offers a causal understanding of marketing impact, surpassing the limitations of MTA and MMM.
• Geotesting with Haus: Provides a standardized, privacy-compliant method to measure incrementality across diverse channels.
• Delayed Lift: Essential for capturing the full impact of higher-tier marketing channels like video and OTT.
• Diminishing Returns: Critical to identify the optimal spending point to avoid inefficiencies in budget allocation.
• Integrated Marketing Strategy: Combining brand building with performance marketing is pivotal for scalable and sustainable growth.

This episode of Limited Supply serves as an invaluable resource for DTC brands aiming to refine their marketing strategies through advanced incrementality testing, ensuring every marketing dollar is effectively driving growth and enhancing profitability.