DTC Podcast — Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue

Guest: Drew Marconi (Co-founder, Intelligems)
Host: Eric Dick (DTC Newsletter and Podcast)
Date: September 10, 2025

Overview

This episode dives deep into the state of A/B testing in DTC (Direct to Consumer) e-commerce with Drew Marconi of Intelligems. The conversation unpacks the critical role of price testing, why profit—not just revenue—should be the focus, the operational realities of building a consistent testing culture, and the fast-approaching future of agentic AI in CRO.

1. Drew Marconi’s Journey to Intelligems

[01:05–02:45]

Early Career: Drew began in consulting at McKinsey before joining a ride-sharing company, where he worked on dynamic pricing and promotions.
Genesis of Intelligems: He realized during early conversations with e-commerce leaders that few brands were A/B testing, especially in pricing and offers.
Company Evolution: Intelligems started with price testing, expanded to shipping, offers, content (CRO) testing, and now personalization and AI.

Quote:
"We realized that people were barely A/B testing anything, let alone dollars and cents of their site." — Drew [02:05]

2. The Core Problem Intelligems Solves

[02:45–03:46]

Brands invest heavily in driving traffic; Intelligems helps maximize profit and yield from that traffic.
They facilitate tests around pricing, personalization, offers, shipping, UX, and copy to iteratively improve profit per visitor.

Quote:
"We help you run tests that figure out what's going to drive more profit per visitor." — Drew [02:53]

3. The State of Pricing in DTC

[04:26–06:28]

Increased Openness: Over the last 4+ years, brands have become more willing to treat pricing as a business lever, driven by volatile margins and external pressures like tariffs.
Broadening Price Strategy: Drew urges brands to see pricing as more than list price—discounts, shipping, and return policies are vital components.
Shift to Analytics: Brands are moving away from “vibes” to a more data-driven, analytical approach.

Quote:
"Pricing needs to be broader than just the list price. ... Your discounts, shipping rates, return policy—they are all part of your pricing strategy.” — Drew [05:12]

4. The Fear and Reality of Dynamic Pricing

[06:12–08:48]

Brand Fears: Businesses worry changing prices will alienate customers or damage brand trust, referencing backlashes like Delta’s.
Practical Experience: Out of 600 million test subjects, Drew's team saw minimal issues from price testing.
Education: Dynamic pricing means more than changing the list price—dynamic offers, segmentation, and personalized shipping rates are elements most brands are already comfortable with at a micro level.

Quote:
"I feel like being a pricing expert is at times like being a therapist." — Drew [06:28]
"Test it and get data and understand how people will react ... Would you rather be smarter going into this risky territory or have less information?" — Drew [07:13]

5. Building a Testing Roadmap — A Practical Example

[08:48–12:21]

Start with Strategy: Identify the business goal—profit, growth, subscription launching, or prepping new products.
Key Levers to Test: Pricing, shipping, offer types/mechanics, offer messaging, and UX.
Structured Example: For a hypothetical “Eric’s Belts” store, start with a pricing straddle test (8% up/down) on a core line—then iterate based on measured profit per visitor.
Sequencing: Begin broad, iterate, and expand into catalog or offer-based tests as data emerges.

Quote:
"A good testing roadmap starts with a good strategy." — Drew [09:13]

6. Segmentation and Personalization in Testing

[13:14–15:23]

Start Broad, Then Segment: Run broad tests (e.g., across all paid traffic), analyze which sources/segments responded differently, and then deploy personalized offers accordingly.
Personalization Rules: Can range from simple content swaps (images, copy) to complex changes in offers or pricing tied to user segments.

Quote:
"We have a whole more targeting possibilities and rules than anyone could ever use. I like to recommend people start broad and look for subsegments where people behaved differently." — Drew [13:24]

7. The 3Es Framework: Explore, Experiment, Extend

[15:23–15:53]

Explore: Analyze your own data to generate hypotheses.
Experiment: Test broadly on segments.
Extend: Apply learnings to specific user groups who responded well to variants.

Quote:
"Don't just YOLO a test ‘cause you saw a screenshot on Twitter. Explore your own data, come up with hypotheses, experiment with it broadly and then extend your learnings.” — Drew [15:37]

8. Real-World Black Friday Testing Examples

[15:53–19:06]

Mechanism Testing: For holiday promos like Labor Day, brands test offer types—e.g., free gift vs. percentage-off—and track which yields higher AOV or profit.
- Example: A beverage company sees equal conversion with a free gift vs. 10% off, but free gift boosted AOV by $10 (profit per visitor rises).
Catalog Discount Strategy: An apparel brand tests discounts only on certain SKUs, adjusting aggressiveness in real time.
Emphasis on analyzing results mid-test and adapting in-flight.

9. Organizational Testing Cultures & Measurement

[19:06–22:08]

Leadership: High-performing brands empower a VP of E-Comm (or similar) to drive testing, with clear ownership and cross-functional participation.
Weekly Routines: Strong teams hold regular experimentation meetings—ideation, active test reviews, backlog grooming.
Unified Metric: “Profit per site visitor” aligns all functions, from UX to pricing, on bottom-line impact.

Quote:
"Everyone is held accountable to incrementality and looking at that same metric." — Drew [21:26]

10. Logistics of Running Effective Tests

[22:23–24:23]

No Dead Time: The best teams always have a test live; traffic is the “budget” for learning.
Multiple Parallel Tests: Larger sites can test in parallel via visitor splits.
Test Length: Minimum two weeks to account for sampling bias and day-of-week variations.
Planned Rollouts: Next test is built before the current finishes, minimizing downtime.

11. Statistical Significance and Test Power

[24:23–27:17]

Bayesian Approach: Intelligems uses Bayesian confidence to weigh test outcomes—look at probability of winning, but also the interval/range of improvement.
Beware Small Samples: Many are tempted to act on insufficient data; longer tests yield more reliable, actionable results.
Pre-test Planning: Agencies set expectations for duration/effect size using stat calculators.

Quote:
"There's a big difference between, 'I'm confident B is better', and 'I'm confident B is 10% better.'” — Drew [25:00]

12. Why Profit Per Visitor Beats Revenue Metrics

[27:17–29:29]

Move Beyond Conversion Rate: Optimize for gross profit per visitor (conversion rate × AOV × margin %), not just conversion or revenue.
Discount Trade-offs: A higher revenue per visitor can mask eroding margins due to deep discounts—focus on incremental gross profit per visitor.
Full-Funnel Testing: Data can account for COGS, fulfillment, and pricing strategy for a true bottom-line impact.

Quote:
"Conversion rate is a dumb metric. ... Pull in cogs, pull in cost to fulfill, and get to a sense of the gross profit you’re making on these orders.” — Drew [27:47]

13. The Future: Agentic AI and “Infinity Testing”

[29:29–33:03]

Short-term: AI agents take over repetitive testing cycles—suggesting, building, analyzing, and rolling out tests for speed and reduced workload.
Practical Launch: Intelligems’ new agent analyzes test results, suggests next steps, examines subsegments.
Wider Adoption: Lower effort means more brands (beyond 20% on Shopify Plus) will implement testing programs.
Long-term: Generative, continuous testing—AI-driven “infinity testing” constantly evolves site variants. Future vision includes personalized, generative experiences for each visitor, though DTC privacy/technology barriers remain.

Quote:
"Infinity testing is a term for it, where ... this section of the site is just constantly being optimized and rolling with the best version but there’s always new things being tried and monitored.” — Drew [31:50]

14. Pricing Meme “Galaxy Brain” and Volatility

[33:03–34:50]

Meme Breakdown: From basic “raise prices and hope” to smart, data-driven price tests that recover margin and maintain performance.
Current Climate: Old attitudes (“I’ll never touch my prices”) are disappearing as volatility affects every aspect of e-commerce.
Final Thought: True optimization is holistic—pricing, UX, offers, and internal silos must all be unified for maximum incremental profit.

Quote:
"Volatility is the new normal, right? ... It's not just one of the things." — Drew [34:10, 34:21]

Notable Quotes & Moments

A/B Testing as Therapy: "Being a pricing expert is at times like being a therapist." — Drew [06:28]
Profit is Everything: "Conversion rate is a dumb metric ... You multiply those three across [conversion, AOV, margin], you get to gross profit per visitor." — Drew [27:47]
Infinity Testing Vision: "Infinity testing ... where without lifting a finger, this page is just constantly being optimized." — Drew [31:50]
Agent Adoption: "Still only 20% of brands on Shopify Plus use a testing tool. ... If we can say, hey, agents are going to bring the effort very, very low, those people are going to start coming into the market." — Drew [30:31]

Episode Structure & Timestamps

[01:05] Drew’s professional journey and forming Intelligems
[02:45] Core business problem and expanding testable levers
[04:26] Trends in DTC pricing strategy
[06:12] Fear versus reality of price/dynamic testing
[08:48] Building a testing roadmap: step-by-step
[13:14] Segmentation, personalizations, traffic source analysis
[15:23] 3Es framework: Explore, Experiment, Extend
[15:53] Real-world Black Friday and catalog case studies
[19:06] Testing culture and ownership in organizations
[22:23] Minimum viable testing cadence, duration, and parallelization
[24:23] Statistical confidence, Bayesian intervals, test length logic
[27:17] Measuring for true profit, not just revenue or conversion
[29:29] Agentic AI: current state and future automation
[33:03] Meme breakdown, volatility, holistic optimization
[34:50] Final thoughts, where to find Drew, and show close

This episode is a must-listen for DTC operators wanting step-by-step frameworks, candid lessons from high-velocity testing teams, and a peek at the AI-driven future of profit optimization.

DTC Podcast — Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue

Guest: Drew Marconi (Co-founder, Intelligems)
Host: Eric Dick (DTC Newsletter and Podcast)
Date: September 10, 2025

Overview

1. Drew Marconi’s Journey to Intelligems

[01:05–02:45]

Early Career: Drew began in consulting at McKinsey before joining a ride-sharing company, where he worked on dynamic pricing and promotions.
Genesis of Intelligems: He realized during early conversations with e-commerce leaders that few brands were A/B testing, especially in pricing and offers.
Company Evolution: Intelligems started with price testing, expanded to shipping, offers, content (CRO) testing, and now personalization and AI.

Quote:
"We realized that people were barely A/B testing anything, let alone dollars and cents of their site." — Drew [02:05]

2. The Core Problem Intelligems Solves

[02:45–03:46]

Brands invest heavily in driving traffic; Intelligems helps maximize profit and yield from that traffic.
They facilitate tests around pricing, personalization, offers, shipping, UX, and copy to iteratively improve profit per visitor.

Quote:
"We help you run tests that figure out what's going to drive more profit per visitor." — Drew [02:53]

3. The State of Pricing in DTC

[04:26–06:28]

Increased Openness: Over the last 4+ years, brands have become more willing to treat pricing as a business lever, driven by volatile margins and external pressures like tariffs.
Broadening Price Strategy: Drew urges brands to see pricing as more than list price—discounts, shipping, and return policies are vital components.
Shift to Analytics: Brands are moving away from “vibes” to a more data-driven, analytical approach.

Quote:
"Pricing needs to be broader than just the list price. ... Your discounts, shipping rates, return policy—they are all part of your pricing strategy.” — Drew [05:12]

4. The Fear and Reality of Dynamic Pricing

[06:12–08:48]

Brand Fears: Businesses worry changing prices will alienate customers or damage brand trust, referencing backlashes like Delta’s.
Practical Experience: Out of 600 million test subjects, Drew's team saw minimal issues from price testing.
Education: Dynamic pricing means more than changing the list price—dynamic offers, segmentation, and personalized shipping rates are elements most brands are already comfortable with at a micro level.

5. Building a Testing Roadmap — A Practical Example

[08:48–12:21]

Start with Strategy: Identify the business goal—profit, growth, subscription launching, or prepping new products.
Key Levers to Test: Pricing, shipping, offer types/mechanics, offer messaging, and UX.
Structured Example: For a hypothetical “Eric’s Belts” store, start with a pricing straddle test (8% up/down) on a core line—then iterate based on measured profit per visitor.
Sequencing: Begin broad, iterate, and expand into catalog or offer-based tests as data emerges.

Quote:
"A good testing roadmap starts with a good strategy." — Drew [09:13]

6. Segmentation and Personalization in Testing

[13:14–15:23]

Start Broad, Then Segment: Run broad tests (e.g., across all paid traffic), analyze which sources/segments responded differently, and then deploy personalized offers accordingly.
Personalization Rules: Can range from simple content swaps (images, copy) to complex changes in offers or pricing tied to user segments.

7. The 3Es Framework: Explore, Experiment, Extend

[15:23–15:53]

Explore: Analyze your own data to generate hypotheses.
Experiment: Test broadly on segments.
Extend: Apply learnings to specific user groups who responded well to variants.

8. Real-World Black Friday Testing Examples

[15:53–19:06]

Mechanism Testing: For holiday promos like Labor Day, brands test offer types—e.g., free gift vs. percentage-off—and track which yields higher AOV or profit.
- Example: A beverage company sees equal conversion with a free gift vs. 10% off, but free gift boosted AOV by $10 (profit per visitor rises).
Catalog Discount Strategy: An apparel brand tests discounts only on certain SKUs, adjusting aggressiveness in real time.
Emphasis on analyzing results mid-test and adapting in-flight.

9. Organizational Testing Cultures & Measurement

[19:06–22:08]

Leadership: High-performing brands empower a VP of E-Comm (or similar) to drive testing, with clear ownership and cross-functional participation.
Weekly Routines: Strong teams hold regular experimentation meetings—ideation, active test reviews, backlog grooming.
Unified Metric: “Profit per site visitor” aligns all functions, from UX to pricing, on bottom-line impact.

Quote:
"Everyone is held accountable to incrementality and looking at that same metric." — Drew [21:26]

10. Logistics of Running Effective Tests

[22:23–24:23]

No Dead Time: The best teams always have a test live; traffic is the “budget” for learning.
Multiple Parallel Tests: Larger sites can test in parallel via visitor splits.
Test Length: Minimum two weeks to account for sampling bias and day-of-week variations.
Planned Rollouts: Next test is built before the current finishes, minimizing downtime.

11. Statistical Significance and Test Power

[24:23–27:17]

Bayesian Approach: Intelligems uses Bayesian confidence to weigh test outcomes—look at probability of winning, but also the interval/range of improvement.
Beware Small Samples: Many are tempted to act on insufficient data; longer tests yield more reliable, actionable results.
Pre-test Planning: Agencies set expectations for duration/effect size using stat calculators.

Quote:
"There's a big difference between, 'I'm confident B is better', and 'I'm confident B is 10% better.'” — Drew [25:00]

12. Why Profit Per Visitor Beats Revenue Metrics

[27:17–29:29]

Move Beyond Conversion Rate: Optimize for gross profit per visitor (conversion rate × AOV × margin %), not just conversion or revenue.
Discount Trade-offs: A higher revenue per visitor can mask eroding margins due to deep discounts—focus on incremental gross profit per visitor.
Full-Funnel Testing: Data can account for COGS, fulfillment, and pricing strategy for a true bottom-line impact.

Quote:
"Conversion rate is a dumb metric. ... Pull in cogs, pull in cost to fulfill, and get to a sense of the gross profit you’re making on these orders.” — Drew [27:47]

13. The Future: Agentic AI and “Infinity Testing”

[29:29–33:03]

Short-term: AI agents take over repetitive testing cycles—suggesting, building, analyzing, and rolling out tests for speed and reduced workload.
Practical Launch: Intelligems’ new agent analyzes test results, suggests next steps, examines subsegments.
Wider Adoption: Lower effort means more brands (beyond 20% on Shopify Plus) will implement testing programs.
Long-term: Generative, continuous testing—AI-driven “infinity testing” constantly evolves site variants. Future vision includes personalized, generative experiences for each visitor, though DTC privacy/technology barriers remain.

14. Pricing Meme “Galaxy Brain” and Volatility

[33:03–34:50]

Meme Breakdown: From basic “raise prices and hope” to smart, data-driven price tests that recover margin and maintain performance.
Current Climate: Old attitudes (“I’ll never touch my prices”) are disappearing as volatility affects every aspect of e-commerce.
Final Thought: True optimization is holistic—pricing, UX, offers, and internal silos must all be unified for maximum incremental profit.

Quote:
"Volatility is the new normal, right? ... It's not just one of the things." — Drew [34:10, 34:21]

Notable Quotes & Moments

A/B Testing as Therapy: "Being a pricing expert is at times like being a therapist." — Drew [06:28]
Profit is Everything: "Conversion rate is a dumb metric ... You multiply those three across [conversion, AOV, margin], you get to gross profit per visitor." — Drew [27:47]
Infinity Testing Vision: "Infinity testing ... where without lifting a finger, this page is just constantly being optimized." — Drew [31:50]
Agent Adoption: "Still only 20% of brands on Shopify Plus use a testing tool. ... If we can say, hey, agents are going to bring the effort very, very low, those people are going to start coming into the market." — Drew [30:31]

Episode Structure & Timestamps

[01:05] Drew’s professional journey and forming Intelligems
[02:45] Core business problem and expanding testable levers
[04:26] Trends in DTC pricing strategy
[06:12] Fear versus reality of price/dynamic testing
[08:48] Building a testing roadmap: step-by-step
[13:14] Segmentation, personalizations, traffic source analysis
[15:23] 3Es framework: Explore, Experiment, Extend
[15:53] Real-world Black Friday and catalog case studies
[19:06] Testing culture and ownership in organizations
[22:23] Minimum viable testing cadence, duration, and parallelization
[24:23] Statistical confidence, Bayesian intervals, test length logic
[27:17] Measuring for true profit, not just revenue or conversion
[29:29] Agentic AI: current state and future automation
[33:03] Meme breakdown, volatility, holistic optimization
[34:50] Final thoughts, where to find Drew, and show close

This episode is a must-listen for DTC operators wanting step-by-step frameworks, candid lessons from high-velocity testing teams, and a peek at the AI-driven future of profit optimization.

Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue with Drew Marconi

Powered by Wave AI

Summary

DTC Podcast — Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue

Overview

1. Drew Marconi’s Journey to Intelligems

2. The Core Problem Intelligems Solves

3. The State of Pricing in DTC

4. The Fear and Reality of Dynamic Pricing

5. Building a Testing Roadmap — A Practical Example

6. Segmentation and Personalization in Testing

7. The 3Es Framework: Explore, Experiment, Extend

8. Real-World Black Friday Testing Examples

9. Organizational Testing Cultures & Measurement

10. Logistics of Running Effective Tests

11. Statistical Significance and Test Power

12. Why Profit Per Visitor Beats Revenue Metrics

13. The Future: Agentic AI and “Infinity Testing”

14. Pricing Meme “Galaxy Brain” and Volatility

Notable Quotes & Moments

Episode Structure & Timestamps

Summary

DTC Podcast — Bonus: How to Build a Testing Roadmap That Drives Real Profit—Not Just Revenue

Overview

1. Drew Marconi’s Journey to Intelligems

2. The Core Problem Intelligems Solves

3. The State of Pricing in DTC

4. The Fear and Reality of Dynamic Pricing

5. Building a Testing Roadmap — A Practical Example

6. Segmentation and Personalization in Testing

7. The 3Es Framework: Explore, Experiment, Extend

8. Real-World Black Friday Testing Examples

9. Organizational Testing Cultures & Measurement

10. Logistics of Running Effective Tests

11. Statistical Significance and Test Power

12. Why Profit Per Visitor Beats Revenue Metrics

13. The Future: Agentic AI and “Infinity Testing”

14. Pricing Meme “Galaxy Brain” and Volatility

Notable Quotes & Moments

Episode Structure & Timestamps