How Incrementality Cuts Waste + What and Where to Test in Ecommerce

Summary

OPERATORS Podcast Episode Summary

Episode Title: How Ridge Uses Incrementality to Cut Waste and Scale What Works
Date: June 3, 2026

---

Episode Overview

This episode of OPERATORS dives deep into the world of incrementality, multi-touch attribution (MTA), and measurement strategies for eCommerce brands. The hosts demystify how the top digital brands are leveraging these tools to cut ad spend waste, drive scale, and gain real insight into what’s actually working. Through a mix of expertise, candid war stories, and data-driven recommendations, the Operators and guests (notably, Austin from North Beam) share what separates emerging brands from industry leaders—especially as platforms, channels, and measurement complexity grow.

---

Key Themes & Takeaways

1. What is Incrementality, and Why Does it Matter?

• Incrementality measures the true impact of a marketing channel or tactic— isolating what actually drives net new revenue, versus what would have happened anyway.
• It’s one of the three pillars of measurement, alongside MTA and Media Mix Modeling (MMM).
• Used properly, incrementality helps brands avoid wasting budget on channels that don’t move the needle.

Quote:
"Incrementality is a great way to... get to the heart of what's actually working. And hopefully lets you get back to the basics of your business, right? Like, you want to be doing the smallest amount of things possible, to do the most amount of money possible."
— Sean, 41:57

2. Who Should Use Incrementality & MTA—And When?

• Small $2M brands can operate on intuition, with basic platform data.
• MTA becomes essential as you diversify into multiple channels.
• Incrementality testing is critical when your spend is large, your media mix is complex (e.g., you’re omnichannel, selling on DTC, Amazon, and retail), or you need to validate new channels before scaling.
• The analogy: Swim trunks for small brands, snorkel (MTA) for growth, and full scuba gear (incrementality) for big, deep-water operators.

Quote:
"You never want to see a guy in the kiddie pool with a full scuba set... And you don't want to see a guy in the middle of the ocean with floaties on."
— Sean, 21:45

3. Best Practices, Pitfalls & War Stories

• Over-testing, bad holdouts, and poor data hygiene can lead to bad—or outright disastrous—decisions. Automation can help, but human oversight is critical.
• Holdout tests must be well-designed; too large or too long leaves money on the table, too small lacks statistical power.
• Changes to your stack, like tracking or site speed, can badly pollute results.
• The team candidly shares stories of both massive wins and failures in experimenting with incrementality across channels and product launches.

Quote:
"There’s some real dangers with incrementality, where you run a test, you think a channel was incremental, and then the MTA data showed it wasn’t... They just blew hundreds of thousands of dollars."
— Austin (North Beam), 08:42

4. How Ridge Uses Incrementality

• Ridge leverages incrementality at the product, channel, and sales channel levels.
• Example: YouTube view campaigns may look weak in MTA, but incrementality testing shows a huge uplift in Amazon sales, justifying further investment.
• They run back-to-back, short-duration tests; always something is being tested. MTA and incrementality together give a constant flow of actionable insights.

Quote:
"It's a culture of testing. Internally at Ridge, there's like a line of 10 people who want something to, like, they all want to test something and they have to get in line. It's like, when do I get to run my test?"
— Sean, 27:49

5. Modern Tech & Automation in Attribution

• North Beam automates the design, implementation, and monitoring of incrementality tests, minimizing human error and reducing labor costs.
• The goal: Give actionable, statistically valid insights without requiring large teams or manual oversight.
• Automated monitoring can stop tests early if data turns insignificant, protecting budgets.

Quote:
"We automate the process of figuring out when and where you should run the test."
— Austin, 09:41

6. Mixing Measurement Methods: MTA, Incrementality, MMM

• Each has strengths:
  • MTA for granular, digital trackable journeys
  • Incrementality for true lift/causality (especially for paid/controlled channels)
  • MMM for large, harder-to-measure initiatives (e.g., podcasts, TV, OOH).
• All work better together as spend and complexity grow.

Quote:
"That's why we have all three. MMM gets really interesting when you're doing a lot of stuff you can't measure with a holdout test."
— Austin, 17:11

---

Notable Quotes & Memorable Moments

• On When to Implement Advanced Measurement:
  "If you're going to spend more, you've got to calibrate it." — Austin, 02:08
• Testing Culture:
  "Just because it's working for me doesn't mean it's going to work for you. That's why you have to test it, right?" — Sean, 25:28
• On Marketers and Shiny Objects:
  "Most marketers want every single possible tool on earth … but you should basically be forced into them." — Sean, 34:31
• War Story:
  "We have to turn through these things. We bought 40,000 Lavender Ridge wallets because the first month we sold 4,000. … Then we got 40,000, and I'm still trying to sell them today." — Sean, 28:44

---

Timestamps for Key Segments

• 00:09-02:08 – Introduction to the realities and misconceptions of running longer incrementality tests; why context and statistical rigor matter.
• 03:54-05:29 – Early-stage brands and what measurement tools (if any) to use.
• 06:10-07:43 – Analogy of measurement tools and brand journey; Ridge’s current playbook on incrementality.
• 08:42-09:41 – Dangers of blindly trusting incrementality and over-testing.
• 10:27-12:01 – Real-world example: Poor test design and lessons learned.
• 14:20-16:45 – Step-by-step: When and how a $5M brand should approach MTA and incrementality as it grows.
• 20:11-21:45 – Opportunity costs, holdout sizes, and best practices for test design.
• 25:27-27:11 – Ridge’s approach to testing cadence, duration, and test types.
• 28:34-29:40 – When results change over time and learning from early “successes” that fizzle.
• 34:31-36:46 – Advice for marketers: Beware tools for tools’ sake; let necessity dictate.
• 39:24-40:50 – Realities of very slow purchase/conversion cycles and the challenge of attribution.
• 46:25-47:15 – The future: Automation, AI, and minimizing labor/mistakes in testing.
• 53:22-54:34 – Tactical test roadmaps for brands: Meta, YouTube, TikTok, Amazon, Pinterest.
• 58:01-61:33 – Industry trends: Creative volume, competition, impact of AI on creative, CTV’s resurgence.
• 64:06-67:59 – The overlooked opportunity in email, SMS, and push notification strategy for retention.
• 68:31-71:22 – What’s working right now at Ridge: going up-funnel, holdout tests, podcasts, Twitter, and rapid creative iteration.

---

Actionable Advice for Operators

• Start Simple: Only layer on MTA and incrementality as your marketing complexity and spend justify it.
• Always Be Testing: Adopt a culture of constant, focused experimentation—don’t over-test, but never stop validating assumptions.
• Combine Measurement Methods: Use MTA, incrementality, and MMM in concert; let each answer what it’s best suited for.
• Automate Wisely: Leverage tools that automate the setup, monitoring, and analysis of testing to save money and prevent human error.
• Beware of Shiny Objects: Tools are only beneficial when the business context and scale demand them—avoid wasting resources.
• Don't Trust Universal Playbooks: What works for Ridge may not work for you; always validate through your own data and tests.

---

Final Takeaways

Incrementality isn’t a silver bullet, but when deployed at the right stage of a brand’s journey and paired with robust MTA and MMM, it allows digital operators to strip away noise, make rapid, reliable decisions, and unlock major growth while maintaining efficiency. But as always, adaptation, skepticism, and a relentless focus on actual business outcomes—not just new tools—separates the legends from the hype.

---

For Further Reference

• Ridge and Caden Lane’s real-world examples illustrate the power—and the dangers—of incrementality and advanced measurement.
• North Beam’s automation-first approach points to the future of testing and insight generation for digital commerce.

---

Essential Quote for the Road:
"You want the leanest team to do the most amount of revenue. Everything comes down to just being like, how flexible can you be as an organization? And incrementality is a great tool to do that."
— Sean, 41:57

Transcript

[A]

We're here with North Beam, one of our favorite sponsors and we're talking incrementality. And Austin promises it's not going to be boring. So Austin, what's the, what's the spicy take, my brother?

[B]

Spicy take is a lot of people have been saying recently on, on social, hey, you need to run longer tests, right? You need to run longer tests to get, to get signal. And that's honestly a nonsensical thing to say. Maybe you need to run a longer test, maybe you can run a shorter test. And that's why North Beam is different, right? Because you know, we've worked with both of you for years on, on mta, right? Multi touch attribution on hourly transactional data. And so yeah, maybe one channel, you need to run a three month test. Maybe one a three week, maybe one a six week. Right. But just blanketly say which people are saying, oh, you need to run longer tests. That's just not, that's not a smart thing to say.

[C]

So you don't have to run it long any. Isn't it have to do with like statistical difference? So it's kind of like the volume that you have.

[B]

It's conversion lag sometimes, right? It's conversion lag. It could be statistical volume, it could be a multitude of things. Right. But because Nor Beam has their MTA data, we can not only do a better holdout that's more precise because, you know, people in the valley where Sean is may purchase slightly differently on an MTA hourly basis than they do in like downtown la, right, where Jason and Hexclad folks are. So we could do a better holdout, statistically speaking, but also understand the right lag. So should it be three months, six, you know, four months, two weeks? Six weeks. So people are just saying the wrong things.

[A]

Austin Habanero Harrison coming in with a.

[B]

Coming in hot, saying people are saying the wrong stuff, wrong thinking.

[A]

Yeah, I think, I think for this episode what we should do is put yourself in the shoes of the average listener. Cody is a guy I know who listens to the show. Not. Not Cody from Jones would be a different Cody doing 5 million, 10 million, $20 million a year. It's an E Commerce brand. You know, they're just starting to figure some stuff out. First, what the hell are we measuring? Why is incremental, incrementality important? Yeah, like let's go like bare bones day one. What are we talking about here?

[B]

Yeah, so when you think about multi touch attribution or just basic analytics, right. We've worked with you all for years. But when I think about people that signed us, sign, signed up for us early, like Groons or Comfort, you know, we all know those, you know, some of the fastest growing companies in world in the history of, you know, our, our industry, they signed up early because they wanted to understand every hour, you know, how are things performing, right? Whether it's their influencers, whether it's, you know, Meta, Snapchat, TikTok, whatever it is. So that's like, did someone click and buy something? Did it take them 60 days, did it take them a week? Nor Beam launched our clicks and deterministic views. We're splitting credit on a view basis. So Katie views an ad on TikTok, view something on Snap, views something on Meta, and buys a widget. We divide it a third, a third, a third. So that's mta, right? And for the listeners out there, when do you need incrementality? Well, you know, I talked to Connor about this, the CMO of the Ridge, about calibrating your MTA number. So if you're looking at like, hey, on Meta and North Beam, I'm looking at like a, you know, $50 cac. But you, you think maybe people converted retail or on Amazon, then you want to calibrate that number to say, oh, well, it's a $50 CAC. But if I include Amazon or maybe conversions, I don't see, maybe it's a $40 cac, right? So it's that calibration layer. So I think when you're small, Sean, you talk about it like, you know, you can keep it pretty simple. Like you look at your MTA numbers, you see your bank account moving, you try to see what, what's what. But eventually when you're omnichannel, right? So you're in retail, you're on Amazon, you're really diversifying your, your revenue where, where it lands. You need to understand your spend better, right? Cause if you're going to spend more, you've got to calibrate it.

[A]

Yeah. And let's talk about those early days. You're a $5 million brand. Listen to this. And you're just running Facebook ads, right? There's, you're probably just running conversion optimized Facebook campaigns, right? Asc, bid cap, some like that, right? You're doing like the most basic, basic stuff possible. You don't need incrementality and you do not need MTA tools, right? Austin. Hold on, hold on.

[B]

So hold on, hold on, hold on. All right, so it's funny, I don't think you need incrementality at that point. I don't. I agree with you. But I remember when like Grun set up early for North Beam and same with comfort. It's like, I think it depends on the scan a that you know how fast you're going to. Right. Because if you don't have good measurement, you're going really fast, you can go off the rails. The other, the other thing where you could use it is nor Beam has a longer window of attribution. Right. So if you're like you said, you're a $5 million brand, you're small, you know, do you have to have mta?

[A]

No.

[B]

Because you can look at your bank account, look at your spend and you know, it's a simple mix. But if you have a product that's expensive, like maybe you know, a 2, 3, 4 or $500 product and your conversion lags really long, you may spend on Meta and in Meta platform you've got a one day view and a seven day click. But what if everyone's buying after 45 days or 60 days or 90 days? You won't see it in platform, right. So in North Beam, if you did sign up for it again, you know, I'm not saying you have to have to, but there are circumstances where you want to understand, do I have lag? Like are people actually buying?

[C]

I think, Austin, it can also be too like early on. I remember the first incrementality test we ever did was branded search with Google, right?

[B]

Yeah, yeah.

[C]

So there's, but I'm with you, Sean. Like if it's omni, I mean Austin and I have talked about this before. Like if you're not Omnichannel and you are driving all your revenue through Shopify, an MTA is of course very valuable. But where does incrementality fall into that? And because if I can, if I can increase my ad spend on Meta, I see the revenue jump up or vice versa. Like when, when do you know is it when you have multiple, like when you're advertising on TikTok and Pinterest and Google and all the things.

[A]

What I think is, I'm going to give you guys an analogy for this is, you know, we're out in the ocean, we're, we're on a little journey, right? And so like the more tools you layer on, it's like you, you're getting a snorkel and then you're getting a scuba suit, right? So like when you're, when you're doing $2 million a year, you can just be out there in your swim trunks. You're not in very deep water, right? And then when you're, when you're doing $10 million a year, you probably need MTA, which is going to be your snorkel. And then when you're getting out there to 50, $100 million a year, right, you're in deep water, you're in uncharted territory. Then you bring in incrementality and everything else, right? Because it's. Yeah, it's just. It's just, it's giving you clarity into harder and harder situations. And we're going to talk about incrementality. It's the point of today's episode. I'll tell you how Ridge is using it, right? We use it on the product level, we use it on the channel level, we use it on the sales channel level. So, like right now we're spending way more money on YouTube, right? We're spending money on YouTube, on view campaigns for tech products. And you know, the MCA data does not look great on that, but the incrementality data looks fant. Looks fantastic. People are buying that stuff on Amazon at 2x the rate of everything else. And on my Amazon business, posted a screenshot, I'm up 2x200% year over year. And it's because of this triangle that we're pushing out, right? It's like understanding, like getting an incrementality holdout, letting us lean harder into this new channel, picking up the sales other places. And that's really who this is for. Like, once you start experimenting with that deeper water, right? This is when you have to start bringing in incrementality.

[B]

No, I agree with that. Also, when your spend's higher, the. The risk is higher that you're making a mistake, right? You know, for you, I know, Sean, you guys really will, you know, accelerate spend dramatically. So that's where, you know, if you don't have good measurement, you get it's. It could be disastrous for sure.

[A]

And you know, Katie brought up Google search. You should test everything in your business, right, either on an MTA level or in criminality level as you start scaling it up because you're overspending on branded search. Like, you know, Google is the most valuable company on earth on any given day. They have a $200 billion ad empire. They don't get that by saving you money. You know what I mean?

[C]

And it's not going to be like too Sean, I think it's not running incrementality test is not going to unlock some massive, like low hanging fruit in ad spin. Like, I think of it as more as a hygiene test on the channels I'm already spending on. So. Right. To make sure that I'm not spending where I don't need to.

[B]

Yeah. There's some real dangers with incrementality where you run a test. I've seen this with a customer where they ran a test, they said a channel was incremental, and then the next day or a few days later, the MTA data showed it, it was not doing well, it tanked. And the customer like, no, no, no. We ran a test like, the MTA data is wrong. Lo and behold, they burned a bunch of money two months later. And I was like, we were like, we told you your MTA data nosedive the next day. Right. Dramatically. And they just blew hundreds of thousands of dollars. Right. So incrementality is good, but you also have to look at your MTA data. Right. It's kind of a dance back and forth. And then to call out Jeremy at Kit. She made a really good point. Is a lot of people over test too? Right? Like at some point you've, you've calibrated your MTA numbers, you kind of have a sense for what's going on. You don't need to run a million tests. Right. So I think it's good to test, but also not once you kind of have locked in where you think a channel's performing, you don't need to test it.

[C]

Do you think people should kind of like test maybe once a quarter or like, what, what do you recommend?

[B]

I mean, it depends. Like, you're launching a new channel, you may need to test more often or you change your ta. Like Connor and I talk about this. Connor, the CMO of the Ridge. Like, you change your tactic. Like you go like Sean is saying, you go from like conversion to awareness or something like that. So it depends on what's happening in your business and how volatile the changes in your channel mix your tactics creative and then you, you know, or, or revenue mix. Right? You know, you go from like only selling D2C. Now you launch Amazon. It cannibalizes your, your, your D2C. Right? So it's like, it's based on all the. And that's a north beam. Another plug. Can't help, but we automate the process of figuring out when and where you should run the test.

[C]

Sean, I feel like Austin setting me up to tell you how I completely up my incrementality test this last month.

[A]

I would love to Hear it.

[C]

I learned first. Okay, first of all, we ran our first incrementality test with north theme. And honestly, it was great. Like, let me tell you what happened wrong and then I'll tell you what was great about it. But what Austin said is so true. So we just, we were so excited. Like, we turned it on, ran it through meta, like went through the whole thing. But in the meantime we had tried. We decided to turn on this new unnamed company that was basically optimizing for site speed. We made UTM parameter changes within our meta count. We did five major things that basically completely screwed up our signal with meta and then ran an incrementality test in the middle of it. And so for anybody thinking about doing incrementality tests, I do think also you need to sit down with your CRO team and kind of be like, okay, we're going to have to reduce the amount of tests we're running right now. Right? Like you kind of need that control group and take out all of the, like, I don't know, Austin. Right. I don't. You weren't sitting on our last call when we went over it. But basically because we had so many signal issues in April, it really caused issues even with our incrementality like results. And we'll run it again because we're certain that other aspects screwed it up. But to your point, Austin, I think it's just so important you do. I'm not the only person that completely screwed it up.

[B]

Now I see it a lot with other vendors where they don't catch it, don't notice it, and then assume a channel is either incremental or not. But the test was flawed. So that's why we try to monitor it really carefully during it.

[C]

But I think that it wasn't the test that was flawed. It was the audience that we had running right into Caden Lane at the time. So it could be a poor. Right? So like, yes, if that's the traffic that we were feeding into our site because our signal was poor, well, then there's no incrementality right. In whatever channels we were testing. But if we had had better signal, then I don't know, it's just, I think there's a lot of ways you can look at it and it, it to me really is like a hygiene test of like, are things running the way that you think they're running and or is in platform? It's like if my kids tell me that they clean their room, right? And. And then like, only like they both say they clean their room. But really one of them probably is just like taking credit for it and didn't really do it. Like, that's attribution and incrementality goes and proves it wrong or right. Right, Austin?

[B]

I think so. I mean, I think we see a lot of data in mta, so I think we have a good sense of like, what a good number is, you know?

[C]

Well, you're MTA. I'm talking about like in platform. Like, what if TikTok is saying, yeah,

[B]

yeah, yeah, like what, the platforms? Yeah, exactly, yeah.

[D]

Am I the guy fingers on keyboard building software with CLAUDE code? No, and I hate when people ask me that, but I love that my team is now able to use Claude and Claude code with Fulfill. They're doing some incredible things that just weren't possible six months ago. I don't know if any other ERP is doing this too, but Fulfill has a new CLI tool and an mcp. So my team can open up CLAUDE on their phone and ask for information out of Fulfill. They can by code, custom dashboards right in Fulfill. We can hook open Claw up to Fulfill now to do work for us. If you're excited about the future of AI in your brand, this is such an obvious place to leverage it. All of our supply chains are complicated beasts. Fulfillment and Fulfill has just made it even easier to improve yours. This is real money stuff.

[A]

Okay, so let's. I still want to go back and put my, my, this conversation in the shoes of a $5 million a year brand. Right? And we're gonna go through that journey as they get in that deeper and deeper water. So, like, they're just, you're a new brand. You crushing on Facebook. You have good creative and, like, you've nailed that. Hey, I have like a landing page that works. I have an ad that works and I got $5 million a year strictly off of Shopify. Then you start saying, hey, I'm gonna test different ads. I'm gonna start testing a couple different channels. That's where I think MTA comes in the best, because it's gonna look at that user journey across the entire thing and give you ad level insights. Being like, hey, even though Meta's taking credit, they're like, this ad's doing better. What we're seeing is this ad actually led to lower CPAs, and that's why it's an earlier step in that journey, is that you're gonna be getting, you know, just more granular data and a different source of data that you can rival against what the platforms are saying, and it can look across platforms. So meta versus Google versus Snapchat versus TikTok. And you can ride that wave. $20 million a year, right? A couple different channels, you're spending that up. But then, you know, let's introduce incrementality. It's like, okay, you're spending money on TikTok. The, the, the TikTok reps are being like, we're driving all your results. Okay? You're spending money on Snapchat. The Snapchat. We're driving all your results. The meta rep doesn't know what'? Because it's a matter of. No offense to the meta rep, guys. No, the meta team is telling you, hey, we're driving a ton of results, okay? And all these people taking credit for everything. So what do you do? You do a holdout test. Okay. So you bring on an incrementality tool. North Beam just launched incrementality. You are going to test them. You'd be like, hey, Snapchat, we're only going to show it to a third of the country and we're going to see if that third of the country has better or worse results than anything else. And then you can actually see the lift of a channel, right? The lift of an ad, the lift of a new product launch or different bidding strategy. Right? Like video view on YouTube versus conversion optimized campaigns on Facebook. Like, holdout's going to do all of that. And then like you guys are saying, it gives you a snap of a moment in time. You can't trust that forever. You do have to continue to be testing throughout your organization. Shorter tests, if you can put them. That's fantastic. Because tests can get polluted. That's what Katie was just talking about. But, like, this ability to always be trying to point your compass in the right direction. Right. Because. Go ahead, Katie.

[C]

Oh, I was just gonna say. And then also, I kind of love that he brings it in. So, like, because the incrementalities ran through North Beam, it brings in the iroads into your platform data. And you can compare it like as if if you're running tests on other platforms, you've got to like, open up both windows, right? And like, analyze both data. But it's kind of. And then because, I mean, we use an MMM too, Sean. Right. So you're kind of leaving that out of. It's really like all three matter together.

[B]

Yeah, that's why we, that's why we have all three. I think MMM gets really interesting when you're doing a lot of stuff. You can't measure with a holdout test. Like, you're like hexclad, you're running a Super bowl ad or you're doing a lot of outdoor advertising. You can't. Or tons of podcast sponsorship. Right. You can't measure that with ink. Shall we test? You can't. Right. So there's some things that mix. Modeling, it really helps.

[A]

Yeah. So let's talk about what can and can't be measured. Right. So MTA is multi touch attribution. It's measuring the user across digital IDs to get to your website. So from Facebook ads to Snapchat ads to whatever else, you put a holdout on top of that to hold out certain users, either geographic users or, you know, lookalike users. You know, with modern technology, you can, you know, me and my neighbor could be in two different holdouts. It used to just be North Carolina versus South Carolina or North Dakota versus South Dakota. Like, that's the way they used to do it, but now it's gotten way more advanced. But to your point, why can't, why can't you do those holdouts on podcasts? It's because you can't control who's going to listen to a podcast, right? Like if it's a host red ad or a TV buy, you know, national TV buy, you can't be like, hey, don't show my TV show TV buy to North Carolina or even more advanced, my neighbor, not my neighbor. Just because it just, it doesn't work like that. It's like it's consumption based media. So that's why, you know, all that's, that trifecta of measurement comes in. And if you're bringing in all three, you better be spending 10, $20 million a year, in my opinion.

[B]

I agree with that. Yeah. No, I think you're right, Sean. And you're, you're trying to protect the, the early stage entrepreneurs are just getting going and make sure that they stay focused and don't waste money, which is smart. That, that's, that's right on also too.

[C]

Like, what do you tell them, Austin, when. Because when they do a holdout, they're technically leaving revenue on the table. Right?

[B]

Because yeah, there's opportunity costs. Yeah, that's what Jeremy. That's what Jeremy at kitsch who's, you know, Jeremy, if you don't know him,

[C]

is the nicest person in the whole world. Is that what you were going to say?

[B]

I was gonna say he's like the best. He's like the best of the best in terms of operating. Everyone knows that. I mean, we, we all know he's, he's the, he's top gun. But yeah, I would say we talk about, I mean, one of the things that Norpee.

[E]

More.

[B]

We're trying to innovate because we were seeing what other folks are doing is minimize your holdouts. Minimize. Create statistical significance and the right test that's maximized for the lowest opportunity costs. Katie, like you're saying, like, not hold out too much, not hold out too long, like back to the earlier part of the podcast. Or people are saying dumb stuff like just run longer tests. Well, it depends, right? Because that's opportunity cost. That's what Jeremy Kitchen, I were talking to. Or he, you know, really intelligently pointed out early is like, hey, like, when I don't spend at my maximum capacity, that, that's costs a lot of money. So, like, that's why it's so dumb that people are saying just run longer. It depends. It depends.

[A]

Let's talk about the opportunity cost because Katie directly pointed out that if you're going to run a holdout test, some percentage of the country has to be removed. And I think, I think the modern. So it used to be 50% is how they used to do it. Then it was 30. I think you can get it done with like about 5 or 10% now. But imagine, like, imagine if you want to do a holdout test on Black Friday sales, should you discount or should you not discount? And then you didn't show your discount to 10% of the country, you're going to be leaving 10% of your revenue on the table. Right. So there's. Yeah, there's certain things you just probably shouldn't.

[C]

The statistical difference on that one is you should not be operating in E commerce. If you run your incrementality tests on Black Friday.

[B]

That's funny, but I guess that's why we built incrementality, because we saw so much nonsensical stuff going on. Either the test is too short or it's too long. The holdout size is too big or too small. And so we figured why not automate that process of making it so that we can recommend what based on your MTA data. So, like unique purchasing patterns by DMA for that channel. Right. Because the current providers will just look at spend across DMA. Sorry, purchasing across DMAs. We're looking at clustering of different behavior in your MTA data per channel.

[C]

Do you know how many. That's what she said jokes you just had right there.

[B]

Oh. So only Katie see I knew I'd never get out spicy. Katie, I'll tell you, I'm trying to

[C]

make incrementality fun for you. Austin, you said, I know people are falling too small. It can't be too. Too long. You had all the jokes.

[B]

Oh, man.

[A]

Oh, man. Yeah, look, and I am. I am probably the biggest incrementality bull on planet Earth, but I just want to make sure people are using the tools at the correct depth of water that they're in right now. You never want to see a guy in the kiddie pool with a full scuba set because he's like, oh, man, you just. You just wasted a ton of money for nothing. And you don't want to see a guy in. In the middle of the ocean with floaties on. You know what I mean? Like, you have to. You have to have the right tools at the right time. Um, so, you know, I could talk more about ridges, incrementality journey, or we could talk about, you know, other things going on inside of north beam and why this is the right time to roll this out. Which direction do you guys want to take the conversation in?

[B]

Yeah, I guess for me, you know, we looked at tests from other providers, right? And they were either too short or too long, or the holdouts were wrong in terms of there were statistical errors in the. In the holdouts.

[C]

Or.

[B]

Or like Katie was saying earlier, they were running the holdouts with a bunch of errors, and, you know, there was a promotion, then it stopped and started, and there was a bunch of things that made the test inaccurate. So we said, well, what if we automated all that? What if we automated doing the exclusions? What if we automated the recommendation of, like, what the smartest test to run is in terms of conversion, like by channel, right? Use the MTA to inform the way to do the holdout in terms of the right DMAs to exclude. Automate the spending, right? So, like, your media buyers don't have to say, okay, let's cut the spend at this time and then start it again. We just automate that. Automate the process of alerting you if things go wrong. Right? So like Katie was saying, if something goes awry and we, you know, one of our customers ran a test for four months, they were all inaccurate, right?

[C]

How did they not catch it before four months?

[B]

You know who? Well, because there's no monitoring, and that's why nor beam, we built in automated monitoring, right? So we're. We're going to be watching these tests, watching to see with a machine, not, you know, humans. And so yeah, they were wrong because there was, you know, mistakes in the data science and actually their in house data science person caught it. But they ran four months of tests that were inaccurate. Right? And they actually told us that we just went with the Norpine MTA data because it conflicted with the incrementality test. So that's why we built it. Right. We saw a lot of human error, a lot of human work and setting up and running them. We're like, what if we eliminated that? So that and use MTA data to make them smarter. Right. So you'd have less opportunity costs. That was our philosophy.

[A]

Implementing AI without trusted data is shooting in the dark and building application from scratch. Dealing with tableau, dealing with data pipelines, dealing with all that. It doesn't have to be your headache. You can hire Saris analytics to handle that stuff for you. Saris IQ gives your leadership team a single shared view of the truth. Contribution, margin, channel, performance, customer, economics, all can be answered instantly with the power of Saris Analytics. Do not fall behind. Saris analytics is experts. They've set it up for Ridge, hexcloud and hundreds of other brands.

[B]

Speak to them today.

[A]

Connect whatever AI tool you already use and you can put that right on top of your personal data privately, securely, and you can start talking to your

[B]

data like your favorite chatbot.

[A]

So we started running incrementality tests like a year ago and it helped us unlock new tools in new ways and new tools inside of the channel. So inside of meta, everyone was just running conversion campaigns forever. We started moving up funnel and so we have custom conversions on landing page view, on add to cart, and we've put way more money into those because they just proved to be way more incremental. Increment. They proved to be way more increment. Increment.

[C]

Incremental. Incremental, Sean.

[A]

So what that led to is like I'm spending, you know, 30% more on Meta, but sales are up 60%, right? Like this this month, sales are up like 80% or whatever. And it's because I was able to add on YouTube because I finally had proof that I was working as a channel. And then I'm spending money on meta in new ways. Either it's reach or video view campaigns or whatever else. And for so long that was lighting money on fire. But if you can layer on tools like incrementality, you can prove that it's working in new ways. With that being said, just because it's working for me doesn't mean it's going to work for you. That's why you have to test it, right? If your takeaway from this is Sean says YouTube works, I should spend money on YouTube. That's a great way to destroy all the money in your bank account, right? So you have to, you have to actually start testing this.

[C]

But were you running all those tests at the same time, Sean? Like, are you testing YouTube at the same time as Meta? Are you breaking them up and each month doing a different one?

[A]

So we, we run. And this is, this is where I disagree with Austin. We run more tests than anybody else. So, like, we, we ran like 25 tests last year. So the average test is lasting 10 days or two weeks or whatever. So shorter tests, but just. There's always something being tested. Because. Because what we think about is, yes, you're losing 5 to 7% of potential revenue by removing those, those GEOs, but the GEOs kind of rotate out, right? Like, you know, it's always different people that are getting removed from those tests. It's not the same 10. You're never going to hit and.

[C]

Well, but couldn't it be like, what if you've got the same zip code, you're holding out on Meta, but then you're not holding out on YouTube and one's feeding the other? Like, isn't it kind of skewed by running them at the same time? Or.

[A]

Oh, well, there's always one test going, but there's one test going back to back to back to back to back.

[C]

Okay. Not overlapping is what you're saying. 25 tests all at different times.

[B]

I don't think I disagree with you, Sean. I think it's more depends on your mix and how much you're doing, right? Because the Ridge is. I mean, you guys are known for a lot of changing and tweaking and experimenting, right? But not because I know our other customers. Some people have more of a steady mix and steady tactics, right? So they don't need to test as much. So, yeah, it just depends, right? You guys are doing so many different things, so I agree with you. I think you should test a lot because. Because you're doing a lot of different things and tweaking all the time. So it just really depends on the business, right?

[A]

I think it's a culture of testing, right? We talk about, like the culture of incrementality internally. At Ridge, there's like a line of 10 people who want something to, like, they all want to test something and they have to get in line. It's like, when do I get to run my test? And, you know, I think incrementality is so great because I think you can still do a B test on your website. Like, there's this. It's going to skew it a little bit, but, like, I think you can do all the a B tests you want on your website. When it comes to price or whatever else, I think those results end up being so, you know, so small to actually tweak the whole thing. So, like, not only are we running a ton of incrementality tests, we're running a ton of a B test. So there's like a hundred tests a year being ran on my website. And that makes this just so much better because we. We run it and then four months later, we run the same thing again just to just double proof everything that we're thinking about.

[C]

Have you ever had to go back, Sean, and like, maybe in the first month in January, it said something was really incremental. You retested it three months later, and it was not.

[A]

Yes, we have totally gotten burned. You know, we launched a bunch of women's products, and women. Women's was crushing it for us. We were doing, you know, like, the first month we did it, we did 600 grand in, like one pink wallet. And we're like, oh, my God, people want pink wallets. The test was showing. It was incredibly incremental, right? So we bought a bunch of them. We leaned really, really hard in, and then it failed. And it's because, yeah, it was like, it was totally incremental, but then we sold it to every person on earth who wanted a pink rich wallet. We could not replicate that success at all. So, you know, we got stuck with a bunch of lavender carry ons. I still got. I'm like, I'm like, we have to turn through these things. We bought 40,000 Lavender Ridge wallets because the first month we sold 4,000. And I'm like, dude, we're killing. We're making a ton of money off this. I should buy a bunch of them. Then we got 40,000, and I'm still trying to sell them today.

[B]

I just think your team is unique. I mean, I know your team's unique. You guys just do an incredible job of innovating and trying and just testing so many things all the time. Not just like incrementality tests, but just any kind of test, like creative tests or influencer. I mean, you guys have been doing that for years, and that's why you've scaled so, so well, you know?

[C]

Well, what about for little loser companies like me? Austin what am I supposed to do?

[B]

Well, you guys are great. You guys do. Do great work too, I think. I think, you know, you know, you guys do great stuff too. I think it's okay.

[C]

Austin, you. You can just tell Sean how amazing he is and I'll just sit here.

[B]

Sean and Connor, you know, they're just unique guys. We all. That's why he's operators podcast.

[A]

So it's taking the industry 10, 20, 30 years to bring incrementality to digital brands. Right? Like, it's not new. It's been around forever. You know, we started hearing about it last year when Meta was talking about it nonstop, right? Maybe a little bit before, maybe two years ago. But why it's taken so long for companies like Ridge to embrace it and Katie, why were you hesitant to embrace it?

[C]

The reason we did it, I'd say, is because fomo, a little bit like, everyone's doing it, so should I be doing it and do I need to be doing it? I mean, the biggest reason, Sean, it took us a while to embrace it was because we were not omnichannel for so long. And because such a high level of percent of our spend went to one channel, Meta, which I knew was working. It's. It. It didn't make sense for me to test a channel that I was not spending a whole lot in. Right? Because the test would have taken longer. It just. It. I didn't. I wasn't. I don't know, I just didn't think it was a big thing. Like an example is a couple years ago when we first ran incrementality testing, we ran1 on TikTok and it actually told me that TikTok was incremental and we were spending a couple thousand a day, so not that much, but my gut told me it was not. And we turned off TikTok ad spend completely and did not even see a penny in revenue dip. So I know that's not like, what you're supposed to say, but I still, I'm skeptical of everything. And especially when you get on to Twitter or X and every other word of everybody is incrementality. Like, it just becomes this like, circle jerk of incrementality. However, we're leaning back into it now because I do think as our. As we scale spend up and lean into other channels, it feels like a gut check. Like, it feels like I'm. I know these things are working, but I don't want to make huge decisions on my gut anymore. Right? Like, I want to know if something actually is performing or not. And honestly running that incrementality test last month and kind of seeing as it was progressing and knowing that what it was, it was telling us very different things, that our MTA was the end platform, was we knew something was broken and it helped us flag it so that we could go try to find the smoking gun. Right. So I kind of like Austin's idea of like, I don't know, maybe for you know, us little guys on the short bus, like I could see a space where I was doing incrementality testing for a couple months, then maybe took a couple months off, then tested again and kind of went through that cycle but. Or did a different. Right. Like do meta one month, do Google another month, lean into Pinterest, TikTok, you know, things like that. But I still don't think it's, I don't know, Austin, I feel like is it brands that are like 50 million and up are doing incrementality? Like where do you see the sweet spot?

[B]

Yeah. Curious to hear what John thinks. I mean I think it's like once you start really, because there's some companies that maybe are lower in revenue but the spend is really high, right. And you're, you know, like for instance you're doing, I don't know, 20 million revenue but you raised money and you decide or you have cash and you decide you're going to go do CTV, they're going to try TV or YouTube. Like Sean's saying you're going to lean into YouTube, right? And you decide you need to know whether it works. Would you agree with that, Sean? Like it's when you're really leaning in hard to something that's view based.

[F]

If you're scaling an E commerce brand today, ads alone aren't enough. After Sell focuses on the one moment that every brand already owns after checkout and turns the post purchase moment into more profit. Monetize every order with post purchase offers and thank you page experiences without disrupting checkout or hurting conversion. Enterprise grade tech used by Gap, Ticketmaster, Macy's and Target. Now driving results for brands like True Classic, hexclad, Ridge and Jones Road. I would know. This is the reason I ended up buying three pans from Hexclad instead of two. After sell has already generated over 1 billion in additional revenue for E commerce brands. Revenue that doesn't require more traffic or higher cac. So check out after sell and tell them that the operator sent you.

[A]

I think, I think as soon as you start adding a big second or big third channel or you have something like you really need to know if it's going to work or not before you invest a lot, lot more. I think Mentality is a great tool. I want to highlight what Katie said because I think it sums up the, the, the, the industry, the best tools for tools sake. If you're marketers listening to this and people in your organization, they love new tools, they love shiny things and it's kind of like if you ever see like a, like a hobby guy in his garage, he'll have like $150,000 worth of tools and he'll never build anything. It's like that's what most marketers want. They just want every single possible tool on earth. And. But the counter is we need these tools but who should use them? Begrudgingly, right? So I wish I never had to use any ad platform to make money. I wish people would just show up to my website and just give me free money. Doesn't work like that. So I have to use Meta. Okay. And then I use Meta to the full extent of its ability and then I can't use it anymore. So I have to go to YouTube. It's the next place, right? I don't want to. My life would be better if I could just go to Meta. Then I have to go to YouTube. Now I'm at YouTube. Okay. I need MTA to measure these things. Right?

[C]

Okay.

[A]

Now I'm going to Snapchat and TikTok and TV. Okay.

[C]

Damn.

[A]

Now I need incrementality. So it's like these tools should be, they're important but you should basically be forced into them. But your marketing team's going to ask for them. They're going to hear about how cool incrementality is going.

[C]

Too many tools and I think Austin, aren't they. I mean really, it's going to be become even more important since like last. I mean we know Meta has changed the way that they're right. Like tracking clicks now. And it's becoming even harder to really give attribution to the. Like you said it has to be within a 30 day window. Like if you've got somebody that shops for six months before they buy something, it's hard to even track that. And the way I think about incrementality is it's. Is that channel needed? Like Sean, you know, I mean really direct traffic is our second or third biggest traffic source. Like people just literally coming to Caden Lane. And so one of the things that's beneficial to us to test with like Meta is do they need to see the ad, right? Like, where they going to come and buy it anyways? So it's not the channels all work, it's just what are you using them for? And where's your, like, you know, where. How high can you scale in your spend on it and how.

[A]

How big can revenue get while spending the least amount of money possible? Because, like, that's really what we want to do. Even stuff like sms, it's like, are you sending too many sms? Right? It's like, well, we got to do holdouts. We actually have to prove that stuff out. So it's, you know, we're adding a tool to hopefully remove negative spend. That's. That's the beauty of incrementality and MTA. It's like if you can save 1% of your meta budget, it's a million dollars a year. So that's.

[C]

You said. You know what's funny is I'm like sitting here thinking about this and like, I mean, I've been doing this 22 years, right? In the last 10 years, specifically on e commerce. But even five years ago, nobody really talked about testing. Like, not even in platform. Like, not even store testing. Sean. Like price testing or like button testing or colors or banners or images. Like, why did testing become such a thing? Is it just because we're all dorks and marketing? Like, why, why now? Why is testing like such a cool thing? Because it justifies us. Like, what do you think?

[A]

No, it got harder. It's like, it's like. What I'm saying is like, the, the. It's so funny.

[C]

Got harder to market online and now we've got to. Basically, we can't just, like, see which way the wind's blowing. Yeah.

[A]

I said we, we layer things on progressionly like it used to be. I knew people, I knew a brand. I'll tell you something about this brand. They're out of business now. They would. They're like, we're a holiday gift. So they would come in in October. That's when the company would, like, every. They would take nine months of the year off. Everyone would be laid off. They'd come back in October, they would crank their budgets. One ad, one landing page, one offer that would make like $7 million. And then January 1st turn all their ads off, come back the next year. That worked for four years. Like, unlike. There was no competitors. There was no one eating their lunch. They didn't test anything. They didn't think about anything. And that was just the reality of, like 2015 to 2019. And then as soon as as soon as Meta got a little bit hard, a bunch of people went out of business and now it's like, okay, the people who survived are methodical about finding the next dollar possible. Right? And, yeah, and like, imagine if you heard somebody only worked three months of the year, made $7 million, you're like, yeah, good luck, Brad. It's like, I, I'm gonna launch your product in two days. That's. I'm coming for you.

[B]

Yeah, yeah, I think that's true. Also, the. I think, Sean, I'm curious also, and Katie, you've seen it for a while, but the, the mix is diversified. Like, people are trying more things. Like, Applovin wasn't around, right?

[A]

Yeah, you know, good point. Snapchat didn't really have an ad platform and TikTok didn't really exist. It was just, it was just Instagram ads, it was just Facebook ads.

[C]

Austin, are we gonna be able to do any kind of incrementality testing on, like, the. On chat GPTs?

[B]

Yeah, that's. Yeah, that's all coming. We're. We're talking to OpenAI about ads. And yeah, I think as the channels diversify and there's more stuff we're gonna need more mta, more view through which I. We didn't talk about that much today, but really excited about that in our platform, the clicks and deterministic view. So viewing, seeing how ads convert on a view basis.

[C]

I'm excited about you guys adding Pinterest. Like, for us, that's a channel that. I mean, Sean, it's probably not a big channel for you guys, because most men aren't on Pinterest, but for us, it's a big one. And there's a lot of conversion lag with Pinterest and it's hard to measure. And so that's where an incrementality test is extremely beneficial because we can really lean into it.

[B]

Here's a fun stat for you. There's a company in the health and wellness space where even on clicks, so not even including view through, something around 23% of the conversions take one year to convert. Convert one year. Okay, think about that for a second. So they spend a dollar on Meta today, and including views, it's probably closer to 30. Almost 30%. Right. A third of that revenue will take a year to convert.

[C]

So their first year in business, they just think they had the worst business in the whole world. They're just like, there's nothing.

[A]

Oh, I was going to say, I'll tell you about two personal experiences with that in the health and wellness space, I bought an Eight Sleep. It's $5,000. So like I've heard about Eight Sleep for three years before making that purchase. I heard about it on podcast. I clicked on ads, the whole thing, because I have to convince my wife to be like, yeah, the bed's gonna heat and cool. It's gonna be awesome. And she's like, oh, another thing in my life, right? And then I have a tonal behind me. It was also $5,000. So like these large ticket purchases in health and wellness, they will just take forever to actually convert, right?

[C]

So everyone that's listening, keep advertising to Sean. In a couple years, he's gonna, he's going to buy your really expensive toy.

[B]

That's where MTA is important, right? That's where incrementality would maybe miss. Like even if you ran a three or four month test, you know you're going to miss, right? So that's where mta. It's a good example where MTA and incrementality need to. Need to work together as a team, right? Because if something takes a year to convert, you're, you know, you're just not going to most likely see that in a, an incrementality test, right?

[C]

So is that company you talked about still around? I don't know.

[B]

Oh, yeah, Everyone on this call would know who they are. Very big company.

[C]

Okay.

[B]

Yeah, yeah, yeah.

[A]

The last thing we'll say about incrementality, you need a weapon to help remove the other tools, right? To help remove the other channels, to help remove complexity. Because a lot of our businesses is just a house of cards. You're stacking this thing on top of this thing. And like, somebody has a favorite way to look at data, so they want this dashboard and somebody really likes this influencers, they want to sponsor them. And like all of that stuff just gets piled onto your business and you have to find a way to get to the heart of what's actually working. And incrementality is a great way to do that because you will get a readout if this actually drives revenue or not, right? And hopefully lets you get back to the basics of your business, right? Like, you want to be doing the smallest amount of things possible, to do the most amount of money possible. And like, you know, you want the leanest team to do the most amount of revenue. Everything comes down to just being like, how flexible can you be as an organization? And incrementality is a great tool to do that. So I know it's counterproductive adding More tools on top of your tools to audit your tools. But that's, that's the world we're in right now.

[C]

I'm just glad we're not playing a drinking game with how many times you've said incrementality in the last hour.

[A]

He does any, any AI talks, we say AI, like 80,000.

[C]

Oh God. That was the whole meta event. You, you missed it, Sean. The whole week last week was AI, every other word. If you didn't, if it wasn't every other word coming out of your mouth, what were you talking about? About.

[A]

But I am excited for the AI ad platforms. Like they are coming. Like chat GPT ads are going to be awesome. They're. I think it's open beta right now. Anybody can sign up for it.

[C]

But everybody signed up. But they're all on waiting list. Did you get in? I'm sure you're in.

[A]

You know, I did get moved to the top of the waiting list, so. But dude, the, the crazy thing is like, you know, only like 4,000 people work at open AI. Like they are crazy. Like they gave me early access and I'm like, oh, cool. How early access going to last? And they're like, yeah, we're going to go public next week. It's like, oh, okay. So I get four days. I got a four day head start.

[C]

Oh, man. Well, okay, so can I just tell my email, if they're listening, they can go ahead and put me at the top because we're still sitting on the waiting list along with everybody else that I know. So that's good. Let us know you test it out, Sean, and let us know how it goes. You could, you can be my first incrementality test.

[A]

There you go. I'll burn the money. And then maybe we should just say if things are. Yeah, I said this a couple times, but a lot of this industry is tribal knowledge. It's like group chats and Twitter posts. What's working, what's not working? And so bad information spreads just as fast as good information. Like I remember, I remember people being like, CPMs don't matter. Like if you have a high CPM, it's good. And I'm like, no, that's, that's not true, man.

[C]

Like, no, it's so true. And just because somebody's posting about their results on their tests that they ran for incrementality does not mean that's what yours is going to be for that same channel.

[B]

Introducing full automation into incrementality testing is I think, the future.

[F]

Right?

[B]

So you're kind of like back to Sean's point about having smaller teams being leaner. We're trying to bring incrementality testing at scale so that it's. It's more accessible, maybe to smaller companies at a lower price. Like you were saying, Sean, because it's automated. Right. You don't need all the human labor

[C]

to execute and, and the human labor to screw it up. Because I'm not right?

[B]

Like, you're just the most important point, Katie. Yeah,

[E]

every SaaS company says they are AI powered, but very few can explain what it actually does for the revenue of my brand. This is why postscripts approach stood out to us. They don't just build AI for demos or buzzwords. They built it to drive real incremental revenue. Postscripts AI called Shopper. It shows up inside of SMS at moments with real buyer intent when shoppers are likely asking questions, hesitating, maybe even about to drop off. Shopper can answer product questions instantly, answer questions about fit, availability, recommendations, order issues, the kinds of stuff that people usually bounce for. This means more conversions, higher a of less loss demand. So you are driving more revenue and doing it more efficiently. Check out Shopper from postscript. We use it at Pela, which is why I am telling you to check it out.

[A]

And it would be so cool to just like inside of your northbeam dashboard. It's like, hey, this channel's crushing it for you. Don't believe me? Test it. And you just click a button to test the whole thing and click it

[B]

and it runs it. And then to Katie's important point, we've seen so many mistaken tests run by the vendors out there. Make sure it doesn't get screwed up and find that because the opportunity cost is huge. Right. And then if you're going to test a lot like the Ridge, you know, we'll alert you if. If you know, when they go wrong.

[A]

Right.

[B]

And say, hey, you know, I know you want to run this test. You guys like to test a lot. This one is statistically insignificant. On day three, you should shut it off so you don't waste your. Waste your money. Right. Automatically. So, yeah, I think bringing automation to the masses reduces cost, reduces labor, helps you run as a lean team. And yeah, it just made a lot of sense to us. And like Katie was saying, automatically injects into your MTA data the calibrate. Right. And Connor at the Ridge, he told me that was what he was really excited about.

[C]

Yeah, because it's all right there. You're looking at it all together.

[A]

Exactly. And look, this is now the big three measurement mta, which is like you guys are the pioneer of. Mmm. Which is like the, the oldest one out there. And then you have incrementality later on there. Is there, is there another measurement vertical that we're going to hear about in three years or whatever that, that I don't know about now?

[B]

Ooh, that's a great question. I think what we're excited about, we announced this two years ago with Meta somewhat quietly. They, they pushed it out where they announced that they were using our data and Google's data at the time, this is a couple of years ago, to train their AI ad serving algorithm. Right. So basically we called it Apex at Nor Beam, which is really, simply put, there's two ads, one with Katie, one with Austin. The one with Katie's doing way better than the one with Austin. To your point earlier, Sean, Meta thinks Austin and Katie are the same. We see that Katie's Nor Beam data suggests Katie's way better. So we, we send that data to Meta and they optimize their AI ad serving algorithm against. We mentioned AI. Thank God we did it. And it optimizes against that our data. Right. So that's something we announced two years ago. Announcing here. We're, we're rolling out a new version of that with, with we're really excited about in the, in the coming months. So that's another. I don't know if it's measurement per se, but it is because we're run a B tests against it to see how does that.

[C]

We saw Austin, I don't know if you remember, but we actually saw huge improvements when we. Yeah, yeah, I remember Apex was.

[B]

Yeah, I remember. Was a big deal for us and, and I appreciate. Thanks. And, and the new versions can be even more powerful.

[A]

You know, the only thing I think Northview is missing that if you guys are doing the big three of measurement is just more testing tools on site and maybe like here's a big, Here's a big Unlock it Rich. We sell a power bank on Amazon. Okay. It's doing fantastic. You know, we thank you. Thank you. We changed about two words in the title and sales went up 10%. Just because you start ranking for more stuff, there's more visibility to it. That's all built in Amazon tooling. Like they have like, you know, AB test. You can just run there. But like that was thousands or tens of thousands of dollars a day just because we changed three words in the title. So like bring that sort of what

[C]

words in the title for anybody.

[D]

Listen,

[A]

I'll send it to you. But yeah, it was just because there's more organic keywords to be had, like, whatever. It just worked way better.

[C]

And to your point, it was working right? You just wanted to test it and then now it's working better.

[A]

Yeah, always be testing. You know what I mean? But yeah, so you could bring the same sort of on site testing. Right. You know, we're using a bunch of great on site testing tools right now. But how do we get those things talking to mta? How do we get those things talking to incrementality, all of that just like full stack sandwich to be considered.

[B]

We are working on that. And I've had. Yeah, we are working on that.

[C]

Yeah. Austin, I want to be able to measure ad. Okay, here we. I want to drive traffic to different landing pages with different UXs and I want it to break it out. Like, I mean, I feel like this is a dash. It's probably stuff you'll already do. It's a dashboard.

[B]

We already do it. We, we could. We already do that now with good naming conven conventions, you can track landing page conversions with good naming. So we already do.

[C]

What if you have bad naming conventions?

[B]

Well, then those have to get fixed. But. Yeah, but we are. We're launching new tools in that arena. But you can already do it today. It just requires setup. So if you need help.

[A]

Katie, how about AI powered automatic naming convention?

[C]

That's what I was gonna say. Yes. Yes.

[B]

Working on that. Sean, are you in our meetings? Are you in our product meetings?

[A]

Yeah, yeah, we're docs in the North Beam roadmap.

[B]

Yeah. Yeah, you're easy. Well, you are first customer, so you, you know, you were there.

[C]

Okay. Okay. So since you are the tester of all testers and I'm failing at it miserably for a day, I'm going to hire you as cmo. What is the first test you run for Kaden? First three tests you run for Caden Lane. Knowing that I already did a meta test. So what's my next one?

[A]

Well, one I'd look. I'd look at the meta test and I'd figure out what we actually tested. Did we test top of funnel? Did we test like, you know, creative style? Like, what was that actual test that we ended up looking at?

[C]

Austin, what did I test?

[B]

I would say just overall channel lift. Overall channel lift.

[A]

Yeah. So then, then I would. If I, if I could do three tests. So you're hiring me for more than one day? You're hiring me for like you know, six weeks or whatever. And to have you come in for six weeks is about $1 million.

[C]

So I have to look at. This is not the question I asked you. Now I feel like you're dancing around because you don't know the answer. Answer. Let's get what. Go get Connor. He'll answer this faster for me.

[A]

That's true. Yeah. Look, I'm, I'm, I'm the showman. Connor does all the work. Right. The first test, I would test higher funnel ad strategies inside of meta.

[C]

So you would keep digging into meta.

[A]

Yeah, because if we could, if we could test page views or add to carts, you could spend 30% more. Right. And what you want to look at is like, what's the. A incremental bonus? So it's like, okay, the MTA data is going to suck. It's going to be like 0.1. But do we get a 3x bonus? Do we get a, you know, 0.5x bonus? Like that helps you just spend more money on people who are, who are colder audiences.

[C]

So you're saying even if I did an overall lift study as my first test, then break it out to maybe even top of funnel, bottom of funnel conversion.

[A]

Totally.

[C]

And the different. Okay, so you would.

[B]

And change the tactics. And change the tactics when. Yeah, yeah, yeah, yeah.

[A]

That'd be the first test. The second test would be, you guys have great content on YouTube. I have a kid now, and if you type in like how to burp a baby, the first video is Katie showing you how to burp a baby. And I think YouTube should be an amazing channel for you. I mean, I, I said my second test is how do we get YouTube to actually work? And I would shoot a bunch of different creative, you know, I would repurpose stuff you have. I would take the best performing shorts, we do some super edits and just be like, can we get this to be the same as conversion optimized meta with the bonus. Right. We'll get some sort of lift bonus. The second test and then the third test. Exact same thing with TikTok, because my wife's on TikTok and it's nothing but baby content. I'm telling you. Like, it's just like a bunch of fat babies being burped. Like, I'm like, our content would do so, so well here. So it, anywhere where there's a young mom, female presence which is going to be meta, going to be YouTube, going to be Instagram, are going to be tick tock. We should be spending six figures. So I'M going to try to unlock that for you. We'll kick it off and then I'm going to try.

[B]

Katie, we'll do a YouTube one for you.

[A]

I'm going to. And I'm going to make you spend. I'm going to make you put more of a catalog on Amazon and I'm going to test some words in the titles and we're going to get some good lift on there.

[C]

Yeah, the three special magic words. We're going to do that.

[A]

And then I would, I would make a whole line of baby products that are not just apparel. You know, my, you know, there's like a little spatula that you put butt cream on your kid. We have to have a Caden laid version of that. Way cuter.

[C]

Okay, that is a first kid problem, I assure you with your second and third. The spatula you're gonna be using to put butt cream on their butt is this.

[A]

Yeah, I'm already there. I'm doing this. But the nanny, the nanny doesn't like using the finger.

[C]

Everyone heard it here. Fancy Sean has to use a fancy spatula spoon because he can't use the butt cream on his finger.

[A]

I'm telling you, grandparents would buy the fancy spoons.

[C]

Teddy's how. He's three months old now.

[A]

Now he's five. Five weeks. He's five weeks old.

[C]

Okay, Austin, so you heard my brand new CMO give you the roadmap of what, what I'm going to be doing at Caden Lane. And you agreed about YouTube. So now is that, is that on? Like, what do we do? Can we, like after this call, we get on and we set it up. Like, what do I do next?

[B]

Yeah, I mean, I think Pinterest could be one that you want to test eventually too, too, right? Because I mean, the testing road map is always predicated on your channels, right? Like where you're investing. So, yeah, I think all, you know, all your top of funnel channels at some point you want to test.

[C]

I'm sure, Sean, but I don't want to have to do all this. I don't want to have to think about it. I don't want to set it up. Is North Beam just going to like, come up, where's my little AI buddy in North Beam, where I can just be like, hey, man, this is what I want you to do. And then is he just going to go, wait, she. It's gonna be a she. Because she's gonna be very efficient. Can somebody just have a female, right,

[B]

like AI Bot and just run the test?

[C]

Can the North Beam AI bot be named Katie.

[B]

Oh, I, you know, who knows? Who knows? It's who knows.

[C]

I don't know. Is that something you're thinking about? Like where I could just go in and the same way that I go into Sidekick and Shopify and I'm just like tell me this and do that. Am I gonna be able to do that on northbeam?

[B]

Yeah. So to you're gonna be able to automate your flight of tests and it's going to automatically suggest what test to run and how long.

[C]

So maybe will it even tell me like hey Katie, I think you really need, you know, judging on your MTA for last month. I think that this month we need to test Pinterest. And this is why exactly.

[B]

That's right. And here's how, here's how much of a holdout. Great call up by Aaron there to, to ask this question. Yeah. Not only when to run it, how long to run it, the holdout monitor it if something goes wrong. So really just automate it from A to Z.

[C]

So what, when is that going to come live? Because I've got to give notice to Sean to like you know.

[B]

Yeah, yeah, yeah, I know. Because in the next month. Yeah, I mean we could test YouTube right now. So we're ready with YouTube. So we'll start with YouTube, right? We'll go from there.

[A]

What's up operators? Welcome to the Rich panel ad read. Rich Panel has been a sponsor for over 12 months. I've been a paying customer for over 12 months and guess what, I just renewed the pay again for another year. We have cut our SaaS bill in half and automation dropped our cost per ticket by 70%. Our CSAT has also improved from 88% which is still really good to 96%. Best in class. All powered by Rich Panel. I told them last year hey you guys need to do the same thing with returns. And now Rich panel has a returns portal. It's built to cut down your tickets and convert more refunds into exchanges. They do the heavy lifting, data import, self service retention flows, team training, all of it. And it'll be live in two weeks. If you want to save 30% guaranteed on help desk and now returns book a demo. Also the mid flight statistical significance monitoring I think is really important because you know we've, we've got to the end of six week test and it's like oh yeah sorry that test, does this mean anything? So just being able to like cut that off sooner re put those resources towards something else. Unlock that, that, that, that, that hold out. Yeah, I think it'd be fantastic.

[B]

I'm with you. I think that's what's. That is what I'm really excited about. Plus the fact that we could do better holdouts because we have your MTA data, we see the way the purchasing by channel is happening. So we can pick the right length of the test. Right. And we can pick the DMAs better because we see all the, you know, the purchasing behavior by, by platform. So yeah, Sean, to your point, monitoring and then also better holdouts.

[A]

Austin, you see more data than anybody else. I'll tell you that Ridge is having an amazing 2026. Is that common? Is everybody crushing it? What's happening tactical and practical sales results right now across the industry. What are you seeing?

[B]

Oh, wow, great question. What we're seeing, I'm going to give you a really interesting stat that the highest performers, fastest growing companies in 26 are outputting two times the creative volume of the lower performers. Right? Two times. So that's just a really. We did some, you know, data analysis and that's the step. Right. On average. Right. Two times. So doesn't mean you want to create a bunch of AI slop. Right. That's not the best way to go. But the people that are intentional, that really work on genuine high quality creative and iterate a lot at 2x the volume of the other people do better.

[E]

Right.

[B]

So. And I think it's getting harder to market because of how crowded it's getting.

[A]

Right.

[B]

You see? Yeah, just I think people getting more competitive in the ad space. Right. Just in general, I think, yeah, with AI tools, people are able to generate more creative faster. So it's kind of pumping the ecosystem full of more creative. So therefore it's noisier.

[C]

And so have you seen a surge of like new. Have you seen a surge of new brands coming on to the market or.

[B]

Yeah, I think, I think Sean talked about it with a couple of ex posts. Right. The idea that like it's a good time to, to, to get in, right. With the tool, you can run as a leaner team. You can, you can use these tools to be, to move faster.

[C]

Right.

[B]

So what does that do? That creates more competition. What does that mean? That means you've got to be better. That means you have to understand your data better. You have to create more authentic creative.

[A]

Kitty and I think our product categories, you know, durables, I mean, baby's still very hot, but like we're still, you know, we're old heads compared to everybody else. I know a company doing Pouches and they're. They're spending 5 million a month on meta and they have a team of four people and it's like, what. What is going on?

[B]

Like, what kind of pouches? What do you mean?

[C]

Like the Zen pouches? Is that what he's talking about? Yeah, that.

[A]

It's like, it's like a. It's not nicotine, but it looks like nicotine. Like, like that whole thing. Right. And it's just crazy that these very small teams are spending tons of money. And because it's like, let's remove all opex. Everything just goes into this marketing machine and we'll generate creative and we have a product one sku and sell the hell out of it.

[C]

And then, you know, that's the dream one skew.

[A]

I'm over here inventing new things to try to sell and you have like a billion patterns a year you have to come out with.

[B]

I think that's why that stat's so important. Is that nailing the creative piece, obviously understanding your measurement, your data. But like what is your creative testing creative volume plan look like? And is it. And it can't just be all AI generated creative. I had a. I did a hot take episode with Cody at Jonesboro Beauty and you know, his point was like, oh, you know, authentic creative is going to be at a premium where it's like really, it cuts through the clutter because of all the, you know, flash in the pan new stuff that's just blasting out. Right. So. So I think doing 2x the creative of your competitors is important. Obviously, that's what the data shows. Yeah.

[A]

You have to do all the stuff you were already doing and then you have to add all the AI stuff on top of it. So it's just, it's just you get, you get way more coming.

[C]

But I still think quality over quantity though, Austin. Right. Like you're.

[B]

I agree.

[C]

That's when you said no AI slop. Like do not anyone listening. This does not mean you build out your Claude code like ad generator and just have crap that you're pumping into meta. That's not the. That's not the easy win.

[A]

Yeah, I've seen people do it though. Make a ton of money. There's. There's a company out there selling men's health supplements just ripping with full AI ads.

[C]

My favorite thing to do now is to play the spot the AI with myself. And it is getting harder and harder.

[B]

I also think that folks are really coming around to see tv. Connected tv. Right. I think that's a trend Where I'm seeing people getting excited about investing in tv. It's hard, right, because you have to have good creative. Back to the creative. But I think, I think from behind the scenes I can tell you there's some very large companies that of course I can't say under NDA, but I think CTV is going to get really interesting.

[C]

Austin, I turned mine on today.

[A]

Oh, there you go.

[C]

Yeah, literally today, like they literally slacked me today and was like we. Because we usually do a lot of linear. And I had tried CTV years ago, but it was a lot of just retargeting and I think it's made a lot of steps, you know, in the right direction for segmenting. And yeah, we're kind of excited about it.

[B]

I think CTV is the thing that people are sleeping on. But it's hard though because you have to have the resources to make creative anyway. Makes sense. Sean.

[A]

You know, Katie, we're in the exact same boat. We always bought Linear because Linear is just very, very cheap. It's like you can get $CPMs, $2 CPMs. Like it's just. Nobody's bidding on the History Channel.

[C]

No. And it's, it's audience, like, I mean, right? Like we love grandmas. So for us it was just a no brainer. But I think if, yeah, for, if your audience is young males, like, I mean, you know, the Today show is not their target audience.

[A]

When Connected TV came out, you know, in 2019 or whatever, it was Hulu running ads and it was so expensive, it was $60 CPMs. Right. So I'm glad there's people like NBC Universal pushing it out and like, you know, even Netflix like has dropped their CPMs. So like now it's in a place where you can actually buy this stuff and be reasonable with it.

[B]

Yeah, that's what I think. I think the people that figure out CTV and really nail it with the right messaging, the right creative, they're good. That's a, it's a good arbitrage, you know what I mean?

[A]

So we have more creative is better. Glad to hear that. Still, that's still the mantra. Connected TV's working. Katie, do you have anything that's that you want to say like, oh, I saw this and it's crushing or Austin, a third thing.

[C]

I mean, don't take testing advice from me because I screwed up my first one.

[B]

So I would say the other thing is that people don't nail email. SMS messaging.

[C]

Right.

[B]

I think it's like the kind of thing where people just kind of do the Same thing that everyone else is doing. And I know the ridge. Like you guys do some neat things with your messaging where sometimes like hey, if this is a hard time of year for you, you know, you know, let us know. We'll opt out of sending you emails or sms. Right. Like you guys do that stuff. I think there's a lot of like opportunity and email SMS to be better and I see. Does that make sense?

[C]

It's our top channels. Like literally we drive so much revenue through both and we do a ton of segmentation really. Like there's no more. Just you send an email to your active audience, you know, your 30 day clicks and opens and. And I think there's a lot of optimization. We've had email and SMS revenue go up, whereas deliverability on email is harder. Right. Like all these new rules about putting things into different tabs. And so there's still is. Yeah. And that's, you know that that's your customers. Right. It's a great retention play.

[E]

Call me nerd all you want, but one of my favorite things is when one of my existing software partners, in this case North Beam, adds something that I historically would have had to pay separately for. Incrementality was broken and North Beam fixed it. So let's face it, most incrementality tests are slow. They're manual. One mistake can validate an entire test and waste thousands of dollars. If you've done it, you know, not anymore. So only North Beam incrementality automates your incrementality test design and monitoring, letting you focus on insights and not logistics. They're the only ones who build the test for you using your MTA data. I love this. Northbeam continuously monitors your test in the background, making sure everything runs smoothly and maintains stat sig. Once your test is complete, North Beam then gives you actionable and precise results fed into your northbeam MTA dashboards. Finally, go to northbeam IO and request a demo today.

[A]

Think about how much innovation has come to paid media in the past five or 10 years. Right. Like, you know, 10 years ago when I was running ads, we weren't even talking about Google Analytics data. Like it was like, like the whole world.

[C]

You couldn't text your customers five years ago there was no sms.

[A]

Yeah, like the world has changed so much on the paid media side, but I think email in particular has been totally a ignored. There's been no innovations, there's nothing cool that's happened. And look, Klaviyo is the big game in town. I'm on Klaviyo I tried to leave, it went horrible. So, like, they're still the best solution, but can you believe how late they were to sms? Can you believe they haven't launched? Like, the cool new thing that they launch is like, we're going to help you copy or whatever. It's like, come on. There needs to be something really cool and innovative happening in email because it's been totally ignored. And as paid ads get harder, email becomes more important.

[C]

I want Apple to start literally selling us notification space. Like, I want to be able, like, if a customer downloads a pregnancy app, like, then I want it to somehow trigger a web banner on their phone.

[B]

I was also just thinking about messaging too. You know what I mean, Katie? Like, I was just thinking about, like, what you say to people, right? Like, because I sign up for a lot of customers emails and stuff and I just think, like, how you communicate with your audience, right? Like, what you send them is. Is something that I think people sleep on.

[C]

So. And I. You guys asked for like a. I don't know, like a easy win or something. We've seen. We've actually, like, drastically increased our traffic and conversion to our app, so which for everybody listening, no, push notifications on apps are free. And if you get your best VIP customers to download your app, which is a better ux and they have special deals and they have special launches and then you get to message them as much as you want because push notifications are free, that's been a huge win for us. And I'm talking, like, a very large portion of our revenue comes from our app now.

[B]

Yeah. Anyway, I think the Ridge, you know, a lot of folks do a good job. Hexclad. They do a good job with the messaging and the emails and.

[C]

Yeah, it's just you don't have to compliment anybody. That's not on the thing right now. You only have to compliment me and Sean.

[A]

Yeah. You know, Kitty and I just had a kid. I didn't realize how important apps are to keeping your kid alive. It's like the Huckleberry. It's like, oh, you got to feed him right now. Okay.

[C]

Jesus, Sean, we're going to have to have a whole parenting episode. You're using, like, the spoon to put the diaper cream on and you have an app keeping your kid alive. Like, I'm just. I'm worried.

[A]

Yeah, the app and the nanny. Yeah, those. Those two things together. All right, cool. Well, I'll tell you guys what's working for Ridge right now. Obviously testing a bunch of different stuff. I think we have to get more of tools we're using. And what that means is going to unexplored places inside of the app and bidding ecosystem. So, you know, if you're at a big enough scale to get out of asc, try going a little bit up Funnel don't have to run reach campaigns, don't run view campaigns, but like get people to add to cart, make a custom event and then run to that. And you will get to be able to spend a little bit more money on meta. Do a holdout test, figure out what your incrementality bonus is. Right. That X multiplier, apply it to that, and now you have a new thing that's like you're reaching new people you wouldn't have otherwise. You have to get people into that funnel. Podcast have been crushing it for us all year. We use a great agency. You know, we're on all. We're on the Joe Rogans, the comedy podcast. Like there's just, there's more podcast space than ever before and they all have a ton of ads on them. So like you can just, you can just jump in there.

[C]

Are all your pot. Wait, are all your podcast ads the actual host reading or are you recording like pre record? Are they like host reading it?

[A]

It's basically 90% of host read, but I think there's some that we have to do auto injection. But, you know, it's like, it's like, you know, name any comedian. We're on those podcasts. Also political shows. We're doing political shows on both sides of the aisle. So yeah, look, man, try to sell wallets any way we possibly can.

[B]

Yeah, you know, you're doing it.

[A]

Another place we're seeing a lot of good results is spending money on Twitter. They have.

[B]

Do you still call it Twitter or you say X?

[A]

Oh, yeah, I still wear Twitter. X, whatever you guys want to call it. They have a new head of product, Nikita, and he's like rebuilt the ad platform and the reason why it's so good is they'll give you a bunch of matching ad credit. So, like if you spend $100,000, they'll give you $100,000 in ad credit. So your money just goes twice as far. And anyway, you can see really good results there. Now we have a very male focused product and the app is probably more male focused than other people, but it's worth trying.

[C]

It's not a male focused app at all.

[B]

I have a funny story for you. I was at dinner with Sean and actually Jeremy Kitsch and we were in Beverly Hills having dinner. And I was looking on my slack and I saw the. The team, the Ridge team and the Norpeam team were talking about some weird numbers on X. Like there's a bunch of conversions that happen in the last hour. And, you know, the teams work hard. It was like Friday or, you know, at like 8pm or something, or whatever, and. And they're like, where's this revenue coming from? And it turned out Sean had. Had posted on X and it sold a bunch of wallets.

[C]

So is that the one about the wall? The.

[B]

The.

[C]

The guy that you sent the suitcase to? That was the. Is that the post you're talking about?

[A]

This, this. This was like a year ago or something. Andrew Tate retweeted me and sold a bunch of wallets off of it. But. All right, guys, look, this is a great episode. I hope you guys, listener of this show, you're doing 2, 5, 10, $50 million a year. You have a couple tools with you, right? You're going out into the ocean and like, you start off in the kiddie pool. You don't need anything, right? Then you go a little bit deeper, maybe bring your snorkel, go a little bit deeper. And that's where you start getting to. Mmm. You start getting to incrementality. You have to test these channels and you have to. The biggest expense in your business should be how much you spend on marketing. I have a 2xmer. Half my revenue goes to marketing. 7% of my revenue goes to people. So it's the biggest thing we spend money on by far. So if you can get a little bit better at it, you'll save a bunch of money and these tools hopefully help you do that. So, Austin, thank you for being here. Thank you for being a sponsor of the operators podcast. Katie, thank you for being the only operator who shows up to record these with me. So I appreciate it and it's always a pleasure talking to you guys. We'll hit a freeze frame real quick.