Why Great Marketers Think Like Data Scientists, with Eric Seufert

Summary

Podcast Summary: Marketing Operators – Episode 100: Why Great Marketers Think Like Data Scientists, with Eric Seufert

Introduction and Reflections on the Meta Summit

The episode begins with Cody Plofker, Connor Rolain, and Connor MacDonald warmly welcoming Eric Seufert to the podcast. The trio expresses their enthusiasm about Eric's insightful keynote at the recent Meta Summit.

• Cody [00:00]: "Thanks for joining, Eric. We met at the Meta Summit and I think right away me, Cody and Connor all looked at each other. We're like, we gotta get this guy on the podcast."
• Eric [01:13]: Reflects on his experience at the Meta Summit, highlighting the anticipation buildup and the value of an unscripted Q&A session.

Eric Seufert's Professional Background and Expertise

Eric delves into his career trajectory, starting from his analytical role at Skype to his fascination with the freemium business model. He discusses his transition into mobile gaming analytics and his eventual move into venture funding.

• Eric [09:50]: "So first of all, thanks for having me. I spent kind of the first part of my career just as an operator... And then I spun up like an investment syndicate that I ended up turning into a full-fledged venture fund."

Unit Economics in Freemium Mobile Gaming vs. D2C

The conversation shifts to the differences in unit economics between freemium mobile games and direct-to-consumer (D2C) brands. Eric explains the challenges in monetizing users in the gaming sector, emphasizing the reliance on a small percentage of high-value players, often referred to as "whales."

• Cody [15:06]: "What are the unit economics you see on the freemium mobile game lifecycle... what's the average lifetime value over 12, 24 months?"
• Eric [15:43]: "With mobile gaming though, the difference is 2%, 1% will ever monetize... you really need this small group of high-value players."

Advertising Strategies in Mobile Gaming – Discord as a Platform

Eric discusses his advisory role at Discord, focusing on the platform's unique advertising ecosystem tailored for gamers. He highlights the distinction between traditional display advertising and incentivized advertising, which offers rewards to users for engaging with ads.

• Cody [23:16]: "I wanted to ask about Discord a little bit because I see that you're an advisor at Discord advising their mobile advertising platform strategy..."
• Eric [23:43]: "The concept with ads is like how could you implement ads in a way that it's actually really beneficial to the end user."

Measurement Techniques: Deterministic vs. Probabilistic Measurement

A significant portion of the episode is dedicated to understanding the differences between deterministic and probabilistic measurement in marketing analytics. Eric elucidates how deterministic methods rely on unique identifiers for precise tracking, while probabilistic approaches use statistical models to infer user behavior.

• Cody [31:19]: "What are the differences between deterministic measurement and probabilistic measurement?"
• Eric [31:34]: "Deterministic measurement is essentially just this idea that you've got some identifier... Probabilistic is you don't have that certainty..."

Incrementality Testing and Media Mix Modeling

The hosts delve into advanced measurement techniques like incrementality testing and media mix modeling (MMM). Eric emphasizes the importance of using a combination of tools to gain a comprehensive understanding of marketing performance, highlighting the limitations of relying solely on any single method.

• Cody [55:16]: Discusses running geo lift studies and the challenge of applying incrementality results across different time periods.
• Eric [56:11]: "You need all of it. Like yes, use the Triple R and Northbeam, definitely... you need a broad set of tools..."

Creative Strategies and Signal Engineering for Advertising

The discussion transitions to the role of creative strategies in advertising. Eric advocates for "signal engineering," where marketers design up-funnel events that better correlate with actual conversions. He underscores the necessity of optimizing creative processes to generate meaningful user signals rather than merely focusing on incremental creative variations.

• Cody [72:02]: Shares experiences with optimizing quiz completions and product page views to attract high-intent users.
• Eric [76:02]: "That's the strategy now that's what marketers have to be able to do... signal engineering."

Discussion on Meta's AI Strategies in Advertising Platforms

Towards the end of the episode, the conversation touches upon Meta's investment in AI infrastructure for advertising. Eric provides insights into how Meta leverages AI for creative optimization and the implications for marketers striving to maximize ad performance.

• Eric [62:32]: "The big takeaway is just how much money they've invested into building that infrastructure... you could mistrust Advantage Plus but there's no way."
• Cody [85:14]: Reflects on targeting challenges and creative optimization strategies with Meta's AI-driven tools.

Key Takeaways and Conclusions

• Holistic Measurement Approach: Relying on a combination of deterministic and probabilistic measurement tools provides a more accurate and actionable understanding of marketing performance.

• Incrementality Testing: Regular and diverse incrementality tests are crucial for adapting to changing market conditions and understanding the true impact of various marketing channels.

• Signal Engineering: Crafting user engagement events that accurately signal intent can significantly enhance the effectiveness of AI-driven advertising strategies.

• Creative Evolution: Continuous and radical experimentation with creative strategies is essential to break free from local maxima and discover more impactful advertising approaches.

• Adaptation to AI-Driven Platforms: Embracing the advancements in AI infrastructure, as demonstrated by platforms like Meta, is necessary for marketers to stay competitive and optimize their campaigns effectively.

Notable Quotes:

• Eric [03:54]: "I really felt like sort of adamant about that. And to their credit... they did allow for Q&A."

• Eric [15:43]: "With mobile gaming though, the difference is 2%, 1% will ever monetize."

• Eric [31:34]: "Probabilistic is you don't have that certainty... it's around like, you probably heard the term like cultural determinism."

• Eric [51:33]: "There's no way you could do better... Just accept it. We're never going back to what it was."

• Eric [76:02]: "That's the strategy now that's what marketers have to be able to do. I mean, that's the job now is signal engineering."

Conclusion

In this engaging episode, Eric Seufert provides a deep dive into the intersection of marketing and data science, offering valuable insights into measurement techniques, advertising strategies, and the evolving role of AI in marketing platforms. His expertise, combined with the hosts' probing questions, makes this a must-listen for marketers aiming to leverage data-driven strategies to optimize their campaigns and drive meaningful results.

Transcript

[Cody]

Thanks for joining, Eric. We met at the Meta Summit and I think right away me, Cody and Connor all looked at each other. We're like, we gotta get this guy on the podcast.

[Eric]

With mobile gaming though, the difference is like, you can't really expect that you're gonna monetize someone right away because it's such a commodified space. You know, discord is particularly important. Right. There's a lot of gamers using discord and so it's a great place to reach them.

[Cody]

You wanted to chat about the astronomer CEO. That's. I guess the first lesson of episode 69 of the Operator podcast is if you're gonna have a fair in your company, don't go to a Coldplay show.

[Eric]

There is a very sharp distinction there between like what you'd consider to be like traditional display advert and what's known as like incent advertising or incentivized advertising, which is very much specific right now to gaming.

[Cody]

Eric, what's up? How's it going?

[Eric]

Hey guys. It's going well. How are you all?

[Cody]

We're good. Yeah, we're good. I think it, what is do is this episode, are we at 70 yet?

[Connor]

I don't think so. I am probably 68, 69.

[Cody]

All right, cool. Well, yeah, thanks for joining, Eric. Yeah, we, we met at the, for anyone who doesn't know, which is probably everyone, we met at the, the meta summit and we really enjoyed your, your punctuation mark to the meta Summit. And I think right away me and Connor all looked at each other. We're like, we, we got to get this guy on the podcast and, and see what we can do to kind of get him fired up like he was at the Meta Summit. So we appreciate you joining.

[Eric]

Yeah, that was fun. I mean I had to, I was like, I, I, there was an, there was like a pre event at like 8 in the morning and so I, you know, went to the gym at like 6 and then went to this pre event and then was just like sitting all day. And so there was just like this, this, this building anticipation throughout the day until my, you know, the, the, the, it was the final, the final piece of content. I didn't want to say that, but if, if, if you say, if you say so. If you say so. But that was like 3pm Right? So you know, I just had the whole day to kind of to get amped up. But that was, that was a lot of fun. I was really happy to, to have been invited into, into had that opportunity. It was, it was, you Know, it was a big crowd. I mean, it was what, 1500 people maybe? It was just a lot of people.

[Cody]

Well, I think the. You could tell, like, everyone at, at Meta was. They're, you know, they're super buttoned up. I think they clearly had like, a talk track and it was really great. I thought the information was really, really valuable. But it was kind of refreshing to end with yours where it clearly wasn't like, super scripted or anything like that. I mean, you probably had your questions ahead of time, but it was, it was refreshing.

[Unknown]

Some of them were sweating a little bit, I think.

[Eric]

I loved it. Well, so, so, so I didn't, I didn't. I told him I didn't want to get the questions beforehand. I just, I just, I just think it's, it's, It's. It just comes off as very stiff when you see, like, a fireside chat. And it's, It's. It's obvious that they, they'd rehearsed it, right? And like, I think just whenever I do any sort of, like, public speaking, I just, I just really like it to be ad hoc and, and you know, because it's just. Just the, the other thing is just like, it's my. Essentially my job to like, follow this space, like, minute by minute. And so if we script something out a month in advance or even a week in advance, I mean, we can't talk about whatever happened that day, which might be, you know, just. Just top of mind for everyone. So I think it's. It's always better for me to, To. To. To just kind of go in without any guardrails. And, and then the other thing was they really did not want to do Q A. They were very opposed to it. I really had to push for it. I. Because I just. I just felt like that adds so much more like it. It. It adds such. Such a sort of genuine edge to, to the actual event. If, like, someone. Not that, like, I have like all the answers or whatever, but just, Just the fact that they, like, they got to participate, right? Like, if I get to go in this room and there's 1500 people and I wasn't just an audience member who was, you know, who was communicated to, but I actually was part of the conversation. It's just such more. It's. It's so much more fulfilling, you know, as, As a. To. To feel like a participant. So, So I, I was really like, sort of adamant about that. And, and to their credit, which I don't think they do very often, they. They did allow for Q and A. And there were some good questions too.

[Cody]

Yeah, it was awesome. And we were in the crowd and it definitely felt more authentic and real. And so. Yeah. So, Cody, you had a. You wanted to chat about the Astronomer CEO, quick, before. Before we get into the.

[Eric]

Yeah.

[Unknown]

You guys see that? What is it? It's the 18th. It'll probably be old news by the time this comes out, but I'm sure everybody saw that.

[Eric]

What a.

[Unknown]

What a funny day he was on. It was on the. The COVID of New York Post as well. So it's like mega viral, like, everywhere.

[Cody]

I watched the video like 10 times in a row because it's just so funny watching him watching both of them realize they were on the Jumbotron and he just ducks below the. The edge of it.

[Unknown]

And like, if he didn't react like that, it's just. It probably wouldn't have been anything. The reaction killed him.

[Eric]

Right. Because people, they, they. They had to dig in to find, like, why did that guy react that way? I mean, if he had just been natural. I mean, it. Maybe his wife would have found out or whatever. I was thinking how funny it would be if that was just a publicity stunt. Like, if he. If he's told his wife and it was like a planned thing and it was just a marketing stunt. That would be really. I would actually be really impressed if that. If that was true.

[Connor]

Oh, yeah.

[Unknown]

Their brand search traffic must have been way up. Yeah.

[Cody]

I didn't know what Astronomer was. Did you guys know before?

[Eric]

I had no idea. No idea.

[Cody]

Yeah. And that was just. Was that just like the. Was that just the big screen at the show? Like, I agree with you. I think if they just wouldn't have reacted, they probably would have liked, panned away and no one would have ever thought twice about it.

[Connor]

Totally.

[Eric]

Right. No one would have recorded it. I mean, they wouldn't have cared.

[Connor]

That's got to be such a painful thing to know, being him.

[Cody]

Like.

[Eric]

Yeah.

[Connor]

Like, it was totally in his power. Yeah. To have avoided this whole thing. Just life altering, to be fair.

[Eric]

It was in his power to avoid it. That in that moment of being caught on the kids, I mean, he had. He had control before that. He had agency before that.

[Unknown]

I think my favorite part about the whole thing were like, all the Coldplay tweets were like, you know, it's like, it's like all the tweets were like, the second half of it has a turn and you think it's talking about the affair and it's like the most embarrassing thing ever being at the Coldplay concert is so funny.

[Cody]

What kind of. What do you think the pushback is? Or like the. The. The. Yeah. Clap back on something like that. Like, does he lose his job? Does she lose her job? Like, what.

[Eric]

Or.

[Cody]

Or is it not that he probably.

[Connor]

Loses his job as public as he.

[Eric]

Was CEO and founder. Right.

[Connor]

He's not the founder.

[Unknown]

Oh, he's the founder.

[Connor]

I don't know.

[Eric]

He's not the founder.

[Connor]

No, he's just been there two years.

[Eric]

Oh, okay. He's probably gonna lose his job, then.

[Cody]

He'S out of there. Yeah.

[Unknown]

And so does the chief people officer.

[Eric]

Yeah.

[Cody]

Bummer.

[Unknown]

All right, well, I was thinking about it. I was asking this morning and nobody agreed with me, but, like, he probably loses his job. I don't know anything about him, but. Right. He probably can be replacement looks bad enough. Like there's probably a CEO out there who would keep their job. Like, if it happened to Elon. Elon was married. Like, Elon keeps his job. Right.

[Connor]

Elon would have never ducked. Ducked from the camera.

[Eric]

Yeah.

[Unknown]

He would have just started like fist bumping.

[Cody]

Yeah. All right, well, that's. That's. I guess the first lesson of episode 69 of the Operators podcast is if you're going to have a in your company, don't go to a Coldplay show.

[Connor]

Yeah.

[Unknown]

I'm looking forward to the TBPN episode with him on it today.

[Cody]

Right, Right. Yeah. Cool. All right, well, let's get into it. Before we do that, thank you to the sponsors, Motion, Pression, After Cell Rich Panel and House. Let's get into it.

[Eric]

Sam.

[Connor]

I want to talk quickly about Motion's AI Creative strategist. These AI agents are built by best in class DTC marketers, including Barry Hot, Jess Bachman, Mirella Crespi, Alex Cooper, and many others. And what's unique is that you get to use agents to analyze your creative using real data from your meta ad account. Connor, have you played around with these at all?

[Cody]

Oh, yeah, absolutely. Our team's deep in there.

[Connor]

I wanted to pull up too. So for those listening, we'll talk through it quickly. One, I think the implementation beautiful. You could see our top performing ad creative last week. What I pulled up was Jess's critique this ads messaging played that progress ran took maybe two minutes and then this is what we got. And I thought it was great. The good fast paced messaging, clean product demonstration, smart focus on MagSafe, what we can push. And I thought this was. This was funny. You know me as a marketer. I'm very precious about our Ad creator. But he's like, Apple's must have accessory is presumptuous and lacks credibility. The smartest wallet you'll ever own is empty. Superlative marketing speak. I'm like, okay, Jess. He definitely, he's definitely roasting the ad. And what I thought was really cool were some of the next steps. He says, reframe around the actual problem. This solves the daily friction of wallet phone management. So the AI has generated what this ad is about and actually reframed what the problem could be. And I thought that was really powerful. Two Iran Motions. Track this month's winning themes, and you see exactly what we're talking about all the time. Durability that lasts a lifetime. Ditching the bulk for minimalist design. Lifetime offers create urgency and cultural partnerships that expand appeal. And I'm like, yeah, look, you basically get what we're trying to do as an advertising business. I think this is really helpful for creative strategy and giving the team actionable insights that we can work on every day.

[Cody]

Yeah, this is saving our team tons of time. Motion is also giving teams an AI adoption cheat code. It's not like they're just throwing this new technology at you and saying, go figure it out. And they are really giving teams what they need to utilize this and get the most value out of this. And last but not least, you do not need a Motion contract to use this. You can try these agents for free, test them out, see how they fit into your workflows, and start getting value out of them without even needing a Motion contract. So if you want to try out some of these AI agents, go try them@motion app.com. so, Eric, we don't. We don't, like, want to spend too much time here. We want to make sure it's like, pretty tactical, but just like, would love to spend, you know, five, ten minutes just on, like, who you are, what's your background, what are you focused on on doing right now?

[Eric]

Yeah. So first of all, thanks for having me. I. I spent kind of the first part of my career just as a. As an operator. So I, I went to. I went to graduate school in, in Europe and ended up getting a job sort of right out of grad school at Skype. So I was, I was an analyst there before the acquisition by Microsoft. And I just became really fascinated with the freemium business model that, that they were using. Right. So Skype's business model was that the product was totally free to use for just VoIP, but if you wanted to call someone's number, like, if you want to actually call like a cell phone or a landline, you needed to buy credits. And so that was the only way they monetized and they added in more monetization over time. But like, when I joined, that was it. And I just thought it was really fascinating because this, this, I mean, this was a long time ago, right? So this was 2010 or 11. And, and this kind of coincided with, you know, just, just sort of like wide scale smartphone penetration, right? So like, even at that time when I was at Skype, I didn't have a smartphone. I had, I had. Or it was like a, it was like a, A mid phone. It wasn't a dumb phone, it wasn't a dumb Nokia, but it was. This is like 2011, but, you know, because the iPhone came out in 2008, right? So I had like a Nokia with a screen, but, you know, and there was some functionality that you could, you know, use it for other than just a phone. But it wasn't, you know, it wasn't like a full fledged smartphone. But, but so I, but I saw that I, I had the belief that, you know, everyone would own a smartphone at some point in time and freemium would be the way that software was distributed, right? You had the app store mechanism, you had everyone with essentially like a computer on them at all times. And, and so it would make sense for freemium to be the dominant distribution model because you've got the entire world essentially as your TAM for. For. As your potential tam, as the sort of ceiling of your TAM for any given product. And so why not just make it free, get the distribution to sort of be as extensive as. As. As is possible, and then figure out ways to monetize users based on their usage habits. Right. And so I saw, you know, Skype's implementation of freemium was pretty superficial, but I saw that there were a bunch of companies. I was living in Tallinn at the time. Talon, Estonia. That's where Skype was founded. And I saw a bunch of people, you know, across the Baltic Sea and in Finland that were doing really interesting stuff with gaming, with freemium. And so I got a job at a gaming company in Finland that was doing Facebook games at the time, but they ended up pivoting into mobile games and freemium, you know, was. There was kind of a question mark around, like, is that going to be the way that games are monetized or is this just sort of like a fad? And I mean, it turns out, you know, with, with. In hindsight, like it. It absolutely is. You know, Pretty much the, the, the, the exclusively how games monetize, but also a lot of other different products. And so, you know, I just got into. You started working in mobile gaming and I was working more in like the analytics side, but it turned out that, you know, the biggest analytical problem, the biggest analytical challenge that mobile gaming companies face. But, but also any of these sort of, like this, this sort of nascent group of freemium companies was user acquisition was just onboarding users, like figuring out how much user's worth when you onboard them, getting that information kind of as quickly as you can based on their usage habits and then using that to adjust the, the price that you pay for them in the form of a bid on a platform like Facebook. Right. And so I, I kind of moved into that side of, of the sort of analytics spectrum and you know, was doing user acquisition, you know, for, you know, freemium games across a number of studios all over Europe throughout like my, my, my time there. Then I sort of fast forward a few years. I was the vice president of user acquisition at a company called Rovio. They make the Angry Birds games. And I, I launched Angry Birds 2 and did the, the mobile marketing for the movie or my team did. And, and then, you know, I had had this idea for a long time of like building out a software platform for predicting cash flows, right? So if you've got, you know, your marketing spend and you've got, you know, your sort of like LTV schedule, you know, that's you could be spending profitably, right? And still go broke, right? If your cohorts are paying back in 90 days, 120 days, 180 days, you could be spending profitably with, you know, kind of significant margin on LTV to CAC and still go broke, right? Because you're waiting to recoup the money, but you're spending it up front. So I wanted to build the software platform that allowed people to sort of project out the cash flows to understand how much they could spend, right? Even if, even if they potentially could spend more kind of like with this theoretical ltv, CAC profitability, understand how much they can spend based on the cash constraint they face. And so I built that, just built it myself, solo device kind of style, and then it got acquired by another mobile gaming company in San Francisco. So I left Europe after like a decade, moved to San Francisco, worked at that company for a couple years, and then I just struck out on my own. I had been writing my blog, mobile dev memo for that whole time. And so I decided to kind of get more serious about it and focus on that. And then I spun up like an investment syndicate that I ended up turning. I ended up, you know, evolving into like a, just a full fledged venture fund. So that's kind of what I do now. I run the blog Mobile Dev Memo. It's got a podcast associated with it, it's got a community of, you know, thousands of, of marketers associated with it. Then I've got the venture fund Heracles Capital. And, and yeah, that's, that's basically it. That's, that's, that's my life.

[Cody]

What are the, what, what kind of unit economics do you see on like the freemium mobile game like life cycle? Like what, how many, like what's the percent of people that start free and end up paying like what kind of average like lifetime value over like 12, 24 months? Like what's, what's that look? I mean this is very, I think for all of our, for us three for sure and for our listeners, like such a different universe than you know, like I know, me and Connor, for example, like we're like very much first order profitable. Like I think we, we get most of our lifetime value in that first order. So I'm curious like what that looks like.

[Eric]

So I mean, I would say fundamentally it's, it's, it's, it's the same, right? Like fundamentally the way that you think about it is the same. The underlying mechanics and the details are different. But, but, but conceptually the way that you think about CAC to LTV is the same. No matter whether you're in mobile gaming, you're in mobile utility apps, you're in D2C, right? It's all, it's basically like you're just trying to map out that sort of expected value of the user, right? With mobile gaming though, the difference is 2%, 1% will ever monetize. Right now, now that changes a little bit when you're talking about ad supported games. Ad supported games, you'll get a broader, you, you, you'll get a sort of like a larger percentage of the user base that monetize just because they're right. But that monetization will still not, will not sort of recover the, the cac, right? What you really need in any mobile game is this sort of small group of high value players, right? People call them whales, whatever you want to call them. That's, that's what sort of accounts for the, the vast majority of your, of your user acquisition spend, right? Like that, that and, and, and they sort of just compensate for everybody else. And so, so that's the difference though. But if you think about like ddc, I've worked with plenty of DDC companies and yeah, to your point, like most of them try to recover the full CAC in that first order. And so it's all about driving aov, right? And so there's a lot more focus on you know, landing page optimization and you know, order page optimization and, and then you know, recovery. That's why everyone gets the, gets the email right away, right? Like recovering a bit card abandoners and that's fine but you just can't really do that with mobile games. Like you can't really expect that you're going to monetize someone right away because it's such a commodified space. No, no, not say D.C. isn't. But like the thing is when you go in a mobile game you have basically no idea what the product is. You need time to sort of acclimate yourself to the actual gameplay experience. And with, with a DDC product, I mean it's pretty obvious what that is like, it's, it fulfills some very specific real world use case with mobile games. You can't understand that and you need to have a moment, you need to have a moment in the game. It could be the first session, but it still needs to be a few minutes deep to understand what the breadth of this game is. How long am I going to be able to play this game? Is this going to fulfill my sort of gaming desires for five minutes or for six months or for two years? Right, because you see some of these games, they have cohorts that are 10 years old and that are still spending, right? And they still represent like, you know, a sort of meaningful amount of the overall revenue. And that's the difference. Like it's that longevity and it's actually the sort of the skew of where the vast majority of the revenue is coming from. It's not from any given user could be expected to pay back their cac. It's coming from. No, very few users are going to contribute anything like in a sort of numerical sense like out of any given cohort, but they need to sort of compensate for everybody else's cac.

[Cody]

Okay, and is this, is this mainly like people that are using the game? They can buy, they can make purchases to enhance their gaming experience like you can buy. I don't even know an example because I'm not like a huge gamer. But is that, is that what's happening? You're like, you're buying Things within the game, like you're buying coins or. I don't know what it would be. I. But is that what's happening?

[Connor]

Connor is the least online among us, so he's like, I'm enjoying him, like struggling through this idea of mobile gaming.

[Eric]

So yes, that is, that is the case. Right. So like, there's, there's, like, there are. So the way that kind of a modern, successful game runs its economy is through what's called like, live ops, right? So it's like this recurring content stream usually of like, events, like kind of tournaments and stuff. And so what you really need to do is get someone embedded in that live ops cadence, right? So, and usually that's through some sort of like, guild system. And it doesn't always have to be, but like, usually that's, that's through some sort of emotional or, or just, I mean, emotional kind of is, is, is maybe like a loaded word, but like, it's, it's through some sort of commitment or investment into the outcomes, right? And a lot of times this is truly like a social experience for people. And so if I'm with my college buddies or I'm with my high school buddies and we have a guild, this is the way that we socialize. Like, we live all over the country, but this is the way that we get together once a week and we socialize or we just keep track of it. Think about like fantasy football, right? Like, what's kind of the upper limit on what you'd spend on that if you're with your college buddies, right? Like, now imagine that wasn't. It's restricted to just football, right? And, and, and, and, and it was a yearly, you know, exercise, right? So like, you think about that and you sort of get a better sense of like, why people monetize the way they do. But, you know, that's it. It's, it's getting people into. It's, it's, it's, it's getting people into like, the habitual enjoyment of that content, and that requires publishing new content all the time. So like, the worst way to design an economy, though, is for it to be paid to win. Like, that's actually like a really bad outcome, right? Because then you actually discourage a lot of people from ever monetizing because there's going to be someone that they can never sort of spend more than, right? But if you, if it's all about sort of like concentrating the focus on this sort of like weekly cadence, then you can see that like, well, someone is investing in this as entertainment Right. This is, this is my form of social, socializing entertainment. And it becomes like a lot more sort of the cap on what they're willing to spend increases. Right. Because it's not seen as like this throwaway sort of impulse spend. And that's actually like a really bad way to design an economy. It's seen as like a true investment into my own entertainment. And like think about what you're willing to spend on Netflix. Think about what you're willing to spend on a ski trip. Like if you sort of think about this as maybe a substitute for those things, like maybe the cap is a little bit lower than that, but it's still quite, it's much higher than, oh, that was an impulse buy, like I, like I had at the supermarket and therefore it was wasted spend. People don't think of it that way. They think of it as something that they truly generated a lot of enjoyment out of.

[Cody]

Got it. That makes sense. Yeah. I was, I was listening to the, to a podcast with one of the co founders of CALM recently, Michael Acton Smith, and I was fascinated by the whole Moshi Monsters story. Are you guys familiar with that?

[Unknown]

No.

[Cody]

Are you, Eric? Yeah, yeah, yeah, that was, that was really interesting. There's a really good podcast on. I think it was how I built this with him and the, the other founder.

[Unknown]

We're about halfway done through the year. H1 is almost over. We're prepping for the second half of the year. We're also prepping for Q4 which is huge for us. And our budgets are going to be the highest then. And when our budgets are the highest, we're going to need the most granularity and the most confidence knowing where we should spend our dollars. And that's why we turned to Prescient. Prescient is an MMM and most MMS, they use 60 year old regression models. They're not really built for D to C. Prescient you can get readouts really quickly if something is changing, which again DTC is really volatile. So you can't rely on an outdated M that you're only gonna get one read out a quarter. One thing I love about Prescient so much, you're able to see the halo effect. So upper funnel spend, which is again is huge. We love to fill the funnel prior to a peak moment prior to Q4 holiday, something like that. We're really able to see the halo effect that has. Because sometimes you're not gonna see great attribution from YouTube campaign or from TV campaigns. Well, prescient plugs into all them and it can actually tell you your base plus your halo. So it's been really helpful for us to understand and actually have confidence to invest in some of these upper funnel channels that are harder to measure with other ways. So I love Prescient. I can't tell you all the technical stuff behind it, but I can tell you it gives me and my team more confidence. Know where we should put our budget, especially in some of these pesky upper funnel channels that are much harder to measure. And again, don't wait till Q4. It's going to be too late then we're halfway through the year but we've really got to all lock in for the rest of the year for Q4. There's a reason we use it. Hexclad Hollow stocks, coterie and 100 more leading brands so highly recommend checking it out. Go to prescientai.com operators to book a demo today.

[Cody]

Cool. Should we. I wanted to ask about Discord a little bit and because I see that you're an advisor at Discord advising their mobile advertising platform strategy, I was curious like hey, what does that look like? Like what's like the, what's the advertising ecosystem on Discord look like? And then what kind of brands are you seeing like Discord advertising work for? Is it mainly mobile games? Are you seeing other, other types of, of businesses advertised successfully with Discord?

[Eric]

Yeah. So I mean I can't say too much and that's not because there's like some restrictive NDA in place. It's more just like I don't believe I've contributed that much. I mean like a very capable team. They have an incredibly capable team. There are a lot of ex snap people and so you know where I can help, I'm happy to help but like you know, they're very capable on their own. But yes, I'm an advisor to their ads effort and it's so the idea here is like, you know, Discord is, is, it's, it's, it's, it's got a broad range of use cases. Right. But like gaming is a, is a, is a, is a, is particularly important. And there's so, so it's like a great way to reach gamers. Right? There's a lot of gamers using Discord and so it's a great place to reach them. And so you know, the, the, the, the sort of, the concept with ads is like how could you implement ads in a way? And this is like kind of how they pitched it to me. When, you know, when when the, the, the advisory board was being formed and they invited me to join. How could you build an ads experience that was actually really beneficial to the end user? Right. Like they didn't. And, and, and, and I think generally personalized ads are very beneficial to users, right? They, they sort of, there's this sort of stigma that's attached to them that like it's a, it's actually like you know, they're predatory or whatever, like invasive or they, they disrupt the, the, the, the core content experience. And, and to some degree that might be true in, in, in, in various circumstances, but like on the maintenance, personalized ads are a really good thing. Like they benefit everyone. They benefit everyone in the, in the sort of ecosystem. They benefit publishers, they make more money, they benefit advertisers, they reach better relevant audiences and they benefit users who, who get on oftentimes access to a product for free because it's monetized via ads, but they also get, you know, sort of more relevant products surface to them that they wouldn't otherwise discover. Right. So like it's sort of, it's, it's beneficial across the board. But how could you integrate that into the content experience to an extent that it's like, it feels like just a totally natural discovery mechanic. And that was kind of what was pushing me. And so right now the, the client base, the advertiser base is very broad based. It's not just mobile games. And in fact I think, I mean, I don't know this to be true but my sense is this probably skews more towards like PC desktop. And the idea is like how can we showcase these games to people and give them an opportunity to get rewards in those games or to, to benefit from seeing the ad. Right. And so that's, that's kind of like the core idea and you know, it's, it's going well. I mean it's still a very early stages but I mean I think you know, with that mentality it's going to be very successful.

[Connor]

I have a quick, quick question on Discord. One thing they were doing that I thought was really interesting for a while was like the gifting component which seemed like a really interesting way to position what felt like affiliate offers. Like at the time I think I could go to gifts and then I was, it was almost like you see Rock doing it now with after sale for D2C brands where post purchase I might get offered a hello Fresh K or something. Discord basically had that, but before but for more like gaming applicable products, that feels like A different form of advertising versus like, I think when a lot of our listeners and myself hear advertising, we think of like more disruptive ads that show up on my Facebook feed. And in Discord, that might just be a banner that's showing up above my voice chat, you know, and you don't have to speak specifically to Discord, but like, is that a good distinction to make? And do you see platforms moving towards both of them or is one more valuable than the other?

[Eric]

Yeah, so I think it is. There is, there is a very sharp distinction there between like what you'd consider to be like traditional display advertising and what's known as like incent advertising or incentivized advertising, which is, which is very much specific right now to gaming, particularly mobile gaming. But I think will become more pervasive. And the reason I say I think that'll become more pervasive is, I mean incentivize, incentivized advertising existed for a really long time and in gaming it was just a very direct path to driving a lot of installs cheaply. Right. So that, the idea being that like you install this game. So I'm in one game and I see an ad for another game and it's an incentivized ad. So what the incentive is is some gift in the game I'm playing currently. Right, right. So now you can understand that that's not very high quality traffic. Like the user is just installing the game to get the gift in the game that they're currently playing. Right. So there's not a lot of intent there. They just, they want to get the gift in the game that they're playing. And, and the whole idea there was like, well, you drive a lot of installs and you know, potentially get some, some uplift in the chart rankings. And that drives visibility. So that drives organic discovery. And that makes up for, you know, the, the, the, the essentially the CAC that's, that's not going to be recovered through these incentivized installs. So that's one form of incentivized installs. But what, what you're seeing now is kind of like, I don't know if it's secondary form or it's like a mutation or it's just an evolution. But you're seeing like real world gifts.

[Connor]

Right.

[Eric]

Accompanying the install, the install ad. Right. And so there's companies like Misplay that support this. So like they have these, these partners and these would be like retail partners. Right. And so it's like you install this game, you get to some level in the game. Right? And this is true for like the, the historical incent type installs too. You, you had to, you couldn't just install, you had to play through to some, some waypoint. But like you, you if you do up to this point in that game then you get some like sort of real world gift, right? That could be a gift card, it could be whatever, it could be some credit. And so that's really taken off. That's become a big part of the mix for most mobile gaming advertisers. I think it was, it's, it's honestly it's, it's kind of like this sort of like revolutionary change in the ecosystem and that was actually one of the ways that a lot of gaming companies recovered post att because gaming was like right, sort of at the epicenter of the impact of att. And this, this sort of allowed, this sort of drove a lot more you know, install spend volume, you know, just, just, just as a result of that real world gift being very compelling. Right. And so that, that, that allowed a lot of companies to sort of like expand their ad spend even because, because Facebook kind of became very challenging to scale a mobile game on Facebook post 80. So, so that might, I could see that becoming that penetrating past gaming. I genuinely see gaming as like the tip of the spear with a lot of different things related to advertising. I think gaming usually is like very head of the pack gaming and not to like, not To Like Denigrate D2C but like gaming is just, it's, it's, it's very, very sort of like analytically challenging. And so you get a lot of sophistication that goes into. And I'm not saying it's not true of D2C but like with gaming it's, it's true universally like in order to scale a game. I mean it's essentially like an analytics exercise in a data science exercise. I mean that's it. And so you know, a lot of the sort of best practices I think are you know, sort of like foment out of the gaming ecosystem and they sort of proliferate outwards into like other sort of like advertising heavy categories like dtc. But my sense is I could see that kind of like real world rewarded style ad penetrate into like into into other categories at some point.

[Connor]

Totally cool.

[Cody]

Sweet. Yeah, that's awesome. Well, appreciate the deep dive there, Eric. All right, so we wanted to, we spent a lot of time, I mean, I know amongst ourselves and our teams and on the show talking about measurement, you know, Talking about multi touch attribution tools, media mix modeling tools, you know, hold out testing, incrementality, you know, the whole gamut of tools and ways to test various things. So we wanted to, we wanted to go that direction a little bit here. So we wanted to talk specifically about, you know, at the Meta Summit Europe we were talking about deterministic versus probabilistic measurements and we kind of wanted to, you know, click into that a little bit. So I guess starting off just high level, like what, what are the difference between the two? Like what's, what's deterministic measurement? What's probabilistic? If you could give any like examples of each. Yeah, I wanted to start there.

[Eric]

Yeah. So like deterministic measurement is, is essentially just this idea that, you know, you've, you've got some identifier and you can be certain that anything attached to that identifier or any sort of like downstream event that, that emanated from that identifier, interacting with an ad in some way was, was always, was always a, attached to that identifier, right? So it's just like kind of like a certainty around the provenance of some chain of actions, right? Whereas probabilistic is, is, you don't have that certainty. You're sort of making assumptions about whether this thing was, was, was belongs to this, this, the, the person that interacted with the ad, right? It's, it's around like, you probably heard the term like cultural determinism, right? It means like, okay, growing up in my culture, like it sort of determines what I'm going to do. It, it like puts me on like a very specific like you know, sort of concrete path. Whereas, you know, like you can think of like a stochastic system. There's always like sort of random, randomness that happens at any given, any given waypoint on a path, right? So, so, so but just to, just to frame this within the, you know, advertising context, deterministic, you think about just having an identifier that you know, always belongs to some, some specific device or person, right? And so like anytime you see the identifier, you know that it's, it's, it, it, it is attached to a specific individual that could just be a device, right? So there could be somebody else using the device or whatever, but you know, it is, belongs to that device. Like when every time you see that, that identifier, you know it belongs to some specific device. Whereas probabilistic, you're, you're sort of like you're working with probabilities that it belongs to that person, that it was that same person. Right. So, so that's just the difference there. It's like probabilistic that you're working with. Like there's a probability sort of function here that tells me like whether this, whether it was the same person who did this other thing. Like. And you know, a lot of times probabilistic attribution is used as a synonym for like fingerprinting, which is not that sort of false. Right. Like that's, that's, that's a, that's, that's a miscast. That's. That, that's sort of like a sleight of hand that's just generally used by measurement companies. It's not really what it means. I mean fingerprinting is, is essentially just taking characteristics of a, of a, of a device and trying to match them throughout time. Right. And so that's not probabilistic attribution. That's, that's just, that's, that's pseudo deterministic. That's trying to turn a collection of things into a deterministic identifier, which you can't do. Especially, especially given the, the, the, the characteristics are often collected, which is based on IP address, which changes over time. Right. So deterministic is. I always know when I see this identifier, it's the same device. Probabilistic is I have some sort of probability function that tells me a probability that's the same person.

[Cody]

Got it. And what about, could you list some examples of, of like common, you know, attribution models or measurement tools that, you know, DTC marketers are using and for, for each.

[Eric]

Yeah. So I mean with probabilistic attribution, you're generally not working with like individuals. You're not trying to do that at the user level. You're trying to do it at like a cohort level or campaign level. Right. And so, you know, you could think of. And this isn't, you know, a media mixed model generally is like a linear regression. Right. But, but the idea would be it's just not like working with like probabilities per se. But you, you'd think about a media mixed model being an example of non deterministic measurement. Right. It's, it's trying to do associations between, it's trying to use the variation in spend and outcomes to determine how much of an impact a given channel has. Right. And so that, that would be an example and, and incrementality testing too. I mean it's, it's, it, it's not true. I mean, like, you know, the thing is, like, what the sort of, the idea that you would try to. So, so the end goal here would be to say, I'm gonna spend a dollar on this channel, it's gonna result in this much revenue, right? And like, and it's. And to truly determine, like, a causal impact, like, when I spend a dollar on this channel, I'm gonna get this much revenue. And so if I got this much revenue from that channel, it's because of the dollar I spent. Right? And so incrementality testing gets you close to that like this. And you can, you can implement incrementality testing any number of ways, but there's still a lot of uncertainty there. Right? And, and, and also, like, one issue with, like, just doing holdouts, for instance, is like, there are a lot of other variables aside from just the ad spend that go into the revenue, right? And so you can't. And you have to, you have to sort of, you can't just ignore a lot of the other, A lot of the other. Call it, you know, sort of characteristics or environmental, environmental variables that also influence the performance of your campaigns. Right? So think about, like, if I do a holdout test on January 3, does that apply to, you know, the middle of August? Probably not. Because I'm in the middle of Q5, right. Or sorry, sorry, December 28th. Right. I'm in the middle of Q5. Right. So that probably doesn't pertain to mid August. It's not germane at that point. So I'd have to have that as like a control. There's a variable there that accounts for. Okay, well, AM I in Q5 or not? You could think about it as just like a, just a binary dummy variable. I mean, in Q5 or not. So that's the control for that. So, so the thing is, like, you do a holdout test, you have to account for a lot of other things. Or I'm doing a holdout test and my biggest competitor is shut off, spent. Well, okay, then that's also going to influence the performance. Right? And so I think that's one mistake that people make with, you know, just, just doing holdout tests. Is that it, that, that, that measures the, the, the sort of, that, that the correlation at a point in time, and that point in time actually might contain a lot of information. And so you actually need to control for that too. It's not just, well, we shut it off and the, the performance dropped by 20 and we had. It was only 10 of our spend and so it was disproportionately contributing to our performance. No, you don't know that unless you control for all these other things, right? And so, and the other thing is, like, a lot of these things, you know, tend to be just, just there's a lot of correlation across them, right? So like, you know, it, you can't, it's very difficult to like, isolate the specific impacts. That's one of the problems with mmm and it's definitely a problem with experimentation. It's not that these are necessarily flawed tools. They can be very good tools. Right? They're very valuable tools. It's just the thing is like, they have to, they have to sit within sort of like a basket of tools and you need to be able to sort of interpret them. And there's some level of common sense that has to go into that. Right? And I think, like, a lot of times people sell solutions as like, this is all you need. And no, it's never all you need. Like, what you need is a basket of tools, an ensemble of tools, and you need the ability to interpret the output. And that's why I think, you know, especially now, like, marketing really has become like a, at its core, like a data science exercise. It's a data science discipline and you need that expertise internally to sort of be able to interpret the output. And just any given tool is not going to give you that. You need, you need a broad set of tools that all sort of like provide a sort of a bouquet of, of. Of information and you need to be able to decipher that bouquet. And it's not just any given tool. Can't, can't handle that for you on its own.

[Unknown]

If you know me, you know how cheap I am and how much I love saving money. This year is all about cutting costs and staying lean. You see me talk about it on Twitter. I cannot stand watching great D2C brands, even mediocre ones, get ripped off by SaaS vendors in 2025. So many of them are charging way too much, over promising and under deliver. I can't stand to see it, so I don't want to see it happen to you. I absolutely won't have it happening to me. You've also seen that I'm all in on AI, and both those reasons are why I love Rich Panel so much. It's about half the price of the beautiful software that we used before, and it was built specifically with AI in mind. We made the switch to Rich Panel right before Black Friday last year, and I almost regret not switching sooner we had zero downside. The implementation was pretty immediate, which actually shocked me. I was a little worried about it. Not going to lie, but it just went off without a hitch. We saved money. We leveled up our CX at the same time. So here's our numbers. We cut our CCX team from 18 to 10 agents. Not before Black Friday. But since onboarding to Rich Panel, with our order volume staying the same or even going up, our CSAT has stayed rock solid at 4.2 out of 5. I'd like it a little bit better, but that's, that's stuff that we can do. AI is handling 70% of the tickets, which is awesome. So our ticket to order ratio has gone way down from about 40% to 14%. So we have fewer full timers. Our revenue employee has gone up. We don't need as many people to get back to people and able to be a little bit more strategic with where we allocate our resources. So if your current CF CX software needs more people, is more expensive than you want to pay, and has some janky AI solution, I highly recommend that you check out Rich Panel. Rich panel guarantees a 30% ticket reduction in the first 60 days or your money back again. It's super easy to switch. They take care of data migration, staff training, all that good stuff. So you can go live in under 14 days. So check out richpanel.com and tell them you heard about them from the Marketing Operators podcast.

[Connor]

I have a question because we talk about this a lot and I think it's a, it's an important point around. Let's talk about like a geolift study. What you are getting in that result is theoretically like an incremental, like an incremental impact at that period of time, at that certain scale under all these conditions. And that's why if you're running in December 28th to your example, you can't perfectly apply that, you know, result to your, to your performance in August. Now my question is, you talk about controlling for these other variables. When brands try to figure out how to utilize these tools over time, do you think we should be focusing on controlling for those variables? So that Q5 test might be valuable to us in August or should we just be running more geo lift studies throughout the year on the same channel? Because the last thing that I would say is when I got into geo lift testing two or three years ago, it was almost first position to me is like test all your channels individually. Now you have the results, now you can go test other stuff. And our experience so far has been those results aren't applicable throughout the year. So we should actually just be doing more redundant testing throughout the year. I found that helpful, but maybe we should be spending more time actually trying to control for those variables to utilize those tests at different times.

[Eric]

Yeah, well, the other problem is there's interaction effects, right? So the fact that I'm running on five channels might actually influence the performance across the board. Right. Or on those channels versus if I was only running four. Right. There's like a halo effect. Right. And so that's the issue. It's like, and I mean I think it's like as I, you know, I think it'd be really easy to like be listening to what I'm saying and think that's like really complicated, it's really complex and it probably sounds that way, but I think the idea would be that you could just run this stuff all the time and you could instrument like a workflow that just always applies some sort of holdout. And in that way you just have to sort of be able to interpret like what do these things mean? And because it's like, to your point, it's like about that. Well, I'm doing the, I'm doing the GEO lift test, I'm doing the GEO holdout test in, in August or whatever. And like what I'm learning is in August across these channels, cutting off spin on one channel results in X drop relative to when I had it on in those two countries. So it's like, it's also like there's also the GEO component at the that right? And like, so a lot of times I would do geotest with Netherlands and Belgium, right? Because they're like similarly size, similar GDP per capita. But it's like, okay, but that only really pertains to the Netherlands and Belgium. And like, and so like if you think about like well what is, and then here's where the common sense comes in. What then could I test that potentially could be valid across the EU or whatever? And it's probably not. I mean, you know, maybe the ad spend, but like if you're, if you're talking about like a lot of times you, you might do things like like app store experiments and stuff like that. It's like, well maybe those apply but like some things just don't like, you know, some, some new product for instance. Like, well I, I, how do I know whether tastes are applicable in the same way in Belgium? Right. You know, as, as France or as the uk? Like there's very different cultural sensibilities There relative to like a product, for instance. And so I think there's just got to be some common sense that goes into this too. It's like, you know, that you have to kind of be able to interpret these results and say, like, does that make sense? Does that make intuitive sense to me? And like that, that maybe in a lot of ways, like, you know, there's a lot of people on LinkedIn or whatever that will kind of, you know, they, they tend to try to promote that there's like the one true way and it's all based on like some tool or some method. And, and like, I just don't think that's true. I think there's also like, there's, there's got to be some sort of like level of comfort with saying that just doesn't make sense. Like that can't possibly be true. And that's like one. You know, I, I wrote this article a while back and it was, I guess it was kind of controversial. I said, look, yes, you want to be doing incrementality testing. Yes, Incrementality testing is the gold standard. That's what everyone says. Anytime incrementality is brought up, someone calls it the gold standard. Yes, you should aim to do that. But if you're spending your first dollar on marketing, what else could the results be coming from? Are you serious, are you seriously trying to tell me that your first dollar of ad spend, right, is, is actually not producing the results you're seeing, but it was some random latent pent up desire to search for that product that just got unearthed as a result of you launching the thing? Like it can't possibly be that. Right? I understand that you could be leaving money on the table and it may not be the most, but just use some common sense. What else could be driving my results? We just started spending, we're only spending in the US we just started spending, we're only spending on Facebook. What else could be causing the conversions that we're seeing? Are trying to tell me it's all organic discovery and we're wasting our ad spend like that. And it's like you, at some point you have to say, look, we need to move forward, we need to move forward. So we have to just take some things at face value and like, yeah, maybe there's some organic, there's some organic discovery that's actually that we're not accounting for in, in the way that we're evaluating our ad spend. And you know, we're, you know, there's, there's 10% of waste or whatever. But, like, just how, how could you explain it? How could you explain the results that you're seeing? Except for we started spending money on Facebook to promote the product?

[Connor]

Totally.

[Cody]

You know, that's. I actually had a misconception about this, and I. You've cleared it up a lot. Because I used to think of it as like, deterministic is just based on actual data and probabilistic is like trying to model out the future that. That hasn't happened yet. But that's actually not the case because you're actually saying that. I think you're saying that a holdout test is actually still probabilistic because you're not actually able to, like, you might have driven 1500 incremental conversions, but you're not necessarily able to like, map out one to one who those people were. Whereas, like, with a tool like Triple R North theme, I actually go in and like, I can actually click on the person's name. I can see exactly the path they took because they clicked an ad and the north beam's tagging them. And then we can actually follow that person over a longer period of time. Like, that's, that's deterministic because it's actually mapped to a single individual person and it has nothing to do whether or not it's based on like, actual data or, or data that you're trying to, like future data you're trying to model out. Is that, Is that correct?

[Eric]

Yeah. Yes. I mean, that just be like forecasting, right? Just. Just sort of projecting out whatever. But, but I think, like, the thing is about, you know, a lot of the attribution companies, and, you know, I respect these companies a lot and they provide a very valuable service. I wouldn't call that deterministic either. I call it pseudo deterministic because they're not actually working with deterministic information. Right. It's not an email or whatever, a device ID that is always going to be the same for when that device is being used. Right? This is. It just isn't. Right. And so they're, they're, they're used. There's some, there's some logic in the, in the background that's being applied to sort of say, okay, it's the same person. Right? But it's not. It's not. It's pseudo deterministic. Right? But yes, you're right. And so the thing is, though, I think when you get into this sort of space, there's a, There's a distinction here between deterministic attribution yes, it, it, you know, is there sort of like more reliably tying something to some individual, but that's not necessarily mapping the causal impact of a dollar of ad spend on the channel that they clicked to their revenue, right? Because that could have been influenced by a number of other factors. And so, like, I think even when you have deterministic attribution, right, Go back pre att, you've got the idfa, it's still a good idea to have some sort of probabilistic attribution that kind of sits on top of that, right? Because then. Because it's. Because things really, you're talking about last click. And the thing is like, well, okay, if I want to understand, yes, I got the last click on this channel, and so do I attribute all of the downstream revenue to that click, or do I say, actually, I have a way of measuring what the impact of my TV campaigns was, or my, my podcast campaigns was, or my influencer campaigns was on that click happening. And if I hadn't spent that money, the click was 30% less likely to happen. And so some of that credit needs to be distributed to those other channels. And those are those interaction effects I was talking about, right? And so, like, you still need a tool that kind of sits on top of everything, right? And gives you some guidance around all these channels that have no click, right? There's no possibility of ever tying a revenue directly deterministically to a podcast, right? Now, there's some ways that Spotify tries to do it, but in the reality is there's the kind of no deterministic link, but that could have an impact on my Facebook campaigns, right? And so I need to understand that impact because if I'm, if I just cut the, if I said, look, there's no clicks on Spotify, it's driven zero revenue, right? But if I cut it off and my Facebook revenue then drops by 40%, well, that was contributing a lot. And that made the Facebook campaigns more performant, sort of like in a sort of like cosmetic sense, right? Not really. The Facebook campaigns were always the Facebook campaigns, but they got clicked more often, more readily because people had heard this, the Spotify ads, you know, and, and therefore they were driving performance. And I need to account for that, right? And so, like, there's no, these are not mutually exclusive, right? And the thing is, like, what I, I mean, I think that the sort of framework that I've sort of settled into now, like when I talk to companies and help them sort of like build out measurement solutions, is like, you need all of it. Like yeah, use the triple beams in Northwest, definitely. You know, use your MMP for app installs, definitely. But you need other stuff on top of that because it's not ATT anymore. It's the diversity of the marketing ecosystem. It's the fact that you have all these channels that really can contribute meaningfully to performance. They're available to you now with self serve, you've got Roku self serve, you've got, you know, you've, you can buy Netflix through TTD. You could buy Netflix or DV360. You know, you could, you could buy, you know, any number of CTV channels. You could buy Spotify. They just launched a Spotify Exchange Saks, so Sax Spotify Ads Exchange. And I told them that like when they do their can party that needs to be called Sax on the Beach. But like, so you could buy all these channels that have no deterministic component, right? And they're, they're, they're incredibly scaled channels. They should be very, they should, they should be a part of like, you know, any scaled marketing teams mix. But you will never be able to measure them deterministically. And so I call this like, you know, there's an instance of Wittgenstein's ruler. This is, you know, Ludwig Wittgenstein was a philosopher in this, in Nassim Taleb came up with this idea of Vicket size ruler. And he said, look, anytime you use a ruler to measure the table, you're also using the table to measure the ruler, right? So you have any sort of measurement apparatus, the only way for you to sort of interpret that, the, sort of the, the, the, the, the, the, the data that it's giving you is to have full, sort of, have full trust in its sort of ability to measure. And so the thing is, if you use deterministic stuff where you don't truly believe that it, it, it's, it the, the values that it's spitting out to you are actually perfectly defensible, like with total precision. Then actually, you know, you, all that's telling you is that your measurement is not precise, right? And so the thing is like, you know, there's, there's kind of a hammer nail syndrome here. And so like if all you've got of a hammer, everything looks like a nail, if all you've got is sort of deterministic infrastructure, then you can only really use deterministic channels. But if you embrace the sort of like probabilistic approach, then you can measure everything and use those in conjunction with the deterministic tools, they're not mutually exclusive. But you can't use those tools to measure Spotify. You can't use those tools to measure ctv. Right? And those are really important channels.

[Connor]

Now if you're listening to this podcast, I know that you're working hard to drive revenue with upsells and across your E Com site generally. But if you're not monetizing your post purchase experience, you're leaving easy profit on the table. Most brands and everyone listening are optimizing the transaction journey. But stopping at the thank you page, and that is a big mistake. After Sell is changing that. We've been using their Rock thanks feature at Ridge and it's a no brainer for us. Here's how it works. You show premium non competing offers from brands like Disney plus and HelloFresh on your order confirmation page. AI driven personalization picks the best offer from hundreds of reputable advertisers. And brands like True Classic, Blendjet and Jones Road Beauty are already using it, seeing 30 to 70 cents lift per order. No disruption, no fees, just extra cash in the bank. And the numbers speak for themselves. After sale customers generate 20 to $30,000 in pure profit per 100,000 orders. There's a 16% average engagement rate which is way better than traditional ads. And it's a new revenue stream without disrupting the checkout. It's an easy win and it's completely free to try. Plus After Sale is hooking up marketing operators, listeners because you can sign up today and you get their entire upselling suite for a year, which we also use very heavily. So visit after sale.com/operators to try rock thanks and see the money rolling in. Stop leaving money on the table. Turn your thank you page into more revenue with After Sell.

[Cody]

So, so what is a. I want to get into like this because we've kind of been circling like the basket of tools. I want to get into that. What's a, what's an example in E commerce of like a pure deterministic way of measuring or a purely deterministic channel?

[Eric]

Well, the problem is like none of these things were ever purely deterministic, right? It just never was. I mean so the only real way.

[Cody]

So much model data.

[Eric]

Is that why there's, there's so much model data and a lot of it's based on cookies and you know, which, which may may or may not be like totally reliable. I mean I guess the only way to do it, you know, would be to, to, to get someone's email at landing page, you know, and then have them, I mean, and then have the email on the advertising channel. That would be the only way to do it. Right, right. And then first party. Right. It's all first party. First party data essentially is the answer. Like first having first party data that ties across not just the, the, the, the public, you know, the advertiser's landing page, but also the publisher's site. Right. And that was, I mean, I guess historically for app installs only, that was the idfa and that was deterministic.

[Cody]

Got it, Cody.

[Eric]

Yeah.

[Unknown]

I find, I find people will try so hard to try to attribute all of these things. Like for example, we had our, our Google reps were trying to get us to spend more, of course, and they're looking at like North Beam data and in platform data and like looking at probably like 8 to 10 different data points to try to like triangulate things and make a case and be like, this is why you need to be spending more. This is how this channel is performing. And it's like, it's, it's. Again, you're not able to properly do this with like full certainty. And there's so much data at that point and it takes so much time and there's so much, there's so much noise and I'm like, guys, let's just run a holdout. Like, we can settle this really quickly. Like, rather than looking at all these 10 different data points, like, let's just run one test. We'll look at one thing. Is it causal? Do we make more money in the regions where we're running this test? And that's where I find it's almost like that like, midwit meme where like, you know, people get super complicated and complex about stuff. And again, I, we still use it, we use North Beam every day. So I think that there is a time and a place for it. But I also think it can get very overly complicated and lead to like, analysis paralysis. So that's why I've really been refreshed by just really focusing on incrementality.

[Eric]

Yeah, I mean, again, it's, it's. People call it the gold standard for a reason. I mean, it's, it's, it's, it's like the best way really to sort of ground your, your understanding of performance and, and you know, the sort of effect of spending money. But like, again, like, there's, there's challenges with that too. I mean, there just has to be some level of common sense here. Like to your point, like, like, let's run holdout Tests. Why are we, why are we trying to, you know, look at this basket of, of of things that may or may not be trustworthy and, and package them up into it kind of reminds me of like, you know, the, the gfc, right? Yeah. Like the mortgage backed securities where it was like, you know, triple B tranches that once you package them together. Aaa. Right. Well, not really. It's still kind of junk. Right. But it's just this idea that. Well, no, we've diversified. There's a lot of have triple B now it's like well, okay, no, but that doesn't change the underlying quality of the mortgages. Right. And so it's like, well, we got a lot of data points that we parse from the, the user's browsers. Like well okay, but how reliable are all of those?

[Connor]

So I've got one question here and then I know we want to go deep on like some meta stuff. Cody, Cody's. Cody and I have been trying to diagnose our meta issues but we've talked about a couple things and I'm trying to think from a, from a listener's perspective what I could take away. One Love your point on common sense. You don't need to run a whole that if you're running one channel like let's just.

[Eric]

On one geo.

[Connor]

Yeah, yeah.

[Eric]

For the first time.

[Connor]

Totally, totally. So, so we. I like that point. I like the point earlier around frequently doing incrementality holdouts because you have all these changing variables over time. And then you talk a lot about, you mentioned operationalizing. So I'm thinking if. Oh. Because the other thing that I'll say at the meta summit, I believe this was your, your keynote. You asked who had operationalized incrementality or probabilistic measurement like experiments and we maybe had five hands up in like the whole building. So like I don't think anybody in E Comm is doing that well and I'm curious if you have any other advice for people like just beginning how, how that might look.

[Eric]

Well, I'd be remiss if I didn't plug one of my portfolio companies as incremental. And so like that is the whole, that's the whole sort of value prop of the product. It just, it does, it looks at these variations all the time. So it's kind of like always on.

[Cody]

Right.

[Eric]

Incrementality testing. But I would say that, you know, it's just a function of like constantly running these experiments. Right? Yeah, these are. The thing is like this is not something it's not Passive. It's not passive. It's not just going to happen over time. Like you have to actually instrument it. And so it's like just being dedicated to like, okay, we can do experiments, but a one off experiment is good for one off information, right? Like, it's good for that moment in time. And, and so, you know, if you want to be understanding like kind of in real time what the effect sizes of, or what, what the sort of like effect of, of the spend is across the board, like in all these different, you know, sort of sets of circumstances, then you need to be testing that. And so it's just, it's just, it's just a function of like instrumenting those experiments like all the time.

[Connor]

Okay, so one, one example that you kind of hit on running a geolift study, let's say I get a readout or. One thing Connor Rowland and I have discussed is in finding that relationship between the incrementality readout and the deterministic MTA data because as soon as that incrementality study ends, you no longer have that information. You do have the deterministic or MTA data on an ongoing basis. Do you think that is a reasonable way to try to action on the geo live study earlier?

[Eric]

Yeah, absolutely. So the way like my preferred sort of framework here is like, I think about it is like concentric circles that just operate on different cadences. So like you're running day to day in Facebook Ads Manager, whatever and you're seeing click through rates, you know, you're seeing, you know, cac, which is models, right. But like, you know, you're seeing, you know, roas and then those are the operational metrics you use day to day. I mean that's just what I'm, what I'm optimizing against. That's the data I have. And then you've got some other concentric circle that might be like a medium mix model. Right? And that's telling me, okay, well when the, you know, given some historical bounds of AB of ad spend, I know that like allocating this much to this channel and this much to this channel, this much is, is, is, is, is optimal. Right? And then you've got, and that might be like a weekly cadence that could be bi weekly. I mean medium mixed modeling is a little bit tough to do weekly, but you could do it bi weekly, right? And then so day to day I'm just operating against the channel information. Then on a sort of like less regular cadence, I'm changing the budget allocation and then you've got like These kind of tactical experiments that you're running on an ad hoc basis potentially to sort of understand like, hey, let's outboard our new channel or let's cut off this channel completely. Right? Like, let's not just say it should be 10 of span, let's say it's 0% of spend and look at what happens there and then just kind of recalibrate, you know, the, the sort of, the sort of budget logic that way. So it's, it's that. That sort of. Those, those concentric circles are, are the sort of the, the, the, the. The. The kind of like the, the gears totally that sort of interplay to, to form like a holistic sort of like marketing measurement apparatus 100.

[Connor]

And I think that, I think that's a great answer and like that's what we're trying to solve at Ridge is we have all these tools available to us. How are we piecing them together over time and for different use cases?

[Eric]

So cool.

[Connor]

Love that answer.

[Eric]

Yeah.

[Unknown]

Connor, I think like, right, like getting an incrementality factor and be like, all right, like we think meta is you know, maybe 1.5 times more incremental than like North Beam Michelle or something like that. Like the way that we'll look at it because again, it's. I know that you can't generalize to a different time period, but like we also can't run a holdout on that channel every single week. So we'll be like, all right, like if my Meta CPA, you know, drops by 30%. Well, like that same incrementality factor I can, I can confidently assume is similar, you know, so like that will just go with it. So it's like it might always be 30% higher or below and like based on how either our meta or like some type of attributable performance is, like, that'll swing. I don't have. If you really want to be confident, you could test and compare incrementality factors at like different times and different seasonality. My guess would be you probably have a pretty similar one where like YouTube's always going to drive more views than a click. You know, you might have a negative incrementality factor on a bottom funnel channel. And so like those will probably be pretty consistent year round would be my guess. Maybe you want to test it in like a promo period to see if you're getting any differences there. Like a launch.

[Eric]

Right.

[Cody]

Sweet. Do we want to. Cody, do you want to go into some of the meta stuff that you were curious about asking Eric About.

[Unknown]

Yeah, for sure. So unfiltered. You're pretty unfiltered at the summit, but you know, no, no meta people amongst us right now, so hopefully we get a little bit more. But now I listened to a podcast episode that you had. I forget who it was, but you had somebody from Meta on. I listened to it right after the summit because some people talked about it where essentially you went pretty in depth with him about just like the meta AI changes, Andromeda Lattice, Gem, stuff like that. So I just, I thought it was an excellent episode, had my whole team listen to it. But would love your take on like, like how you feel about the, the meta ecosystem right now in the direction of you're not necessarily the generative AI, but more of like the AI backbone and the delivery system. Yeah. So just would love, would love your general take on. On the direction of that stuff and how you think about it.

[Eric]

Yeah. So I think, yeah, I mean he was incredibly transparent in that, in that pod. That was Matt Steiner. He's their VP of AI infrastructure for ads. I would just, you know, I just not to. Not to be self promotional, but I would recommend people listen to it. I learned a lot. Certainly it was a good one in the.

[Cody]

Wait, what was that, what was that show? Where can people find it?

[Eric]

Mobile dev memo, podcasts. And then the, the guest's name was Matt Steiner. But I guess like the, the big takeaway is just, just how much money they've invested into building that infrastructure. Right? And like there's, there's kind of a couple takeaways for me. It's like one is, you know, if you think about. So he was talking about. So, so, so Lattice was like, okay, how can we basically do like transfer learning? How can we. How can we learn from things? We. How can we learn from like, you know, specific categories or specific types of ads and like apply that elsewhere. Right? So like, okay, that's, that's, you know, technical but like that seems like an obvious thing to do. Like Andromeda was. It was a different thing. It was like, okay, we've seen this massive explosion of the number of ads that people are using, right? And like, which is, you know, is, is. Is the case when you sort of build out the fully automated end to end kind of black box type thing. Then creative becomes like the sort of. This, the most significant lever to drive performance because everything. All the other decisions are made for you. Right? So you just have to kind of feed the beast. Right? Feed it with lots of creative. But also they started launching Gen AI creative, which is just going to obviously increase the amount of the volume of creative that people are using. And so it's like you've got this massive bank of creative and anytime they want to fill an impression in, they can't like kind of in real time, you know, distinguish or like sort of comb through this, this massive amount of creative that, that is targeted to that person. They've got to do some sort of like pre curation, right? And so that was Andromeda. They do pre curation of like the millions of potential creators they could use to fill that, to find like the top 10 best. And then they run the model for that is like a more precise kind of, you know, classification routine to determine like what the best one is to serve there with the highest probability of, of of delivering the outcome. And so, and, and, and the thing is like that was hardware optimized. I mean they worked with Nvidia to build hardware to, to, to, to, to, to sort of run that most efficiently. And so like, okay, if you think about that, you think about the effort that went into that and the resources that went into that and you say, no, I don't like Advantage Plus. I could do better. You're fooling yourself. Like you're kidding yourself. There's no way. I mean it's like you could, you could mistrust Advantage Plus. And I've talking about, I've talked about this at length, right? I've talked about the advertiser dilemma. I've talked about satisficers, regret. I've talked about the idea that like, I know they could be giving me better performance, but they're meeting my standard and that's. There's some regret there. I wish they were giving me maximum performance. I know that they could give me better performance, but I told them what my target was and they delivered on that. Right? And could I have done that myself? Probably not. And so it's like you're so, so in the, in the reality is like, like people overestimate their abilities here. I know that like, you know, probably catch some, some backlash for this because I, there is a lot of resistance to Advantage plus. Right? I would say it's like it's not even resistance because I think most people intuitively understand that they're better off with it, but I think there's like some animosity towards it because they still kind of wish there was total transparency and they still kind of wish the company wasn't trying to maximize its own interest, but actually trying to maximize their interest. Like a nonprofit. But unfortunately that's not the world we live in. Right. And if you look at the resources that were invested here, like it's, you just kind of have to accept it. Like it, it just is, it's. We're never going back to what it was. And actually the way things were relative to today was a lot less performant. Right. And so if you want to on board 10 million more advertisers, you need to it it it you need to invest in, in this direction, which is just to make it drop dead simple to press a button and reach customers that are relevant to you. Right. And like I talked about at the summit, like I just gotten my haircut the day before and like I was at my mom's house in Houston just visiting and you know, I didn't know where to get my haircut. And so I just went on Google Maps and found this like local place and it was like, it was really, it was just some guy's thing. And, and like that guy will be an advertiser at some point. Like that guy will be advertising to audiences, like local, like local audiences about his, his barbershop. Like that will be where we're at. It's like that next sort of tranche of entrepreneurs, solopreneurs, local businesses, they will be able to advertise because it's just so drop dead easy. And so I think it's like, it's easy to get hung up on this idea. Like, yeah, I wish there was more transparency. Do I? Yes, of course. Do I wish that, you know, there wasn't this sort of conflict of incentives. Like they, they want to maximize, they, they want me to maximize spend. I want them to maximize out outcomes and performance. And like, well, they're gonna win out on that. They're gonna maximize spend within the constraint of my, my, my ROAS target. That's just what they're gonna do, right? And so like I wish there wasn't that conflict, but there is, it's just the nature and it's the same is true with pmax. The same is true, you know, with Amazon's blanket on the name. The same is true with Pinterest Performance plus. The same is true with TikTok Smart Plus. I mean that's just the direction of travel here. And so like, I think it's easy to sort of of to be bitter about it. But the reality is this is obviously in the best interest of the platform. You got to hope that if the platform is, you know, prosperous and successful, that it's going to result in sort of like more downstream benefits to you with respect to like innovating even more into this, into the infrastructure. And so, but at the end of the day you just kind of have to accept it. And this goes back years and years. I mean this goes back like there was Google's initial like black box solution was called UAC Universal App Campaigns. It was only for app install, but they did this in 2018 and people were up in arms about it and it was like, okay, fine, but this is obviously what's best for them. And you know, it'll onboard a lot more advertisers because they don't need to have all this infrastructure to scale campaigns like the, the black box will do it for them. And so it made sense that they just essentially just expanded that out to all categories, but particularly ASC and you know, ecom. But it, I mean it, you could sort of like hate it all you want, but that's obviously what the future is. And so I think the, the sort of like more productive reaction is to sort of respond to it and to adapt to it and to build your own tools to sort of get the most out of it.

[Cody]

I keep hearing the same things from people lately. I'm being asked to do more with less. I have big goals to hit, but my budget is tight and every marketing dollar needs to work harder. I know I'm hearing this at hexclad. I'm sure a lot of operators are hearing this at their brands. Super relevant right now. Incrementality testing is the best way to figure out what's actually moving the needle so you can move around your ad spend in a way that's backed by science. All three of us use House for incrementality testing. We all love it. And House is now working with more than 40 of the top 100 DTC brands, which is pretty insane. I can speak personally for Hexclad. The amount of insights that we've gotten, especially on our view based channels and how those are driving impacts and efficiency and revenue for our business is not only super valuable, but only possible through House. You know, channels like Connected TV, YouTube, TikTok, these very view based channels that don't garner a click the same way that Facebook and Google does have been really only measurable through GEO incrementality holdout testing. And the actionable next steps from these tests in terms of deciding to scale up or down or keep spend where it's at is, is. Is really amazing. House helps you run experiments on your channel so you can Confidently answer the questions you've always wondered about. Things like what channels are actually driving my business? How much should I be spending on each channel? What's the impact of my ads on Amazon or retail sales? How should I structure my Meta and Google accounts to make sure they're spending dollars in the right way? The beauty of House is that it's built for marketers. The science under the hood is rigorous, but the platform itself is simple. So you pick a question, launch a test in minutes and get real results fast. Every customer is paired with a measurement strategist, someone who understands growth and brings a clear strategic point of view. They will get to know your business, help you build a roadmap of impactful tests and guide you as you operationalize incrementality in your day to day. Even if you're brand new to testing, you're not doing it alone. It is very much no in no way. Like go sign up for the platform and figure it out on your own. You have a partner in these customer success reps from House who's been in your shoes. And now more than ever you need to make every marketing dollar counter want up level your measurement with House by going to house I.O. forward slash operators, that's H A U S I.O. operators and start allocating your budget with confidence. So do you have any thoughts on. Because this is something that we us three talk about a lot and I think all three of us and, and a lot of other advertisers have even I, I don't want to say hacked the system, but kind of hacked the system by optimizing towards events that are not necessarily a business objective. Like I don't really want someone to come and just look at my product page. But what I found is that if I optimize for that I can actually get high intent people that will ultimately convert at some point because the, the problem we've seen is that just the ASC campaigns and just our, our ad accounts in general are a lot worse at reaching new people. And obviously if that dries up then we're in a big problem at some point down the line. So do you have any thoughts on, on like what, what's the right balance right between like, like using the technology tools that these, these cutting edge ad platforms are developing and making those work for your business and making sure that you're not like not reaching enough net new people to keep your top of funnel full and juicy and converting people over time.

[Eric]

Right. So, so that's, that's actually, that's the strategy now that's, that's what, that's what marketers have to be able to do. I mean, that's the job now is signal. People call it signal engineering. Right. So how do you make sure that you engineer top of funnel signals that are actually meaningful and actually do correlate with, with purchases? Right. And they're good proxies for value. And the thing is, because if you say just give me purchasers that, well, they've got a bank, they've got, they know who the purchasers are and they're going to continue showing your ads to the, the sort of more limited group of people. If you want to expand that group, you've got to optimize towards top of funnel stuff. But you got to optimize towards the top of funnel stuff that is correlated with purchasing. Right? That's the analytics exercise for the marketer. They can't do that for you. I think at some point they will actually. I, I, I genuinely believe at some point Google and Facebook will be managing landing pages for you and they'll be saying, look, that in that, that event doesn't tell you anything about their likelihood to convert. We've, you know, we've, we've got the sort of data to support that. Actually what you need to be surfacing is this event, right? And it might be something that you never would have imagined to be the case, but nonetheless is a good proxy for someone purchasing. And so that's, that's really the challenge. It's like looking at that entire onboarding funnel, you know, from the moment of landing at the landing page and testing things and, and, and looking at the correlations between like, you know, purchasing and doing this thing and actually, and then if you want the tail to wag the dog, it's building a landing page such that you are surfacing these moments of intent. Like how so think about, like if I had to build a landing page that basically tested someone, it was a test of their intent. How would I do that? And in doing that, finding events that were really good proxies for someone, converting later, you know, down funnel and then, and then sending those back, because as of right now, the platforms can't do that for you. But if you were being really like strategic about it, it's not just using the landing pages that you've always been using and the events that you've always been using and going up funnel or down funnel, it's building new landing pages that lean into this idea, that, embrace this idea that like, if I send them really strong signal and it's up funnel then that's going to be better for prospecting.

[Cody]

Yep.

[Unknown]

That's actually, that's actually so smart. So like we're all running some like mid funnel. We all struggle with this. We run a lot of our stuff because we're a beauty brand through like quiz funnels and so we've had success and we're testing right now optimizing for like quiz completion events. Connor from Ridge is running funnels where they're optimizing to view content which is a product page view but they're, they're having people go through a landing page first. So it's definitely more intense. But I still think we built these funnels more for a purchase objective and we didn't kind of reverse engineer it. I think that's actually genius. Just like build a funnel where you're getting a higher intent signal that's not a purchase event and then being able to optimize towards that, I think that's really smart. I know a guy, I talked to a guy who's got a mobile, I guess it's a mobile workout app and they have a quiz funnel and they've always found Bettis performance, especially on TikTok optimizing for some like not even somebody that quit that completes a quiz but like if you give a specific answer in that quiz like that was even how they're doing and like they always found better perform even on like a, an attribution level. They found better performance on those which I always thought was wild.

[Eric]

Yeah, I mean the quiz, the quiz format is the, is the sort of canonical approach but like there's any number of ways. Right. I mean you can imagine having like a really challenging captcha like you just for no reason, you just put it there just to make sure like, like this is what I'm talking about the tail wagging the dog. I'm going to create hurdles that the user has to clear that I will test as proxies for purchasing and like that that capture serves no purchase other than to tell me whether this user is really serious or not. Right. And you can think of any number of ways to do that. But like that that's. I think that's ultimately that signal engineering idea is, is kind of what marketers have at their disposable other disposal other than the creative which creates like a big, big component of it. Right. But if you think about like post click, you've got a blank canvas, you have any number of things, any infinite number of things you could do to sort of surface the best possible signal and send that back. The better the signal is, the better the platform is going to be able to reach relevant users. But the thing is, if you want to break out of that sort of, you know, like that, that, that sort of like local maximum, you've got to go up funnel. You've got to go broader and then, and that's how you do it. It.

[Unknown]

But with still keeping some intent most likely, rather than just going completely up funnel and like serving everybody.

[Eric]

No, no, no. That's the point though. You're testing those up funnel events for the likelihood of conversion.

[Unknown]

Yeah, that's genius. And, and, and because you have pretty good knowledge, at least I, I think of like the ad delivery thing. So again one, we are seeing that a lot. Do you also think, you mentioned creative. Like if you were us in our shows and you're trying to kind of break out of this local maximo, we're like, well, we're finding, we're spending more, we're reaching the same people. Our rolling reach is just not getting better unless we're going on purchase events. Like, like, is this just the, the AI system and these like relevance engines? If you have the same creative, it's just going to show things to same people over and over. And the only way to break out of that is just like massively different creative. Is that what you would do?

[Eric]

It could be. I mean, I like to shake things up like pretty significantly. I mean, I like to. I just, I just, my, my sort of approach is like just radical experimentation. Right. And so I think like, if you are finding yourself stuck in a, in a groove, then then yeah, just, just, just whatever you can do to sort of like break yourself out of that is, is. Is worthwhile. And so that could be like a total overhau. Could be a total overhaul of the creative strategy. Different, you know, different concepts. But I think you should be experimenting to that stuff all the time. I mean like my, my sense is like I wrote about this the other day and like people push back, but I, I still kind of believe it's true. The, the creative testing process is. Doesn't tell you anything about the underlying creative. You shouldn't care. You shouldn't care about what's in the creative. The process tells you how well the process is working. Working. Right. The creative performance tells you how well the process is working and that's what you care about. And so if the creative performance is bad, it doesn't mean the creatives are bad. It means the process is broken. You're not Producing enough stuff, you're not producing enough variation, you're not producing enough concepts. That's what you can't learn anything from any given creative. Any given creative working is essentially. Should be essentially random. It should be okay. We stumbled upon this random combination of things and it triggered someone's lizard brain, right? To. To click. And that's what you should care about. You shouldn't care about. Oh, well, I think what drove performance here was that the dog was in the right corner and the sun was in the left corner and the shadow was cast in this angle like this. And so all of our creatives like that should follow that pattern. That doesn't know that's meaningless. What you should care about is like, hey, that creative work, that means the process work. So how do we actually improve the process? We get even more output, we get even more variation. Right. And if we were in a. If we get into the steady state where the performance is always good, then that means the, the process is good. And if we're really lumpy, that means the process is not working. But I can't learn anything from any given creative that is essentially random. It's not information.

[Connor]

I would push back on that a little bit.

[Eric]

Lots of people. Lots of people. Yeah, yeah, yeah.

[Connor]

Like, I do, I do think marketers are extremely guilty of like over extrapolating random occurrences in the creative. That part I totally agree with. I have said that I think my toxic trait is I think we can just solve it with volume, where it's like, yeah, like better process, more creative. Let's get it in there. When in reality it does go down to more critically thinking about the creative. And I do think in that process we're more critically thinking about it. There are learnings to be had. Like, what are, who are we speaking to? What is the message is so key to us selling wallets to men that like reflecting on what those learnings could be and then putting that into the process of, of, of generating more creative. I think there's a lot of value to be had there.

[Eric]

There. Yeah, I, I think, I think that's, that's not necessarily so. So I don't necessarily disagree with that. Right. But I guess my point is like, if you are really embracing the black box, you have no control over who. Yes. And obviously probably show it mostly to men. I mean, it's a product for men. Right. I mean, you know, I don't know what percentage of women or your product sells women, but like, my assumption would be rich is mostly men. Right. So you've Got, there's all, there's like an, there's an automatic disqualifier there in terms of targeting. Right. And I think Facebook has learned that or Google's learned that. Right. I guess my point would be you'd have no control over where it's shown, like in terms of the placement. You have no control over the sequencing. And, you know, I, I, I, I get the point that, like, okay, I want to appeal, we're trying to appeal to men, and that's the whole, that's the whole sort of core demographic here. And, and how do we get into the brain of a man to sort of best resonate? I guess my sense would be that, like, that's like a deeply psychological exercise and it's like, maybe impossible, like maybe it's not impossible, but like, it's, it's definitely out of scope for me. Right. I'm not a psychology. I guess maybe if you had a psychologist on, on staff and he could get into how those synapses fire and you don't need to experiment. You just knew then. Yeah, just do that. But my, like, for me, I would just test everything totally. And, and then I guess, like, you know, which isn't to say like, okay, this theme continues to win. Well, there, I guess there's some learnings there, like broad learnings, right. Like, this theme continues to win. This theme continues to be. I guess the question is, what theme have we not tried yet that would actually outperform that? And that's why I still feel, I still think it's like, okay, well, yeah, we know something about that theme and there's knowledge there. But have we, have we sort of like, explored the entirety of the concept space? And if not, then that's great to know, but there's things we don't know that could be even more impactful for our business, and thus we keep experimenting.

[Connor]

Totally.

[Eric]

Where people go wrong is they think they're experimenting, but they're just producing variations of the same thing. Like, you know, that's not experimentation. It, that's just producing variance. Right. And so, like, what you're going to do there is you might optimize like a little bit. You might optimize at the margins because, like, oh, we got a different background color here, or we change the color of the, the person's shoelaces from blue to red. That's not really experimenting. When I talk about experiment, I talk at the concept level total.

[Connor]

And we talk about local and absolute maxima all the time. And what I think what you're Describing is like aggressively testing for the absolute maxima. If you find something that works, just try to test into something that's better. And I think people under index there significantly. But there is a bit of both. Like you're not always at the peak of that, the, the local maximum. Right. Like I think there are incremental improvements that could come from reflecting and analyzing what is working and continuing to improve there. But we're like splitting hairs here.

[Eric]

Yeah. And I also think like we're talking here about like high scale channels where it's very easy to deploy a lot of creative. I do think there's a lot of value in borrowing learnings for other channels. Right. So like think about like a Spotify, okay. Like if this ad works, let's just keep doing stuff like that. I mean there's no way to just do rad experimentation there. Like we're doing TV ad, you really only get one shot at goal, right? Like this. There's not a lot of opportunity to experiment there. So let's, let's take what we've, let's take what we've seen with the historical performance of these themes and say, well, okay, we don't know if this is the absolute best, but we're going to do a TV ad and it's going to cost us 100 grand. So let's just pick whatever has worked the best historically and like use that information to sort of craft the, the TV ad.

[Connor]

Totally.

[Unknown]

Okay, so you said one thing, you said you're talking about like Ridge and, and you know their demographic and like Google and Meta probably getting trained on that. So that's one thing we struggle with. I think we both do where it's like even when we produce creative that is trying to reach a different demographic. So like for us like we're mostly like 50 plus and we're really trying to reach younger people and even when we produce creative with younger people in it that we're speaking to them that we think is decent, like it still delivers to an older audience. What would you do to break out that like I think there's some signal engineering that has to happen as well that like the creative alone is not enough. Like what would you think about or if Ridge wanted to reach women and creating ads for that. It seems like Meta just gets very trained in a loop on like an account level as well.

[Eric]

Yeah. So there was a blog post that was going around about that some time ago, right. Where they had. It's. They, they saw that like they were just getting their ads shown to Older people and. Which would make sense, right, because they more disposable income, probably more likely to convert on any given E Comm. Product. Right. But it was just absolutely like the wrong demo for this particular product. You know, I think there, I mean, you, you got to figure out a way to break out of that, you know, that, that, that's, that that machine learning is from the saddle or the valley. Right. Like you're, you're stuck at some, like the, you're stuck at. You stuck, you know, at some unproductive, you know, gradient. And you got to shake things up. I mean, if you have to get young people to convert, once young people start converting, then. Then the system will start targeting them more.

[Unknown]

But you got to do enough because you can't just like you have to get some signal there and then get them convert, if that makes sense. Like you have to give it enough of a push.

[Eric]

Right. You got to get the stick, you got to get the signal. And so then maybe you just bid very aggressively and do creative that, you know, would just be absolutely, you know, sort of antithetical to any old person's sensibilities and, you know, and just try to break out of that. That saddle. Yeah.

[Unknown]

So things like bid multipliers, running specific campaigns when you test different ad accounts or you just like take a hypothesis and just like rapidly test pretty much all of it.

[Eric]

Yeah, yeah, I probably. I'm trying to. I'm trying to recall off the top of my head, like what, what the solution was in that blog post. I don't quite remember, but yeah, I was trying to do something radical that, that, that basically like precluded any old person click. I mean, it should be open, but they precluded whatever, you know, sort of like, you know, not target demographic from clicking on it. Yeah.

[Unknown]

So Connor, if you want to reach women, don't put Sydney Sweeney in your ads because that actually would probably backfire for you guys.

[Connor]

Agreed.

[Unknown]

Do you remember who. Where that blog was or who wrote it? I'd love to read that.

[Eric]

I don't remember. I think it was on. I'm gonna say I saw it on Hacker News or something. It was. This was like a. More than a year ago, I think.

[Unknown]

Got it.

[Cody]

Okay, sweet. That's a. That's a pretty powerful packed episode 100.

[Connor]

Eric, you have so much fun to wrap it there.

[Eric]

Yeah, let's do it.

[Cody]

That was really fun, Eric. Appreciate you coming on.

[Eric]

Yeah, cheers, guys. Nice to. Nice to be in touch and it was nice to see you all at the summit and hope to hope to stay in touch.

[Connor]

Likewise.

[Cody]

All right, that's a wrap on our episode with Eric Sufert. Thank you to the sponsors, Motion Prussian after Cell, Rich Panel and House. And as always, if you're enjoying the show, make sure to, like, subscribe and share with your marketing friends. We'll see you next time.