Season 7, Episode 26: Demystifying open source MMM (with Michael Kaminsky)

Summary

---

Mobile Dev Memo Podcast

Season 7, Episode 26: Demystifying Open Source MMM (with Michael Kaminsky)

Date: July 28, 2026
Host: Eric Suefert
Guest: Michael Kaminsky, Co-founder & Co-CEO of Recast

---

Episode Overview

This episode centers on "open source MMM" (Media Mix Modeling), specifically demystifying open source projects like Robin and Meridian developed by Meta and Google. Eric and Michael clarify misconceptions about these models, discuss their strengths and pitfalls, and provide grounded advice for marketers considering advanced measurement solutions. The conversation goes deep into the philosophy, practicality, and future of MMMs, particularly when used for mobile and digital-first advertisers.

---

Key Discussion Points & Insights

1. Introduction & Context ([00:37]–[01:50])

• Michael Kaminsky introduces himself: Co-founder/co-CEO of Recast, with a statistics, causal inference, and marketing science background.
• The discussion is sparked by a controversial article in the Mobile Dev Memo Slack: questioning whether Meta and Google (through open source MMMs Robin and Meridian) are “grading their own homework.”

2. What Does It Mean for an MMM to Be Open Source? ([01:50]–[04:53])

• Transparency & Verifiability: All code is open for scrutiny, modification, and use by anyone.
• Michael:

  "I can download all of the underlying source code for both Robin and Meridian, I can read every single line of it, I can change that source code, I can run it on my own data and I can see how the results come out." ([03:27])

• Meta and Google’s rationale: Increase transparency, allay fears of bias, unlike their proprietary reporting tools.
• Eric:

  “If you're going to make that claim...why would you make that claim against the open source library where all the code is just available to you to vet?” ([04:53])

3. Addressing Bias: Can Meta/Google “Grade Their Own Homework?” ([05:27]–[09:18])

• No Hidden Bias: Open source MMMs are model frameworks, not models pre-trained on data.
• Any claims of inherent bias can be tested (e.g., swap channel labels, observe no difference).
• Michael’s test:

  “Run the model with Facebook activity labeled and Google activity label to run it once and then run it again and just switch the names and you'll see that it's no different.” ([06:59])

• Why skepticism persists:
  • General skepticism is justified given ad platforms’ incentives.
  • Lack of understanding of open source and MMM mechanics fuels misplaced suspicion.
• Eric and Michael agree that alleged platform bias is an artifact of past proprietary models and legacy agency-driven MMMs, not these open source models.

4. Why Did Meta/Google Release These Open Source MMMs? ([09:18]–[11:19])

• Traditional agency MMMs often bias toward legacy media (like TV), not digital (FB/Google).
• Open source MMMs released as a “counterweight” to models that systematically undervalued digital channels.
• Michael:

  “...other media mix models...have been historically structurally biased against digital channels...Facebook and Google felt like, look, these MMM results keep coming back saying that Facebook and Google are really bad. We feel quite strongly that they're good.” ([09:18])

5. The Shortcomings of Open Source MMMs ([12:07]–[14:20])

• Complexity: The default assumption should be that MMMs are wrong.
• Causal inference from aggregate data is nearly impossible to get precisely right.
• Model results can be very unstable, sensitive to input settings, and often contradicted by direct incrementality experiments.
• Michael:

  “It is not able to find the true causal signal in that data. And that is the fundamental problem with MMMs.” ([14:08])

6. Last Touch Attribution vs. MMM ([14:20]–[16:07])

• For most small/medium businesses, MMMs can be net negative—introducing more noise than signal.
• Despite known flaws, Last Touch Attribution is often “good enough,” especially for direct-response, performance-driven businesses.
• Studies show Last Touch can be “85% as good as only doing incrementality experiments.” ([15:04])
• Michael (Spicy Take):

  "Most businesses do not need an MMM...an MMM will be net negative for them because it will actively mislead them." ([14:20])

7. Common Sense in Measurement & Decision-making ([16:34]–[20:03])

• Scale and complexity should dictate measurement sophistication.
• At low scale (e.g., spending first ad dollars), causality questions are less pressing.
• Debates about attribution and measurement often ignore practical business common sense.

8. How to Evaluate and Onboard Open Source MMMs ([20:50]–[24:49])

• MMMs are not interchangeable. Small config/data tweaks can drive vastly different results.
• Evaluation needs structured checks:
  1. Parameter Recovery: Test on simulated data where the ground truth is known.
  2. Predictive Accuracy: Can the model forecast future results when given historical data?
  3. Cross-validation with Experiments: Do MMM estimates align with known lift/geo experiments?
• Michael:

  "Everything else doesn't matter if the model isn't correct. If the model is going to mislead us...the dashboards don't matter. Nothing else matters because it's wrong." ([21:12])

9. Practical Feedback Loops and MMM Uncertainty ([24:49]–[29:13])

• Feedback from MMMs is slow, especially with long effect windows or minimal spend changes.
• Only large, sudden changes in marketing activity yield actionable insights.
• Organizations are often risk-averse—large budget shifts are rare, slowing learning cycles.
• Confidence intervals around channel performance are wide—uncertainty is intrinsic.

10. The MMM Vendor Ecosystem Explosion ([29:13]–[32:04])

• Technological advances and open source tools lowered barriers to entry for new vendors.
• Regulatory/tech changes (making digital tracking harder) increased demand for alternative measurement.
• It's hard for marketers to judge MMM “quality” (causal accuracy) vs. surface features (dashboards), so poor models proliferate.

11. The MMM Illusion and Incrementality’s Importance ([32:04]–[33:57])

• MMMs are oversold as “crystal balls”; reality is messier and more nuanced.
• Value comes from deliberate experimentation, not reliance on model outputs alone.
• Michael:

  "A lot of vendors in this space...have pitched MMM as like it's a crystal ball. You don't need attribution anymore. Just run an MMM and it'll tell you the true incremental value...That is just not true at all." ([32:59])

12. The Evolving Measurement Landscape ([34:45]–[37:13])

• Peak interest in MMM appears to have passed—now in a period of disillusionment and recalibration.
• AI-powered tooling has democratized data analysis, empowering non-analyst marketers to scrutinize their data.
• Incrementality testing—deliberate holdouts, geolifts, etc.—on the rise again as the practical gold standard.
• Michael:

  "AI happened in the last three years...it's going to be huge." ([35:05], [40:41])

13. AI and the Democratization of Marketing Analytics ([37:13]–[40:41])

• AI enables faster, deeper data interrogation—VPs/Directors can “do” data science, not just consume dashboards.
• This exposes faulty assumptions more rapidly, but brings risks if users lack statistical fluency.

14. CTV & MMM: The New Frontier ([40:41]–[46:10])

• CTV is now a “scale channel” for many direct response advertisers—triggers first MMM usage.
• CTV’s addressability (e.g., geography) allows for more robust incrementality testing vs. linear TV.
• Measurement remains challenging: Wide uncertainty persists, true incremental impact is fuzzy.
• Michael:

  "You're never going to know. Oh, is the true cost per acquisition from CTV $64 or $68? Like, you're just never going to be able to answer that with very much precision." ([44:43])

15. The Problem of False Precision ([46:10]–[47:38])

• Legacy tools gave a misleading sense of certainty (a single neat ROAS number).
• Real-world marketing impact is always a range, not a point estimate.
• Teams must “make beliefs consistent” with results and focus on outcomes, not panel metrics.

---

Notable Quotes & Memorable Moments

• “The problem is that the distinction needs to be drawn between the competence of the economists and the correctness of their analysis.” – Michael Kaminsky ([00:37])
• “An MMM will be net negative for them because it will actively mislead them.” – Michael Kaminsky ([14:20])
• “If you're going to make that claim...why would you make that claim against the open source library where all the code is just available to you to vet?” – Eric Suefert ([04:53])
• “You can always run an MMM and always get some number out on the other side. But how do you know if it's right? That's the hard problem.” – Michael Kaminsky ([18:33])
• “AI happened in the last three years...it's going to be huge.” – Michael Kaminsky ([35:05], [40:41])
• “A lot of vendors in this space...have pitched MMM as like it's a crystal ball...That is just not true at all.” – Michael Kaminsky ([32:59])
• “Your beliefs in terms of marketing performance should be consistent with the top line results that you see at the end of the month.” – Michael Kaminsky ([46:39])
• “I look at people who are, again, these, like, fairly small companies running these MMMs, and they're getting what effectively is just a random number generator out on the other side.” – Michael Kaminsky ([19:45])

---

Timestamps for Key Segments

• [01:35] – Michael’s background & intro
• [02:41] – What is open source MMM; why transparency matters
• [06:59] – Debunking the “inherent bias” myth with practical tests
• [09:18] – Why digital channels needed defense against old-school MMMs
• [12:07] – Core flaws with MMM as a causal method
• [14:20] – When MMM is net negative vs. Last Touch Attribution
• [20:50] – How to evaluate (and stress-test) MMMs
• [24:49] – MMM feedback loops, uncertainty, and practical inference
• [29:13] – Explosion of MMM vendors and industry confusion
• [32:59] – MMM overhyped as “crystal ball”; caution against snake oil vendors
• [35:05] – Evolving landscape: AI and experiment-driven measurement
• [37:30] – AI’s impact on the data-fluency of marketers
• [40:41] – CTV as a catalyst for MMM and its measurement quirks
• [46:10] – Perils of false precision; forecasting over attribution tables

---

Takeaways for Marketers & App Developers

• Open source MMM enables transparency, but it’s not plug-and-play nor bias-proof by default.
• Model outputs are only as good as the evaluation framework you use.
• “Quick wins” with MMMs are rare; chasing black-box outputs can do more harm than good.
• Common sense, incremental testing, and forecasting rooted in business reality outperform “shiny dashboards.”
• AI has democratized (and complicated) robust data analysis but doesn't eliminate the need for statistical savvy.
• Marketers should use MMMs as one tool—best deployed when complexity and spend justify it, and always validated by live experiments.

---

Where to Find Michael Kaminsky

• LinkedIn – Michael Kaminsky
• Recast YouTube Channel

---

[End of Summary]

Transcript

[Sponsor/Ad Host]

AI is changing everything, but it's only as good as the signals behind it. Branch connects customer interactions across paid, organic, offline email, web and app touchpoints and turns them into the trusted context you need with links and attribution that capture the full user journey. Learn more@ Branch IO and while you're there, check out Branch's AI search and discovery report, covering insights from more than 300 enterprise, marketing, growth and digital leaders to understand how the industry is responding to the rise of AI search. That's Branch IO.

[Michael Kaminsky]

The problem is that the distinction needs to be drawn between the competence of the economists and the correctness of their analysis.

[Eric Suefert]

Welcome to the Mobile Dev Memo podcast. I'm your host, Eric Suefert and I'm joined today by Michael Kaminsky. Michael welcome back to the podcast, Eric.

[Michael Kaminsky]

Thanks for having me. Happy to be here.

[Eric Suefert]

So, the occasion for us speaking, although I'd been intending to bring you back on the occasion for us speaking was an article that I shared in the Mobile dev memo Slack recently that was kind of questioning whether open source mmms and in particular Robin and Meridian were allowing Meta and Google to grade their own homework. Right. And so this kind of, I don't know, catalyzed a conversation and I just thought it'd be interesting to get you on, to get your take on that, but know also to talk all things. Let's call it open measurement. Before we get into that, could you please reintroduce yourself to the audience? Yeah, absolutely.

[Michael Kaminsky]

So I'm co founder, co CEO of this company Recast. We do marketing measurement, planning, analysis, including media mix modeling. I have a background in statistics, causal inference, econometrics, but have been really doing marketing science marketing measurement for the last 10 or 15 years.

[Eric Suefert]

Okay, got it. That was very efficient. Okay, so. So let's talk about Robin and Meridian because that was the source of this article and I'll link to it in the show notes so people can go read it if they want to get more context before listening. But what does it mean for an MMM to be open source? Why do you think some people view Robin or Meridian as an attempt by Meta or Google to grade their own homework? And why is that viewed flawed? I mean that was the point I made in the mold amendment with chat just to kind of not bury the lead here. That's a flawed view in my mind. It doesn't allow these companies to create their own homework. There's no baked in preference for Meta or Google, and there's no way for these open source libraries to artificially inflate purposefully, deliberately, artificially inflate the performance metrics of Meta and Google. But why do you think that view persists and what does it mean for an MMM to be open sourced?

[Michael Kaminsky]

Yeah, I think the question's one at a time, so why does the view persist? I think it's reasonable and justified for lots of people to be skeptical when these sorts of ad platforms propose measurement methods. Right? Like they are obviously sort of deeply and broadly incentivized to try to make their own platforms look good. So like that is very reasonable to me. I don't think it applies in this case. I agree with you. But if you log into Meta's ad platform, you should always be asking yourself, like looking at their reporting, how is this biased? Why is it biased? Meta is obviously very incentivized to make their own ads look good. Google is as well. Like that should be baked into how you're thinking about looking at the reports coming out of any of these platforms. And so it's reasonable for a lot of people to be generally skeptical. And those who don't understand how open source software works or how media mix models MMMs work, they probably just apply that same skepticism without thinking deeply about what's actually going on under the hood here. So what does it mean for an MMM to be open sourced? What does it mean for any software to be open sourced? What it means is that all of the underlying code is available for anyone to look at, read, and in the case of both Meridian and Robin, download, change, use and even resell based on the open source license they have. So that is to say that I can download all of the underlying source code for both Robin and Meridian, I can read every single line of it, I can change that source code, I can run it on my own data and I can see how the results come out. Being open source means that anyone in the world can do that. And the reason why I think Meta and Google wanted to open source these tools is because they wanted to give that extra layer of transparency to the world to be able to say, look, we are not biasing this towards our own channel. You can look at the code under the hood and, and confirm that for yourself. That is not something that you are able to do with their, you know, default ad platform reporting based on Look, Windows or even based on their, you know, quote unquote incrementality driven conversion optimization. We can't see any of that underlying source code, but for Robin and Meridian we can.

[Eric Suefert]

Yeah, I mean that's, that's what strikes me as so odd. And in kind of making this particular accusation because it's entirely verifiable. I mean this is just, this is, it's by nature of being open source, you could look through everything and the actual platform based attribution tools are not. If you're going to make that claim, once you make it against, once you make it against the opaque platform attribution tools, why would you, why would you make that claim against the open source library where all the code is just available to you to vet?

[Michael Kaminsky]

I fully agree. I mean again, it's, it gets eyeballs, it gets people talking. This is like, this just feels like a thing that someone wanted to publish to get attention, which you know, works pretty well in, in the attention economy of today.

[Eric Suefert]

Well, so I, I want to kind of just kind of hover on, on kind of two aspects of this. So, so the one. Yes. So the code is visible to you. There are no like starting weights here. Right? I mean like, it's not like, it's not like sharing. It's not like an open weights model. Right? Like an open weights model. Like the model's trained. Right. This is just the model framework and so you train it on your own data. So there's no like starting weights that you might apply to your own analysis out of the box. This is a model that you run on your data. It's meant to be trained on your data to start with it. So, so there's no way for them to bias or you know, inflate the kind of performance baseline of their own channels. Cause it's, that's just not how the models work.

[Michael Kaminsky]

Work.

[Eric Suefert]

They're like a blank slate. It's just a code.

[Michael Kaminsky]

That's exactly right.

[Eric Suefert]

Right. So there's, there's no like starting point. There's no priors here. Like, it's just, it's, it's meant to be run on your data. There's nothing, there's nothing to it until you run on your data. But you could, I mean, you could just, you know, clone the repo. Tell Codex, hey, could you just look through this repo and see if there's anything in here that would bias meta performance. Right? I mean that would be a very sensible thing to do. I, you should do that with any repo, to be honest. Look for vulnerabilities or anything like that. That would take five minutes. Right? I mean this is not like, it's not some big mystery here. Right? It's not, there's nothing that we need to uncover. Like, we're just, we're not, we're not necessarily arguing in the abstract. You could just do that. Yeah.

[Michael Kaminsky]

And I would again recommend that people do that. There's other very simple tests that you can run. One that I often suggest is like run the model with Facebook activity labeled and Google activity label to run it once and then run it again and just switch the names and you'll see that it's no different. Right. Like it's not keyed off of the names or off of anything special about any of the different marketing channels. You can split the marketing channels any way you want want. You can give them all different names and the results will not change. So it's very unclear to me how you would imagine that there could be some inherent bias in these models when you can do these very easy tests to demonstrate that that's not true.

[Eric Suefert]

I just said the word bias, but I've been trying to not say the word bias because there's, as you know, there's kind of two interpretations of that. You could think of bias as being some inherent prejudice against something that would be more like a cognitive flaw. Or you can think about bias in the context of like machine learning. Like the bias variance trade off. Right. So like bias meaning the risk of underfitting and variance meaning the risk of overfitting. And there, yeah, I think you could make the case actually. So this is, here's where I'll kind of contradict myself a little bit. But it's not about Meta or Google per se, but it is a reason, I do think, why they did release these as open source, because I do think that absent this kind of framework, absent a probabilistic attribution solution, you might actually underfund a Meta or a Google. They might get less credit than they otherwise would and you might be over crediting things like linear TV or radio or out of home. And if you are primarily like a digital property and you're doing those kind of legacy, you know, you're, you're investing in those legacy channels, you might actually be under attributing these primary channels that are driving a lot of top of the funnel awareness, even if they're not necessarily capturing every click. Right. And so I do think that's why that was their motivation here. It was like, well, we think you're undercounting. We think you're undervaluing our contribution. And so if we give you this tool, it'll give you the ability to see actually Meta or Google or whatever is generating more value for you than, than what your measurement apparatus is telling you at the moment, is it, is that a reasonable assertion here or is that, is that a weird thing to say?

[Michael Kaminsky]

I think it's a little bit more complex than that. So let me sort of talk through what I think is actually going on here. So I think Meta and Google are great advertising products, probably the best ever. They are very, very good. I think Meta and Google, what they found was that a lot of their advertisers, the brands that advertise with them, were using other types of media mix models, either ones that they were managing in house or ones that were run by other big consulting firms or even their agency. And a lot of those other media mix models are in fact or have been historically structurally biased against digital channels. So they would do things to put their thumb on the scale for the model to make specifically channels like TV look good. And there are a variety of reasons for that. A lot of agencies historically made a lot of their money specifically from TV advertising. That was a cash cow for them. They wanted to make it look good. A lot of CMOs and even, you know, higher ups in marketing at the brands themselves felt strongly that they wanted to do a lot of brand advertising because it's fun and it's more interesting than a lot of the sort of conversion based advertising. And so I think Facebook and Google felt like, look, these MMM results keep coming back saying that Facebook and Google are really bad. We feel quite strongly that they're good. We think that it's because the MMMs are biased against us. And so they wanted to release these open source MMM products as a counterweight to what they were seeing coming out of these other mmms that in fact were structurally biased against digital channels because the analyst or whoever was managing them was putting their thumb on the scale for the channels that they wanted to look good for a bunch of different reasons not actually related to performance. And so when I think about why are they doing this, it's more about that reason combating other mmms which are actually of even lower quality versus necessarily sort of trying to give a better way of measuring versus a digital tracking type measurement.

[Eric Suefert]

Right, okay, got it. Yes. I think that I would be much more inclined to trust one of these tools than to trust the output of an agency's tool because the agency really is incentivized to have you think certain ways because they're taking a cut of spend as their fee.

[Michael Kaminsky]

Yeah, I think, I think that's exactly right. I think, I mean, again, there's lots of problems with these open source mmms and I'M happy to talk about what those problems are, but versus some random regression that some analyst at a random consulting firm or a random agency is running, the open source mms are going to be way, way, way better than that. And again, more inspectable, more interpretable, especially in the world of AI and LLMs. Have codecs, have Claude look at it, have it pressure test it. That's just going to be a much better experience for most brands.

[Eric Suefert]

Well, so let's walk me through, like what do you think the flaws are? What do you think the shortcomings are of these open source mmms?

[Michael Kaminsky]

So I think mmms in general, like the default assumption should be that they're wrong. The problem that an MM is trying to solve of looking at aggregate data and sort of variations in that data and then trying to attribute causality to that is just an almost impossibly difficult problem. It is so, so, so difficult. There are so many ways for it to go wrong. Easy ways to demonstrate this to yourself are to simulate data that you think is realistic based on how you believe marketing works in the real world. Run that through any MMM and then see if you get the correct results back that matched your simulation. When you make sort of very obvious assumptions in the way that you simulate the data, for example, that marketing performance can change over time as creative changes or as the market changes, that you know, there's a lot of correlation between your different marketing channels, that there's seasonality. You run the data that you simulated so you know what the truth is. You know truly how effective every marketing channel in the mix is. You run it through the MMS and you get results back that don't match that at all. And so it's just like obviously on its face the MMM isn't working right. It is not able to find the true causal signal in that data. And that is the fundamental problems with MMMs. And I think people, over the last couple of years people got really excited about marketing mixed models, especially because these open source tools were released. But I think as people are using them now today, the shine is starting to wear off because people run their data through the model and they run it one week and then they run it the next week and the results totally change or the results just like don't make any sense. They're clearly contradicted by well run incrementality experiments, either geolift or user level ones. And so it's just like, well these tools just clearly aren't delivering on the promise that people think that they're making. Right. This Idea that we can connect or generate a causal relationship or identify a causal relationship between the marketing activity on one side and the business outcome on the other side.

[Eric Suefert]

Right. And that's really the fundamental issue. It's just, you know, people misinterpreting what these tools are supposed to be able to tell them or what what they do tell them, and assuming it's some kind of replacement for attribution when it's not.

[Michael Kaminsky]

I think that's exactly right. And I think to hear this, I'm a vendor in this space, like, we sell a marketing mix model, but most businesses do not need an mmm. And in fact, an MMM will be net negative for them because it will actively mislead them. Whereas, look, Last Touch attribution has lots of problems. I'm very familiar with them. But for a lot of businesses, it's way better than anything else that they have. And an MMM would actually be worse for them than just following the Last Touch attribution.

[Eric Suefert]

Oh, wow, that's a spy. That's a spicy take. We got to let that hang in the air for a second. That's a spicy one.

[Michael Kaminsky]

I mean, look, you can generate data that shows this. Again, if you imagine that, you know, most mmms are wrong and will actively mislead you because the model is just incorrect, it just is learning the wrong things from the data, then, like, it is a net negative.

[Eric Suefert]

Yeah.

[Michael Kaminsky]

And I think Last Attribution, again has a ton of flaws, but especially for smaller businesses that are really performance marketing driven, it works really well. And there's been a couple of studies that have been released recently. I'll see if I can track down the link to share it with you so you can put it in the show Notes that showed that Last attribution was like 85% as good as like only doing incrementality experiments. Like, that's great. Like, that's awesome. That's an amazing tool to have at your disposal. And I've seen this over and over again. Like, I work with, you know, lots of very small businesses, just like personally that it's very clear that they spend money on Facebook, they spend money on Google, they get a lot more conversions and you can sort of see it just by eyeballing the data. And that lines up really well with what you see in Last Touch Attribution. That makes a lot of sense. Last Touch attribution starts to fall down when you have more complex mixes, when you have much larger media budgets. But again, an average MMM is actually going to be net negative. And so it's like, it's worse to have the MMM than not to have it.

[Sponsor/Ad Host]

Mobile game developers no longer need to Pay up to 30% in major app store fees. With Xsolo Webshop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards and analytics. Start today at Xsola.com that's X S O-L-L-A.com or use the link in the episode show notes.

[Eric Suefert]

Do you remember that furor that erupted on LinkedIn a while back about Last Touch? And I had written something, I can't remember what the title was, or something like the need for Common Sense and Digital Advertising. And my point was, yeah, okay, fine, Last Touch is not perfect. But if you're spending your first dollar, are you really concerned about the causality of the conversion? You know, like, if you're launching a product and spending your first dollar on digital advertising, are you really worried about misattributing that? Like, is that, is that really a concern? Is that really something that you should be investing time and money into in terms of doing a holdout or just some sort, some sort of, like, deeper incrementality analysis that really a pressing concern for you? And then, okay, well, if it's not. And you say like, categorically it's not, and then like, well, okay, then there's a. Now we start talking about this second dimension here of just common sense of realistically, how likely is it that I'm misattributing the effects, right? And my dollar spent is not truly incremental. And that gets like, to your point, that gets more realistic as you scale spend and you've got word of mouth and you develop a brand and you develop loyal customers and you start diversifying your mix and you start, you know, you start having multiple digital channels being run at the same time. But even if you're spending, let's say you're spending a million a month only on Facebook, do you really need to do some disruptive incrementality study where you just shut off and you know, you do. And like, you might say, yeah, you do need to do that. A million's a number that necessitates that. And okay, fine, we could, we could probably find a number that we both agree on. But the broader point is there's that second dimension of just like, well, I've got to use some common sense here. How likely is it that there are competing claims for these effects at the first dollar of spend? Maybe we'd Both agree there are none. And I think most people would agree there are none. And then it's like, well, then what's the number? What's the dollar number?

[Michael Kaminsky]

Yeah, I think that's right. And I think every business needs to sit down and think about this. But I think spending a million dollars a month mostly on Facebook, it's like, probably your whole business is driven by Facebook. That's probably 100% incremental, and there's, like, not very much baseline. So that sort of thing can push people towards doing some sort of incrementality experiment. You know, again, shut off or spend up additional dollars in some number of geographies to try to get some signal. Like, how far off are we? Are we off by 5% or 10% or more than 20%? And, like, if it's more than 20%, okay, yeah, let's, like, dig in and try to figure out how much. And if there's, you know, what the size of the pie here is. But if it's like, 5%, does it matter? Like, is it worth spending a bunch more time and energy on this? Probably not. And again, like, MMM is even worse because it's so hard to validate if an MMM is correct or not. You can always run an MMM and always get some number out on the other side. But how do you know if it's right? That's the hard problem. That is like, actually very difficult to answer. And especially it's very difficult for, like, a marketer to be able to answer. And so is that actually adding any signal? Is that actually helping you make the next decision? In most cases, no. And so that's why. Why I. I look at people who are, again, these, like, fairly small companies running these MMMs, and they're getting what effectively is just a random number generator out on the other side, and they're frustrated by that, and they're like, this doesn't make any sense. And I'm like, of course it doesn't make any sense. You haven't thought about how to evaluate if the results of this model are even usable or not.

[Eric Suefert]

Right, right. And I want to. I want to make something clear. I'm not claiming that no one should ever run an incrementality test. I want to make sure that that's very clear. Incrementality? Yes, the gold standard. That's the word you're supposed to use. It's the gold standard. You know, you should be doing incrementality testing. It's important.

[Sponsor/Ad Host]

My.

[Eric Suefert]

My point is more about scale. And when that becomes mission critical for some companies, of course it's mission critical. You have to be doing it. It's a necessity. But my point is more about the, you know, the scale question and relating back to, like when, when an MMM actually might be net negative in terms of measurement value. Okay, so that raises a good point. So. So there's these flaws in the open source approach and the flaws in the open source, call it the interpretation or the utilization of these open source models. How should a marketing team evaluate an open source mm? Like, what are they looking for? Are they all roughly, you know, are they just fungible? Are they interchangeable commodities?

[Michael Kaminsky]

So the answer is no. And the way that you, the way that you know that the answer is no is again, you take your data, you run it through two different mmms, whether they're open source mmms or even Meridian with two different settings, right? You can change some of the settings in it and you can run it and you will see that you will get two very different sets of results generally. Run one will tell you Facebook's your best channel, you spend a ton of money on Facebook. Run 2 will say TikTok is your best channel and Facebook is terrible. So immediately you sort of know that, okay, this isn't really reliable. We made one small arbitrary change to the model configuration, or we switched between two different libraries that supposedly do the same thing and we got totally completely different results. So the first thing that any team should be thinking about is they're thinking about whether or not they should use an MMM or which one they should use. And this is true of whether it's an open source approach or working with some other vendor, is how are we going to evaluate if the model is correct or not. Everything else doesn't matter if the model isn't correct. If the model is going to mislead us and tell us that a channel is good when it's actually bad and vice versa, then the shiny reports, the dashboards don't matter. Nothing else matters because it's wrong. So it can only be valuable if we think that the model is actually right. And so every brand needs to think about how are we actually going to evaluate that before they start on any MMM project at all? I think that there's a couple of ways to think about evaluating an mmm. The first I mentioned earlier, which is parameter recovery. We're going to simulate data where we know what's true based on what we believe about how our business works, and we're going to see if the MMM can get the right answer right. That's a good way to check if this could even work at all. So that's check one, check two is being able to predict the future. Can we train this model up to two months ago and then have it predict what's going to happen over the next two months?

[Eric Suefert]

Right.

[Michael Kaminsky]

That's a second way to evaluate. Okay, could this plausibly work? That is a better check if you were actually changing your marketing budget over the last two months. Right. Because what you want to know is, can this model predict what is going to happen as we make changes to our marketing budget, as we spend up on Facebook or spend up on TV or whatever, we want to know that the model can predict what's actually going to happen. That's what it means to have a causal understanding of the data. And then the last one is verifying the model with lift tests and experiments. Right. Can we say, hey, this model says that TV is really good. If we go run a lift test, does it actually line up with what the model is saying? Or historically, we have a bunch of lift tests. Does the model get that right? Those are the ways to actually think about evaluating an mmm. And those are what you need to be doing before you embark on this journey again, whether it's open source or whether it's working with a partner.

[Eric Suefert]

Right. And so how should a company approach onboarding a tool?

[Michael Kaminsky]

So again, I mean, I would say like, make the checklist and start running these checks. Start by simulating data. How do you think your business actually works? How do you think marketing actually works in your business? How long are the time shifts? How much does marketing performance change within a channel over time? Stimulate the data. Do the parameter recovery exercise. If you're onboarding a vendor or working with your internal team, have them do the holdout predictive accuracy check only. Send them data up to two months ago, ask them to forecast the next two months and then check and see. Like, is this right or is it terrible? If it's terrible, then you shouldn't trust it for anything else. So just go through and actually do the checks. Unfortunately, it's a lot of work. Like, this is hard stuff. It requires writing a lot of code. There aren't great open source tools for this. But this is where the actual valuable work lies is in setting up the framework for how are we evaluating it. It's very similar with machine learning, right? Like running, you know, XG Boost is really easy. Setting up the framework for how are we validating that there's no leakage between our training and our test set. That when we actually deploy this to production, it's going to work the way that we expect that we're actually getting the estimated benefit from deploying this model into production. That's where all of the really hard work lies. And it's the same thing with an mmm, but even worse because the feedback loop is often much slower than, than it is with like traditional machine learning applications.

[Eric Suefert]

What are you seeing as the like a reliable feedback loop there? I mean are you seeing companies that can do this with like, like weekly feedback or is it just, it has to be a month or it has to be two months or a quarter

[Michael Kaminsky]

or whatever, so you can start to get feedback after a week. The problem with something like an MMM is in general the forecast that it's making are generally fairly uncertain. And so within seven days it can be very difficult to like you have to make very large changes to a marketing program to be able to invalidate a forecast that an MM is making over the course of a week. And so what that means is that in general, often you just have to wait longer than that in order to be able to see, hey, did these changes that we make, are they consistent or inconsistent with the forecast that we made in a number of days ago over only a seven day time period? Unless there's a huge change in the, in marketing activity over that time, and specifically huge changes in marketing activity that we believe only has short term effects or has most of its effect in the very short term, that's the only way that you can actually invalidate a forecast of an MMM over such a short time period. But if you imagine that you have some sort of awareness type channel that you believe have fairly long periods of effect, 30 days, 60 days, 90 days, well then it's very difficult to invalidate an MMM's read over a time period that's much shorter than that.

[Eric Suefert]

You're right. And then, not to mention other effects, right, so like spillover or whatever, but like which, that's another whole nother can of worms, I guess. But if you're running a fully digital regime, I imagine you can get, and I mean, I mean like all your ads are digital and you're selling digital stuff, right? So like you're set, you're an app, right. And you're not fulfilling rideshare, it's, it's your game or something, or a fitness app. It could be a shorter feedback loop, right? I mean, because it's not, you could, you could make those adjustments faster and you could see the effects fast. I mean like you probably See the reactions faster. Right. I mean, you could have a shorter feedback loop than a quarter, probably even shorter than a month. Right?

[Michael Kaminsky]

Yeah, that's absolutely true. Especially for like very short term purchases like games or fitness apps. Right. I work with, you know, a whole range of businesses that range from like very expensive luxury products that are not bought on that schedule. But like, you're right, if you have an all digital business with like a very cheap first purchase price or even a free first purchase price, then you're going to expect much shorter time shifts and you can get that faster feedback. But you do still need fairly large changes in your marketing budget in order to see it in the mmm. Because of the uncertainty bounds in the estimates around channel performance.

[Eric Suefert]

Right.

[Michael Kaminsky]

These econometric models, inherently, they have a lot of uncertainty. They're not very precise measurement tools. Right. You might have an estimate that the cost per acquisition for Facebook is between 50 and 75 dollars.

[Eric Suefert]

Right.

[Michael Kaminsky]

That's a pretty wide range. And so you need to see a fairly large, you know, movement in Facebook activity to be able to see the result. To be able to say like, hey, is this right or not? Does that make sense what I'm describing? Like the way that the uncertainty in the parameter estimates makes it so that it's difficult to invalidate just because the uncertainty is so wide?

[Eric Suefert]

No, no, of course. Right. And so like, you know, you would need to dramatically. But that's, I think that's not really that different from just using a platform's own tools. Right. I mean, if you, you know, if you reduce the budget by 5% or something and you know, you've got this, these massive confidence intervals, like, okay, well, you're not going to get any information there. You'd have to like cut the budget in half or drop it to zero or something. But I do think like, if you're running a fully digital regime, you're in a position to do that. It's not like you've got iOS that you send out six months in advance with like a TV campaign or something.

[Michael Kaminsky]

Totally. If you're willing to do it. Right. A lot of organizations do not are scared to make such large changes in their marketing budget. Again, this is just based on, I think a lot of like, oh, Facebook has always been our best channel. We're very nervous to make changes larger than 5 or 10% because what if that makes it so that we don't actually hit our target for the quarter or whatever. So you're totally right. That is like theoretically possible. I think in a lot of organizations Practically, it's very difficult to do to make these like big swing changes. Obviously we see it happen. There are some organizations that are very willing to do it, but a lot of organizations, especially as they get bigger and they're at the level where like an MMM actually makes sense, you get a lot of very natural conservatism that starts to kick in and people are very unwilling to make those sort of changes because they are very risk averse.

[Eric Suefert]

Why did the MM space become so crowded? I feel like it's, it's just, I see an overwhelming amount of mmm content on LinkedIn and I have no idea when that happened because it seems, it seemed to be recent. It seemed like there was just some kind of moment where every, everyone runs some sort of MMM shop now.

[Michael Kaminsky]

Yeah, well, so I think, you know, I'm obviously part of this, but I think a couple of things going on. So one is technology made it possible to like do mmms much faster than what had traditionally been available. And so again, if you imagine that like sort of legacy MMM vendors, they did everything by hand, it was a lot of analysts time. It was a lot of like putting together decks. It was like, oh, we don't actually have to do that. We can run these models automatically. We can do it in the cloud. We're able to take the work out of the analyst hands and put it into the computer's hands. And so that opened up the ability to do this in a really interesting way. Second, open source tools come out. And then third, you know, a lot of people start, started to get worried about digital tracking methods, attribution, last touch attribution, and started looking around for other tools that could potentially solve their measurement problem. So all of those things come together. There's like a big appetite for a new way of doing things. People see the new ability to use MMMs to do it and so they start talking it up. And so I think that is what drove like this explosion in MMM vendors. And again, some of them are sort of only or MMM first and some of them are other like attribution tools that tack on an mmm because they're like, well, why not just like run Meridian in the background alongside our, you know, MTA solution and we can say that we have an MMM as well. I think the big problem in the industry is that there is no good way for marketers to evaluate the MMM's quality in terms of what matters. Right? This is the thing that we just talked about, that what actually matters is, is the mmm, right? Is this, you know, this MTA tool that is running Meridian or Robin or their own homeworld thing behind the scenes, is it generating like actual useful output or is it just a random number generator? And because it's very difficult, slash impossible for marketers to evaluate that, then they don't evaluate these MM vendors on what actually matters. They evaluate them on a bunch of like the shiny dashboard type features that are easy to see and easy to evaluate, but aren't actually the thing that matters for actually using the tool to drive profit into the future. And so that's a big problem that we have in the industry right now is it's really easy to run an mmm and it's very difficult for buyers to evaluate the different mmms on the axis that actually matters. Which is, is it actually learning true causal relationships in the data or is it just spitting out random numbers in a fancy dashboard?

[Eric Suefert]

Well, yeah, I mean that's, that's the, the challenge here with MMMs. I mean, you've got to deliberately instrument the experiments. I mean you have to be able to do that. And if you, if you don't do that, you're not going to, it's not going to mmm. It's, it's going to operate across like this historical data set. But it's, it's not like prescriptive in that way. And so you're always gonna like. It was just weird to me when people complain about not MMMs in particular, but like when there was this transition happening on the advertiser side from just pure play, last click, especially in the app space, to the probabilistic methods or. Well, how do I know what my ROAS is? You don't. That's the point. You have to test that, right? Is just a measurement tool. You have to test it. You have to create the experiment that gives you the two things that you compare. That's the whole point. And it was just like that. Okay, well that's new operating model, right? For a lot of people. And it doesn't fit with how they function.

[Michael Kaminsky]

And I think, I mean that's totally right, it doesn't fit with how they function. But also it doesn't, unfortunately a lot of vendors in this space and like hype people on LinkedIn, they have pitched MMM as like it's a crystal ball. You don't need attribution anymore. Just run an MMM and it'll tell you the true incremental value of every single channel and campaign you have. Like that is just not true at all. It totally oversells the capability of this technology. And it sets people up for failure because CMOs or VPs of marketing, they hear that and they're like great, I'll just buy whatever shiny. Mmm. It's going to tell me the truth and all my problems are going to be solved. And that's just totally unrealistic. But again, for marketing leaders who are not steeped in the history of econometrics and the history of mmm, it's easy to see how they have gotten misled by people, you know, snake oil salesmen effectively, really dramatically over claiming what this technology can do.

[Eric Suefert]

Yeah, right. And just selling it as something it isn't exactly. Selling it as attribution, selling it as forward facing attribution versus just measurement which is very different. Lets get real.

[Sponsor/Ad Host]

Half the time you're not sure if your campaigns work. You just kind of hope they do. Attribution numbers look good, but what if those conversions would have happened anyway? Incremental helps you find out for real. No tracking people, no guessing games, just clear answers on which campaigns actually move the needle. If you want to stop guessing and start knowing, go to Incremental. Because good marketing isn't about getting clicks, it's about, about getting results. Check them out@ Incremental.com that's I n c r m n t a l dot com or click the link in the show description.

[Eric Suefert]

Okay, so we last spoke. I forgot to look this up but it must have been like two, two and a half years now.

[Michael Kaminsky]

It's been more than one year for sure.

[Eric Suefert]

Yeah, it's been a while. So let's say it was three. I don't, I think it must have been roughly three because I was in Austin when we spoke. How has the measurement landscape changed since we last spoke? Since, since from three years ago. How, how have things changed in that time?

[Michael Kaminsky]

It's a good question. So I think, you know, I already alluded to this at the top of the call but I think we've seen, we've, we've already seen peak. Mmm. And I think we are now on the downward slope again. I think a lot of people got really excited about marketing mixed modeling. They over promised what it could do, they brought it into their organization, it failed fairly dramatically and now they are sort of either writing it off or starting to realize like oh actually this is a, this is not a magic, you know, a crystal ball. This is not actually going to solve all of our measurement problems. And they're starting to really rethink what the role is in their organization. So that's sort of one big macro Change, obviously, the other big change is like AI happened in the last three years, which has put a lot of power into a lot of people's hands to be able to run their own MMMs, set up their own measurement stack and be able to really interrogate the numbers in a way that really wasn't possible before. And so I think that's really interesting and exciting in that it allows some companies, at least those that are willing to put the work in, to do it, to really start to evaluate these tools on the dimensions that matter and be able to see up close and personal. And hey, if I run this MMM with two slightly different data sets or two slightly different assumptions, I'm getting different results. And that means that it can't be a crystal ball and we need to think about a different way to use this tool and to use this paradigm. Maybe the last sort of major trend is the rise of incrementality testing. Again, a couple of new vendors have popped up and are having a lot of success, which I think is great that people are starting to think about, okay, where does, like, deliberate experimentation fit into the way that we strategically want to operate a marketing program? How much budget are we willing to commit to, like, testing budgets and to learning about the true value of additional investment into these channels? And I think that's been, you know, really great and really positive for the industry overall and a much better, like, foundational base to build on.

[Eric Suefert]

Right. Talk to me about the AI analysis piece because that's really interesting. I do think that you've got a lot more people empowered with the ability to do, to be like a marketing data scientist than could have done that in 2023. And so that, to me, does feel like a material change. Is that something you're seeing?

[Michael Kaminsky]

Yeah, absolutely. And I think again, you have all of the problems that people, at least the influencers on LinkedIn, constantly talk about. There are cases where data analysis can be challenging, it can be confusing, and LLMs get it wrong. And so you have to pay attention. But broadly, I see huge amounts of people who historically weren't even analysts, like, you know, VPs of marketing, directors of marketing, sitting down and really diving into the data. And I think that it's really positive to the extent. And it's like, this is what I look for. And what gets me really excited is they start to find flaws in their own assumptions and they can do that much faster than they were able to do in the past. So in the past you might imagine that like a lot of VPs of marketing, the Data that they get comes through in PowerPoint slides. They see it once every couple of weeks. They aren't ever really getting into the weeds, and so they aren't ever really forced to confront the potential misconceptions that they might have or the inconsistencies in their own beliefs. But as they're able to actually get into the data, they can see that much more clearly where they're able to put the pieces together to say, like, wait, if Facebook is actually that performant and we just increase spend on Facebook by 30%, why didn't our sales go up? And they could start to really dig in and like, dive into those types of questions, which I think historically they like, maybe vaguely are aware of, but don't have the time or the ability to actually ever dive in and check that out. And now they're able to see it, like, face to face with the data via cloud or codex or whatever. And I think that's a really, like, that is a really powerful new thing that is happening that is getting people to ask themselves and their organization more of the really hard questions that they need to be asking.

[Eric Suefert]

Yeah, I mean, you know, there's obvious risks there. Right. I mean, you get somebody diving into the data who doesn't have, like, the fluency with, you know, just even like basic statistics, it could go awry. Right. But I think in general, it's a very good thing. And I've seen that. I mean, you're seeing that across the board and not just marketing, but in marketing, there's a very acute application which is just like, can I. Can I understand, like, what the impact this is? I understand the incremental impact. And then I think that it just the ability to do that, I think made incrementality much more real to a lot of teams. I think maybe they. They viewed it as like this kind of like, abstract, nebulous concept that didn't really apply to them. But. But now the fact that you can interrogate that question very easily. Right. With a couple prompts and maybe seems like there's more of a. Like for some marketing teams that, where it didn't exist before, there's. There's more of this nexus of scrutiny which, you know, which is nice and that's good. And that I think does, you know, probably at least nudge people away from last click where they might have been totally reliant on it before.

[Michael Kaminsky]

Yeah, I think. I think that's exactly right. And I've seen things even where, like, marketers are like, oh, you know, we haven't been able to like get our data to run an mmm. And like that was the big blocker. And then they're like, I just asked Claude to do it and Claude pulled all of the data, now it's there. And so it's like, it just, it went from being a thing that felt totally impossible to them to like 30 minutes later it's done. And that it's, it's hard to understate how much that opens up the ability to like get to the next interesting strategic step. And I think that that is, we're only starting to see the impacts of that, but I think it's going to be huge.

[Eric Suefert]

Yeah, right. How are advertisers adapting their MMMs to CTV? Because I feel like that's the big growth channel for a lot of direct response marketers. They're just seeing CTV as like the scale channel. And that necessitates maybe the use of an MMM for the first time, right? Or, you know, it necessitates, like you said, I mean, it, you know, you onboard a new channel. Well, you've got to adapt. You know, the numbers are going to change, but they're going to change more slowly. And so you've got to just sort of be prepared to like invest while maybe the MMM adapts, right. Over the course of a month or something. How are, how are they adapting to CTB while using an mmm?

[Michael Kaminsky]

Yeah, that's a, it's a great question. So broadly, right. One of the very nice things about MMMs, at least in theory, is that they are sort of channel agnostic, right. If you can get the data about the channel activity, whether that's spend or impressions by day, like you can just slide that into an MMM and it will sort of just work. Whether it's CTV or some other future new channel that hasn't even been invented yet, like you can always just slide it into the MMM and everything will sort of just work. So that's a very nice feature of mmms. You're totally right that as companies are moving into doing more CTV measurement, that often spurs them to consider like, should we be using an mmm? I think, you know, again, the answer is like maybe depending on, depending on the business, depending on how big the investment is. I think for most businesses, one of the nice things about CTV connected TV in particular versus something like, you know, traditional linear television is that it is possible to run like geographic level experiments with it, which is what I would always recommend doing before starting to do an mmm. Linear TV is very difficult to experiment with because of the way that it's bought. Like national TV has a very different buying mechanism than local tv. And so it's very unclear. Even if like a geographic test on linear TV bought locally extrapolates to national, it has all of these other crazy problems. But like CTV doesn't, you're able to like naturally target at the geographic level. And so that makes it very easy to test and learn into in a way that's going to give you valid incrementality results. And so what I recommend to most businesses is like, look first, just launch it and see if there's any signal at all. Look at your, you know, look at your vendor's measurement panel. Always take it with a grain of salt. But like take a look at it. Look at your post checkout survey results. Is anybody saying that they've heard about you from tv? If there's a sign of life there, start to think about scaling it up and then start to think about, okay, how do we measure the incremental impact via something like a geographic lift test. And then only at the point where it's like, okay, we can't run tests fast enough to be able to, you know, continuously measure this channel. Or we need to be able to, you know, incorporate all of our different results into something that we can use for forecasting and planning into the future. Only at that point should you really be thinking about, okay, yeah, let's go do an mmm and let's use it for all of these different things that aren't already solved by our touch based attribution system, post checkout survey and some amount of like incrementality testing.

[Eric Suefert]

Right? Yeah, I mean that's, that's the, you know, the nice thing about CTV is you stop and start it, right? You can scale it up, it's great.

[Michael Kaminsky]

You can do it on a dime. You can do it in different geographies. Like it's a, it's so much easier to work with and to get started with than like traditional linear television.

[Eric Suefert]

Is it harder to parse out the effects? Like, because, I mean, you know, you've got the second screen phenomenon right. I mean, is that, is that, like I said, I mean, it's, you know, you really can only do measurement here with a probabilistic model. How are you seeing companies deal with that? Because you might end, you might just end up over attributing to meta, right? I mean, because if you're, someone's seeing the ad and then they're scrolling meta and they see another ad, that's it, right? That's what it looks like that drove the convert. Like, how are you seeing the marketers be able to parse that apart? Or like what, you know, because I mean, I understand the, I understand how to do it, but like what, what challenges, I guess are you seeing marketers face when doing that?

[Michael Kaminsky]

Yeah, I mean it's a challenge and it's like the bad news is that like it's not, it's not really solvable to the degree that people would like it to be. Like there will be some amount of leakage with an MMM or with an incrementality experiment. You're going to have really wide uncertainty bounds around what your estimate for the impact of that is and you're never going to be able to get it very precise. So like the answer is just like, you're just never going to know. And what you have to do is you have to sort of cobble together signals that are all going to be imperfect as best you can and then try to use that to make a decision. But like, you're never going to know. Oh, is the true cost per acquisition from CTV $64 or $68? Like, you're just never going to be able to answer that with very much precision. You're going to get ranges of like, well, it's probably between 50 and 75. But our post checkout survey responses around TV has been trending up. So we're going to have to just like jump and say like, okay, you know, we hope that it's actually on the, you know, the low range of that cpa. So it's more profitable for us and we're going to keep doubling down into it because strategically we believe it's important and we want to diversify away from Facebook. Those are the sorts of decisions that you just practically have to make. You're never going to get the sort of like really fine grained precision around the estimates of the performance that most marketers would like and that would make the decision really easy.

[Eric Suefert]

Well, I mean, I would argue that they were overly confident in those metrics beforehand anyway. Right, Absolutely true. You know that Roas number was never some discrete number. Right. And it, you know, that's, that's part of the issue. It's with like the last click attribution tools in particular, they just acclimated customers to seeing a single number and having a false sense of, of certainty around that. And it just, that was, I was never justified.

[Michael Kaminsky]

Yeah, that's totally true. And again, I think, I think a lot of people, at least now intellectually recognize it, even if they don't like, like it in their heart of hearts. But what I point people back to is that you should be thinking about doing forecasting and if you're using these numbers, the like last touch number, and you're plugging it into your forecast and you keep missing your forecast, that means something is wrong and those numbers are wrong. And you need to like, go back to the drawing board and think about like, just make your beliefs consistent. Your beliefs in terms of marketing performance should be consistent with the top line results that you see at the end of the month. And if they aren't consistent today, you need to like go back and think about what has to change to make those consistent. And that's really where, if you put your focus there, you put some amount of attention there. It starts to alleviate a lot of the problems that I think we've seen in the industry of people overly focused on last touch attribution when it isn't actually delivering, you know, the results that people think that it's delivering for them.

[Eric Suefert]

Yeah. Well, amen. Michael, this was great. Thank you so much for coming on on short notice. How can people interact with you? How can they consume your wisdom elsewhere?

[Michael Kaminsky]

Yeah. Follow me on LinkedIn. Michael Kaminsky. That's a great place. We also have a great YouTube channel that we've been, you know, putting out a lot of great content on the Recast YouTube channel. So check me out on LinkedIn and YouTube.

[Eric Suefert]

Michael, appreciate your time.

[Michael Kaminsky]

Eric, thanks so much for having me. Great conversation as always.