← Always Be Testing
Always Be Testing
#75 Unlocking Affiliate Marketing ROI with Media Mix Modeling | Pranav Piyush, CEO | Paramark.com
00:46:22
Loading summary
Transcript90 lines
- [00:00]
Pranav Piyush
Foreign.
- [00:06]
Tygrange
Hello, hello, hello, ladies and gentlemen. Welcome to another episode of the Always Be Testing podcast. I'm your host, Tygrange. I'm really excited to talk to Pranav Piyush today. Pranav, how are you doing, man?
- [00:18]
Pranav Piyush
Doing great. Yeah, thanks for having me. Love your background and excited to chat about all things testing. This is obviously very near and dear to both of us, so it'll be fun.
- [00:29]
Tygrange
I mean, I can't think of a better guest to have on the pod. We've had some phenomenal people in the world of measurement. Mmm. Incrementality on the podcast before, and I've just really enjoyed Pranav, our conversations, how you think about things. And really excited to dive into stuff with you today.
- [00:48]
Pranav Piyush
Let's do it. And saying the same thing, affiliate partner stuff is, I would argue, the hardest part, part of the puzzle. And so I'm also looking forward to how you all think about it and learning a little bit about the space and pushing it forward together.
- [01:04]
Tygrange
Yeah, Maybe we'll have a solution at the end of this pod. Probably not, but I'm going to give that as a teaser for everyone to listen.
- [01:10]
Pranav Piyush
Let's see, let's see. Let's get into it.
- [01:12]
Tygrange
Let's do it. So give us a little action on the awesomeness that you're building with Paramark. Tell us what it is.
- [01:20]
Pranav Piyush
Yeah, good question. I'll keep it super short. I hate plugging my own things. Here we are, you know, a holistic measurement platform for both B2C and B2B. That's the thing that's interesting and unique about us. Marketing, mix modeling, incrementality testing, forecasting, and a lot more. And that lot more is all about how do you make all this actionable for marketers, not just data scientists. Right. Like, somebody has to take action at the end of all this and drive growth. And so we are essentially an extension of your team in that pursuit of driving growth through these methodologies.
- [01:55]
Tygrange
I love that. It's music to my ears. We talk about it so much in our team around being data driven and what does that actually mean? And are you actually bringing an insight to a client? Are you actually giving someone something actionable to do? And anytime I do my curiosity of, like, what's happening here, what's happening there internally, as we look at data and performance, it's like, okay, what am I going to do with this data? Is it actually going to help us move things forward? So I love that context and clarity.
- [02:23]
Pranav Piyush
Yeah, exactly. I often say, you know, creative is Batman and measurement is Robin. And so our job is letting you know, creative just shine. And that's what our clients and customers should be focused on for 90% of their time. And they use us for the measurement once they have launched that new creative, new channel, all that fun stuff.
- [02:44]
Tygrange
That's amazing. Obviously. Mmm. Hot topic media mix modeling. Give us your sense of maybe definition and just jumping into like how you think about that is it's gotten, you know, significantly hot button topic across partner affiliate marketing. It's been a, it's been around for a bit. I'd love to hear your perspective on it, definition and kind of how you guys are approaching it.
- [03:05]
Pranav Piyush
Yeah. First thing that I would say is media mix modeling. Marketing mix modeling. There are some nuances and differences and the purists will come out and you know, talk about, well, are we just talking about paid media? Are we talking about the full, you know, four P's of marketing, right? The price, the product, the placement, the promotion. And some modern folks will also sort of add other factors to it. So the way I think about it, simply for a marketing audience, right. I'm not talking to data scientists here, so data scientists, please don't kill me. What we're really trying to do is help you understand all the various factors that may be impacting your business metrics. What is most correlated versus not, what is most predictive versus not, and use that as a way to inform a roadmap of experiments that you should be taking on to drive more growth. That's how I think about marketing mixed modeling. And I think the big reason for why this is kind of coming back into focus. Right. You said it. It's been around since the 70s and 80s. P&G famously has been doing it, you know, spending 20 million, $30 million a year on MMM. But it hasn't been accessible to most brands until very recently. Lots of great open source stuff happening from Google, from Meta, that's making it more accessible. Lot more vendors out there like ourselves who are entering the ecosystem. So the price of doing this has gone down, the accessibility has come down, it's much more faster. But there's other things that are happening from a tailwind perspective. Right. So all the privacy conversation is getting people to actually think about what they're missing in their multi touch attribution models and that's causing the shift in discussion. There's also the high cost of capital that has changed the conversation. So there were some days where you could growth at all costs and that's no longer the case. Outside of A.I. right. And so everybody in CPG, D, 2C, B2B, whatever acronym you come from, you just have to be a lot more, you have to pay more attention to your metrics from an efficiency and from an effectiveness perspective. So all of these different things are making. Mmm. You know, kind of have a massive comeback. And I think the next 10 years this is just going to be attribution. Right. We're not going to think about multi touch attribution as the, as the word for attribution. Mmm. And incrementality testing are attribution.
- [05:35]
Tygrange
Yeah, that's really interesting. There's a lot to digest and kind of unpack there for folks that are maybe like less in the init as you are. My thinking around it and you know, bear with me a little bit, is like attribution, incrementality are, are two different things, but they're highly correlated, highly related. Right. Is that, is that fair?
- [05:54]
Pranav Piyush
You know, the word attribution, what does it mean? If you look up the dictionary, there's at least one definition that will say it's cause and effect. So the word really means what is causing your sales. And when you think about the whole industrial ecosystem of multi touch attribution, it has nothing to do with cause and effect. So the word has been bastardized, unfortunately. And that's why people think about attribution and incrementality as two different things. But when you really understand the words and if you're being true to the language, then they're the same thing. But the methodology of mta, you know, it's just, it doesn't get you cause and effect. And that's how I think about the two words. Hopefully that that helps.
- [06:43]
Tygrange
Yeah, no, it's, it's, it's funny because I almost picturing that like that diagram where it's like the simpleton on one side and the, like the, the Yoda on one side and the Yoda on other and the like brainiac person that's going crazy in the middle. It's like a silly meme that's been circulated is like, that's kind of how I communicate memes, which is sad and funny, but like, it's like they're two different things. And then the middle, well, they're the same because someone who doesn't really understand. And then there's the middle where it's like they're two different things and it's like the most advanced people in the world, philosophically, they're the same. It's like, it's kind of a funny Paradox, if you will, but I'm kind of getting into a. Going down a rabbit hole.
- [07:22]
Pranav Piyush
Totally, totally. Now, I use this analogy sometimes, right. I ask people, the sun rises in the east and sets in the west. Right. And everyone will say yes, they'll nod. And then I ask them, but in reality, that's not what's happening. The sun is not rising and setting at all. Right. The earth is. Right.
- [07:41]
Tygrange
Yeah, yeah, yeah.
- [07:42]
Pranav Piyush
And so I think about MTA and incrementality in those ways. Right. Something MTA makes it really easy for us simple humans to put words to what's happening in front of our eyes. But when you really ask the question of what is actually going on, on in space, it's a completely different thing. And are these things related? Yes, they are, but they're very different.
- [08:09]
Tygrange
No, it's. It's awesome. And it's like we're kind of getting the 3D chess snapshot of it a little bit with you. Which is, which is fun. Yeah. And I just to like, flow with you on this. Like, it. It's always interesting to me. Like, I think it's important to like. It almost feels like the debate slash discussion for us when we talk to folks about affiliate and influencer. It's almost an analogous where you're like, well, philosophically they're the same thing, but how we staff them, how we manage them, how we scope them and price them are different. So anyway, I also think it is good, as we kind of alluded to earlier, for some people to kind of go, okay, look, when I think about, I need to kind of understand my crediting system, I need to understand my tracking system, and then it's going to help inform some of the testing we're doing that in the holdout world or the geo, that allows us to kind of get closer to that incrementality question or that mmm, am I getting that right? Or how does that resonate?
- [09:08]
Pranav Piyush
Yeah, I think so. I think there are two schools of thought here. There, there was a movement where you would go straight to incrementality testing. And like, hey, you don't need mmms. Mmms are correlation. Like, you know, we can go straight to experimentation. And part of it was ideological because they positioned incrementality testing as something that is superior to mmms. And I think that's a little bit of a misnomer or a misstatement because there are things that are valuable and interesting about incrementality testing. There are things that are interesting and valuable about mmm, and they're very complementary and you should think about them in partnership. That's our approach. And I'll give you an example. I mean, affiliates and influencers are a perfect example of this. To do incrementality testing, you need to be able to create a good test and control bucket. And guess what? You just can't do that with influencer, right? That tech doesn't exist because you don't control how the influencer's activity is going to show up in the real world. You can't say, oh, I just want to limit my organic social post to California, right? Like that. That's not how it works. So in those cases, MMMs are a fantastic solution. And that's where, you know, I think about sort of using these things together. You know, affiliates is interesting because I think that the same thing applies to affiliates because you don't control, you know, how you think about incrementality and geotargeting of affiliates. But there are some more controls that you have with affiliates. And the controls that you have are if you have. I'm going to give you a simple explanation here. Let's say you have a target of. I'm just making this up. 50 affiliates that you want to work with. You can actually split this into, you know, half and half and say for 25 of them I'm going to try this incentive strategy and for the other 25 I'm going to try this other incentive strategy. And you have to make sure that those are random groups and that allows you to now to run an actual test of Is one driving more. Whatever your conversion metric is on the other side or not, that's a very simple and easy, in fact, a much more controlled test because you can randomly control for it, right? You're not doing it by geography or what have you. So I know if I was answering your original question, but I love MMS and incrementality testing, both. We typically recommend start with an MMM to inform the roadmap of testing rather than throwing spaghetti at the wall and just doing experimentation for the sake of experimentation.
- [11:45]
Tygrange
I like that this is phenomenal and I love, I love where it's going and to zoom out and give people some light into mmm. My understanding is essentially you're. It's an econometric model and modern tech has kind of made it more accessible to more brands. With your tech, with your technology, AI is involved to help support and create correlation and understand inferences. Is that accurate?
- [12:10]
Pranav Piyush
Yeah. The word AI, similar to the attribution, can be interpreted in many ways. In the 50s, AI was something in the 70s and 80s, AI was something. And now most people, when they Hear the word AI, they refer to generative AI, LLMs, what have you. The machine learning, the data science that's happening for MMMs is not generative, it's much more mathematical. So it's a statistical sort of concepts, regression concepts, causal inference concepts. So you're right that in the broader sense it's AI, but just so that your audience gets it, it's not LLMs, it's not ChatGPT, it's much more mathematical.
- [12:49]
Tygrange
In nature, more data science centric than like LLM. Generative AI centric. Yeah, I like that. And I think just to underscore the value, my understanding is that with an MMM exercise, a brand can kind of look at a lot of the factors and levers that are creating the impact to the, the buyer behavior to say, well, the weather caused this, or we have correlation that the external event caused this or, or that you actually were running a significant promo on your homepage and that was causing it. Is that, is that kind of directionally where brings in those factors?
- [13:29]
Pranav Piyush
Yes. So in summary, the way it works, and I'll give you an example, right, you've got your meta, LinkedIn, search, SEO, affiliate, email, all of these different channels running at the same time. And the models are able to understand the relationship between each of those channels or even sub channels. If you break it down. And the metric that you are trying to move, right, Whether that's signups or trials or purchases, whatever it might be, not just those channels. But then to your point, if you run promos all the time, or you've got a different discounting strategy, right, you run a 25% off of the first month and then you bump it up to 50% sometimes for black Friday or what have you, you take that data into account as well to understand the relationship between that promo and your sales, whether it's price changes, whether it's discounts, whether it's your own product itself might have changed. You change the way you do trials. So you can take a lot of different variables. A lot of businesses in the real world might also get impacted by macroeconomic factors. If you're in the financial business, you might have, you know, interest rates or forex rates might impacting your, your metric. So that's where human judgment comes in as well. It's like, you know, anybody who's doing mmms for you should be asking all of these questions when we're onboarding to make sure we understand the full context of the business. And it's not just, you know, a simple, okay, we got your LinkedIn and your Google and your Meta and let's just run that through the models. You might get some false, you know, positives or negatives there.
- [15:08]
Tygrange
Yeah, no, I think it's a really great reminder for folks that are not as familiar with it and haven't experienced it. Give the audience a little bit of background on like, for brands that are kind of close to this. I kind of have some good senses of like, okay, you're not ready. If you are here, can you give people a sense of like, if, if you're at this stage, if you're running these channels, if you're running this spin, kind of speaking to the data you need and such to have a proper MMM program. Can you. What's your take on that?
- [15:37]
Pranav Piyush
Yeah, it's a good question. There's a lot of misinformation about it. Some people throw out numbers like, you need a million dollars of spend a month. You need, you know, to be a $50 million business. Like, none of that is really true. There's probably three or four considerations. First, if you are doing incrementality testing, you can do this for your first hundred thousand dollars of investments. You don't really need to have any sort of minimum threshold to do incrementality testing. You may not want to pay a vendor to do it. You can do it yourself if you understand the concept. So invest in educating yourself of like how to run an incrementality test. Here's such a funny example. You can log into Meta today and run a conversion lift study on Meta, which is basically an incrementality test. Yeah, I was just about to suggest completely self serve. Right. And use that to educate yourself. So there's no minimum threshold on incrementality testing. On mmm, there is a threshold because it relies on historical data you need. We typically recommend two years of data or more.
- [16:38]
Tygrange
That's a lot. It's a lot of data.
- [16:40]
Pranav Piyush
Yeah. So this is not a good answer for like startups that are just getting off the ground. Doesn't make sense. But it also doesn't mean that you have to have 20 years of, you know, history. Two years, three years is good enough to start. And again, when you combine, combine MMM and incrementality, get some benefit from the last two years of data and then run incrementality tests. So that's the threshold. The other thing I would say is, like, if you're mostly just on one channel, you probably don't need to invest A whole bunch in any of this, right? You should be able to DIY this. You can observe the trends. The real fun stuff happens when you are two channels, three channels, four channels, and it's hard to piece apart what's working, what's not working. And so those are some of the things that I would say. The final thing that I would say is most vendors or solutions are charging at least 30, 40, 50 grand a year for these types of solutions. So if I were a CFO or a CEO approving this budget, I would be like, okay, so that means we should be spending at least couple of million or more for this to even make sense that I'm going to dedicate 50k to measurement, right? And before that, you know, marketer, you better have a simple, easy way to do it yourself. And so I say go educate yourself on the concepts. Figure out how much of this you can do in a spreadsheet yourself. And then when you reach a couple of million dollars and spend, start to think about, you know, how aggressively you're growing and do you want to invest in an actual dedicated measurement platform and.
- [18:14]
Tygrange
Would you say that couple million, is that per year? Is that on a different cadence?
- [18:18]
Pranav Piyush
Is that I'm thinking per year and even that is light, right? Like I'm not suggesting every. It's not a lot.
- [18:25]
Tygrange
It's not a lot relative to our world. From two years of data, couple million in spend per year, that's, that's a lot of customers out there for you guys to help support. And just, I think it's funny, I think it was before our chat, I think I shared some thoughts and predictions as, you know, it's become all the rage or whatever. But I do think that Mark you and what you're doing represents this continued democratization of mmm incrementality. And I think that's just not a rocket science observation, but it's that following that trend line of seeing more people be able to access it and it's exciting because then it's like it adds more credibility to digital marketing, it adds more credibility to marketing, adds more credibility to what we do. So I think it's a great thing for people to lean into and let the best creative marketing campaign win.
- [19:17]
Pranav Piyush
A hundred percent agree. I think that democratization is going to continue happening. That's a tough word for me.
- [19:22]
Tygrange
Me too.
- [19:23]
Pranav Piyush
And the next five to 10 years, I think the landscape is going to look very different, right? There's going to be more vendors in the ecosystem, you know, more accessible price points faster, better sort of Models will come to the market. So it's a really exciting, interesting time for sure.
- [19:39]
Tygrange
Yeah, no, it's awesome. And it may be a good segue to kind of dive into the challenges with partner marketing. Affiliate influencer. We obviously kind of referenced it previously. You know, I'm curious to hear what your thoughts are and we can kind of maybe ping pong back and forth on that a little bit to see if we can get closer to the solution. I think it's been a really hot topic people have brought up. You know, Google's mmm, obviously meta has a solution at its core. Affiliate makes it very difficult because it's. You're paying, you're basically paying after the fact of the action happening for most of it, not all of it. And so by its definition it sort of runs counter to, you know, a lot of the correlation and MMM methodology is my understanding.
- [20:23]
Pranav Piyush
Totally. So let's talk about the influencer stuff first. That's the easy part. So we can kind of, you know, carve it out and get that done. I think challenges on the influencer stuff are fairly straightforward, which is like actually having good tracking in place. So most of our customers are actually just tracking in spreadsheets. That's fine as long as you have a good sense of when you did an influencer post and how much did you pay for it, the date that you published it, and some type of reporting about engagement impressions reach from that post over the first seven days or 14 days or what have you. It's not a bad start. Start there, keep it simple. And when you've reached a point where, you know, spreadsheets aren't cutting it, you want to get better data. There are so many tools out there now, things like, you know, Grin and Aspire. And I just heard about another one, I think it's called agentio that are doing some interesting things where you can get much better reporting for your influencer marketing. If you're sort of really programatizing the whole process and you take that data, that's what feeds into the mmm. So we're very happy with how our customers are dealing with that data and they're really growing the influencer stuff and we're able to kind of show them how that's working and even split their influencer into sub buckets. YouTube vs Instagram vs some are sort of macro influencers, some are micro influencers. That whole strategy so that tagging is happening within those spreadsheets. So fairly sort of straightforward. The challenge is like you just have to invest in setting up the Processes to collect the right data, to get that data to your measurement partner on time, et cetera, et cetera. Is that your experience or what's your thought on reporting and tracking of influencers?
- [22:12]
Tygrange
Yeah, on Influencer I think you talked about some of the, the platforms that make the most sense. You know, I think we've, we've had, you know, some success and we've actually looked at a lot of them and have kind of settled in on one that we're focused on. And so I think that that's an important piece in terms of like identifying and getting some of the basics in. I think where you can get some UTM link passed back, where you can leverage unique codes that is tied to your affiliate tracking, it can be valuable.
- [22:43]
Pranav Piyush
Which platform are you using? Tyre?
- [22:45]
Tygrange
We're working right now with Creator Co and finding a lot of success there. I think there's their balance of like tech and tracking and value and the types of brands and influencers that they're tapping into has been very positive. We're using a number of affiliate platforms you might expect leading affiliate platforms you'd expect. So trying to get those to kind of sing together in terms of, you know, having UTMs in place, having promotional codes in place. I think you're right. There's. At the early stage, you don't want to let perfect be the enemy of the good. But as you, as you do get more spend and more volume, you obviously want to improve that visibility and that fidelity. So I think you're on point in terms of the quote unquote simplicity around an influencer. I think that there are a lot of things that you can utilize to just triangulate and make sure that you're seeing that full picture. Ideally, when we think about affiliate and influencer, you really want to be able as best we can to see kind of that journey from the various touch points to understand kind of where that purchase kind of interacted along the way. And I know it sounds obvious, but I think that's kind of need that visibility and ideally we can see that within affiliate, ideally we can see that within an influencer. Ideally we could see that within the two and then, and then the ideal state is that you've got kind of some of those other channels layered into that, that kind of not so linear path towards a purchase. Right. So I think you're accurate in your assessment.
- [24:10]
Pranav Piyush
Okay, that's good. So I'm going to check out Creator Co and one other thing that I would say is the UTM tracking and the code tracking is super helpful for understanding. So the immediate response when we look at the data that feeds into the mmm, we actually care less about the UTM code and code tracking and look at overall engagement with those influencer posts. So. Right. So that's the way to think about it. The other thing I would say about Influencer that's interesting is when you think about understanding the incrementality and the diminishing returns. The way we advise customers is vary your spend month over month. So if you're spending 50k a month on Influencer and you want to see what happens, you know, or what can be possible, well plan it in such a way that you can vary your spend from 50k to 200k for one month and then bring it back to 50k. And when you do that variation, you are able to do run, you know, essentially a time based test where you see do your actual business metrics follow that trajectory. Are you able to drive a bump as you know, between Jan, Feb and March to see that spike in your numbers. And that'll give you a very clear sense both from a UTM tracking perspective, but also from an overall perspective that you know, yes, there's even more upside in influencer or not. Obviously operationally that's hard. You have to do the hard work to make that happen.
- [25:28]
Tygrange
But yeah, yeah, it's a quick up ramp in that hypothetical, but then it's doable. And I think that we also want to advise with the right product, the right types of influencers, the right perspective as we've talked about, like Influencer affiliate is generally a longer term play. And so also coming up with an appropriate window. Right. Like there are oftentimes a little bit of a slower burn. So sometimes we might look at a little bit longer window. Maybe it says 60 day window, but coming up with something appropriate that finance leadership team is on board with, you still need to kind of have some read fairly quickly, which I understand totally.
- [26:03]
Pranav Piyush
Let's switch to affiliate. And I'm super curious about this from your perspective. I think it's the harder beast 100 and to make it sort of really clear for everybody, the challenge is that when you build an mmm, the channels or the metrics that feed in, these are the inputs into the model are typically things that are happening before a conversion event. Right. So you're saying I have these many impressions on meta or LinkedIn or Google or YouTube or what have you. Right. And impressions as a proxy could be video views, it could be plays. Sometimes people like to feed in the actual spend. That's happening on those channels. And looking at the relationship with conversion with affiliate, the reason that's not possible is because, well, how do affiliate commissions work? They work after the purchase. So if something is happening after the purchase, how do you include affiliate in the model? That's the challenge. That's the question. And the first thing that we essentially look at is, and this is, I don't know if you've come across this, but there are some affiliate channels that do have impression data and unfortunately they make you pay for it. So I came across this where you can actually look this up, right? You can go to Impact and ask them to enable impression data for your account, but they're going to charge you extra money for that. And I don't know why they do it. They should make it accessible to everybody. And the reality, I think, is that all of these affiliate platforms actually have the impression data because that's how they do the tracking. Right. So if an affiliate is promoting your product on their website, then Impact actually needs to know and has a tag on that website to do the full sort of loop of conversion. So they can see the impressions pixels.
- [27:58]
Tygrange
Firing in the pixel tracking example.
- [28:00]
Pranav Piyush
Exactly.
- [28:01]
Tygrange
I think there are cases where perhaps they're not getting that visibility. Like the a platform might not be getting that full visibility. In particular, the publisher or the affiliate can, you know, has an ability to kind of set up in a particular way. And so I do believe that there's a lot of data there to just for the most part be able to pull out more impression data than we're getting.
- [28:25]
Pranav Piyush
Precisely. So my push to the industry is make that data a little bit more accessible so that brands can sort of include that in their models. Well, 90% of the customers, 95%, 99% of customers are not going to have that data. So what do you do? And the reality is there's no good answers for some people, including affiliate spend is not a terrible answer. And I'll tell you why. If you think about the nature of the conversion and this is where it's a little bit more nuanced, right? So if you are commissioning your affiliate on an event that is somewhere in the buyer journey, but it's early enough in the buyer journey where you're not conflating it with actual, you know, conversions, then you're actually okay. It's kind of the same as branded search or non branded search in the sense that, yes, it is. The model is likely going to, you know, bias towards affiliates just like it's biasing towards brand or non Brand search, but it's not the end all be all and like it's fine.
- [29:31]
Tygrange
Is that an example where you are you alluding to a case where a brand is paying for like multiple actions along a path point to a purchase kind of a thing?
- [29:41]
Pranav Piyush
Yeah. So if you're, you know, I'll give you an example. Like I'm incentivizing the affiliate to get me to a demo request, just making.
- [29:49]
Tygrange
This up for sure.
- [29:50]
Pranav Piyush
As opposed to incentivizing an affiliate to a self serve trial convert. Right. Which is a very different sort of equation. And so there are some nuances in understanding the business, the buyer journey and using that data accordingly. Another one that I often think about, this is more of a strategy question of what is the affiliate strategy and is it exclusive from your direct strategy or are you dipping in the same bucket? You see what I'm saying?
- [30:22]
Tygrange
When you say direct, what do you mean exactly?
- [30:24]
Pranav Piyush
Your non affiliate paid media Strategy, Google.
- [30:27]
Tygrange
Meta, LinkedIn, whatever you're using. Yeah.
- [30:29]
Pranav Piyush
So are you going after the same audience or are you trying to reach a new audience through affiliate? And that also helps you kind of understand how to think about this in a model because if there's a high likelihood that you're going after the same audience that's a little bit more challenging.
- [30:45]
Tygrange
You're actually bringing up something that makes it very interesting nuance that affiliate is not necessarily a channel I like to say. And like I think it's kind of becoming more common in affiliate to recognize that it's more like this multi channel lever. I think of. You've got media buyers on meta, you've got paid search SERP exposure from affiliates in the search engines. You have email, you have some programmatic. So that adds to that complexity. And yes, you have deal, you have coupon, you have lots of content opportunities across the board. Yes, you have some influencer, there's tech and card linked offers, there's like 10 different business models within there. So that's where it's. It gets a little tricky. I do think that not to jump topics but like yeah, I almost want to get down the path of like Google Meta and how much they're claiming. But I'll, I'll. We can dive into that later. I think people are understanding that a lot of models by default sort of overcount those two but bringing it back to this, you know it is really interesting. I would, I do think there's something interesting around when you do have flat fee payments, we do have some brands and advertisers that have to run on a cpc. There's not a lot of them. Some of them do an affiliate because of legal reasons or constraints. The flat fee dollars are seem to be only going up like going back to influencer. The percentages that are flat fee and upfront are obviously even higher. It's, I mean, is it 75, 25? Probably. Maybe it's inverted for, for affiliate a little bit or maybe closer to, to even Steven. Right. In terms of flat fe per sale or cost per lead.
- [32:21]
Pranav Piyush
Totally.
- [32:22]
Tygrange
I'll pause there and pass it to you there.
- [32:24]
Pranav Piyush
No, that's a great thing. And you reminded me of this conversation. Right. So when we work with our customers, we're trying to get that click data or that referral data, and that is the thing that we're actually feeding into the model. Now again, sometimes customers don't have that readily available. They only have the commissions available for whatever reason. But the click data and the referral data is very easily tracked. Right. You have that. And so if that is the feature in technical terms or the value that literally feeds into the model. Now you're a little bit better than spend because you're saying this is how much traffic affiliate is driving and that's a better representation of or proxy for how many people were exposed through that affiliate to your brand. And that's not perfect, but it's one step better than looking at just the spend. And then you still have the spend data that's used for the cost per calculations coming out of the model. Right. So you get click data or traffic data that tells you, you know, finding the correlations between that and your, your metric. So therefore it's contributing 3% of your, you know, trials. And you spent $100. And so, you know, you take that 3%, you divide that by $100 to get your cost per trial sort of number. And so that's a reasonable way of doing it. The challenge is that some models are built on spend. This is one of the reasons why we don't like models that are built on spend. And what I mean by that is they're taking spend data as an input variable rather than impression or engagement data as an input into the model. So our strong bias is towards using actual impression and engagement data and then using the cost data to do the cost calculations rather than using the cost data to find the correlations themselves.
- [34:09]
Tygrange
Yeah. And if I'm hearing you correctly, you feel like you can gather, if you have enough cost data, meaning some upfront, some pay after, if you will, you still can get some good Directional data and sense of value out of the MMM model is what you're saying for sure.
- [34:27]
Pranav Piyush
And I've had this discussion with our engineering and data scientists who are also like, hey, if you really want to get super precise, you have to be willing to run an on off test with affiliate. And that's hard for everybody involved. But I would argue that it's not crazy, like you turn it off for a month to understand sort of the baseline and then you turn it back on. That's a good enough sort of answer if you are sort of really dealing with lots of debates internally, right? It's like, hey, let's see where we are. Right? Let's see where the baseline is.
- [35:04]
Tygrange
And are you kind of saying, going back to your original point of allow MMM to kind of bake enough and then that informs. Okay, we should run this holdout. We should run this lift test. We should run this pause, pause, start, test.
- [35:21]
Pranav Piyush
Exactly right. So if the model comes back, and I'm just gonna give you a wild example here, right, where you've got 10 different channels and affiliates is like, oh, affiliates is driving 40% of conversions. Okay, everyone's gonna look at that and go like, that can't be true.
- [35:35]
Tygrange
Scratch their head.
- [35:36]
Pranav Piyush
Yeah, exactly. All right, so what's going on there? Right? It's a false positive. And we can, you know, everyone needs to have a reasonable conversation around it. The vendor needs to be able to explain that, hey, it's possible that, you know, because of just how affiliates work, that it's picking up a false correlation here. Let's run an on off test that will give you a better estimate. You can feed that test result back into your model as a prior. And now you have a calibrated. Mmm, right? So this is the new hot thing that's happening in the ecosystem now is you can actually calibrate your model.
- [36:10]
Tygrange
Yeah, causal.
- [36:12]
Pranav Piyush
Yeah, I hate to use the words causal, et cetera, et cetera. But it's basically. And this is not a new concept like this has existed in the past. You could do it through other methods. But, you know, this is a good practical way for most marketing organizations to do. And this is why I say combine MMM and incrementality testing. Right. When you have both of these concepts, you get a much better output.
- [36:36]
Tygrange
Yeah, no, it's amazing. And I think I've run holdouts. I've seen brands do them. I've talked to a number of affiliate folks around this world not to totally take affiliate take over this one conversation. But it's interesting to See it vary and it's not always crystal clear. It's not a consistent theme of like oh yeah, it's always incremental or this particular partner is always, you know, found to be incremental or incremental enough. So there's, there's some nuance to it around the pricing involved that you've set up for it. The, the way, the way it's interacting with your overall strategy. The longer term view of kind of like what are those, what are those customers interacting with as well, how long are they coming in and what kind of that behavior or AOV are they driving when they do? So it's not as I think simple as people, you know, want it to be. And I think. But I also like that the real data scientists approach of complexity that you are bringing in and understanding is like no, we're making it more available, we're making it more accessible. So it's almost like simplifying this magic oz behind the curtain stuff that was very few could do. And I think it's, it's, it's again not to like cheerlead it too much but it's very positive that it's being more accessible and you're kind of educating people on it.
- [37:59]
Pranav Piyush
Yeah, I think the, you're 100% right. And I give a lot of credit to everybody in the ecosystem, right. Our competitors like everybody's trying to do the right thing which is make this accessible, make it affordable and you know, drive the right type of conversation. So full respect to everybody who is trying to make this happen. The one thing that I would say is if you are talking to an MMM or an incrementality testing vendor, the thing that you have to look for is are you going to get that actionable advice on an ongoing basis. And it's very easy to look at a fancy dashboard and get a whole bunch of numbers and data and you do nothing. So if you didn't achieve an action at the end of that that drives growth, you just wasted money on measurement. So look for that in your partnership with a measurement vendor.
- [38:52]
Tygrange
I love that it's good counsel kind of bringing us, bringing us around here to, to get some more good intel and then wrap it up. So you know you've had some amazing experiences in your career leading up to Paramark. There's some kind of marketing measurement or testing learnings out of, out of the build.com experience. I know bill.com has been an amazing case study example of growth and just you were there for a significant period. So excited to Hear if there's any thing that you're able to share about your learnings there.
- [39:21]
Pranav Piyush
You know, Bill was interesting. So when I was there, you know, the startup that I was at got acquired by Bill.com and there was another startup called Divi that got acquired by Bill.com so really a powerhouse. And each team was very different in how they did marketing. So a lot of my experiences were sort of understanding and learning from all the different sort of functions, all the different sort of teams. The big lesson that I would say from Bill was how impressive their partner distribution was. So they go through. Bill.com is essentially a financial platform. They help SMBs and mid market companies with account payables and other things, payments. And they had built an amazing network of accountants and other financial advisors who would refer Bill.com, guess what, affiliate. And so they had built their own in house systems and processes to encourage and incentivize that partner network to drive growth. Massive success. Bill.com wouldn't be where it is today without that network and that was massive. The second thing that was incredible for them was just what I would call sort of product led growth virality where if you receive a Bill.com driven payment you're like oh, what is Bill.com and you know you're going to go Sign up for Bill.com to pay your vendors. There was a whole bunch of like virality there which was, you know, pretty impressive as well. And that's this. These concepts are being copied like basically by pretty much every, you know, the new competitors, ramp, Brex, so on and so forth. So I don't know, I thought partner was really interesting for them. Massive, Massive.
- [41:03]
Tygrange
I love that. Yeah, that's huge believer in kind of the partner marketing and tech ecosystem for B2B and for consumer. For us personally, we're just seeing amazing learnings and action out of it and we're heavily invested and excited about that space. So it's cool to hear one of your big learnings were positive from that field.
- [41:21]
Pranav Piyush
Oh, 100%. Yeah. I'm a big believer in partner. Yeah.
- [41:24]
Tygrange
And I'd say that it kind of dovetails a little bit into, you know, you had a chance to teach at Reforge. I was a huge proponent of them for many years. An early student and involved heavily in the community. And were there any kind of AHAs or learnings that you had from those dialogues and conversations?
- [41:42]
Pranav Piyush
Reforge is an incredible platform and community and they're going through their own sort of set of changes. But I used to Send everybody on my team, like go take a reforge course on X, Y and Z. Here's your L and D budget type of thing. And so when the opportunity arose for me to lead a course, I was like, yeah, this is a no brainer. And we put a course out there. So if you are looking to learn about marketing measurement, go check out that course. It should be easy to find. Just search for my name and reforge. Just great conversations. I had an opportunity to do this like twice or thrice now and now it's on demand. So I would say just very positive. I love teaching and also learning through teaching because it forces me to get better at what I do. So it's, it's great. Reforge is awesome.
- [42:27]
Tygrange
Love that. Yeah, I love that. Totally. What's the biggest mistake you think brands come to is you kind of alluded to some of that, but when setting up mmm, what's something that you kind of caution people to avoid?
- [42:38]
Pranav Piyush
Oh my God, that's a long list. But one thing that I would say is looking at incrementality testing and MMM separately and thinking of them as separate point solutions I think is a big mistake. The second thing that I would say is getting into debates about my model is better than your model I think is a little bit of a wasted time and effort. The reason I say that is, you know, it's kind of like is GPT better than Llama, better than anthropic? Yeah, the fringes. But everything is kind of the same at the end of the day. And so it's not about the underlying models as like they all are pretty good, they all are pretty sound. It's what's the UI and UX of how are you going to actually use it and action upon it. So those are the two things that I would focus the evaluations on.
- [43:29]
Tygrange
Yeah. Was there a test or an MMM or holdout or incrementality test that just kind of like was mind blowing or notable or one that you want to share that's kind of like counterintuitive or interesting?
- [43:40]
Pranav Piyush
Yeah. Young brand in a part of the world. In Japan, consumer mobile ran a TV ad, had 50% to 70% incrementality coming out of that doubled the cost of other channels. But 50% incrementality, which means, you know, they spent a lot for that, but they ran it as a holdout. So only ran it in like specific parts of that country, but just massive. Right. Like this is a tech brand selling a tech product. It is consumer, it's Mobile. So I get it. So don't overlook tv.
- [44:14]
Tygrange
I love that. I was on a connected TV panel about a year ago and did a ton of deep dive. We're seeing some exciting results for our clients and starting to kind of become more available through affiliate because as I said, multichannel and it's taken off. It's a good one to look at. And I've actually had conversations with a lot of people that have looked at pretty objectively from kind of a fractional CMO perspective, like watch out for partner marketing, meaning affiliate influencer and watch out for connected tv. Not to say they're end all be all for every situation, but definitely a great call out. Coming down the wrap up here. Getting to know you Pranav. It's just been amazing to chit chat with you. What do people not know about you?
- [44:54]
Pranav Piyush
Oh my God. I'm a huge fan of marketers. That's all I can say. So if you do marketing and you want to sort of, you know, get better and brainstorm ideas, I do a bunch of networking sessions. I do lots of, you know, one on ones with people just because it keeps me on my toes and I learn from every one of those conversations. So if you're ever interested, just reach out on LinkedIn and we'll find time to chat.
- [45:20]
Tygrange
You're the man. I love that. Just a couple more here and rolling through any hot book tips or suggestions or resources for people to look at besides maybe your reforged course?
- [45:30]
Pranav Piyush
I just got a book yesterday from somebody. I'm going to put this up right here. This is from one of one of the LinkedIn founders over just a touch. There we go. Let's do this one.
- [45:43]
Tygrange
There you go. There it is. Oh, I like it.
- [45:45]
Pranav Piyush
So check this one out. Yeah, it's about what the future of agencies might be. So I'm going to read that one.
- [45:53]
Tygrange
Add it to my list. Might even make the RBL reading list.
- [45:55]
Pranav Piyush
There we go.
- [45:56]
Tygrange
Amazing, Pranav. I'm just grateful to have you on. You nailed it. So many good things to talk about and a lot more for us to talk about. Obviously in the world of mmm. And really appreciated it today. It was awesome.
- [46:07]
Pranav Piyush
I'm glad. Thanks for having me. This has been great.
- [46:09]
Tygrange
Have a great rest of your day.