A (4:51)

Yeah. Well, I, I think you've just created a new rule of thumb. I like that, that you should spend 80% of your time on the best practices and learn because like you said, if 20 people tried it and it failed, maybe it is going to fail for you. But then take 20% of your energy, your budget, your efforts and do the exact opposite of the best practices and figure out for yourself why, why they learn. And to your point, like you know, if you're early and you don't have a lot of resources, you don't want to over index on being a contrarian. But I mean that's the whole thing, right? It's a, there, there's alpha in the best practices and that's why they're the best practices. But then there's also Alpha in the things people aren't trying or the things that certain people failed at. But you're going to succeed in that for a very, for something that's very specific to your app or your niche or your market or some other thing like that. So, yeah, the new rule of thumb, follow best practices 80% of the. And try crazy stuff 20% of the time. I like that. Let's go through some of these, like, more specific contrarian takes. And one of the ones we were discussing is that sometimes you just need to take something as a fact and not consider it a problem until it becomes a problem. And the example you gave me, and I'd love for you to elaborate on this, is that if your trial conversion is low and you're like, oh, trial conversion being low is a problem. Well, is it a problem? Like, are you getting cheaper installs? Tell me about that.

B (6:21)

Yeah, like, typically people would say like, oh, we have, I don't know, installed to start trial ratio, I don't know, 5%. And you know, usual healthier benchmark is more like 15 to 20% and such. And I'm like, okay, what have you done? Like, what have you tried? And how did you interpret the results? How did you build on those results? They're like, oh, we didn't because it's a problem. You know, so that's not a problem, it's just a fact. I think something becomes a problem after you systematically test it, iterate on the results, and try many different ways from, let's say, radical to iterative, and you're still stuck at the same level. Then I think I can consider that as a potential problem. Instead, you should always see what you can do or understand the root cause. So diagnose any factual problem that you consider as a problem and then come up with an action plan and then measure the results to see the impact. It's as easy as that. It might sound very straightforward, but I can tell you, many people bypass this kind of step by step funnel in figuring out stuff. Because when you follow such an approach, you realize that, oh, actually it's not a problem, or I don't know, it's simple as people say, like, oh, we tried, I don't know, SNAP and it didn't work. I'm like, okay, how did you try it? Oh, we spent, I don't know, 5k in a week and it didn't work. I'm like, yeah, that's not a problem that SNAP didn't work. You just didn't test the channel in a proper way. That is optimal for the channel. You shouldn't test the channel just for a week. And same goes for, I don't know, creatives to channels or the markets. Oh, it's too expensive for this market or that channel didn't work. So people like to come to conclusions fast. And I think because of the priorities, because of this scarcity, kind of a mindset, they want to reach conclusions really fast and they want to move on to the next. So when they're in this kind of mindset, I think then they're missing out on testing things properly. Like, I think a lot of people focus on the execution, which is on paper grade. But I believe the success of a test is at least 50% dependent on how you plan it, how you strategize it, how you hypothesize it and how you execute it. And execution is just the what part. Maybe it's even sometimes less than 50%. But basically how you execute matters more than you execute or not. Sometimes people also say, oh, we tried, I don't know, let's say lookalike, it didn't work. Because the way they tried lookalike was at 10%. Run it for a week, that's it. No seed testing, no different ranges of testing and such. You cannot just call the whole thing doesn't work if you don't slice and dice and try like different things. That's kind of my approach and that's how I deal with day to day business. To understand and prioritize it based on, okay, how can I say I tested something and what information I need to call the shots, that it doesn't work or it does work. So this kind of thinking I think will force you to think of all the preliminary steps on what you should be considering and so on.

A (9:54)

Yeah, I like that kind of root cause thinking. I actually did a talk at MAU a couple of years ago about the top three subscription app metrics that really matter. And that was kind of the thrust of my talk was that you need to look at the whole funnel, not just individual metrics. And like what you were saying, it's like if you have a low trial conversion rate or if you have a low install to trial rate or if you overly focus, focus on any one metric and you're like, oh, this is a problem. And you like grind and grind and grind on that quote unquote problem when maybe you can get like really cheap installs and the really cheap installs are why you have a low conversion. But hey, that's not a problem. It's not a problem when you get really cheap installs and your conversion rate's not great. Like, that's maybe just a natural fact of your business that for whatever reason you're getting incredibly cheap installs. And the example I always use with this is like, you know, my apps have gotten featured a ton by Apple over the years and getting featured is fantastic because it's totally free, zero cost of install. But guess what? Those cohorts convert really poorly. So do I have a conversion problem? I don't know. I know that those cohorts don't convert very well, but like, should I overly focus on that or should I be more focused on top of funnel and like finding other ways? And so a great reminder to not get overly focused on any one specific thing to the exclusion of like all the different moving parts in the funnel. And I like the way you phrased it too. Like it's just a fact. That's a fact. Right now, that's a fact. Is it a problem? We don't know it's a fact. Figure out if it's a problem. Don't just assume it's a problem. Yeah, so another thing you mentioned you've been thinking a lot about lately is creative interpretation. And you know, we had Eric Sueford on the podcast recently and he's intentionally a bit provocative saying like, don't even try, don't even try, try to interpret the creatives. But you were telling me you do think there are things that you could learn from trying to interpret why things won, why they didn't. So what is your process for looking at the winning creatives and finding some value in that process?

B (12:09)

I think looking at the output of a creative performance definitely matters because that's how you will iterate and find the next winners and or the losers. It is important to look at the relative metrics as well. So maybe what he's referring to is more like don't take it as your source of truth because platform is doing the modeling and such. But at the end of the day, the output, the performance of the creative matters and the approach should be more like what can we learn from it regardless of the output? In a way like good or bad, it's all about the learning. Okay, this creative for whatever reason performed better and other creative performed for whatever reason worse. Let's understand why. What is the hook rate, how's the thumbnail? What is the audio, what is the background and this and that. So once we understand and then we start changing one variable at a time, then it becomes A lot more clearer on how we can engineer the success. Right? I mean, that's the whole point of. That's how you create winners from the winners because you have a winning concept and thanks to the interpretation of the output, you're able to create more winners because you look at the relative metrics. That's a guidance for what you should be testing and what you should be not testing anymore because whatever you do in that particular case doesn't work. However, I have seen from the same concept, like the third variation worked, but the first and second failed. It's the same concept. It's just slightly different messaging and such. So that's why you would always test with few variations just to avoid such cases where you don't just jump to conclusion of a whole concept and so on. I think that's also important. So there could be of course false positives and so on. Nothing is perfect. But also, I'm not also saying like, oh, you should trust algorithm blindly and stuff like that. What I'm just saying is we should be able to interpret the results and then drive conclusions to shape the next creative iterations based on that. So that's when you need to interpret the output. And then what matters at the end of the day is your success rate and not the amount of creatives. Because I think LinkedIn nowadays is full of oh, you know what, we just tested 500 creatives. Oh no, we tested 700. How much did you test? Kind of race. That is not so meaningful because those numbers alone don't matter. If your success rate is very low, I would more look at your cost per successful creative that is beating the BAU ADS performance. And it doesn't matter if you test 20 versus 200. Of course, okay, 20 versus 200 might matter, but it doesn't matter. If you're testing a very high amount of creatives with a little success, it matters more. You test less amount of creatives, stronger hypotheses, stronger investment from an analyzing and hypothesizing way with a higher success rate that what matters more than just the total number of creatives that you are testing. I see potential issues when the creative amount is on the higher end because then the mindset is like, hey, look, I just have to test these. Let's just move on. And then having less emphasis on why it worked, why it didn't work, and so on. And then sometimes there's a disconnect because typically a company that is testing 500 creatives a month tend to have a team and usually there are, I Don't want to call it silos, but the guy who is running the UA is not doing the old briefs of the creatives and then analyzing and then giving the briefs and all that. Typically in the smaller startups, yes, but in the bigger teams there is that. And then, so then the UA person's job is just to run them and maybe analyze them. If they don't have the analyst team, then to give the creative, let's say the optimization manager, to put them into briefs and get the next production. So when the quantity is so high, like I think the level of detail and sometimes also the quality of analyzing and shaping the next input is also being jeopardized a bit because of the high quantity, which is not necessarily a win or something to be proud of alone if your success rate is low. Because also then the mindset shifts, like, hey, Look, I have 500 creatives, I can't spend 5k on all of them. Sometimes this gets into, oh, we only spend $100 on these creatives and that's it. Okay, but how did you decide on that threshold? And I think finding that threshold is such an important thing for every account. For example, there's no universal number that I would give. It's very different from business to business, depending on your cpi, depending on your other CPX metrics. But what I would suggest is to look at your cumulative spend and cumulative CPX metrics, whether you're testing on a CPI or a cost per trial or just cost per subscription and such, and see when they stabilize. So because I see a lot of accounts with high spend, they're sometimes just spending 10k, 20k only to realize, oh, this is actually not a good creative. I'm like, oh, really? I could tell it after 500 bucks. Like, look at the data. It was already bad. It was already screaming, like, why did you spend so much? So in order to avoid that inefficiency and scaling it more efficiently, you could easily look at cumulative spend and cumulative performance over time and to see where your data stabilizes. Because we're not going after the statistical significance. Like no one has kind of money for that. But we also don't want to spend just 20, 30 bucks to see, or we don't want to just let the algorithm decide which one to pick. So there the problem is the following. Like, just because algorithm decides to deliver one creative over others doesn't make the other creatives bad creative. I think that's such a misconception or interpreting in a, I would say in a wrong way. Like that Particular creative that got the 90% of the spend was maybe lucky, we call it like lucky 5000 impressions. Because meta does some sort of a decision making around first five to 10,000 impressions. And if one of those 5,000 people who saw the ad didn't engage or whatever signals that Meta didn't receive, then it will not deliver anymore, but it will deliver to other one. We just got maybe 12 clicks and stuff like that. And then I've seen that multiple times where you take out that dominant ad from that ad set and all the other ads like suddenly bloom and you're like, oh, they were also great ads but because of the other better ad or however you want to call it, which is not necessarily better ad as well, because sometimes Meta called the shots way too early. No conversions, nothing. Like it doesn't even give a chance. So in those cases we don't call that creative that got all the spend successful and all the rest bad labeling. I think this is not universal and this is really, I think case by case basis. I've also seen ad sets with 50 ads live. Oh, let's see what algorithm likes and we will run it for a week and then we will see what happens. And then what happens in that week? 90% goes to the same ad. And because they have this rules of thumb, oh, we run it for seven days and at certain, I don't know, daily budget. But then that's useless when the 90% of the budget goes to one creative. That's not a test. You should in those cases, for example, break the rules and like, hey, look, this reached my threshold, this is my decision threshold. Now I want to see what other ads gonna perform for that minimum threshold that you decide based on your cumulative spend and cumulative performance over time. Yeah, so that's why there is no one rule that is valid for all. But I personally try to get spend on the other creatives. If I see a dominant one, I believe that every creative should have an equal chance. So I democratize a bit within the ad set. Right. And because I've seen it countless times that the decision that made by Meta was way too early. Yes, we trust algorithms, but we want to guide the algorithms. When we see 80, 90% of a dominant asset, let it be a creative, let it be a country and such, you're not fulfilling the potential of the other variables. And I think this is important. For example, if you put the US with all the other smaller, let's say European countries, there's a 90% chance that at least 70% of the spend will go to us because of its size and everything. Also you should never do that anyway. Like you should always tier buy like CAC or LTV depending on what you're optimizing towards and ideally without those dominating variables. And if there's a dominating one, then you can always take it out, test it separately, give a chance to the rest. Because it's not meaningful to have, I don't know, let's say like just a couple of thousand dollars spent with a really good cac, really good roas. Well, if you cannot scale that and then you try to scale by increasing that existing campaign's budget, but that country that you wanted to actually scale is not increasing its spend necessarily because if it's getting dominated by the other bigger countries or based on whatever logic depending on your setup. So in those cases then yes, you can experiment with the value base, the rules and stuff like that and that we can talk about later in the testing and the learning side of things.

A (22:29)

What's the tactical move then there when you do have that winning Creative that's getting 90% of the spend, is it better to pull the other creatives out because that ad set already knows how to target that specific ad to the right people, to the right geography and everything else. Is it better to pull the quote unperforming ads and give them a shot somewhere else or is it better to pull the performing ad into a different set? What's the best way to actually implement?

B (23:00)

That depends on on whether this creative is in a creative test ad set versus in a BAU ad set. If it's in a BAU ad set, then I would create another BAU ad set and then I would include those creatives that were under delivering into the new one and I wouldn't touch that something that is working well. But if it is within the creative testing ad set and if it already reached my threshold, then I would already pause it and then I would put it on the bench to get ready to upload to BAU adsets whenever the team will do the creative refresh. So it is case by case basis. But in general rule of thumb is don't touch something is working well but depends on what case we are talking about.

A (23:54)

So yeah, that makes sense. I did want to step back and not to put words in Eric's mouth. Maybe his point is more to not overly focus on a single hypothesis for why a creative one. So like going back to the whole like you know, should you learn from creatives? Because I do want to dig a little deeper into Kind of how you think about what you can learn. And the example he used was you look at a creative that has a dog in it and you're like, oh, like people love dogs. And so that's why the creative one. And you maybe get overly focused and then to your point, then you go create 200 new assets all with dogs. But maybe it wasn't the dog. Maybe it was just that it was something cute and a cat one would perform just as well, or a panda one would perform even better, or ocelot or like who even knows? Right. How do you think about forming a hypothesis around that? Because it seems like that's really the important step is, is like not being overly confident in why you think something won, but instead just allowing it to help you form a hypothesis for other things to test based on a multitude of reasons why that creative could have won.

B (25:10)

Yeah, I mean you should Never just produce 200 variations of just from one signal. I think it's always in phases. So let's take the dog example. Okay, let's have the same dog, let's have a puppy, let's have another breed, let's have another animal. So just to test and see if it's the animal or not. So what we want to understand is what makes the difference? Is it the dog, is it an animal, is it something else? And once we figured it out, then we would slowly roll out the production and so on. So it's never about, oh, the dog won. Now we need 200 variations of the dog. It's more about, okay, like, is it the dog? We want to validate that. So in every step we want to do that and we want to do in both directions. That's why we have both iterative concepts and the radical concepts iterative would be, oh, let's put another breed, let's put two dogs maybe, and it's even more effective. But you know, in the radical one, you can have completely different hypotheses than what doesn't work based on what works today, but also on based on other sources. So when it comes to what creative to test, I think you should utilize both the first party data and the third party data. It's not always about the first party data. So first party data, meaning your meta creative output, that's one variable that's a valid, let's say input. But you also have, let's say for particular copy or let's say if you're testing particular, let's say visuals and such that also exist in your app. Right. Okay, what is the Most popular thing in the app that is more valuable for the users. You basically look at within your data set. You look at within your in app analytics, what you can learn, what resonates with users. You look at your high LTV users like why do they stick around? What do they use the most, what messaging resonates with them. And then because our goal in performance marketing is to find more of those high ltv users. So testing, like test results from test campaigns is one variable. But in app analytics, data is another data input that we can utilize. And then we have the third party data where we have competitor concepts, what works for competitors. We have overall trending memes or concepts or audios and such. And then you could have category level concepts. They're not really direct competitor concepts, but other big players that are doing well within your category. What are they doing, how do they advertise their product? So what you can learn from them. So then you bring and merge all these data sets together. Then you decide on okay, you know what, of course you give a lot more weight to the meta test results, but it is not a 100% hey, this is it guys. Dogs, let's go. So it is about merging that. So when you hypothesize the next iterations, it also should come from what worked and what didn't work, not only for you, but also for others. You can check the metaed library to see how long competitors are running a particular ad and how many variations of that. If there is one concept with 15 variations and that is a particular the only one that has 15 variations compared to other ones having just two and three, you should be able to interpret that likely that creative is performing well for them. That's why they're exploring more angles. So it is important to analyze and put enough resources and attention to the planning part of the experiment. Exactly because of this way and not just act like I don't know growth bro, and hey, this worked well, let's do 200 of them and so on. So I think this balanced effort, planning and executing matters here.

A (29:00)

How do you think about analyzing competitor ads though? Because to your earlier point, there's a lot of people out there just following the playbook and creating 15 variations because they formed a bad hypothesis around. So how do you actually dig into these competitor ads and try and figure out like if it is actually working for them? You know, that's something I just see throughout the whole industry broadly is that it's too easy to over index on something you observe without the data and say, oh this must be working. These 15 creatives must be working because they've created 15 variations that may have been like a dumb move. And you just don't realize how poorly it's performing. And they don't even realize how poorly it's performing because they don't have it instrumented. Well, they're not tracking deeper in the funnel. How do you think about looking at competitor data and getting any insights from that when you don't have the full picture of why, how, what their process is, how good their stuff is instrumented, how smart their people are, et cetera, et cetera.

B (30:09)

Yeah. So anything you see online, let's say, should be only as inspiration and not treat it as this. Because I've also seen and heard people saying, we tried it, it didn't work, but I don't know, Leather is using it. So then they justify and then they defend, oh, but it works for them. So as you said, it doesn't mean anything for the brand. Exactly. Because of the reasons that you mentioned. So it should be just treated as inspiration, as another input. That's why I talk about a pool of inputs that will shape together the next iterations. It is never about, oh, let's copy one to one, whatever Compare is doing, oh, let's copy this trend. It comes down to how I can customize this to my own Personas and in my own ways, based on what worked for me before and what didn't work for me before. So at the end, this needs to all go in that processing machine that considers the Personas, the past performance, the current performance, and all your learnings. And then you test it. So it's never about, oh, Compare is doing this great, let's do the same. I think that is probably why many people are getting upset. I don't know when they just copied the same paywall or same creative, and then finding out, oh, it doesn't work for us. Well, why should it work? It's different audience, different funnel, different pricing, different messaging, different promises. So although, for example, on the web funnels, many of them look very alike, especially in the health and fitness space for many years. And then everyone copies from each other. So pretty much a lot of the health and fitness web funnels are similar. Now. Some are missing, of course, some, I don't know, smaller elements here and there, but the conceptually they're the same. At the end of the day, you should just run to see. So they should be treated as inspiration and as a variant in your whole test hypothesis and not as a test hypothesis itself. Yeah.

A (32:13)

And then one other kind of elephant in the room in all of this is making sure that you're actually instrumenting things well and actually testing well and can actually interpret the results properly. Bahe recently had a tweet that I thought was amusingly provocative. He said said I won't quote it exactly, but it was something along the lines of like half the people who say the blinkist paywall works for them just aren't testing it properly. So how do you think about like in all of this experimentation, informing hypothesis and like running all these tests, how do you think about like even just validating your own process and testing and tooling and that side of things to make sure you're even drawing the right conclusions from the data.

B (32:59)

Yeah, I mean that's also I think what we covered earlier, like 50% of the issue with something working or not is about how they tested. So I can like totally get like why he mentions that because I hear like similar stories. They're like, oh, we tested XYZ pricing because competitor is the same but our conversion rate dropped and our revenue dropped. Like you shouldn't take these kind of stuff as the hard truth or your strategy. It should be only as a variant and as a source of inspiration. And in terms of how to interpret the results, like everyone has their own in app analytics tool or their third party tool depending on what test are we talking about? If you're talking about let's say the paywall tests and stuff like in app analytics is the way to go. If you're talking about creative tests and so on, then we are talking about MMP as a source of truth, relatively speaking here. But if you're talking about incremental impact of a channel, like we are not looking at the last click obviously. So that's also one thing that there is not one single source of truth for every answer you're looking for. So if you want to compare creatives against each other like MMP could be your best friend. But if you want to compare the impact of UAC versus Meta, MMP is likely not your best friend because one is a push channel, one is a pool channel and the competitiveness of the vertical and such. But typically UAC is a lot cheaper because of the search and which of those higher proportion of those searches also coming from the brand search and so on. So you wouldn't do a budgeting decision or forecast just based on MMP and such. So you might want to rely on incrementality and mmm and conducting experiments around the geolift and the holdout and the blackout test and so on to come to such conclusions. Knowing your baseline, knowing your seasonality problems get different at different stages. Like you don't have to worry about incrementality when you're just spending 100k a month. I think it might matter, but it matters a lot more once you get bigger. Because a lot of the decisions at such stage are driven by MMP last click, which is normal because when you are at that stage you don't have all the data infrastructure set up to measure incrementality and so on. Like I'm maybe generalizing a bit, but typical startup don't have all those kind of tools in house or third party like to make it happen. And once you get to a certain level, I would say 500k plus then these are becoming more, a lot more important and prominent in decisions because then you start with the yearly budgets and then the forecasts and everything and the profitability starts mattering a lot more than before and so on depending on, let's say your investor vision as well. At the end of the day you should just stick to one tool for certain decisions, while you should also cross check with other tooling time to time. Just as a baseline. I'm not saying for example you should use App Store Connect as your baseline. No. But if your MMP and meta are very different, but your App Store Connect, CPP data and meta matches, that should give you also some signals. Take it as there are source of truth that you should be sticking to, but don't take them blindly as this is it, like UAC is the best channel, let's double the budget only to see potentially the impact is output is not the double and so on. So that's why then it comes down to multiple tools and multiple ways of interpreting it. For example mentality and last click also conflicts or MMM might be conflicting with day to day decisions as well. Depending on at what level you do MMM or at what cadence you feed the MMM and how you calibrate it.

A (37:23)

Having worked on so many different apps and audited so many different accounts over the years, do you have any horror stories of like the data team's SQL was wrong and for a year they were just like making really bad decisions or, or the MMP was not hooked up correctly. Or like do you have any like just crazy horror stories from all these accounts that you've worked with? Or multiple crazy horror stories of just making bad decisions because the process wasn't set up?

B (37:53)

Yeah, I think story was something like this. They switched the mmp and then they didn't implement the events on the neve mmp. I think there are probably plenty, but there are also some must avoid at all cost type of things. Such as I remember one project switched MMPs and they contacted me after a month and saying like hi, we need some help. We cannot figure out what's happening. And the issue was so obvious. So they switched MMPs, they didn't implement any events on the new MMP and the old MMP was still connected to Meta. They still had campaigns optimizing on Meta and this is like a sizable company. But the problem is they have a different team handling everything mmp. They have a different team handling meta only kind of a silos and stuff. And I was like, how did you not realize this for almost a month? They're like, yeah, because not our thing and so on. So this stuff is like I think on a very extreme level. Other than that, I don't know. I've seen as crazy as someone spend 100k on a creative with hundred plus dollars and did not stop it before. I think it was like 90k. Something didn't stop. Like I was like really?

A (39:19)

With $100 cost per install?

B (39:21)

Yeah.

A (39:22)

Whoa.

B (39:23)

Relatively it was not crazy high compared to other ones, but it was high enough that you should have killed it after $2,000 max and not wait for a hundred thousand. But so that happens like when? I mean it's not an excuse, but it tends to happen in large accounts with no automation guardrails. Yeah, let's keep it this way. Like no guardrails in place and you're just really focused on execution and day to day business and you're missing out on hygiene checks, rules of thumb and all that kind of stuff because you're too focused on testing the maximum amount of creatives and so on.

A (40:01)

Maybe this is another good rule of thumb to come out of this podcast about not following the rules of thumb. But I don't know how you would divide it up, but maybe 80% of your growth team's efforts should be focused on testing and moving fast and all that stuff. But you want to leave like 10 or 20% to like be continually auditing your processes, continually auditing your data structures, continuing to audit your events structures and all of those processes that are in place to make sure that you don't end up in those kind of situations.

B (40:34)

Yeah. Like another example I just remembered is on the positive side of things, we were doing creative testing and then we did find the winner at the end thanks to interpreting the output and iterating on those output so it actually works when you do the iteration and find more winners. And we were able to, I think 5x the campaign budget within like two weeks with the same and even decreasing cost per purchase. And we were not applying 20% increase per day. If you see that your daily budget is 2k but algo spends 3k and you get the same performance, I was like, let me handle the bid and budget for this campaign because I see something, I see something very unusual. And I was so excited about it that we literally went hockey stick and the cost per purchase literally on a downhill. And that's like any UA manager's dream. And in those cases, like. But also the budget at some point was spending the double of the daily budget. But it is no problem as long as you have the performance. And talking about as a problem, like there are also companies who put, let's say overspend on a Sunday is a problem. But then again, it's just a fact. My approach is, okay, what do you do for it? Have you decreased the bids and the budgets? How did you try to overcome this problem? And then they're like, oh, it's just the algo decides on that. I'm like, yeah, but you can guide the algo. If you put a lower budget, if you have some automation in place, then algo be like, okay, they don't want me to spend more and so on. And there are multiple ways to control it. In a way you cannot control it 100% of course, but we can guide the algorithms in that way. Like another example is the capped inflated budget strategy where we have a million dollars a day budget at the campaign level. Which is not a good advice unless you put a campaign spend limits in place. Right? So as long as you have the guardrails in, you won't be spending 1 million a day. Of course, you will only spend up to your total campaign cap. And we tested this to bypass the pacing function, although we were still using the bid caps and the cost caps. And this was like earlier this year when we tested it and it was also recommended by Meta. And basically it did work quite well because we were able to capture a lot more higher quality traffic. We did not spend a million dollar a day. The spend varied across the days, but we were still able to get a stable performance with such a crazy budget that the algorithm is not pacing the day anymore. Because it is very large. Right? Because the way the daily budget works, it divides the budget to 24 hours in a way that of Course it doesn't. It never does it equally in a way. You know, you always have this kind of fluctuations, the evening times, it picks up and so on. But by putting a million a day, again, always with the budget caps and spending limits at every level.

A (44:00)

So, yeah, this is one that could get really dangerous. So I think if I'm understanding this correct, is it at the creative or the daily budget would be at the entire account level. Is that where you're setting the daily budget or.

B (44:15)

No, it's at the campaign level.

A (44:17)

Okay, at the campaign level. So you set the campaign level budget to a million. And the reason you do that is that Facebook or Meta, Meta will try and throttle your performance to not overspend in the morning. So let's say if you're. Let's say instead your budget was 100k and 100k is what you actually wanted to spend for the day. And the problem would be that Meta's like, ooh, they only want to spend 100k. We shouldn't serve too many ads this morning because we know the night's going to pick up and so we're going to like kind of throttle the morning ad delivery. And so what you're saying is like, at the campaign level, you set that to a million, but then you very carefully go through each bid set and set caps there. But then how do you balance those bid caps so you still don't end up spending. Or maybe you, you want to give it some freedom to maybe spend 200k or 300k or like. Yeah, so then how do you dial in those big caps where it does still allow the algorithm a little more flexibility in more aggressively spending?

B (45:20)

Yeah, so of course you start conservatively, right? Like if your cost per purchase is 50, you want to start, I don't know, around 45. And so like, or 10%, 10 to 20% below, and you might not get delivery. Then you play it slowly. And by the way, this is not for, let's say, not for beginners. So why I'm saying that because this works well at large accounts with large learnings that they're spending 50k a day anyway type of a thing, and they want to see if they can spend more than 50k a day, if they can spend 100k a day. And this is one of the ways to unlock that. And again, with the campaign limits. So, for example, how do you ensure that you don't spend 200k but you want to spend only 50k? You set a campaign spend limit and then you put the 50k as a, you say like regardless of how good is my performance, I don't want to spend more than 50k. So you put that 50k. But that 50k is not stopping your auction dynamics. Because auction dynamics are based on the bids and the budget and the cap is like as far as I know it is another control mechanism that is not going into the whole auction in terms of pacing and the bidding and all that stuff. So it's just a guardrail. So that's the way you control it. So you have the bid as a control mechanism and then you have the safety mechanism by setting up a max limit and then you want to spend another 50k, you're happy with the first 50k, then you increase your campaign limit to 100k, then you give that room again and then now you want to spend more so you increase it. So basically you are still in control. You're just bypassing the auction dynamics in this way. As an example, and we tried this for multiple accounts and we had some great results. It didn't work. For also some they didn't get delivery or when they get delivery, it was too quick, too much and then they had to start at 10k and 20k and so on per day and so on. I don't know what happened later. I know the cases that it worked and then the cases that didn't work. I didn't follow up on the latest. That is because I'm also busy with my full time job. But yeah, there are many this kind of things you can test and this is of course one extreme. You should not test stuff only for sake of extreme testing. But there are simple stuff as like the value based rules. You can test those bit modifiers that exist forever. But now it used to be available now only through FMPs. Now it's available within the UI. Like if you have performance difference above 20, 30% among the, let's say the placements or demographics and such, it is worth testing it. And yeah, so there are a lot of tests to do across all channels. So when you want to look for room for growth, like you should look at both vertically and horizontally. So horizontally will be like at the channel level. Okay, Are we really maxing out at every channel? Do I have the response curves of all the channels? Can I really double on this channel or rather spend x percent on this channel and Y percent on another channel? This usually will come from the MMM output because MMM will consider all your various sources of data. Although mostly it's looking at the impression span and conversions and such. But it does know the seasonality because you upload your last 24 months data and such and then when I talk about the vertical side it is about I don't know on Meta I gave already like a few examples on. We talked about the audience testing. We talked about like clustering based on CAC or LTV depending on how you optimize. We talked about the value based. There are like 50 plus I think tests that you can do. But prioritization of this test should come from your current data and then on UAC it should start as basic as okay, what are you bidding on? Are you bidding on Firebase events or your MMP events? Are you still optimizing towards an install? Are you optimizing towards a lower funnel? And have you tested excluding your brand keyword, what is incremental value? So at the different stages. So starting from, I think we talked a lot about the recent examples were more about like large advertisers with large budgets where they can invest into in house teams to build all this mmm or get a third party tool to basically utilize the outsource the mmm. But at a small scale you can also just turn on and off. Like large advertisers cannot afford to do that. But if you're a smaller advertiser, like if you're just spending, okay, I don't want a number now because I said small advertiser. Let's say if you can afford to turn off the channel regardless of your size, let's put it this way, if you can afford, just do it. If you cannot afford, then you have to use these kind of either the geolift or the blackout or holdout or there are tons of scientific statistical studies you can run for the sake of incrementality and such. But the most basic one is sometimes switch on and off and then observe the outcome. And again, there are no one right way to do things. It really depends on your size, it really depends on your market vertical and such. So I know some of our folks within our industry like they have great advice that are applicable to most accounts but they're not applicable to every single account as well. So that's why I mentioned at the beginning of the talk like take them as inspiration. Take them as like that. If you hear something this didn't work at all, be like great, I will try it. But later but don't be like oh, we should not even try that unless you have also evidence that it will also not work for you. Or there are cues because at the end of the day we have to prioritize how many things we can test. And yes, there are 100 things to test. Yes, it's good to utilize someone else's experience instead of you finding out by yourself. It's good to leverage other people's learnings. And that's definitely a good point. But what I'm just saying is just don't take it as, don't take out that test from your roadmap because someone said it didn't work for them was my kind of main message here.

A (51:38)

Yeah, this is a really fun. I feel like this was also a fun contrast even, or not a contrast, but a compliment to the signal engineering discussion in that Thomas talked about. And the industry's been talking more about signal engineering as a way to kind of manipulate the algorithms, if you will, to coach the algorithms, to influence the algorithms. And then so many things that you talked about today are also ways, especially those last few comments around the manipulating big caps and things like that. The point is there's just so many ways to tinker and tinkering with bids and caps and all those kind of things. And, and maybe for some, like, focus on creative, let the algorithms do their thing and you get enough performance and you don't need a big team and like, whatever, it works. And again, that's kind of your point, right? It's like, don't feel like you need to test all of it. Don't feel like you need to try the bid cap, but at some point, maybe you do need to scale and you're hitting a wall. There's a hundred different things you can try to break through that wall. And I think you've shared a lot of really fun, kind of contrarian or just maybe less talked about tactics to help break through those walls when you are struggling, when you've done the test and realize you do actually have a problem, it's not just a fact of the business and that there really is a problem. There's like so many different ways to tackle it and just try crazy stuff. So it's been a super fun conversation.

B (53:03)

Yeah, I mean, I also want to add one more thing to what I said in terms of the testing. Like, sometimes people hear something and they're like, oh, we heard. I don't know, someone said mapping trial to purchase works better. Like, oh, should we also do that? I'm like, what is your reason? What is your diagnosis on how did you come up with that hypothesis and why would you change stuff? So that's why also there is a misconception on, oh, let's just test everything. But that also sometimes contradicts with don't touch if something is working well. And the particular example I'm talking about is event mapping for example because you mentioned also single engineering and stuff. It's also at the basic level like sometimes you don't need to come up with specific demographics triggered events. Sometimes it's also about optimizing what event you map on meta side for example sometimes. So there are typically three options I see people do. They either map their trial to trial, they map their trial to purchase or they map their trial to subscribe to give better signal to the algorithm. And some people just do it purchase because everyone does it. So it is great as long as it works for you if you're happy, if you're profitable, if you've been using it for five years, if you scaled like don't touch it, don't try to change it because your whole learnings is based on that. On the other side, if you're not happy with the purchase optimization because your cost per purchase is almost higher than your LTV and then if you're not running the account because of that, then it's time to calibrate slash try something else. And of course this should not be the main thing to look at. You should first analyze the account, understand why things didn't work and what. And one of the levers also could be on the event mapping, the signal engineering side of things because when you optimize towards a purchase where the meta has a lot of information on and which is great because it goes after people who actually purchases things. But also you should think about okay, these people are purchasing anything. Just because someone bought a skirt doesn't mean that they're gonna try your app. You know, there's also a bit of the relevancy there. So I remember like I think it was sometime last year had an app that had a really high CAC and because they mapped their trial to purchase and then all we did was mapping their trial event to trial and their cost per purchase slash the actual cost per trial went down by I think around 35% only with that change, same creative, same campaigns, just different event optimization. Also their CPMs changed as well because if everyone is optimizing towards a purchase so and then the guy is, let's say this brand is optimizing towards purchase because his AOV is $200. He's willing to pay up to. Let's assume $200 versus a cost per trial's worth for your business is $15 let's say or $10. How are you going to compete with the guy who is willing to pay 200? From the auction dynamics point of view, that guy is spending more. And the way to give the auto bid a signal is you increase your budget, you indirectly increase your bid when you have the auto bid, when you don't have the bid cap and cost cap and then that guy keeps increasing and then you're just getting crushed and he's maybe getting the highest intent and you're getting the lowest intent purchasers and stuff like that. So there's a mismatch on what you want to get and what you tell the algorithm. And then on the other side, when you map your trial to trial, you're honest about your kind of intentions and the app because the only way for algorithm to know your business are the events you're sending to your events manager to your meta events manager. So also the events that you map should also logically make sense and should ideally have a funnel logic as well that I've seen. The other day someone mapped subscription cancellation to add to cart trial converted to something else and there's a subscribe event purchase event, there's everything. So for meta like what is this? Is this an e commerce shop, Is this a subscription app? Is this whatever. And all these advertising businesses that are worth billions are built on machine learning because machine learns from all the signals. And if you send this kind of wrong signals that doesn't make a funnel. And then the algorithms are calibrating the whole auction dynamics because they're optimizing their own ecpms with likelihood to convert based on your typical funnel and typical behavior. Like these are like my personal opinion, but I'm just interpreting a story here. Like algorithm is not confident to show that advertiser's ad to that user because there is much clearer signal on another advertiser's funnel based on this user's historic activity that is known to meta. The meta algo will rather show advertisers B's ad to that user and not A because he doesn't know if this user will convert or not. And he knows that if this user will not convert on the advertiser's A's ad, he will not increase his budget. And meta optimizes towards the spend the advertiser value. Right. So and for meta to work at optimal level, signal engineering is one thing like to improve what you're sending, but fundamentally you should not send mixed signals for the sake of what you're potential thinking as signal engineering as well. So in general, keep it simple, keep it logical. Don't send all possible events because you can. I think that's what also people do. Oh, like why don't we track also this? Because look, we can send this event. It's difficult to quantify what I just said, but it is just talking to Facebook product managers, knowing how this platform learns from the data, learns from the signal and so on. This is my interpretation of that. And I see that the simpler accounts also do better because of that. Right? It used to be different, like you had 50 types of campaigns and so on. Now you have the broad interest and the lookalike that is all you need. Of course there are more, but I'm just simplifying past versus now because algos know exactly whom to go after based on the signals that it gets and based on the signals that it has on the user. So the signals that your app has is if it's totally different than what users engage with all the other apps and all the other businesses, then there's a mismatch, right? And that's a whole business model of meta. It will find exactly the user, it learns from the users that it showed impression to, it gets the conversion as a feedback and then it will try to find more similar people. So when you send this mixed signals, mixed funnel, that doesn't make sense from the auction calculation point of view, then you're also potentially losing out on this kind of opportunities. So having said that, like if you're happy with your current event mapping, don't try to just challenge it for the sake of challenging. If you are unhappy and thinking there is a room to try something different, I would suggest create another event that gets triggered at the same place. For example the trial start. Add another trial start event in your mmp. Then map that to, let's say to subscribe. Then run a split test purchase versus subscribe to see which event is more resonating. Of course the history of each will not be the same. So I would at least wait a week or two so that the events manager will build some history and understand what does this event mean for their algo and so on. Again, it's difficult to quantify, wait one week versus two weeks, but it's just to build some data history and then run the test and then see how it is. And that's exactly what we did for this particular project and then we saw a 35% improvement just like that. So sometimes you shouldn't think too complex on okay, but maybe they also come back on Day three, then I will fire this super high LTV predictor event that you can also do that, but also just step back, zoom out. Okay, what am I telling the algorithm? How does the algorithm know me? Like is algorithm xyz. So it should start at this basic level and so on.

A (61:55)

Yeah. Which was to your broader point that there's so many things to play with. But don't break things just because people say you should try and break them. Like form a hypothesis for why this would perform better and then systematically test it versus assuming because everybody said signal engineering is going to work, that you should be doing that exact form of signal engineering. So yeah, it's been so fun chatting through all of this. Anything you wanted to share as you wrap up? I know you do take on contracting work, but you're also working on your own app studio if people did want to get in touch or are you taking clients or anything like that.

B (62:30)

Thank you also for your insights and always happy to help. Happy to chat when it comes to, like, MarTech, when it comes to UA challenges, when it comes to attribution CRO aso like, that's what I've been dealing daily for the past 12 years because of my main project. And always happy to chat. Always happy to learn from you as well and learn from the community. So that's how we all grow.

A (62:56)

What's the best place to reach out? Just LinkedIn or what's the best place to get in touch with you?

B (63:01)

Yeah, LinkedIn would be cool. All right.

A (63:03)

Well, such a fun conversation. I really appreciate you joining me.

B (63:05)

Yeah, thank you. Thank you so much.

A (63:07)

Awesome. Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.