A (3:20)

How would you recommend actually tracking that? Or like in your ideal state, is that just in a notion doc in a Google Doc or is it in a spreadsheet? And like the revenue changes that happen, like what's the ideal state of something?

B (3:32)

Like that look like basically the revenue bucket. So to say to me it's like Excel, Like I would track in Excel, like how are moving and then from that maybe have a little presentation for me or like then to share. And then you map, okay, here, you know this line, this happened because of this thing. And then you can like link to whatever you got to take or do whatever documentation that assists the company or you can put like within the same Excel file, if you stay in Excel, the links of what happened when. But yeah, I'm. I mean I'm a bit of an Excel person, but yeah, like everyone needs to find their own way.

A (4:07)

When you're trying to optimize conversion retention, when you're working on as you do revenue strategy at a company, all these little things add up to become the product in a way that is hard to untangle from each other. And so having this record means that, you know, when you ran this experiment, you can then go back and look at that cohort. Did it churn at a higher rate? Did they engage in the app in a different rate? How did they engage in the app differently? And so all of that leads to kind of making better decisions over the long haul and being able to look back historically and understand those decisions and maybe retest some of the assumptions from the past. But one of the big things is conversion over optimization. I think a lot of apps are falling into this trap these days of getting too focused on early funnel metrics at the expense of down funnel metrics. So what are some of the pitfalls you've seen in that?

B (5:03)

I think there's two major things that I keep seeing over and over again. One of them is this not letting enough time for the cohort to evolve and see what happens later. Right. What you mentioned, like what happens with the renewal rates. Like, okay, we have a Price increase. Oh, great. You know, we did amazing. We rolled this and then six months later you look at the cohort and say, oh, actually the control group actually is performing better than the test group because they are renewing more. So one of them is like this, not looking, giving the time. Sometimes it's fine to roll out, right? It's fine, you have a win, you roll, but then look back, right? And then the second one is within the same moment in time, like the same snippet in time, but not looking at the right metrics. Like, there's some times that we forget to look at some metrics. I gave like a really clear example sometimes with a windbag windback campaign, okay. Or like people that cancel the auto renewal, okay, when they do that, we're going to send an offer, right? Okay. They cancel auto renewal. We send an offer because then maybe we get them back. Okay, great. We do this. Great. Amazing numbers, okay. You look a bit more deep and you realize, oh, a lot of people are canceling, you know, like putting the auto renewal of right at the beginning. And what they are doing is they're asking for a refund and then they're getting the offer. So you are actually like net negative. So I think that's also something that tends to happen especially with refunds and chargebacks and because people tend to forget about those. Like they never happen. And yeah, you might be, even without waiting in time, you might be messing up with your system. So you really need to understand your whole set of metrics and how do they all work together. Because most of the cases something goes up and something goes down and you need to make sure you know what is going on.

A (6:53)

I hear this all the time with price testing where you double the price and you exactly match the revenue. So conversion cuts in half. Sometimes you'll get that win where you double the price and you do get a 25% lift in average revenue per user or something like that. But you always gotta be looking for those downstream things. Like you said, one thing goes up, another goes down. You know, when you look at the entire lifecycle of a subscriber, any one movement here can have downstream effects if you're not really carefully looking for it. And I think we're just in this mode as an industry of chasing payback as quickly as possible. And sometimes it's like it's just a business decision you have to make. You have to make that decision of like we're going to sacrifice long term revenue for being able to hit that roas at day, whatever 7, 30, 90, whatever you're targeting. But I think what's really important and what you're hinting at is like you got to know what you're sacrificing. You got to know as you push this number up, what number is going down and are you willing to make that trade? Because a lot of times people aren't tracking those down funnel numbers and don't know the trade they're making. What are some of the specific examples you've seen of that kind of over optimization and how, how it does impact the long term?

B (8:11)

It happens also a lot when introducing new plans. Like when you didn't have the yearly subscription or when you didn't have the life. I mean lifetime is a huge example because the value that you get from the get go like really high. And then every time, like every time you do a test like lifetime versus something else, like always, there's something else is going to lose because people are looking at what is the LTV of that something else, right? Like if they would have bought a tournament subscription, they are not bringing you the €100, they are bringing you, bringing you like €300. So don't compare it with the you know, 250 compared to 350 compared to 350 compared that. So I think like every time there's like a new plan that always like people please look at the lifetime value that this plan would have had. And then another thing that happened I said before, like a lot like refunds and chargeback is something that people tend to forget about. So if you are introducing a new payment method and things like this also like especially like look at those metrics, look what is happening there.

A (9:10)

Any other specific places you think people should be paying especially close attention to? I know you know paywall optimization is a huge thing and I've been frankly surprised at how big it moves. Folks have been able to see on the paywall like over this past year one of the like really big paywall optimization things that people have been doing is having a toggle to enable the free trial. So by default you don't have a free trial. You, you know, tap a button or slide a slider and enables a free trial. Now Apple has started to reject some of that. So I don't know if that's going to be something that is allowed long term. But I've been shocked at like you know, what a big lift that can have. But again I haven't talked to folks who looked at that cohort 6 months, 12 months down the line. I mean we're Just now, I mean I think that that started about 12 months ago. So you're going to start seeing those cohorts maturing. Anything like other things like that that you've seen that, that, that you saw very specifically lead to down funnel problems.

B (10:10)

Sometimes it's not so much what we said like down funnel problem and more like you are just putting the revenue at the beginning, right? Is this done more revenue than at the end or not? But one thing that happens that is really clear, like with churn, people usually tend to just think churn is a product thing, like churn is a product problem. But most of the time I'm not most of the time, let's say most of the time. But a lot of the time the like the commercial strategy that you have, what people buy, when they buy it, which price they buy it are going to have such an enormous impact on your renewals and extensions. And what happens is that people like in the marketing side they don't really, you know, see in the indexes and then the product side they also not have a view on what is happening at the beginning of the fun in a way. For example, something that I think is going to happen right now is we have now the web checkout, right? In iOS you can send people to buy in web. What I've seen in different companies is the web renewal rates are significantly higher than the upgrade. So what is going to happen now is a lot of teams, a lot of product teams are going to be like wow, like our product is amazing. We have increased our renewals by whatever 20% if they were to slice. For what is a cohort of users that bought natively in apps? What is the cohort of users that work through web? Maybe the renewal rates are flat and what just happened is you are acquiring users through other means. The same can be said of course for the subscription plans. Like if you have like your share of users that buy a one month subscription is bigger or like it grows for whatever reason, your renewals are also going to get better if you look at the overall number. So I think when you're looking at data like cohort of the different users, what they bought, when they bought it, if discounted or not discounted is also another clear example. Like sometimes you do a discount, you bring a lot of revenue, but then those users, they are not going to renew or sometimes you do a price increase, you bring more revenue because your conversion goes down but not as much as your, as the pricing list goes up. But then those users you have like less pool to upgrade users and things like this. So there's a lot of, a lot of different colors to this problem.

A (12:26)

So what are the specific metrics and ways you dive into the data to better understand that? Are you looking at both subscription retention and usage retention of very specific features like how are you watching that entire funnel? What's your preferred way to look at it? What are the metrics you most look at?

B (12:46)

So what is most important to me is which plan did they buy discounted or not and where. Like you know, web versus app. Then sometimes in some and this is something that you need to like look for your company, right. Sometimes the marketing channel might make up for different users and cohorts. Right. So understand what are the different ones. And once you have that, then you just look at that, you just look at that cohort and then if you see that the cohort is going up or down, you know, okay, probably separate vaccine. What usually happens is that this, the share for those different cohorts is changing over time and this is what bringing you fluctuation on renewal rates. It might be different when you're. Yeah, when you're doing a specific experiment, you know, that might change that. But I think it's really always good to look at what are those things that make a difference that are upper funnel, as upper funnel as it gets, so to say. And then look for that. And then when it comes to when I'm looking at a test that I do in the paywall onenote then I'm looking at revenue per user of course. But then I'm also having a look always on refunds at Charlesback so I always wait like a couple of weeks what is happening there because that's like an immediate effect that you can see. And then always after some months recheck the cohort, you know, like what has happened with these people. Sometimes it's a bit frustrating because you have the majority of users buying a 12 month subscription so you won't see it until later on. But maybe there's something that you can do if you see that the one month behaving in a particular way, maybe you can assume that the same would happen for the 12 months. You can also look when you have the result, make some scenarios of renewal rates of the different, you know, if the one month was to renew at this, at this, you know, like 10% less or 5% more, whatever you want based on the result of your test, what would happen. And then you can know once, you know, like you have some like really easy way to check in A way like in a sense like you can you keep an eye? Okay. If it goes below 10, then I know we are in trouble. I know that probably the. The winner is actually the loser.

A (14:48)

And this gets back to what we started with was keeping a detailed history. Is that when you do run an experiment, do you set like an alarm like you know, three months from now?

B (14:59)

Yeah. My Google calendar up there has this experiment recheck with my laby data analysis.

A (15:07)

Yeah, gotcha. So you are looking back at specific intervals and do you do that for. I guess you wouldn't do that for every test. What's the criteria for the things that you're most care to look back on?

B (15:18)

I think like anything that is price changes I really care about. Because for example if it's about oh the layout is different or the copy is different, then sure maybe there's a difference in renewal by why would it right. Like so there's certain things that are like probably not so I don't know. You shouldn't be spending a lot of time. The thing is also the more you do it, the easier it gets. In a way it's just like another part of the of the analysis. So to say it's already set up that way so you have already a cohort is like okay, you pull back okay here. And sometimes it's even with some tools it's even there that you can just recheck this test how it perform. So it doesn't need to be like really time consuming. But usually everything, everything that is with price or discounting or things like this I would always recheck after. Yeah. Depending on what we are selling, you know, like three months, six months. And then if you have to wait

A (16:18)

a year, it seems like too maybe the priority would be to make sure you're checking back on the things that made the biggest move. So maybe there was a copy change on the paywall button that did a 25% lift. Which sounds crazy but it happens sometimes where you have these like crazy lifts. And so maybe those kind of things would also kind of fall into that bucket of like anything that made a surprisingly big lift mark that one to go back on.

B (16:47)

Because especially if then you are not seeing. Because something that happens a lot is oh, we have all these ferments all positive. But then when you're looking at the numbers, you know you're still flat, you're still improved, but not that much. Especially if you see that then it's time. It's more time to reject. Like actually we'll say if you see that the growth of your company follows all your ab retest increases, then it's less of a, you know, like red flag. Maybe, like maybe you don't need to retest. You seem to be doing fine. I think that. Because I think a lot of the times we're like, yeah, all this, like, you know, I look all the list of, you know, experiments with like 5%, 10%, I'm like, but wait a second, like what, why we are not growing, you know, like 30 year on year if we have all these wins. And I think this happens also like with product, like with, with you know, like a new feature, whatnot. Yeah, this lift. This lift. But then when you look over time, it's not really holding together. So I think there's also like, I don't know why it happens. Right. But there's this like mismatch between what your A B tests are telling you and what the long term effects is telling you.

A (18:00)

That that is such a great point. It's like you can stack all these wins. Oh, we got 10% higher completion in the onboarding step. Oh, we got 15%.

B (18:09)

And it's sort of compound. It should even compound. So it's even like way more than, than that and it's not.

A (18:16)

So yeah, tracking and making sense of all this is such a challenge. But that's why you need to do it. You're not actually getting that 10% increase if you're not actually getting that 10% increase. And you're never going to know if you do these experiments in isolation and aren't tracking everything. So it was so fun to chat with you about all of this. Thank you so much for coming on the podcast. Anything else you wanted to share as we're wrapping up? I know you just started this new role at yousician. Any job listings you want to shout out or anything else like that?

B (18:46)

Yeah, we are, we are looking for people. It keeps changing. So I will just go to jusician.com careers but yeah, it's a really fun company to work at. So like helping people, learning to play an instrument, that's a great thing to do. Yeah.

A (19:01)

Awesome. Well, thank you so much for joining me. This is great.

B (19:03)

Thanks a lot.

A (19:06)

Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.