A (6:29)

And I guess you weren't there at this time, but I did hear there's a story out there around a venture capitalist pushing this direction. What was the story there?

B (6:37)

This story I think is when Capital G invested in Duolingo. Leila, who was also on our board after that investment for a while, she told Luis at the time, this is probably the last investor you will have in this company before you figure out monetization. So in a rounded way to say you have to figure out monetization as a growing company, you at some point have to do that. But I think also the common wisdom at the time for consumer companies like Facebook is like, just grow audience, you'll figure out monetization later. So I think Duolingo followed that, but until a point where I think Capital G and Layla had this big push to be okay, it's time. And I guess that was around 2016. So when I joined, the mindset shift had moved to Luis. Like first thing he told me was like, we hired you. Your sole job is to figure out how to monetize Duolingo. And that's what I tried to figure out.

A (7:31)

That's awesome. What a great story. So ads, they're tricky. How did you introduce ADS1 in a way that didn't totally piss off users and tank retention And I mean, I'm sure there was user pushback and bad reviews in the App Store. This is part of the story I think a lot of folks can learn from because any monetization change you make will spur some level of like bad reviews or negative feedback or churn. So how did you balance that as you started to roll ads into the product?

B (8:05)

This is a very good question. I think that there's the. I'll give the obvious answers and maybe the harder to navigate parts first. For a consumer product, you should obviously dog food your product heavily. So. So that whatever you're doing to the experience, you have firsthand experience on how it feels. Meaning you can easily say, oh, we'll put an app open ad. And if you're just watching metrics, I'm sure that might look fine. But when you test it, you're like, what the hell is this? Then you should know what you've introduced and obviously avoid the things that you wouldn't want to see as a user. Hence our first test. And honestly, kind of even, that's our only ad placement is after you've computed a lesson. We show you an ad not during, not before, but always after because we believe that doesn't interrupt your learning. So one is the qualitative. It should be a good user experience and you should know that by testing the app yourself every day. The second one is quantitative, like measure everything you can. Like, is retention fine? Is app closes fine? What's happening to your reviews? You might have to make some trade offs. Ideally you wouldn't. But realistically, if you go from zero ads to some amount of ads, there is some risk that some curves might come down. But I think you should know what that cost is anyway. That's like through a B testing. That's how we measured it. To be honestly, our retention was maybe a slight hit, but mostly neutral for most of our ad tests over time. That's one. The third one is there is of course you go from a free product to selling something users won't always say, thank you for doing that. Like this. Imagine if Uber started completely free and, and one day they said, okay, now we're going to charge for our Ubers. Users would hate that too. And that was, that's kind of what we did. I mean, in a way we provided a free service and we kept A lot of the elements free, but we said, hey, actually you can buy a streak repair now and you might have to see an ad at the end of your session. We got some complaints. The thing that helped us have clarity was one retent Retention was fine. Like retention was neutral on most of these experiments. And when you looked at our community's complaint volume, if you quantify that, for example, per a million daus, how many ad complaints are there? Or ad negative reviews, however you want to count that versus other topics. Like for example, I realized I did this quantitative analysis in 2016 because we were at this pivotal moment of like, do we launch ads or do we not launch ads? And we looked at for example, the number of complaints about why we didn't have the Finnish language on Duolingo was 3x the number of people that are complaining about duolingo having ads. So that starts putting it into perspective. Like it's not zero complaints, but actually in the grand scheme of things, this is not going to break the app and our community is not going to run away tomorrow. So I think understanding the volume of what you're doing to the community as much as you can gives you perspective. Otherwise, every negative comment as a product builder hits home really hard. Like everything looks like the world is falling apart. That's why it helps to like look at the broad picture of like all of our issues as a product. And all products have many issues.

A (11:09)

Yeah, that's exactly what I was going to say is that it's so hard as a builder, it's really hard to get those few negative reviews. And I think that's a really helpful insight to look at that bigger picture and understand. Like I got a bunch of negative reviews from my weather app for the audacity of charging $40 a year for a weather app. And I did get probably over focused on those negative reviews. But then when you look at all the people who were we're subscribing, were staying subscribed, actually have pretty good retention. It's like you got to like zoom out and not let those like that small subset of users who are going to complain tank what's ultimately going to help drive the business forward. How did you work on ad quality? I believe you implemented AdMob first and again in my weather app I had AdMob. Eventually I ended up pulling it because it was like gambling ads with like, you know, 777 Kevin and animations and like all this crap. How did you work with AdMob and how do you think about ad quality? As part of the experience and how it can actually be detrimental to the experience.

B (12:14)

Yeah, I mean, ad quality, I would say we are like incredibly strict on. But with Programmatic you'll of course have some amount of control. Not full control, but with anything that is honestly, it could be even slightly dicey. We blocked all those categories. Like if it's gambling, even if it's like games that look kind of sketchy, pick your category. Any of those that we didn't feel feel was also young audience safe and or just brand safe, we blocked. So the starting point is we used AdMob first and then we integrated other sources like Facebook ads or at the time Flurry. And now we use also, I think multiple networks. We use Unity. That was to mainly broaden demand. But for category and what kind of ads we allow. We tried our best to block off any categories that are dicey, even though it might be a revenue loss. We were like, we don't care. The most more important thing is good user experience and ad quality. So we did it. We did a lot of category block, category blocking. And if we needed to do custom blocking, we went and tried to work with ad networks to be like this type of ad has like 3x buttons to exit. We don't want this in our app. Please don't give us this thing. So we also did some of that back and forth as well. That's our stance this year also to improve ad quality. We're also doing our own direct ads more and more because then we control quality entirely. Like it's an advertiser that we know like we shook hands with and the format is what we've created. So it's beautiful and it's a great advertiser. So one now new lever for us is to reduce programmatic volume and increase direct volume because it's just a better user experience.

A (13:43)

Yeah, I love what you are doing. I mean, it's hard for smaller apps. You don't have scale to do this kind of thing. But tell me more about how these partnerships work to improve that ad quality so that it does improve the user experience. Because you'll be incorporating duolingo characters into the ads with the brands you're partnering with.

B (14:00)

Have you seen them?

A (14:01)

I haven't seen them yet, but I read about them.

B (14:03)

It's very cool. It's currently only in the U.S. so we're focusing on growing these in the U.S. because obviously we created a great format. But to be able to scale this, we also need to be working with companies. And right now we're just focused on companies that would want to advertise with us. In the US we created multiple formats, but the one I love the most and that seemed to be working really well is one of our characters. So we have eight duolingo characters on Duolingo that you see is it introduces the ad. For example, pick a brand. Let's say it's a detergent. I'm making this up. It's not like we had exactly this, but one of our characters, Lily, would make jokes about, like, oh, like, I need to wash my clothes. I wish I could do that. And we kind of write a custom script to introduce that ad that's very short. And then the ad itself plays so that at least we are building a little bit of our character world further. And it's not jarring as, like, you ended a lesson right now. That's what our ads are really a little bit where, like, you see an ad, it feels like you're easing into the ad. We're trying this right now. It's when you see it as a user, it's actually like a. As an ad, it's a very delightful format. Our challenge, I would say, is scaling it. I think we. Since it's a custom format, well, you got to go up to advertise and explain how this works. So it's. It's a bit of a slow adoption process, but we're going to. I mean, our goal is to basically increase the volume of these ads and reduce the programmatic by a good amount.

A (15:24)

In 2026, one of the ads you do often show is for the subscription itself. How do you balance from an LTV perspective and monetization perspective, how do you balance when to show those ads for the subscription and then when to show programmatic or these new brand ads.

B (15:43)

So we just. To set the stage, if you do a duolingo lesson, and this includes any of our lessons, so you could do language, math, music, and chess. At the end of that lesson, you see an ad and we decide, like you said, if we should show you a programmatic ad versus a one of our internal ads, which could be an ad for a subscription, or it could also be an ad for our other products, like the Duolingo English Test. But we call those basically internal ads that we decide we've produced. And it is a duolingo product. Majority of them are going to be either super duolingo ads or Duolingo max ads. The decision logic is mostly now controlled by ML, actually, but the ML uses various signals that is trying to figure Out LTB maximization at its core, and it is looking at every signal that we feed it. That could be an indicator for someone that is more likely to buy super. Or it also balances out, of course. Like if, for example, it's shown you the same. It doesn't really show you the same super at three times, but it's showing you super ad three times that clearly you're not interested in, it backs off and shows you other ads. So it is basically built to optimize LTV with a likelihood to purchase being the objective function.

A (16:57)

A lot of apps that have ads in them don't do this kind of thing. But, like, as cheap as the CPMS are these days, I think all developers should be experimenting with putting ads in for the premium product. It's a perfect way to kind of balance that monetization of if you have a freemium product and you're not making that much on ads, but you just don't and actually follow up with a question there. But it's a great opportunity to remind people, hey, there is a paid product. There are features you can get. And highlighting the benefits. I've seen quite a few of the ads early in the journey when I was first using Duolingo, and they're really good. They really highlight well, hey, you're doing great, but you can do even better. And they really highlight the product features. And so I think that's a great opportunity for freemium developers who do have ads to be exploring that more to kind of push that upgrade to the subscription 100%.

B (17:48)

And I think the other reason I like our ads is they're just better than programmatic ads. Like a user.

A (17:54)

Better user experience.

B (17:54)

It's just a better user experience. I mean, I think if I had a magic wand, I'd rather have no programmatic ads within my app and only internally produce ads because they look better. And we're also building kind of lore for our world characters through. Even though they might be talking about super, they're also cracking jokes about their own. For example, we have this kind of controversial super ad. I don't know if you've seen this one where junior, one of our characters is in a bathroom stall and he, like, peaks and, like, makes, like, a roll joke because there's a toilet roll that he's, like, peeking under. It's kind of weird, but it's like we're kind of also entertaining a little bit with the ads we produce because we have produced them with that intent. It's way better than showing programmatic ads.

A (18:35)

How does ad monetization do per user compared to subscription? I would assume part of the idea with these partnerships and other things you're working on is to continue pushing that monetization better and better for those free users.

B (18:50)

So for us, if you look at Duolingo's statistics, subscription revenue is far and above the biggest chunk of our revenue. So I think it's above 80%. And ads is a much smaller percentage. But if you look at payers among our DAU base, actually MAU base, it's about 10%. So 10% of our users subscribe. However, subscription is more than 80% of our total revenue. So we focus naturally when it comes to what we think about for monetization mostly on the subscription side of things, because we believe it has one much. It's already a much larger business, but also much higher potential because we still feel like with any of these actually, but certainly still, even with the subscription that has now become a sizable business, there's just so much more to do that is making the subscription better. And also with the freemium app, you realize over time, and this is a lot of experimentation we've done, one is like building the product to be better, so making our subscription features more valuable. But there's a lot of work to be done. And also making sure you can communicate those well, we touched on videos. Videos certainly help because that gives you a format where you can actually say, hey, this other thing that you could buy has all these features. But because there's not a lot of other easy places to tell that story, on top of that, there's like how you design your purchase pages make a difference. For example, how users enter that. Enter that flow makes a difference. So anyway, that's in my head how you're communicating your subscription. So we're putting more and more of our monetization energy into those pieces, meaning make the subscription better and more useful and make sure we can communicate it well, that tends to have higher return for us in general as a business than ads. So most of our efforts is on subscriptions.

A (20:29)

Yeah, I mean, you know, Facebook's numbers are just insane. You know, I think I saw like 250 per user per year in the US which is about what a duo plus subscription costs. But the time in app, the way they're able to do their bid structures and things like that. But I imagine there's still a lot of room to grow. But it's interesting that at Duolingo scale, the subscription opportunity is still so much better than the, than the ad monetization I think so.

B (20:56)

And I think with social media apps the reality is it is fairly easier, I think as a product problem to integrate ads into a feed and in a product where users are there not for a particular utility, but they are there a lot of times to pass time or they don't even know why they're there. So I think it's way easier to increase ad load in a way that is not disruptive to how the user feels when you are on an app for a particular utility. And for us, a lot of users are there to learn a topic. Well, not a lot. All of our users are there to learn something. I think increasing ad load is not particularly that tolerated. I guess so I think that that creates a challenge for a challenge and opportunity. Honestly for us, subscription just fits as a better business model than increasing ad load the way Facebook might be doing.

A (21:47)

Yeah. One last question on ads and then we'll move on to the subscription. But one of your, the product managers on your team came on the podcast I think 18 months ago and he talked about holdouts that y' all long term holdouts. Did y' all do a holdout on ads? Are there people today? Is there like 001% of subscribers that still don't have ads that you can measure against the people who do have ads?

B (22:12)

Not anymore. But we did run I think almost like a year or two year long holdout for ads. We do run long holdouts, although I don't recommend year long holdouts because what also eventually happens is you forget that you have a holdout and then eventually you introduce. You're not testing it every day, you introduce a bug to it. So like even though you have a year long holdout, you realize at some point you actually polluted it because you like introduced bugs there anyway. So very long holdouts we are anti just because of a practical perspective, but we do way more like three month, six month holdouts on things that we know have different short or we assume will have short to long term effects that might be different. One of those, for example, this is not exactly monetization but anything that has to do with our leaderboards we try to run long period experiments or long holdouts because on the Duolingo leaderboard set, if you start from the lowest league, it takes you 10 weeks to get to the highest one. If we change the dynamics of how the leaderboard works, we might not see the actual result in two weeks. So we kind of let these run for longer. Another one is push notification experiments where every of course new Push notification you introduce has amazing returns because it has amazing novelty. But then if you wait out six weeks, sometimes it doesn't just come back to neutral, it actually turns negative because you're sending too many push notifications. That turns people off. So some stuff that we assume not just monetization might have longer term adverse effects. We run long holdouts or just run.

A (23:40)

A long experiment the way Duolingo tests things. We could talk hours just on that. But I did want to get back to the monetization journey. So at what point did it become clear that you needed to introduce a subscription? And this was pretty early in the, you know, Apple famously opened up subscriptions to all apps in 2016. I believe you launched a subscription pretty soon after that.

B (24:03)

17.

A (24:03)

So what was the story from. Okay, ads are starting to work. We haven't pissed off all the users, but now we're going to like actually start charging.

B (24:15)

So the journey we went through was ads was the first experiment. And then after ads was introduced as a good pm, I was reading every review that was written on on Duolingo that included the App Store or Reddit or our own forums. And there was a significant amount of people that are that were saying, can I please pay you guys? I don't want to see ads. And I love Duolingo. I get so much value out of it. It wasn't like a hateful I hate ads. It was more like I would love to give you guys a few dollars just to remove my ads. And. And we thought that was a good idea. And subscriptions were quite early for mobile apps at the time. You're right. Majority of the app wisdom was generally you want to sell IAPs and I think games had obviously kind of popularized that and that was pretty common. Subscription apps were earlier there was the big like streaming and entertainment, but outside of that there wasn't a lot of digital subscription behavior. So we took that idea to heart. But we said we're not sure if the subscription thing is for us. Our mind was still like, we probably going to have an IAP model. So we tested an IAP which was honestly like looking back, it's so janky and it's not clear why people bought this. But we said you can buy an ad free month of duolingo. So for 30 days we'll turn off the ads, but it's a one time purchase. And we tested different price points. We tested $3, $5 and $10. And everyone thought this kind of wouldn't work. Like, it's like, why would people pay this one off bizarre exchange. Honestly, the uptake was kind of good and way more than what we expected. So we were like, clearly there is some demand for turning off ads. Even though we created the most bizarre package to sell. People want to buy this, let's make it a subscription. But to make it a subscription, just no ads felt slim as a subscription service. So then we built offline capabilities. So we said you can use Duolingo fully offline and turn off your ads. And then when we designed the purchase page, we're like, this looks really weird with two features. We need a third feature. But we couldn't actually at the time figure out what the third feature should be. So we kind of wrote a line there. That was the thing that users were telling us, which is support Duolingo in its mission of free education. I think we just said support Duolingo's mission for short. And that was our first entry into subscription. That was Duolingo plus. And honestly what I think we saw happen was probably majority of users early on bought it for the fact that they enjoyed Dueling us so much and wanted to contribute back the other half. Or maybe the half of the demand was from no ads and minimally for offline. And that kind of started us giving us a run rate obviously. I mean, I'm happy to talk about it more. But from then to today the subscription has evolved a lot. But that was the very first version.

A (26:57)

Yeah, that's fascinating and I do want to go through that journey. So that is pretty light. I mean again this was 2017, so best practices and everything else hadn't fully evolved and this was just a experiment. But how did you start evolving that over time? What were, what were some of the. And maybe just kind of give us the highlights of the last almost decade now. Eight, eight years. The kind of biggest unlocks, when we added this to the paywall, when we added this new tier, what were some of those big unlocks over the past eight years that really kind of moved the business forward?

B (27:31)

There's a few, and let me Try to think, 10 years is a lot of time. There's a few levers that have, looking back have really either caused step function increases immediately or over time. It really compounded. So one of them, I guess now, if you work in subscriptions now this is like kind of a no brainer. But at the time it really wasn't. Having free trials is one. I actually as a pm, I resisted free trials for a long time. I assumed they would like completely flop.

A (27:57)

Product was already free.

B (27:58)

Yes, Exactly. It's like, why would people want to try free trials for the exact same product they've used for years? The product is free. Like if we were a fully premium product, free trial in my head made a lot of sense, but for freemium, like you said it didn't. I thought it didn't make sense. Turns out I was wrong. Free trials are really helpful for the obvious skeptic user that might not want to pay upfront, but they are interested in trying it out. It just makes it so easy to just jump in and try. It's very intuitive. Now as a user, I love free trials because I'm like, well, I don't want to commit to pay. Let me give it a go.

A (28:33)

Were there specific things that were then behind the paywall that people needed to experience?

B (28:39)

Not really.

A (28:40)

Really?

B (28:41)

You can say offline, but it's offline is the same as online. It just works when you're in airplane mode. So if you imagine this subscription which is no ads and offline, you didn't really need to experience it. But I think it just removes the payment hesitation so it lowers the barrier to entry. So you're not committing payment. You're like, I'll give it a go. And I think that's the real benefit of free trials. So at this point we have, we're very pro free trials and I think our purchase flows are now designed to highlight how our free trial works as a mechanic. We even have designed pages which like communicate, hey, there's a seven day free trial. On day five, we're gonna send you a notification to see if you still want your subscription. Like, because I and this has performed well in a B tests because I think users have a subscription aversion and I think in a lot of countries, maybe not as much the U.S. there's also subscription scam aversion where you click on something you somehow like can't get out. I think if you highlight how what a free trial is one it's like you don't have to pay anything really. And what the mechanics of it, it relieves the subscription aversion.

A (29:44)

Yeah. While we're on the topic, I know you recently did an experiment with a try for free button.

B (29:49)

We've been experimenting with a lot on everything our purchase pages and maybe it's worth talking about. You mentioned AB testing. Maybe I'll take us on a tangent for 10 seconds. So Duolingo concurrently runs somewhere around 400 A B tests. So the Duolingo app itself is heavily evolving real estate and a good chunk of those, not majority, but a good chunk of them also happen on our purchase flows. So if you look at any element, whether that's like, Duo's flight path on the super page or what the copy we write, how it's highlighted, how it's underlined, what the button color is, what the button copy is, all of those elements, we have teams trying to figure out how to make them better. And of course, for the record, we're talking a lot about monetization. That's not the only place in the app where we're doing this. Like, our onboarding flow is the same, our session end is the same, our lessons are the same. So we take kind of like maximal care into improving all these flows and. But monetization is no exception. But that button copy, we've experimented a lot, and that is one learning that I honestly didn't have or didn't really expect. What you say in the purchase button makes a massive difference on your revenue curves. We have tested many things. There's a couple magic strings that were like, you look at the graph and you're like, there must be a measurement mistake. Like, why are the numbers so green? The two specifically were. We went from, I think, start my free trial. So I think we said something like, subscribe, that's your baseline. And then we tried many things, but one of the big up list was start my free trial. That was very good. And then we tried. What did we say? Try for free. That was better. And then we tried try for 0.00. Weirdly, that is the string that I think hacks your brain to really grok that that button is entirely free. 0.0, like two 0 decimals now. I think that's what's live in the app, if I'm not mistaken. And that performed really well.

A (31:43)

And just changing try for free to try for dollar sign 0.00. You saw a meaningful uplift in conversion, correct? Yes. And it was, I guess, the start trial conversion, but then also the trial conversion to pay.

B (31:58)

Correct.

A (31:59)

So, like, both numbers went up.

B (32:00)

Correct.

A (32:01)

Wow.

B (32:01)

Yeah.

A (32:02)

That's so crazy.

B (32:03)

Yeah, it was unexpected. Honestly, it's like you wouldn't think this makes that much of a difference. I think one other thing I've learned with freemium and honestly, any subscription and paywall behavior is a lot of. When you go and design these flows, you are designing with logic, right? You are saying, let me communicate my features, so whoever is going to buy should read them and understand what they're buying so they can do the logical exchange of if I Part with this many dollars, I will get this many features. That is how you go and design a flow. But the reality is humans do not experience mobile flows that way at all. Actually, it's way more based on emotion and reaction than, oh, let me read and think about the value exchange of what, what is happening here. It's way more emotional. So I think these, these things that are kind of sound like, you know, they're saying the same thing, right? 0, 0 and 3 are the same thing, but they create different emotions make a difference.

A (32:55)

That's fascinating. Well, we've dive down multiple levels of rabbit hole. But to surface us back up in thinking through the past 10 years, were there any other kind of key unlocks, specific features that you put behind the paywall that really move the needle?

B (33:12)

Definitely. So we talked about free trial. Another thing for us that was quite a meaningful turn in our evolution was honestly putting more useful features behind our subscription. So I started with saying, okay, we had ads and offline, we started adding. We had, we added two features that were basically transformational to the subscription. One of them was Duolingo had this pacing mechanic called Hearts. Now it's called Energy. But Hearts was not integrated with our subscription at all. The way it works is you can go through Duolingo and do as much as you please, but if you make a certain number of mistakes, you start running out of Hearts. And then before you can do new content, you have to regenerate Hearts. And you can do that by doing practice lessons or you can do that by watching ads or at the time we said you can renew your Hearts with an in app purchase. We decided to integrate the Hearts mechanic with our subscription and said, hey, if you subscribe, you actually get unlimited Hearts. That was a big turning point because turns out a lot of users don't want to practice old content, they just want to do new content. Even though practice is good for your learning, we said, okay, if you would like, you can choose your optionality if you subscribe. That was a good turning point because it made the feature set of Plus Duolingo plus at the time way more integrated and way more useful. Another feature we created was again around Practice. Custom Practice. So we created a feature that is a separate tab called Practice Hub. We made that part of super at the time. Actually it's now became fully free. We now gave it all to free users because we realized it should be free. So we did that. But at the time we said, if you would like to do very custom practice on the things you've learned On Duolingo, here's an extra place where that's served to you as an inventory so you can go back and practice anything you'd like in whatever order you like, rather than the order we prescribe. Anyway, this is a very long winded way to say feature improvements that expanded the feature set of Duolingo plus was naturally a win on two fronts, both conversion and retention of the subscription, which is great. We love that and that gave us a lot of scale. Other than that, honestly, it is rare to have put like one silver bullet that improved things for us. A lot of this has been continuous a B testing improvement. I think we talked about the copy version as like one thing, but imagine that happening across every bitten piece of our purchase flow. And I think that's why Duolingo's purchase flows get copied a lot. Because I think people have heard us say this, I say this a lot, which is like we did a B test many things. So if you want to copy something, probably you're copying something that has been tested, at least for Duolingo. It probably won't work for most apps, but you can look at us because.

A (35:55)

We'Ve tested stuff that's fascinating. Before we move on, I did want to talk one more angle of monetization. I know by the time this podcast airs, you'll have taken something that was previously paid and moving it into the free tier. Why? Why do that?

B (36:09)

So let me talk about actually two features that we. One of them we just did and we are about to do so. Practice Hub I talked about. We realized that number one, as a goal, we would like to provide more value to our free users. Because free users, there's. There's the missionary reason why we want to teach more people, but there's the business reason, which is free users are our marketing engine. Duolingo grows as a entire user base and as a business through word of mouth. Word of mouth is much stronger if your free product is stronger because you might or might not pay. Irrelevant. If you're using Duolingo, you tell your friends about Duolingo and say, this is awesome. I'm learning Spanish. They try it, they do the same. So that's how Duolingo really grows. That's the majority of our growth engine. And that growth engine, of course, gets stronger if we can provide more value to free users. So it's a delicate balance if you're a freemium product, to decide how much value to put on each side. Right. And we certainly want to put value as much as possible to our Free user base. So that's the overarching reason on why. But the two features that we did was Practice Hub and the second one is Explain my answer. These are really, I mean, I love these features because they are really additive to your duolingo experience. So Practice Hub, we talked about. I'm not going to repeat it, that is now actually already live with free users. And explain my answer is moving from being a Duolingo max feature, which is our highest tier, to free users because we have realized the revenue value is not that high. So we want to. Meaning it doesn't make that much revenue.

A (37:43)

How did you determine that?

B (37:44)

We tested it. It. We tested giving it for free and compared it to a B condition or a control condition where it is part of max. The impact. Well, there was impact, but we decided to kind of, you know, reduce our revenue, but rather provide value for our free users because we believe that's the right thing to do in the long term. And there's one more reason, which is when we first introduced this feature, it was financially impossible for us to make it free because it uses LLM calls to explain. Basically what it does, for the record, is you give an answer, you answered something, and we basically use LLMs to describe to you what you got wrong for your specific answer. So it's very like tailored. That's why there's an LLM that is used LLM costs were really high, so we had to put a part of a subscription just to provide the service. But costs have gone down significantly and we've improved how we cash things. So since costs came down, our only decision was revenue loss and revenue loss. We decided to take to provide this honest, this service for.

A (38:45)

Yeah, it seems like that's a real theme, is being very strategic about what's free and what's paid because of that balance, because you want to create that amazing free experience, but you still monetize. And so that tension must be difficult. As a product team.

B (39:02)

Yes. And you know, honestly, the hardest part is, for me, it's really easy. I think for like CEO, it's very easy. We always want to do the right thing for the user. It's not hard for us to think about, okay, how do we improve the free experience? There's some obvious things the hardest part is taking a loss in an experiment is not intuitive. So the teams that are working on these are finding it hard to be like, I understand all this good stuff you said, but we're going to drop our metric. It's like, it doesn't feel intuitive. So I think convincing the organ, convincing the teams, and explaining why this is the right thing to do long term is the hard part, because you can imagine the positive side of what I'm saying doesn't really have a metric, but the cost of it has a very real metric. So I think that they run hand in hand. But I think you have to realize, especially if you're a freemium business, is if you only always go by positive revenue metrics, you will burn yourself to the ground. Like, I don't want to call out specific companies, but you can see a lot of subscription businesses that have effectively nickel and dime their users if they're freemium, and then burn their growth engine even though their revenue kept growing. But that only works for like some amount of years. But eventually your growth stops. I think for us, what's most important is our growth engine, our user growth engine keeps growing. That's why you have to make these hard calls sometimes to be like, we're going to take a loss, it's okay because we're helping. Our growth engine is the convincing part, is the hard part.

A (40:30)

How do those calls get made when an A B test shows a benefit? And I imagine this happens in more than just these, like, monetization features. But how do you balance the kind of art of product management against the science of product management when you run 400 simultaneous A B tests?

B (40:48)

This is a great question. I think this happens a lot for us, I would say, because we have this other side of our equation as a business. So we talked about user growth and monetization that's the same for every app. But Duolingo has one more unique element, which is learning and learning efficacy. So we run a B test to try to improve our learning efficacy. And the problem with that is it doesn't have a single metric you can put on a graph. Even if you could, I mean, we use proxy metrics. There's no perfect metric for it. But even if you have proxy metrics that are going up, let's say your learning goes up, but your revenue comes down. Should you exchange that? Should you not exchange that? There's these trade offs you have to make. And I think that's where you move away from mostly purely science to, like you said, the art of product management. How do these decisions get made if they are really consequential? It's like, all right, the revenue loss might be large or there's a DAU hit, whatever it is that the clear business hit. We have to bring in most senior people to decide if As a company, we should take this trade off. Like that means our CEO would be certainly involved in saying, okay, committed, let's do this. If it's at smaller scale, I think most of our teams understand these trade offs fairly well. Meaning if it's an exchange of. For example, clearly the thing we're doing helps you improve your conversation skills, but in turn maybe it makes our content a little harder. So maybe it hurts some of our business metrics. But if the amount is small, teams can independently make those calls. If the magnitude is large, but it's a strategic decision we should make, then I think we bring in senior people to sign off is kind of how we operate.

A (42:19)

And this is why Duolingo is such a well loved app. I mean, hearing you explain that and understanding how many product decisions have been made over the years to provide that better experience at the expense of DAU or monetization or other things, it's why people love it. And that's like you said, it's hard to put a metric on that love and the social sharing and the feeling a user gets when they open the app. But it sounds like that's something you protect very carefully as a company, which then pays dividends over the much longer term, even if you can't always a b test your way to that.

B (42:54)

That's very well said. We have this operating principle that I love and we repeat a lot, is take the long view. Meaning you can make business decisions very differently if your goal is to be a very profitable or a very fast growing company for two years, versus if you want to build a company that will be around in 100 years. I think those are like, like very different decisions. And to be specific, you could monetize the Duolingo feature set very heavily if you wanted to. And for two years you'll make a lot of money. But one, that's not our goal and two, that would be terrible for free users. But three, we will burn ourselves to the ground if we do that. Like, we cannot build a hundred year business by taking more and more from. That's what a lot of freemium apps fall into this premium trap where like the easiest win you can always do is like, oh yeah, let me take some free features, make it paid and immediately all your metrics are green. And that does kind of work for six months or maybe a year, but eventually it stops working. The other thing it does is it creates opportunity for someone else to do it for free. The Duolingo story, the very early one that we didn't really talk about is why Duolingo grew so much early on. At the time there were tools teaching languages, there were the Rosetta stones of the world, but they were paid and expensive. So being free and fun may created a massive differentiator. If we were to tomorrow wake up and say, all right, we're going to charge for all of our great features, very likely in two years another great app will do it for free. So I think preserving the free value is just a moat that we have to keep and we want to keep for missionary reasons but for business reasons it also makes a lot of sense.

A (44:27)

Yeah, that's such a great answer. Well, I did want to move on to the new products that Duolingo has been introducing and kind of becoming a multi product company. For a lot of apps it's a big leap. The most famous example is Calm finding sleep stories and that secondary product market fit being this new lever of growth that really moved the needle for them. So tell me about why you decided to start adding new courses and we can talk through how that went and what you added and everything.

B (45:00)

Definitely. Honestly, it's been a really fun journey and it's been now. It's been four years since we introduced our new subject, maybe three to four, but now we have math, music and chess on top of languages. The reason we chose those specific subjects, they're all varied and they have their own thesis, but at its core they have a few properties that carry across. One is they could be taught on mobile in a fun way. Not every topic qualifies. For example, coding we considered but it's really hard to teach on mobile so we decided not to invest there. That doesn't mean we won't ever do it, but today it felt too hard to do. Another one is they all have large markets of people that are already learning these subjects. So we're not trying to create the market necessarily, but we are. There's already existing demand, that's 2 and 3 and I think this is connected to mobile. Our expertise in language, which is at its core, if you think about it, a bite sized fun experience can carry over fairly well to these subjects. The anti example of this would be stuff that is done offline, like running could have been something we decided to do, but that is an offline experience that you tie with an online experience. That is not exactly how we're used to doing bite size exercises. So we wanted to make sure our competencies played well in these subjects so these topics fit really well. We actually looked around quite a bit before we invested in each of them. Maybe the fourth internal reason these topics is each of them had one or more people inside the company that were quite tenured, like tenured builders that really wanted to build these. So they became kind of the seed for these projects.

A (46:38)

So I did want to get into the staging of this. Because you first launched math as a separate app. Right. So what was the thesis behind doing that? And then what didn't work out that you now brought it back into the app.

B (46:51)

Yes, our thesis changed actually over time. So when we launched math, we assumed our future would be separate apps, like our strategy, because at the time we also actually had. We still do, but we had another app called Duolingo ABC that is meant to teach literacy to younger children, so teaching their first language rather than second language that Duolingo teaches. But our strategy was headed in a direction where all these new topics would be separate apps. We thought at the time. This is partly true still, but our thinking certainly evolved that different audiences would seek different apps rather than be in one mega app for multiple topic learning. So if you want to learn math, we thought the overlap between you and the language learner would be slim and that's why you would probably want a separate app for doing one job. And I don't know if this was common wisdom at the time, but it felt really good as a strategy. We did realize one thing as time went on, which is actually two realizations happened. One, turns out a lot of people want to learn multiple things at once. At least Duolingo learners want to learn multiple things at once. So it became an inconvenience actually to switch between apps. And for two apps, maybe you could survive, but if it's three, four apps, you're just not going to do it. And there's just this inherent human demand that we realize exists within Duolingo, which isn't just like, like, I'm here to learn Spanish. There's certainly a lot of users that are there for one thing, but also a lot of users are there for self improvement in general. So it was a natural behavior that we just saw happening, which is, I'll learn a little bit of math, I'll learn a little bit of chess. I am there to spend my good screen time on Duolingo to improve myself. It is less relevant exactly what that topic is that might change over time. As long as I'm improving myself in something that kind of led us towards, well, let's make it easy for these folks to do that. So that was one. The other one is just the business realities of Growing a separate app is extremely expensive. While we were like, why are we playing hard mode when we can play easy mode? Because in the same app you have a user base. It's way easier to get a user that is in the main app to try something new than get them to download an entirely new app. App.

A (49:03)

It's so crazy to hear that from you. I mean, of all the companies with tens of millions of users and beloved and everything else, that the separate app was hard mode.

B (49:12)

It's hard mode for sure. And I think a lot of companies have come to this conclusion. I think there is successful and unsuccessful examples of both of these, actually. But the famous examples are I think Uber and Facebook have done both separate and combined apps in multiple forms. And that's why I don't think there's a right answer for every company. But. But for us, if you think about user behavior, which is generally there's a lot of users that are around self improvement and let me learn something and that will change over time. It just made sense for them to be in one app.

A (49:43)

Yeah. How did you solve the product problem? I think one of the challenges of mobile is that you have such tiny real estate to communicate everything, to shoehorn a feature into this tiny space that's already crowded by other things. How did you solve the product problem of it?

B (50:04)

A lot of. With a lot of work is the short answer. And before we launched anything, these projects all took probably from six, like somewhere from six to 12 months to go from the first idea to full launch. So that means in the meantime, we probably designed, tested, prototyped, redesigned, prototyped, tested again on ourselves and some beta users maybe many, many times, until we got to the state where it's like, okay, within the confines of this mobile experience, this feels fun enough. It's actually teaching something useful and it's good enough to go in the app and be on par with the quality of where the language courses are. Because language courses have been improving for 13 years, whereas this new thing is starting from scratch. So anyway, a lot of dogfooding and prototyping I think is answer number one. But the other one is I think you can eventually teach a lot of things on mobile. There are some courses that are easy, there are some courses that are harder. Music is really hard. I would say chess is fairly easier because you have one visual element that is the chessboard. As long as you fit that into the mobile experience, you can build a lot of stuff on it, but really it's now contained. Music is way harder because we're for example, teaching you piano skills on a screen. And we have to think a lot more creatively on how to do that. And you can imagine the amount of stuff we've tried and still are trying to improve, which is you can't fit an entire keyboard on a piano, for example, and you also can't fit it when the phone is vertical. You have to turn it horizontal to fit more key space. We actually tilted the keyboard a little, a few degrees, so it gives you a bit more angle for touch space. So all that stuff is like innovation that you end up having to figure out. And I would say music is the hardest. Chess and math are easier because math follows a very similar visual style to in exercises language.

A (51:49)

How do you surface it to users in the app in a way that doesn't get in the way of the people who just want to learn a language or are just there for music or, you know, the crowding the screen, real estate with these multiple things. How do you solve that problem?

B (52:04)

We've made it fairly integrated, so it doesn't feel like you are crossing into a different app, but it feels like you are just switching what you're learning. So we had already built a system basically to switch courses. So if you were learning Spanish, you could switch to learn French or Italian. Any other language in that system. We now also offer with, with some different hierarchy. Of course, unlike, you could switch to French, which is the second language you learn the most. Or there's your other topics now which are math, music and chess. We integrated there and honestly it's worked fairly well because we do already have a lot of users that learn multiple languages per day and they started discovering it. We also did some light promos just to raise awareness on the topics. That said, this remains a bit of an unsolved problem, if I'm being honest, which is a lot of our users have no idea that we also teach math, music and chess. So next year we're still going to try to figure out how to make sure that at least our user base that's coming to the app, every bay is aware that these topics exist.

A (53:03)

Yeah, it's a tough problem. How do you think about competition with Duolingo? Do you think of YouTube and Netflix and other screen time as competition or who competes with Duolingo?

B (53:17)

I think any screen time is probably our real estate competition. If you think about the natural idea of competition, you would probably think, well, education, any education app. But realistically one, we are already way bigger than any education app, so at least user base wise, so we don't particularly see behavior of, oh, I stopped using Duolingo and now I'm using an education app. Most of any churn or any user that we lose goes to spend time on Instagram or YouTube or anything that is fun in, in their phone. So for us, I think real competition is good screen time or I guess good or bad. I think that's subjective. But anything that is spent in screen time is Duolingo's competition. And that's inherently why we spend so much time on making Duolingo fun. It is also, I would say it's headwinds to try to take something that is brain intensive. Like learning a language is not an easy activity. It's not as passive as scrolling on your feed. It's actually like burning your brain a little bit if you do it for a long time. And we try to make it as easy as fun because otherwise, well, you'll go to Instagram, you'll go to TikTok. There's many engaging things that are just pulling you. So that's how, if I'm being honest, we don't really think about competition because it's so weird to think about screen time. But if you were to like define it, it's really screen time.

A (54:35)

And I think that's honestly the case for most apps these days. Unless it's a specific utility where you need to like get in, get out, solve a very specific problem. If it's any kind of even habit forming app, I mean almost any app, when you launch your home screen, are you pulled to Instagram or are you pulled to Duolingo to do your lessons? And it's tough. How do you think of AI as competition? And will AI just subsume everything? Nobody learns a language because They've got their AirPods automatically translating. I imagine that's a hot topic right now.

B (55:08)

Well, I get this question a lot and I actually really love it because I think we have a pretty clear thesis with two parts of this question. So one part is, will AI remove the need for learning languages? And in that case, will Duolingo. Well, what will Duolingo do if that happens? The other one is, I think will AI disrupt Duolingo because AI has language teaching capabilities? I'll start with the first one. I think the reality is being able to teach a language is not particularly a new part of a new ability for a tool to have. What I mean by that is textbooks also can teach you a language. Now AI can, which is, for a technology perspective, it's incredibly impressive. It can do that, but text. It is not enough, unfortunately, to remove the need to learn languages. People we've learned, and I'm only talking about languages, we can talk about other subjects that the reason people learn languages is incredibly varied. So you might be learning to connect with your heritage. AI doesn't really solve that at the end, you still will want to learn languages. This is similar to when calculators came out. We didn't stop teaching math. It's quite similar. And the need to learn languages does not disappear even if you have massively good simultaneous translation. And AI can do that really well. So that's. I think we don't see language learning demand going anywhere. And we certainly don't see it post LLMs either. The other question is there are now other tools that are non duolingo, like the chat GPTs of the world that can actually teach you language. If you ask it to teach me Spanish, it will do something. And I think that's. That's very fair. But I think that my textbook analogy still applies there where it can teach you a language. But is it exciting? Is it fun enough? And until that is the case, a thing being able to teach you a language is not a threat to our business. I think we don't see that much. And we continue investing in making things fun for that reason. And maybe the third thing I will add is we see AI only as an accelerant to duolingo because we use AI to do mainly two things that make duolingo better. One, produce content at scale, at higher quality and higher volume. The other one is introduce features that we couldn't do before that solve some of the teaching problems that we really care about solving, like conversation, practice. So in a way, AI has helped us a lot and I think it will continue to. And we will continue building better products with AI. So we're happy in everything that's happening.

A (57:39)

Well, let's move on to a few of your contrarian takes. So don't build for a Persona, build for the general population. Why? I mean, it's the classic PM thing. Build a Persona, figure out who you're talking to, all that kind of stuff.

B (57:54)

Yeah, this is. Any PM framework would say don't build for a general audience. Always pick who you're building for. I think when building consumer products, at least this is certainly duolingo. I don't want to generalize too much, but we have, we are like very anti Persona. So if someone comes in from the outside world, joins our team, most of the time their instinct is like, oh yeah, Like I'm going to build some Persona maps. I'm going to call this person the Wanderer, the other one the Explorer and which one am I going to build for? And we're immediately, please don't do that. Like stop. Because I think if you want, it really depends on what kind of product you want to build. If you're trying to get to a billion users, which we, we have kind of the, the, the ambition to, you have to eventually build for a general population. That's, that's, that's like. I think my theory number one, theory number two is with language learning, if you pick a Persona, you're actually really cutting off the rest of your audience. And language learning is such a long tail of reasons for learning. Someone could be learning to get a job, someone to be learning to move countries, someone could be learning to connect with their heritage. These are extremely varied. So if you pick one and say, oh, this is really important, yeah, like the common PM framework would say, that's how you succeed. I think for us, that's how you fail. Because then you've ignored basically the entire long tail of why people are learning languages and you've built for a very specific one. So if the use case of a product is varied, which a lot of consumer products have that property actually you have to figure out how to build for the general population.

A (59:21)

That's great. Next one. People don't read.

B (59:25)

Okay, this is another favorite one. This kind of connects to something I hinted at most early career or honestly even people who join Duolingo with experience, product designers or PMs fall into this trap of what they write on the screen as copy and text think is read by the user. The reality is on mobile apps, people's attention span is already shrunk. It's like you might be like cooking with one hand and doing your app with the other hand. You might be on the toilet. There's all kinds of situations where now your attention span is like shrunk. And the other reason is you are kind of going through an experience by not kind of paying attention to every detail, but the core maybe like the 30% of a screen. So we always repeat this. Of course it's an exaggeration. Of course people read some things on mobile, but really in a graphical interface they read very little. But to simplify, my like extreme hot take is people don't read. So don't rely on text, rely on great design to guide the user. And maybe a better said version of this is if you need to write a tutorial on what to do, you've already lost it. So it needs to be design that leads the user, not the text and the labels that you wrote.

A (60:35)

Yeah. All right, well, let's close out with the three questions I ask every guest. What was the most impactful experiment, change or something you implemented this year? Your biggest win?

B (60:45)

It's not exactly this year, it's been two years. But this is worth calling is the introduction of Video Call on Duolingo. I've been using Duolingo to improve my Spanish for nine years now. And this is the one feature I can confidently say it's helped me speak Spanish way more than before. And I mean, we're very excited for what it can do. It's. It still feels very early, but already if you use it, you can just speak Spanish better and I feel great about that.

A (61:12)

That's such a great example of building a truly great product experience with AI. A lot of people are slapping it on and trying to put a chatbot in their app where it doesn't make sense. And the Video Call with Lily is such a great example of taking this new technology and actually creating a great customer experience. That's cool that you brought that up as such a win to. What about the biggest fail?

B (61:36)

Biggest fail is this is not. I don't think this went very public, but actually what a lot of our community doesn't remember or don't know is before Video Call, our first attempt at like a big speaking feature was a human based tutoring product. So it was a higher tier subscription because it was quite expensive for us to logistically provide. But you would press a button, I think we called it, did we call it Tutors? But it was our higher tier subscription. But you press a button, you would connect with a Spanish tutor on demand, no booking required. Honestly, it was pretty awesome. Like for speaking practice. It was really awesome. It had this one massive problem, which is no one wanted to do was really scary. Duolingo. Imagine this super fun experience and we would tell you, hey, you are going to connect with a human. You're going to practice Spanish. People to press it, they would dial in, we would see their faces come online and they'd be like, oh, shit, get me out of here. Because one, it's really scary to speak in Spanish like on the language that you're learning. And two, there's another real human on the other side that was extremely jarring. We iterated a lot to ease you into it, but eventually we gave up because humans just did not want to do that and actually this brings me to another hot take that I do want to mention. Do not take user feedback at face value. For years, this is what people asked us to build. Like we're like, please build me a system where I can talk to a human tutor. We built it, no one used it.

A (63:05)

It's such a great example of a great feature, but just it doesn't work. And like as product people, we can still get into our heads, take feedback and build something that's truly great. But it, it doesn't work. So that's such a great one. All right, last question. Growth would be easier if I think.

B (63:25)

Growth would be easier if two things were true. One is pixel perfect design didn't matter in consumer because I think consumer app design bar has become so high that if you want to make progress and lead product LED growth, you have to get pixel perfect design that will go out to users. It can't be just, just a wireframe looking button that you put in your app or your illustration is like not really cared for. This increases the time to ship products. So that's one. Pixel perfect design does matter and growth would be way easier. Maybe this was the case when apps first came out, but it matters and it makes a difference. And I think too if AI really contributed to productivity in both prototyping and coding and what I mean by that is I think we are seeing today in the ranges of 10 to 20% productivity improvements, that's really impressive. But like when LLMs first came out, my hope was that they would be kind of 10x improvers. And I think if you look at the Silicon Valley hype today, that's what the messaging is. But really I'm looking at what we're doing and what every company is doing. Really the speed up of the product pipeline is in the 10 to 20% range. Growth would be easier if we got what was promised and it was 200% improvement, but I don't think that's the case.

A (64:46)

All right, well, as we wrap up, anything else you wanted to share? Any. Any jobs you wanted to shout out or anything?

B (64:52)

I will definitely do that. We are currently hiring for, honestly we're always hiring, but currently we're hiring for quite a bit of product manager positions. I think the audience of this show probably is interested in monetization since we also talked about it and we're also hiring for monetization product managers. They're all on our creators webpage.

A (65:10)

Nice. Yeah, I think that would be a great job for this audience. So if you're really into sub club you're probably really into monetization, so apply. All right, Jim, it was so much fun talking to you today. Thank you so much for joining me.

B (65:23)

Thank you so much for having me. This is awesome.

A (65:27)

Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community. Hello, I'm your host, David Barnard. My guest today is Jem Cantu, Chief Product Officer at Duolingo. On the podcast, I talk with Jim about the premium trap many apps fall into, why free trials work, even for freemium products, and how try for $0 actually outperforms try for free.