B (2:27)

I'm like, oh, my God, I did not want to learn German. He did not want to learn Spanish. We were very worried because we're like, okay, well, if we ourselves can't get it, you know, we can't get ourselves to do it, then we can't expect anybody else to do it. The solution was to turn it into a game as much as possible. And so by the time we launched, it was pretty fun. But it's mainly because we were trying to get ourselves to do it. And part of the reason that that was the case is neither of us likes learning languages. We actually are not language lovers. We don't like learning languages. And I think because of that, we made a product that works for the average person as opposed to for people who are obsessed with learning languages so much there.

A (3:09)

I would like to unpack, but I want to go back to just initial confusion. What makes this, like, a good PhD topic? Like, what was the computer science problem you were interested in?

B (3:19)

The computer science problem was basically trying to teach things to people.

A (3:25)

Is that a computer science problem?

B (3:27)

Well, you know how to get computers to do it. There's a lot there. You know how to do adaptive. You know, how you use the data to adapt to the student. There's also motivation problems like human computer interaction. So there was a lot there that we could have used. And by the way, also at the time, this was, you know, AI at the time was just starting to get to the point where we could have thought, you know, thought about training models to. And this is not large language models. This is just training classifiers to kind of teach better or something like that. So there's a lot there that could have been a PhD thesis topic.

A (4:01)

You've decided that you need to make it easier for normal humans who do not have infinite motivation to learn languages from computers. What was the first thing that you did that worked that, like, helped people get over the motivation hurdle of it being so boring?

B (4:17)

The first thing that we did that worked was making lessons not 30 minutes long, but 2 minutes long. It makes a big difference. And it's not because people were spending less time. It's that two minutes is amazing because at any point in time, you just think to yourself, oh, it's just two minutes. But the time investment for 30 minutes, you know, if you're right now and you're like, if I ask you, do you want to do something that takes 30 minutes you're like, oh, man, I don't know about that. Even though when you think the time investment is only going to be two minutes, you may spend the 30 minutes, but it's just like, yeah, I can start right now because I can end anytime. So that's the first thing that worked. We did many other things that worked big, but that was the first thing that really was a big game changer.

A (4:58)

Is there anything that really surprised you from a behavior perspective? That worked.

B (5:02)

A few things. I did not expect streaks to be this powerful. I mean, a streak is extremely powerful.

A (5:07)

Can you explain what that is?

B (5:09)

It's just a counter that measures the number of days you've done something in a row for duolingo. The duolingo streak is the number of days that you've used duolingo in a row to learn a language. It's incredibly powerful. People really talk about their streaks. We have 10 million active users that have a streak longer than 365. So that means they haven't missed a day in the last year or longer. So I did not expect streaks to be this powerful. The other one that I did not expect this. We have these notifications to get you to come back to the product. I did not expect the passive aggressive ones to work so well. This is how it went. We were sending you notifications that just said something like, hey, come back to learn Spanish. If you did not come back the first day, we would send you a notification. If you did not come back the second day, we would send you a notification. But after five days of inactivity, we stopped. At some point, it occurred to me that if we're stopping to send you a notification, we should probably let you know. So we started sending this notification on the fifth date that said, these reminders don't seem to be working. We're gonna stop sending them for now. It turns out this is very powerful at getting people to come back because they feel like we've given up on them. I did not expect that. I mean, we didn't even do that with the goal of getting people to come back. But it turns out that's actually very powerful.

A (6:25)

I have not studied education, and so this doesn't come from a place of knowledge, but this idea of, like, making it easy and, like, making the, you know, barrier to entry lower. I feel like there's some controversy around it. It raises questions like, okay, learning is hard, and, like, there is no way around it. Like, a smart friend of mine compared learning to, like, going to the gym. But for your mind like, it's very simple. You just have to do it. And there's no other way besides, like, being willing to put the effort and think really hard about it. How do you react to that point of view?

B (7:02)

I would be very, very happy if everybody who is working on an education app thought that way, because then we will have no competitors. It just turns out that the hardest thing about learning is motivation. By the way, I believe that to be true of the gym, too. You know, like, a lot of people talk about, like, you know, you can spend all your time debating whether an elliptical machine is better than a treadmill, but in the end, what matters is that you do it. That's like 90% of what matters. It's the same with learning something. You can spend all your time debating whether you should read that book or this book or whether you should do it through an app or you should do it through a tutor. There's, of course, differences in how effective each of these are. But what matters the most is that you actually do it. And if you want to get people to actually do it, you have to make it as easy as possible to get started, to get in there, and to be motivated. I believe that is the reason why we have grown so much. I mean, and in fact, it's funny, I mean, we see a lot of language learning apps that pop up that say, oh, we're like Duolingo, but without the gamification. And every time I see that, I'm like, great, you do that. Carry on. 99% of the world's population just is not that motivated for any activity. And, you know, there's the few people that are extremely motivated. Good for them. The vast majority of people are just not.

A (8:26)

If you are an engineer or, you know, in many other types of knowledge work, the idea of flow is almost holy, right? Like, you need to be, like, in the context and understand the code base and, like, be really thinking about it for a while, or you won't make any progress, much less, like, learn something new. Do you think about that at all when. Because if people are like, I'm on the subway, I'm going to Duolingo for two minutes, like, how does that factor into the experience?

B (8:49)

We don't really think about that too much. I mean, I understand the concept and it makes sense, but I mean, for us, the single most important thing is to get people time on the app. You know, we know, Turns out for an English speaker, getting to a pretty good spot in Spanish takes about 500 hours.

A (9:03)

Oh, wow. Okay.

B (9:04)

We just got to clock those 500 hours, we just got to clock those like, that's it. And you know, is it 600 hours because you're not, as, you know, in flow, or is it 400 hours because you're like, really in flow? Sure, there's probably a variance there, but in the end, we have to clock something like 500 hours for you to get to a good spot in Spanish. And by the way, that same number for Chinese is 2000 hours.

A (9:29)

Yes. As somebody with a master's degree in Chinese lit, it's not a language meant to be learned.

B (9:37)

It really. It doesn't seem to be.

A (9:38)

I find your perspective on the human behavior around education both, like, a little bit dark, like, not particularly idealistic, and yet incredibly inspiring to me. Me as a sort of end user, because I am working, I'm engaged in the work. I have three kids. I theoretically do some other things. And yet the thing I would like most be interested in doing with my free time is skill acquisition. Right? Like be a better chess player, improve my Chinese as like, you know, fruitless as that seems to be, or learn more math. Many different things. And yet, if you ask me, are you going to carve out, you know, an hour and a half at the end of a day where you've been attempting to be productive for 16 hours and do more of that in a really painful way? The answer is clearly no. Right? And yet, if you ask me, could you imagine putting 600 hours into it two minutes at a time over the next few years? Like, actually that seems doable.

B (10:36)

After working enough in consumer products, you realize a number of things. I mean, people don't read, they just don't. People are lazy. And if given the choice, they may tell you all kinds of things. But the reality is that if given the choice, they'd rather scroll on Instagram or TikTok. That's just reality.

A (10:58)

The thing that is positive in all this is you can take some of that behavior and do something useful with it anyway.

B (11:04)

Perhaps that's what we're trying to do with Duolingo. I mean, what we're trying to do is to use a lot of the. The same tricks that are used to keep people engaged with, with mobile games or whatever it is, but to get them to learn something. I mean, that's. That's really at the crux of what we do at Duolingo.

A (11:22)

Duolingo has a pretty unique position in that you were a company with some scale when the capabilities in machine learning started sort of accelerating in their progress and expanding. How did you handle that and sort of, how did you react internally as a product organization?

B (11:43)

I mean, for us, it's been really positive. The fact that, you know, they are language models, you know, and we teach languages, it's a really perfect application. They've been really positive in two ways. To teach something, one of the things you have to do is create a lot of content. This is the content that you're trying to teach. We used to make that content kind of half by hand, half automatically, like it was kind of by hand. But there was all kinds of automation around it to make people, to make the people that were creating the content go faster. But still, this was going relatively slow. Over the last couple of years, what we did is we've retooled this whole content creation pipeline to be entirely based on large language models. I mean, it's all based on AI. So humans are not involved any longer or very little, if at all. And what that has allowed us to do is just create massive amounts of content that were just not possible before. So we're just, we're creating many more courses. So, for example, that's one of the things that we're. That we're doing right now, and we're just launching it right now. We used to have, we teach 40 languages, but the way we teach languages is we teach, for example, we teach Spanish for English speakers, which is a different course than Spanish for German speakers, which is a different course than Spanish for Chinese speakers, et cetera. So we used to teach 40 languages, but for English speakers. So if you were a native German speaker, you could only learn, like four of them. What we're able to do with AI now is because we can go so much faster. At this point, we basically teach the 40 languages to every base language, and that's a major increase in the number of courses. So that's an example of something that we can do. The other example of something that we can do now that we couldn't do before is practicing conversation. Historically, we could teach you vocabulary, how to read, but this actual conversation, the only way we knew how to practice it was with another human. And we experimented with doing that, but it turns out most people don't want to talk to another human in a language that they are not very comfortable.

A (13:46)

With because it feels bad.

B (13:48)

It feels bad, it gives you shame, it gives you, you know, it's not good. But now with large language models, you can actually practice conversation with an AI, and then you do so without feeling judged. And so we're seeing a lot of uptake on that. So it's been really good for us.

A (14:06)

What else are you excited about in terms of AI changing the product itself, like role play and explain my answer and these things. What do you think matters or work so far?

B (14:17)

You know, I'm excited about generally practicing real world language, like conversation. I'm excited about that. Outside of language, I'm really excited about teaching math. We can do a much better job at teaching math with large language models and we have a math course, but it is, you know, we're completely retooling it to be a lot more like a tutor now. You know, again, I'll go back to tutors. Tutors can be really good at kind of learning outcomes. They have this one big problem. They're really boring. So we're trying to find ways to make it a quote unquote tutor, but that is also gamified. Just turns out most people would rather play Candy Crush than sit there in front of a tutor. So we're trying to come up with something that is as effective as a tutor but as fun as Candy Crush. The reality is we'll probably come up with something that is 90% as effective as a tutor and 90% as fun as Candy Crush. But at least the combo of this will be a lot more effective.

A (15:10)

So math, music, chess.

B (15:13)

Chess now?

A (15:14)

Chess now? Yes. How do you decide what duolingo can teach beyond languages?

B (15:20)

What we're looking at for subjects to teach are one, we're looking at very large audience. So we need something that hundreds of millions of people want to learn. There's a reason for that, and it's because we're an app at the end of the day and we cannot charge $30,000 to each user. We charge you 10 bucks. In order to make a significant amount of money from a new subject, it has to be learned by a lot of people. Otherwise, these very niche subjects just, you know, if we charge you 10 bucks and there's only 100 people learning it, this is just not worth our time. So they have to be a large potential audience. The other thing is we look for things that take a long time to learn. You know, if you can learn something in two hours, you probably should just go watch a YouTube video. And that does a perfectly good job. If it's just like two hours. We look for things that really take hundreds of hours to learn and then things that we think are good for the world and we can do a good job with a mobile app. Oh, there's one extra thing which is internally we need to have Somebody that's really excited about it. So this has been true for all of these other subjects. There's somebody that's very excited about music, somebody is very excited about chess, and that's what has done it.

A (16:37)

Okay. Even beyond what makes sense for duolingo as a business from a technology perspective, do you think there are things that are, like, hard to teach with computers for now?

B (16:49)

I don't know if there are. There's things that are different that certainly different from what we do at Duolingo. So, for example, history, the way you teach history is not with these drills. You know, the duolingo is really good at drills. The way you teach history is probably with really well produced videos. It's probably the best way to do it, and maybe I can get better at that. But so I think there are just things that are different than Duolingo. But ultimately I'm not sure that there's anything that computers can't really teach you. I think overall, we'll be able to teach everything really well with computers.

A (17:20)

Do you think there's any version of the world where AI is a threat, like large language models are a threat to Duolingo?

B (17:27)

I mean, sure, they're a threat. I mean, you know, that's one of the things that is scary about the world that we live in. With AI and large language models. We're undergoing a platform shift, I mean, of some sort of, um. I don't know what's gonna happen on the other side. I'm not super worried, but you just never know. And it's not just for Duolingo. It's gonna be all kinds of things, right? I mean, could be a threat for Netflix. Like it could be that just a large language model just press a button and it makes you the perfect movie. And then so, you know, I don't know if that will happen or not. Right. I just don't know. So it's kind of a similar thing. Like, who knows what will happen. At the moment, it doesn't look like it's a major threat, but your guess is as good as mine, I think.

A (18:08)

I mean, you guys have a huge amount of distribution and a huge amount of data and the interest in engineering culture to go after it.

B (18:16)

So the way we see it is we have large distribution, we have data on how people are learning languages. That is unique. I mean, we have a lot of people. It's also the case that brand ends up mattering quite a bit. And we do have a good brand for language learning. So the combination of all of this we're hoping will be good for us.

A (18:34)

Can we talk about brand for a minute? Because it is. Duolingo has a very unique one. There is perhaps more risk and more distinctiveness in brand voice than the vast majority of public companies might take. You are, of course, a consumer company, but there are plenty of companies who don't. Don't take those risks. Like, where does that come from for you guys? Why did the owl die?

B (18:59)

He didn't die. He just faked his death.

A (19:01)

Okay, that's even weirder.

B (19:04)

It's not like we on day one decided, you know, we're gonna have a brand that is unhinged, that where the owl does weird stuff. It just kind of evolved over time. Our brand voice evolved over time. What happened was in the product, the owl was a little pushy to get you to do your lessons. And then the Internet just started coming up with memes about the owl doing crazy stuff. Like, you know, they started coming up with the owls willing to kidnap your family for you to do your lesson. That all was invented by the Internet, not by us. But as we saw that that was happening, we started leaning into it. We thought, why not? And the more we leaned into it, the better it worked. And so we just found something that resonated, you know, that got us going. Then a few years ago, we had a very junior marketing employee that we had just hired who said, hey, we have this old owl suit here that we had used for, like, events inside the company and just said, hey, can I make some TikToks out of it? And I actually was against it. I'm like, I don't. I don't know if anybody's going to be interested in this. And it turned out that when we put it on TikTok, and they all was. It really was just the owl doing really dumb stuff, like, you know, twerking and falling down or whatever. A lot of these videos started going viral, and it was interesting because none of the videos said, you know, learn a language on duolingo or subscribe to duolingo or anything. It really was just the owl doing weird stuff. It just, you know, it took a life of its own. We formalized a lot of it, and we have a pretty good idea of the types of risks we're willing to take. But we do take more risks than most companies, certainly than most public companies. One of the things that I think helps us so it helps us that we have a mascot, but I think it also helps us that we're an education company, because, you know, ultimately Education, you can't really say that education's bad. It's hard to argue against education. So that allows us to. That allows our marketing to do things where we can't be criticized that much. If we were a company that was trying to, like, just really extract your money or like, you know, we were like a vaping company or something, probably people would push back more on our marketing. But given that we're an education company, I think it gives us a little more leeway.

A (21:21)

Given this unique scale of duolingo, I think it's more than 100 million people monthly. Is that right?

B (21:27)

Yeah. Yeah.

A (21:28)

What do you know about how people learn that other humans don't that you think is interesting?

B (21:33)

A lot of it is just codified on what the algorithms have adapted to. I mean, we have adaptive algorithms that try to figure out how to teach. And I think it's hard to verbalize a lot of it. I mean, we know a few things that you can verbalize that may not be that surprising. But the farther your language is from your native language, the harder it is to learn. We have a model that can predict whether you're gonna get an exercise right or wrong. And we're actually extremely accurate. So when we give you an exercise, we know if you're gonna get it right or wrong. We're very accurate. And the way we do that is we just watch everything you're doing and we see what you're good at and what you're bad at. And for example, we know for each user, this particular user is bad at the past tense. One of the things you may think at first that the right thing to do is because this user is bad at the past tense, we should give them more past tense. And that's roughly true, but it's not exactly true, because if all we did was give you lessons of things that you're bad at, these would be very, very, very horrible lessons for you. So there's a whole, you know, I was going to say arts, but it's actually more of a science. There's a more of a science about when to give you the things that you're bad at. And for example, whenever we give you an exercise, the right thing to do is to give you an exercise that you're about 83% chance of getting it correct. Turns out that maximizes enjoyment and maximum.

A (22:54)

Enjoyment means I'm going to get to my five or 600 hours, and that's really the outcome.

B (22:58)

Exactly. It just keeps you. It keeps you motivated and. And it's funny, it's not 100%, because 100% is too easy, but it's also not 50%. It really is closer to 100%. You gotta mostly win, and that seems to be what works.

A (23:15)

What do you think are some of the implications of what you guys are doing or learning for traditional education and schooling?

B (23:24)

First of all, I do think that education is going to change over the next some number of years. I can't say one year, but it's probably less than 20 years. Something's gonna change. And the reason for that is it's just a lot more scalable to teach with AI than with teachers. And by the way, that doesn't mean the teachers are gonna go away. You still need people to take care of the students and you still need. I also don't think schools are gonna go away because you still need childcare.

A (23:48)

In your view, like schools could be childcare, but everybody's duolingoing.

B (23:52)

I think it's gonna be something like that. It may not be duolingo, but I think it's gonna be something where there's one teacher and like 30 students. Each teacher cannot give individualized attention to each student, but the computer can. And really the computer can actually know with very precise knowledge. It can have very precise knowledge about what you. This one student is good at and bad at. That the teacher just has no chance of having because there's 30 students and they just kind of give you that. So I do think that it would be more effective if some of that time is being spent with essentially an AI teaching you. I do think it's going to get to that. It's also the case that there are extremely good teachers for sure, but there's not very many of them and certainly most everybody in the world doesn't have access to a good one. So I think that there's going to be some change like that. I do think it's going to take a while because changes in education are very slow. But I do think it's going to be like that. I think a lot of what we're doing will apply in terms of keeping people motivated, et cetera. I do think that in a formal education setting, some things do need to be different. You have a little more control over the students actually doing this stuff when they're in school. So you can probably make them kind of. Instead of expecting that they're only going to do two minutes, you can probably expect that they're going to do 20. So there's some differences in the school setting, but I think a Lot of what we're learning here will apply.

A (25:17)

Is there anything you feel like people misunderstand about Duolingo?

B (25:21)

When we became a publicly traded company, investors thought we were a Covid phenomenon. Turns out we were not. The biggest misunderstandings are how important motivation is. I really don't think people understand the. I mean even our competitors, even. Even not just competitors for language learning like people who do education companies. It is amazing to me that really most everybody that we talk to does not seem to understand how important motivation is. I think that's one thing. I think another one is just how much sophistication there is. A consumer may not know because the app is so cutesy and like, you know, little animations and everything. There's a lot of sophistication about when to give you even the animation. So there's a lot of sophistication about that. And I don't think people understand just how many, you know, there are. Duolingo's the result. I checked the other day. We have run over the history of the company. We have run 16,000 AB tests and I don't think people understand that it has taken 16,000 A B tests to get to this point. Yeah, I think that's probably the biggest misunderstandings.

A (26:24)

Any predictions about what the large scale changes in learning mean for society or even how you might encourage people to think about learning for themselves or their children?

B (26:35)

The good news, I don't know. Good or bad news maybe, probably bad news for Duolingo. But good news is this change is going to be slow. I don't think you're going to see a change where like next year everybody's learning completely different. So I think we'll have some time to adapt.

A (26:48)

What's the drag force? I feel like that's actually a controversial point of view that it's going to be slow.

B (26:53)

Oh, go to a real school, they're doing stuff from like 30 years ago. The drag is just, you know, it's like governments, it's just slow. If you live in, you know, New York or Silicon Valley, you'll see that, you know, there are schools that are doing really progressive stuff, etc. But the vast majority of the education system, it's very slow to make changes like this. School systems are regulated. You know, in Texas they're trying to not teach evolution. Like there's just weird stuff that goes on in school systems. So I think in general it's going to take some time to do this. You probably will see some private schools move there faster. There's Also an interesting thing about private schools, they're very fancy private schools. You pay 50,000 bucks a year to go to a private school. It's hard for them to say, well, what our kids do is use duolingo. Because it's like, well, why am I paying you 50,000 bucks? So it's an interesting dynamics that's going to happen, but I do think that some of the private schools are probably going to be the first ones to start really moving towards this. And I think you'll just see a lot better learning outcomes in general. The other place where you may start seeing is there may be some countries that leapfrog some countries that are at the moment probably a little behind. For them, this is the only way in which they can scale their education. So you may see that.

A (28:19)

Yeah, I tend to think very tactically also about impact on the entrepreneurial ecosystem. And maybe it was all about finding the idea. But you studied math and then you studied CS and you got a PhD and you were a professor and had done some body of work before you became an entrepreneur, including starting another company. Actually a really interesting one. Recaptcha. But one of the things that is interesting to me, even like, let's say 15 years into the technology ecosystem in my own career is people are becoming experts at much younger ages.

B (28:53)

Oh yeah.

A (28:53)

I think obviously if you can set yourself on a learning journey, like first it was just because of, you know, the Internet. Right. You can learn from just content and forums and finding community and whatever else.

B (29:05)

Right.

A (29:05)

But when I think about some of the dropouts that we work with who are like, well, I wrote a textbook on wireless technology as a sophomore and I'm like, man, I. I waitressed at Outback Steakhouse when I was 16. Right, right. One of the things that I, I think would be really exciting as, you know, we look forward five or ten years actually, if people can get to skill acquisition and learning outcomes that are much cheaper, broadly accessible and easier to motivate, you know, you will get experts that are doing really interesting things by the time they're 15, 20.

B (29:41)

Yes. And not only will that will be true, but I think there'll be experts and they'll have the tool of AI. So I think it'll be the case at least in the foreseeable future, that'll just magnify their expertise. So they're experts and in addition to that they can go a lot faster or they are. So yeah, I mean, compared to, you know, waitressing at Outback Steakhouse or, you know, whatever dumb thing it is that I was doing when I was. I mean, I did not have access to. The first time I had access to the Internet, I must have been like 15 years old. I don't know what I was doing before that. I think I was playing with like, Legos. I mean, that's basically what I was doing. So, yeah, I think there's. We're going to see a lot of that.

A (30:27)

So it sounds like course authoring is a place that Duolingo's gotten a lot of leverage. When you think about just using AI broadly as a tool within your company, be it building product or everything else you do, where else do you think it can have the most impact?

B (30:44)

The visual style of Duolingo, we're very unlike most. Well, I mean, a lot of apps are kind of, you know, well designed, that's for sure, but we're very animated. Like, we're like a cartoon. I mean, Duolingo is like a cartoon. Making that has required a lot of human effort. It turns out we can do a lot of that with computers now. Now, that doesn't mean we're not going to employ the artists. The artists are still here, but they are going so much faster. When artists could do before in like a month. Now they can do in like a day. It's really unleashing their creativity because they're not spending their time on the mechanics of like, is this shadow just right? No, they're just unleashing their creativity. And I think that's. That's pretty awesome. So we're seeing a lot of that, you know, a lot of our animations and a lot of illustrations are now computer generated.

A (31:39)

That's not what I expect you to say, but I am really thrilled for the future of math and chess and 140some new languages and vibe cartooning for Duolingo. It'll be great.

B (31:50)

Yes.

A (31:51)

Thank you so much for doing this, Luis.

B (31:53)

Yeah, thank you. Thank you, Sarah.

A (31:55)

Find us on Twitter nopriorspod. Subscribe to our YouTube channel. If you want to see our faces, follow the show on Apple Podcasts, Spotify or wherever you listen. That way you get a new episode every week and sign up for emails or find transcripts for every episode@no-priors.com.