← ASU+GSV Summit Sessions
ASU+GSV Summit Sessions
Native Speakers: What First Mover Language Apps Can Teach Every EdTech About AI
00:39:43
Loading summary
Transcript51 lines
- [00:03]
A
This session was recorded live at the 2026 asu gsv summit in san diego.
- [00:14]
B
Hello everyone. Thank you for joining us in our discussions about language apps and language learning and language assessment. I am Alina. I am with Duolingo and I lead the RD for measurement and assessment for our high stakes assessment called the Duolingo English Test. And I have here people with experience with learning apps that help people learn and people with experience with AI. And we are also fortunate that we are going to learn from them and from their experiences. So my first question is that all of you have had a successful trajectory launching and maintaining language learning products. What are the three main lessons, lessons you have learned from this journey? And I will start with Max.
- [01:09]
C
Thank you, Alina. Max here from Novakit. And I would say there's definitely more than three lessons that we learned. I'll probably limit it to three. So number one is when we were starting there was a lot of doubters that a full immersion language learning is actually going to work for kids. So basically everybody would say like, you know, my kid doesn't speak a word of English and you're just going to immerse them straight into the like with a native speaker and they would not speak a word of their native, you know, native tongue. How would that work? Well, guess what, it actually works really well. And we see like a really good response from kids, like 96% of them actually improve their English really quickly after they start. So this was one really great encouraging learning that the full immersion does work. Then we also learned that I guess to surprisingly or not that parents actually don't really want to be all that involved in learning for the most part. I mean, unless they're Chinese of course. So it's. Yeah.
- [02:16]
B
Or Romanians. Ask my son.
- [02:20]
D
Yeah.
- [02:20]
C
So basically designing for kid independence is actually a big part of what we do nowadays that we learned this lesson. And the third one I would say is around AI. Just recently when we started experimenting, we realized, well, on one hand kids actually love talking to AI, especially when it's a cute character, obviously who would not prefer to speak to a cute pandy compared to a middle aged adult. It makes perfect sense. But there is a little caveat and we actually see that the speaking time is actually about two times higher than with the live teacher. But the caveat is that the sessions are pretty short. So basically if you look at the total time on task, humans are still far, far superior to AI. So basically that leads us to roughly about five times superior. So basically this leads to conclusion that you do need a hybrid environment. So AI and humans. This is kind of the best golden standard I think going forward. So these are my three lessons.
- [03:23]
B
Thank you. Thank you, Ken. Karine, what do you think?
- [03:25]
D
Yes, hi, Corinne from Chegg. Actually three lessons and I would say the first one almost is, is a myth, right? The first myth was oh, speaking another language will not be necessary.
- [03:37]
C
Right.
- [03:37]
D
We're going to have translation in our ears and we can almost like have telepathy and understand, you know, the different cultures from each other. The fact is English is not a soft skills anymore. It's becoming an infrastructure for an organization. We have more and more change in the way organizations are going to be set. So gartner is saying 20% of organizations are going to lose their middle management. So we're moving towards a diamond organization shape and 74% of organization are even finding it more difficult to keep the talent or find the talents that, that they want that know how to use AI and have the expertise that they need. And so cross working, cross border, cross functionally. Understanding cultural challenges is actually becoming even more important than not. The second learning that we had is, you know, AI native is very different than AI features. So I started in 2016 when I, when I was at Global English and we had about 20 years of, of English oral assessments and that were graded by teachers and so we used AI and neuro linguistic programming at that time to automatize what we know how to do. Today it's very different today it's not about having a feature of doing something we know how to do. It's actually leveraging AI for doing something we don't know how to do. So we don't know how to have super quality teacher available 247 with immediate access that know me at a level of intimacy that you know, now we know ChatGPT and Claude maybe sometime knows us more than our best friends. And so those are, you know, like I would say two of the key lessons that I would call out and features now when a new model is coming in can be obsolete when you build the intelligence within your platform and really the AI is learning from the behaviors from the users and also then defining the next steps. That's where we're seeing is AI native versus not.
- [05:50]
B
Thank you so much Hugh. Please share with us your lessons.
- [05:54]
A
Yeah, this is Hugh from Lingoace. Actually we started our language teaching business from eight years ago from a very, very small apartment in Singapore. So we teach English and also Chinese globally to the students from 180. So as Max mentioned About of course we have more than three lessons but actually I would try to really prioritize what are the top three.
- [06:16]
C
Right.
- [06:17]
A
So number one, lessons from our own personality is really about like, you know, how to provide quality and a consistent experience. So before actually and we teach, of course there are many, many different ways to learn English, many different ways to learn the Chinese. But actually over past eight years I think we can deliver the, you know, and the quality and the consistent experience. So I had a conversation with one of our parents I think still reminds me of many years ago. So she mentioned about actually and I'm really surprised by your company because I tried five teachers, all of them are very great. So that's why actually I think the consistent and quality are very, very important to parents, right? So because it's not about money they spend with us, it's about the time, the trust they give to us. So that's why I think number one is really about the consistency and also the quality. So the number two, I think is really about and it's kind of the two side of the coin and from the business side is called retention. But from the, you know, on the learner side is perseverance, right? Because actually language learning every knows that it's not about one day, it's not about actually overnight you can acquire a new language. So unlike maybe, you know, math or other subject, you can get the idea very quickly. But for language you need to practice. So that's why actually, and you know, in our platform, many students learn with three years, five years or even, even maybe 80 years and from day one with us. So that's why actually that part is called perseverance because only when they show the interest, when they have the perseverance, they can see the learning outcome, right? So from business perspective it's retention if they stay with us longer enough so it makes the business more sustainable. So that's kind of the two side of the coin. So the last but not least is really about like, you know, technology enablement. So from day one we are an online education provider. So that's why actually we, you know, leverage a lot of, you know, firstly we use online technology so we can hire 5000 Chinese teacher in China and 3000 English teacher from US and Canada. So through Internet we can deliver the seamless experience right to the doorway, right? So this is the first step, but after that we think about for the young learners, they need a different experience from adults. So that's why gamification is a very big part of our learning. Even we have the live tutor, but it's not really a boring experience. So the gamification still play a very, very essential part. So like today we think AI play even a big part, as Max mentioned about and AI and human being together, the combination which is very, very powerful. So that's why early this year we launched the AI tutor as like, you know, a component. For example, like, you know, the young kids can learn with our lab tutor twice or three times a week, but after that they couldn't access the tutor anytime, anywhere. But AI tutor can do that. Right? So that's why they can practice with our AI tutor with the same content and in different way. Actually 24 by 7. So that's why I think technology enablement is a very, very essential part of the business and also of the learning journey. Yeah, that's kind of the. I try to prioritize the top three. So they are a long list.
- [09:30]
B
Absolutely. Thank you so much. I will also share a few things from Duolingo, so I'll try to be very brief. One, of course, by now everybody knows is engagement. So philosophical concept behind it is if you make it all really engaging, then people will come back and they have a chance to actually learn something. If it's not engaging, even if it's the best product ever, they won't come back and therefore they won't learn. So engagement is truly important for us. Another lesson that we have is micro content. So the type of teaching that we do is pretty much whenever people want to do it. We do not have a formal way to teach. And that means that micro content is relevant for teaching. And we emphasize that. As many of you I know are users of Duolingo. So you've seen that the third one is using AI for good. As you again, many of you know, our learnings are all free and they are not impacted by ads. The only ads come at the end. So of course there are memberships for fancier approaches. But the basic education is free for all and that wouldn't be possible without the technology. And I think that's a good example of how AI can help all of us be more inclusive all over the world. So, yeah, these were my three lessons. Trying to be really quick here. But I will move to the second question, which is how did you scale your products? What worked well and what failed at scale? So I will start with Karim.
- [11:20]
D
Okay. I would say what worked well. You know, Buzu is the brand that you would knew for language learning. For us, what worked well at Busuu is AI not Only helped us scale content volume, but it really helped us scale high quality interactions. So you were just talking about engagement before? Right. And so by adding more gamification, we augmented by 48% the engagement from the team, from, from, sorry, from learners. But really at the end of the day it's just the entry point. If they're here, then we really drive them through structured progression because really at the, at the end of the day it's about learning how to perform a language. And so that was the first learning for us, which was, okay, let's get more engagement, but let's make those interactions really drive an outcome for our learners. If we had a lot of engagement and not outcomes, then we knew this was not getting to the mark. The other piece I want to highlight, which it was, you know, it's kind of we have adult learners, we don't have kids and they don't really need a puppet or something, but they need psychological safety. And so AI really created psychological safety for practicing rehearsing across any times when the tutor was not available. So we're going to 80, 20% roles where 80% of your rehearsal and practice is done with AI, but we have 20% where you need that specific human interaction, behavioral mirroring that you only get with a human in the loop. What didn't work is when we were just looking for more activities. More activities do not mean progression nor outcomes. And so really keeping the outcome and progression in mind in any of the, you know, the additional features, tools that you have in your, in your, you know, in your portfolio with AI is critical.
- [13:32]
B
Thank you, Hugh. Do you want to share?
- [13:34]
A
Yeah, of course. Actually. And I think for our business we have like, you know, live tutors. That's why the scaling problem for us are the two side problem. Right. So one side is really about how to acquire customers. As I mentioned, we have students from 180 countries. So the past few days when I talk about, about our business with other friends here in this conference, everybody asks, how do you acquire customer? How can you take this kind of acquisition cost? So I think from the growth perspective, I think one thing works very, very well with us is word of mouth. So because we sell our product directly to parents, to students. So that's why actually if one family is satisfied with our product, so they can recommend our product to their friends, to their classmates. So that's why actually the word of mouth is the one, the most, the most powerful tool of us. So Today more than 50% of the new customers of Lingoas are from referral so this is very, very powerful like you know, and a tool to us. And but from teacher side I would, you know, mention about because scaling is not only about students, customers, but also about your supply because we hire more than 8,000 teachers globally. So how to really provide very consistent experience. So that's why we spend a lot of the, you know, and you know, time a lot of the like, you know, talents on the teacher training and we build like, you know, our own content. So we are not a marketpl.
- [14:51]
B
Right.
- [14:51]
A
Mark Brace means that we put tutor and you can find tutor by yourselves. If the quality is good, you are lucky. Quality is not good. Hey actually and you are not lucky.
- [14:59]
B
Right?
- [14:59]
A
But in our case actually we spend a lot of effort to train the teachers. We provide a very great teaching and a framework the content. So that's why we can provide a very consistent experience. So without very and highly standardized experience, I think we are not able to scale. So this is other area. But one lesson learned I will share is really about during like, you know, and after 2021, we raised a lot of money, right? So actually we raised more than 100 million in a single year. So that's why actually we burn a lot of money on paid marketing that something didn't work, right. So actually, you know, in a very short term we see result, we see the business grows very fast. But actually and the cost is so high and also actually and it doesn't work at all. Right. So that's why actually after we slow down and for a few years now, we focus on content, we focus on teachers, we focus on quality and we see the growth come back again. So that's kind of very, very expensive lesson we learned.
- [15:53]
B
Awesome. Max, do you want to share?
- [15:56]
C
Sure. I would probably note that what didn't work is basically kind of trying to stick with the same product and kind of trying to keep selling what you have. And basically the space we're in, you constantly need to evolve and what's. What actually does work is actually meeting your potential customers, meeting parents where they are. And that is applied both financially. So we actually had multiple lessons where we discovered that there were significant growth opportunities for us by offering less expensive product, in some cases, in other cases more expensive product. And also in terms of friction. So as I mentioned, parents, a lot of parents don't want to get very involved into the learning process and basically having to kind of supervise your kid make sure they actually learn. For a lot of parents, it's a friction. I mean, they just don't want to do it. So they would rather just kind of us magically motivate their kid and kind of bring them the result at the end of it. And we really see AI as a huge enabler for us to actually deliver in this promise on both angles. So right now we're rolling out products that are significantly less expensive. With help of AI, we can actually make it happen. So we can provide both some live teacher component and AI and do it at a significantly lower price point. But it also reduces friction because a lot of experience becomes on demand rather than pre scheduled. So we see that that's basically like a double win and double kind of tailwind for the business growth. So if anybody of you is still kind of treating AI as a threat, then kind of maybe it's time to rethink and really think a bit about it as a tailwind for you.
- [17:46]
B
I absolutely agree. So thank you. On the Duolingo side for the app, I guess we scaled by also the word of mouth, so that was definitely important for us and still is. And the second element I would mention on the learning side is the work that my colleagues did on making sure that our communication is aligned with the style of communication of Gen Z. So being in touch with the cultural elements, being able to respond in real time to mems on the Internet and all of that was extremely helpful for the company. Now, I want to point out that I work on the high stakes test, so that's a very different environment, that's a different market. It's regulated high standards for accuracy, for quality, for validity. So what did it help the Duolingo English test? Well, it was quality. On one side, we had to show that our test matches the standards, and on the other side was the test taker's experience. So the test is taken from home, anytime, anywhere. It's remotely proctored, and we had to show that it works well and it's secure. And also we, as I mentioned before, we had to show that the test is valid despite the fact that it's much shorter than any other test. How did we do that? The test is adaptive. And when the test is adaptive and it adapts to the level of ability of the test taker, then you just don't need that many items. You can quickly zero in their ability. All right, so let's move on to the next question. So fortunately, I need my glasses. Engagement versus learning. We've been talking about this for a bit. Language apps are definitely masters of engagement. How do you balance the two? Where have you seen engagement tactics that conflict with Actual learning. And how did you resolve it? So, Hugh, do you want to start?
- [19:57]
A
Oh, yeah, I can start, actually. And it's a very interesting question because, you know, at Lingoaids we always focus on more for the serious learners, right? So as I describe the business to others, so it means that actually our customers, our parents, will spend thousands of dollars for one student with us in a lifetime. So that's why we always care about the learning outcome. So I can give you a few examples. We always put learning outcome as the highest standard. So for example, in certain country we partner with Cambridge, so we use their CFR standard textbook. So we provide very, very high standard for the learning. But at the same time we also personalize experience like Learning Journey. It's fun, there's engagement, gamification, but actually we never compromise on the quality. So that's why when we sell product for young kids, so it's very, very challenging because there are two buyers, right? One is parents. Parents need to see, they don't want to get involved, but they want to see that learning outcome, right? But actually for young kids, they lack gamification, they lack engagement. So that's why we always balance these two. And for Chinese subject, we start from 80 years ago. So we already publish our own textbook because. And we believe it's very important to really and provide very, very high quality content to our peers, our competitors, so we can improve the industry standards. That's why we publish our textbook, so everybody can use our textbook. And also we, through the gamification, our content can be delivered more effectively. So that's why I think we need a balanced engagement and also learning outcome. If there's one thing we need to prioritize, I would say that is the, you know, and the quality of the learning, not the engagement. But engagement is kind of the tool to deliver the high standard.
- [21:40]
B
Thank you, Max.
- [21:43]
C
Yeah, I would say for us, probably one of the challenges, again, as a company that started originally from live tutoring and now kind of extending more into AI augmented learning, the challenge is to extend engagement from the day that the tutoring happens into other days of the week. So basically the challenge for us is not so much to kind of limit the engagement, but to transfer it to another day because it's so easy. So after the tutoring session ends, it's much easier to kind of to keep them there. So we actually tested like very similar experiences to YouTube. So basically like right after your lesson ends, we show, okay, this is next. So you're going to watch this, this and this, and it Works great. But again, we see that on the efficacy level, it is not really what we need. So it does not really make a difference. There's very quickly, you can see diminishing returns. And actually AI tutor is what allowed us to unlock this kind of the other day engagement. And we see already today about 10% of students engaging daily with the product rather than kind of twice a week that they would otherwise be engaged. So that's kind of been a really great unlock for us.
- [22:54]
B
Thank you, Karin.
- [22:56]
D
Yeah, so I was sharing just a little bit before, which is, you know, at Buzu, we pair gamification with structured progression and we don't let learner actually move ahead without any accuracy or capabilities. And so we're really using more of a sandwich strategy where we start somewhere where they feel comfortable and then we bring something that is challenging and then end up again on something comfortable. So we're always, we're seeing, as you said, engagement is the door to be able to have them go through a learning experience that is a bit more challenging. At Chegg skills, we don't even look at engagement at all. We look at outcomes. And so 67% of people going through the courses of students are actually using it on the job. And so for us, outcomes and value is really number one. And so of course on the app and then overall is how do we get gamifications and the right level of difficulty to engage and keep people in the learning process.
- [24:03]
B
Thank you. Thank you for sharing. On our side, perhaps one thing that comes to mind is, and maybe some of you have heard or experienced, we used very unhinged sentences. And actually there is a Reddit like as the Duolingo says, which is kind of funny, where people put all the sentences that they get to translate nowadays. And that was on purpose because if you are presented with a sentence that doesn't make sense, then you will check those words again and again trying to make sense of them, and that helps with learning. So it was on purpose. But now with AI, people assume that it was just AI hallucinating. So that's a really interesting dynamic. So we are trying to say, no, this was on purpose. We really wanted to tell you that the horse is actually, I don't know, reading the newspaper. We meant to say that so that you think about it and you remember those words. So it's an interesting phenomenon going on with AI now that people don't believe us. They said, oh, this must be be poor AI. Yeah. So on the test we have not a challenge, but when I build A test. I look at something that I called and again is more technical. I look at the element of information per unit of time. So when we have English for academic purposes, many people would say, well, how long? They should be able to write like essays and sit there for, you know, two hours. And that's not how we think about it. We think, as I said, as information per unit of time. So we've done experiments and we've done piloting to look at what type of items and interaction will give us the same prediction of quality engagement at a university class based on the type of item that we use. So are different types of metrics that we take into account when we build a test. So but we did have to work with this engagement time per test versus the measurement accuracy and all of that. So thank you. The next question is speed versus trust. In a fast moving AI infused market, how do you decide when to ship and when to validate? So we'll start with max.
- [26:37]
C
Sure. I'm definitely very much action biased. So usually when the team is hesitating, I'm telling them to ship and figure it out later. But you know, obviously to make sure that we maintain the consistency and high quality because, you know, we have more than 100,000 students that depend on us. You know, we can't just sort of throw things out and see what happens. So we do very extensive A B testing, randomized trials. This is extremely important. I think this is actually a huge advantage we have over kind of a legacy publishers where a lot of content is expert driven. You know, they just basically somebody said it's okay, so that must be okay. But there's really very few data points to prove that. So I think this kind of new age content providers such as Duolingo, Novakit and you know, old all the present, we have this advantage of actually proving it with data that hey, if we switch content in this lesson, there is actually two points, efficacy improvement. And that's how we make sure that we can move fast but not break too many things at once.
- [27:50]
B
Thank you. So Karin.
- [27:53]
D
Yeah, so we actually separate those deliberately on our site. There are. When decisions are reversible, we move fast. When they're not, we go slow. And so in some ways there are things that are reversible that are related to access and usability. So your onboarding, your, you know, some of those experiences, the moment we touch outcomes, this is where we go slower. So for instance, at chick skills, the 92% of the learners stays at their employers for over a year. If we lose trust There it can have a massive impact. And so we really this is a place where this is a tension in our product team saying in which camp are we, is this reversible? Not reversible, how much is it impacting the progress and outcome of our learners? And this is where we're making, you know, this is how we're making decisions.
- [28:52]
B
Thank you Hugo.
- [28:53]
A
Yeah, actually I can share a little bit about my own background actually. And my first job, you know was engineer with IBM. So that's why actually and I am truly believer of AB test. So actually in the company actually we move very fast as a young company, 80 years old startup. So every ideas actually when we have the idea we can move very fast. But actually I echo to you know, max we are doing education, we need to care about the learning outcome and also parents satisfaction. So I can give you one example which is our AI tutor. Recently we launched so actually in January we launched the AI tutor. So we only invite like less than 100 students. So we have more than 100,000 students, you know, active user with us. But actually currently we only invite 100 students to test the AI tutor and we see the percentage of the completion, how many like they finished like you know, five minutes, like you know, practice and also the satisfaction of the young learners, the satisfaction of the parents. Only when we see we reached a certain bar and then we will scale to maybe 1000 and then with the 1000 sample we can test another one or two months. So basically now we scale to more because we are very, very confident about the completion rate. So firstly if they don't complete means they don't like it and also do they come back so the other day they do it again. So actually we think the feature is good. So that's why actually and we measure like you know and completion, we measure like you know and the retention and all these are very, very important in the metrics. So internally actually our company, our culture is really about data driven, right? So in any meeting we need to see the data, tell me the you know and the completion, tell me the retention, tell me the like you know, satisfaction and NPS and then we can make decision right to scale. So also actually I can give another example. It's really about in one product we launch and for six months we grew very fast and then actually we see some problem about retention. As you know we need to slow down so we can sacrifice the growth for another six months as in another six months we slow down, we don't grow, we fix the retention issue and Then we can scale in six months time. So that's kind of the. We move fast but we are very, very careful about learning outcome and the customer satisfaction.
- [31:06]
B
Thank you, thank you for sharing that. I definitely agree, agree with everyone here. We do the same. We have a B testing. We are data driven, metrics driven for the learning app for the test. As you probably imagine, if it's a high stakes test, we cannot do AB testing. So what we did instead was to build a platform where people are invited to participate in research studies. Obviously they are very motivated to participate. The request comes after the practice test and we are able to collect data and do the analysis. So that works for us really well. But we have to be very careful and obviously on the test, that being one of those doors that closes if you get it wrong, as Karen said. So we are going to be very careful there. The next question is different. So it's about ethics, but it's ethics with a twist. In what case it's unethical not to use AI. Who wants to start?
- [32:19]
C
I think in all cases this is just such a powerful technology. To be honest with you, when I first saw it I could not really believe my eyes because I'm an engineer, I know what's possible. And when I, you know, just had my first chat with ChatGPT 3.5, I remember two years ago, I was like, holy crap, what is this? Is this aliens invading? So I think it's, it's definitely. It has to be used with the right guardrails. It's enabling technology. It just, it has to be used. That's my opinion.
- [32:51]
B
Thank you.
- [32:52]
A
Yeah, I can share my view. And firstly, I'm a parent. Right. So. And before my role of entrepreneur, I think as a parent I really have my own view about how to let my young children, it's not about adult, I talk about children and for example, I have a daughter, nine years old. I truly believe, I don't want give her too much exposed to AI and because she's too young, she doesn't have the very fundamental skills, critical thinking, problem solving, all the skills. I think she's not able to make the judgment by herself. But for my son who is like you know, 16 years old, I'm very confident, like you know, he can make reasonable judgment, you know, you know, the teenage.
- [33:32]
B
Right.
- [33:33]
A
So actually we still. I couldn't believe him as himself as well. But anyway, so I'm comfortable to let him to get a certain level of exposure and so that's why we translate that into my business. A Daily work I really think about depends on age group. So we need to give them different level of exposure. So for example, at a younger age, which is our major target audience, so we believe AI can be used as the tool for teacher, not very directly to every single learners because they don't know how to make own judgment. They still need to build their skills, reading, writing, critical thinking, problem solving. They need like you know, and see some failures and, and also they need to see some success. Right. So all this is very important. But AI can be a tool for like you know, and the teachers. So that's why we build AI tutor for them. Right. So they can use our AI tutor instead of the visit like you know, ChatGPT, Gemini or maybe you know, cloud by themselves. So but you know, for the older one, so they can use AI tools by themselves. So that's why actually, and I think, and you know, it's not about whether use AI or not, it's about how to leverage AI in certain extent.
- [34:47]
B
Makes sense. Thank you.
- [34:49]
D
Yeah. And I'll probably bring something that not most people are bringing generally, which is AI is actually here to provide more access. But most of AI, not in China, most of AIs have been trained on English. So to actually take the most out of the intelligence that AIs can provide. Actually speaking English is still very important and we can widen the gap if we're not empowering people to use those tools in English. And so it's how using AI to help people learn English is even more ethical. Otherwise we're actually getting even more of a gap about who can harness this new intelligence to do their work in an augmented way. And so today if we don't use it, we'll just increase the divide.
- [35:47]
B
Thank you. Thank you. Yeah, I agree with the presenters. I also am a strong believer in AI. On the Duolingo English test, we launched the first complex task for reading that was built with AI in April 2022, which if you may remember, that's about eight months before GPT was launched. So definitely I am for the use of AI. However, we use it with a lot of guardrails on the test. So again it depends on the purpose. On the learning app it's a bit easier. We have mostly adults on the website, so we have filters in place and it works fine. But for the test we need also compatibilities because it's a standardized test as well and therefore we constrain AI, but we use it throughout the test development from developing the content to administration and proctoring. So yeah, let's move on to the last questions. We only have a few minutes left. Personalization versus measurement. As experiences become highly personalized, how do you maintain a stable interpretation of proficiency and of progress? Are we over personalizing learning
- [37:06]
C
once I can start? This is Max. So we have from the very beginning we have been very, I would say methodical about measuring efficacy. So we're really kind of breaking down our curriculum in what we call the micro skills and there is thousands of them. So basically I think kind of as long as you do it in a data driven way and as long as you can see that personalization actually moves the needle and that comes back to AB tests actually, then it's all good. So I think there's definitely an optimum. Just need to be very data driven around it.
- [37:42]
B
Thank you.
- [37:43]
D
Yeah, and in some ways the way I'm looking at it, which is probably very close to what Max, you just shared, which is personalization of the journey, not personalization of the destination. So the proficiency standards are very clear and that's why we have Istax test and everything. But personalizing the journey to get to that destination is probably the best gift we have right now in terms of changing the way we're learning as individuals.
- [38:13]
B
Thank you, Hugh.
- [38:14]
A
Yeah, I agree with both of them. So I think measurement is very, very important for language learning. Right. So actually that's why actually you can see how we make decisions. So we hire our head of curriculum, which is the PhD from Stanford and also expert of assessment. So that's why actually we always believe. I know. Yeah. The learning experience can be personalized because every learner has their own learner profile. But actually in the end states the destination should be standardized. That's why assessment plays a very, very big role in lingo ways. So even just maybe one hour before this panel, I talk with our vendor of assessment. That's why assessment plays a very, very big role in our platform. So our head of curriculum is the expert of assessments and we use our third party provide very great, like you know, world class actually assessment tool for our learners. So that's why I think measurement is super, super important is the how to define the common standard, the golden standard for everyone. So that's super critical to lingoist.
- [39:13]
B
Thank you for saying that because that is also true for us. So we use assessment and learning measures for evaluating people in addition to personalizing the experiences. And with that we are almost at the end of our session. I want to thank the panelists. So please join me. Thanking the panelists and thank you all for participating.