← Edtech Insiders
Edtech Insiders
Week in Edtech 5/20/26: AI Backlash Grows, Anthropic & Gates Launch $200M Education Push, MasterClass, Chicago Booth & OpenAI Launch an AI-Native Business Program, and More! Feat. Angel Chung of The Wharton School & David Rogier of MasterClass
01:41:12
Loading summary
Transcript107 lines
- [00:00]
Alex
This season of EdTech Insiders is brought to you by Cooley LLP. Cooley is the go to law firm for education and ed tech innovators offering industry informed counsel across the pre K to gray spectrum. With a multidisciplinary approach and a powerful edtech ecosystem, Cooley helps shape the future of education. This season of EdTech Insiders is brought to you by Starbridge. Every year, K12 districts and higher ed institutions spend over half a trillion dollars. But most sales teams miss the signals. Starbridge tracks early signs like board minutes, budget drafts and strategic plans and then helps you turn them into personalized outreach Fast win the deal before it hits the RFP stage. That's how top edtech teams stay ahead.
- [00:53]
Ben
It's only a matter of time before AI automated weapon systems are making decisions about human life, period.
- [01:00]
Alex
And that is definitely one of the many scary things about AI that those 42% of Gen Z is worried about. They're worried about their jobs, but they're
- [01:08]
Ben
also worried about why they're booing AI.
- [01:10]
Alex
Yeah, I mean the story's happening in Meta right now where they're basically trying to force thousands of their employees to have AI surveillance on their work so that it can basically train them to do a lot of the work like they do. And the people are rebelling against this. But I mean that's exactly the kind of autocratic top down decision that could be made inside a company or it can be made inside a country that is extremely dehumanizing, to put it but lightly. Welcome to EdTech Insiders, the top podcast covering the education technology industry. From funding rounds to impact to AI developments across early childhood, K12, higher ed and work, you'll find it all here at EdTech Insiders.
- [01:54]
Ben
Remember to subscribe to the pod, check out our newsletter and also our event calendar. And to go deeper, check out EdTech Insiders plus where you can get premium content, access to our WhatsApp channel, early access to events, and backchannel insights from Alex and Ben. Hope you enjoy today's pod. Hello EdTech Insider listeners. We are back. It's old school today. Alex is back and in full effect, y'. All. Welcome back. Alex, Week in EdTech has missed you and we're so excited to talk about everything that we can fit into an hour. I mean, I don't think we're going to be able to do it, but let's see what we can fit in. Welcome back. And how's like coming off the high of ASU gsb, it was just so huge that this Year was like none I've ever seen before.
- [02:47]
Alex
Yes, it's so big, so many people there, so many sessions happening, so many events happening. Obviously being within the GSV ecosystem makes it a different experience. You're sort of seeing behind the scenes and helping volunteers and doing some, some really important and sort of unseen activities. But it was incredible to see so many people from across the edtech and education and higher ed and workforce ecosystem all in the same place, all at the same time. Lots of educators. It is truly a high. I mean you walk around, I think anybody who's there, you sort of walk around like a flutter the entire time and then go back and look at all your, the people you talk too and think about all the follow ups and it is, it's quite an experience. It's been amazing. And then it's already been a month, which is crazy. But now we're looking into the summer and thinking about what's next in this world, in this crazy edtech world.
- [03:38]
Ben
Yeah, it's been a really, really interesting time. I feel like we've gotten to the depths of edtech winter. So while it's heating up outside, it is really getting cold in some parts of the edtech landscape. But we also have really interesting articles coming out. This new series we're doing on efficacy research. I know you've been a champion of that for a long time, but actually unpacking what does it mean practically some of the Stanford scale insights, how does it apply to ed tech and education and how we see all of this playing out. We also have David Rodger from Masterclass on the pod and then we've had a bunch of great events including incredible happy hour. So great to see everybody there. If you haven't checked it out, our newsletter, kind of recap of ASU gsv. It's kind of the seminal like exclamation point on that whole event. So before we go into too much stuff, any other updates that you have, Alex, before we whip around the world of AI?
- [04:42]
Alex
Yeah, well, just building on that, that research and efficacy piece, you know, one of the interviews in this episode, at the end of this episode. Stay tuned for it. David Rajer is amazing at Masterclass is doing really interesting work. We also talked to angel chung who's a PhD candidate at Wharton who just put out some really interesting research using about, you know, 700 Taiwanese high school students and doing a comparison study about using AI or not. And they found some really interesting stuff out. I think we're maturing as a field, you know, as the Edtech winter as people are sort of getting a little nervous about screen time as a number of states are starting to pass laws basically banning laptops in school, except with exceptions. There's some pretty interesting things happening out there, and I think there's a very vocal backlash against edtech, often because there's not enough efficacy research. There's some really interesting research coming out. So I think that the Stanford Scale initiative is incredible. I love the series that we're doing with Stanford and Scale and the Overdeck Foundation. It's incredibly important right now. And that newsletter was one of our most read newsletters of all time. It's just worth talking about.
- [05:45]
Ben
I mean, the backlash that we predicted in. In December, it really has come to full fruition. One of the stories from this week is really about the new college graduation ritual, which is booing AI. We'll put the link here, but a number of commencement addresses, including Eric Schmidt, former Google CEO and former lead of Schmidt Futures, which unfortunately no longer funds education, got a rousing boo from the audience at University of Arizona's commencement.
- [06:19]
Angel Chung
And.
- [06:19]
Ben
And overall, Gen Z is really raising their hand to say this isn't great. 42% of Gen Z says AI will harm job opportunities and wages for people like them, compared with 33% of millennials and 39% of Gen X and 37% of baby boomers. I will say AI is still pulling ahead of the Democratic Party, pulling ahead of Trump. So, you know, it could be worse. But I do think that we've entered that phase where the technological change is upon us and some of the dislocation and disruption, we're starting to feel it, and anxieties are riding high.
- [06:58]
Alex
Yeah, I think an important piece of that is that I think the perception for a while was that there was this big gap between the people who are extremely AI positive, many of whom were entrepreneurs or tech folks of various kinds, and educators. And there was this thinking that, okay, tech people want to lean in on AI, but educators and principals and administrators and professors are a little nervous. And now there's actually a little bit of a change in that narrative because a lot of the people complaining and worried about the future of AI are students. You know, I think we saw this. There's another terrific editorial that came out this week in the Times from a Stanford student talking about, you know, basically saying, I'm the first AI generation. I came in my freshman year, AI came into existence, and now we're graduating and we're using blue books now. And every single person I know has ChatGPT opened every class and the research is not yet showing that it's positive. But we know that professors have all these different reactions, students have all these different reactions. Even the students, which I think we had this sort of hypothesis that the educators were going to say no, but the students were going to say yes. I think even the students are starting to say, what does this actually mean, this crazy change? And like the stat you just said, over 40%, 4 out of 10 Gen Z think that AI is going to really affect their job opportunities and wages. So no wonder when you have the likes of Eric Schmidt talking to a graduating class, they're not booing him, they're booing AI as a concept. And the reason we know that is they're booing almost any speaker who is talking about AI as a main part of their speech. And they're saying, hey, AI is coming. It's going to be part of our life, it's going to be part of their world. And people are like, I don't want it. I'm afraid of what it's going to be. That is a huge change.
- [08:35]
Ben
I mean, New York Times in their podcasts have also put out some interesting things about how China is rolling out AI versus how us is rolling out AI. And China's really rolling out AI in a practical way where it's just integrated in things in a very small lowercase AI way, where it's just like, let's make this thing incrementally better, like our matching for delivery drivers or our ability to improve factory efficiency rather than this race to AGI. And overall sentiment in China is much more positive about AI making life better as a result. Part of that is a perception. I think the other part is that we are in a trough of trust with tech companies. And I think we've believed the hype before that, oh, social media is going to make the world better, or having infinite access to content and democratizing it through YouTube is going to be great. I think we're now grappling with some of the social impacts of that. And this is where edtech is kind of getting swept up into for better or for worse. And there may be some ways in which we in EdTech haven't had a quality or a research or an efficacy bar that's sufficient. But the number of people that are equating like social media, a cell phone, screen, time, a learning tool, and AI, that's one big bundle that they're just frustrated with. And the trust is very low. And I think that that's a theme that's only going to get worse when we get the Toy Story movie here at the end of the summer. That's going to be the toys versus the iPads versus the screens. There's a cultural zeitgeist here of like anti tech that I think that wave is. It's not cresting, it continues to build.
- [10:23]
Alex
I totally agree. And you start to see educators on LinkedIn literally talking about edtech, like, hey, note to edtech. Hey, ed tech, you should know that, blah, blah, like literally Talking to the EdTech community as if it was a monolith, as if it was big tech or big pharma or, you know, we've talked about it before, but like, you know, really angry and really, I think, lumping in some of that unfulfilled promises of the sort of Silicon Valley Messiah movement that we've seen over the last decade and a half. Jira Daras Jiddaras wrote an amazing book about it about, you know, this narrative of we're going to save the world and meanwhile inequality keeps getting bigger and some of these celebrity CEOs keep, you know, start buying islands and disappearing from the society. It's crazy. We also saw this week, I think it's really highly relevant. There's an interesting paper came out from the University of Colorado Boulder, and this is covered by our great friends at the EdTech Innovation Hub, I think becoming one of the most important hubs for news and edtech that basically says, hey, other technologies that have come into the classroom have sort of come in and usually been gatekept by the educators. But AI is this arrival technology. We've all heard that before. And it's really completely changed the adaptation. It's completely changed how it works. And they basically posit a framework of all these different elements of how you can sort of put it together. But it may not be a total revelation, but it is, I think, really worth noting that AI, unlike almost any other tech in edtech history, really came from all sides, right? It came bottoms up through student usage. It came from the side through educational tools that started integrating it. It sort of just landed on the entire education ecosystem at the same time between like 2022 and 2023. And even now, three years later, I think there's almost no clarity about the proper policies or how the grading should work. And you see places like Stanford, which are, you know, models of how to do this, going back to blue books and saying we throw up our hands, it's integrity is just out the window. If they say that, how would a community College or a state school, why would they do anything different? So it's just a. It's a really confusing time. And I think that the combination of the students and the professors. We always hear about students being worried about false positives for AI identification. Right. If they write something that's identified as AI, we've seen parents suing schools for, for that. This combination of things is so explosive right now. And I think the evidence is one of the only things we can do as an ed tech community. But another maybe trying to differentiate between ed tech and traditional big tech. I wonder what that looks like. What do you think, Ben?
- [12:54]
Ben
Well, it's harder because big tech is coming into ed tech. Like there's a blending, emerging. And one of the sessions that I'm hoping to do at New York EdTech Week is. But Steve Shapiro is like this kind of merging of big tech and edtech. And where's the room for the edtech entrepreneurs to operate and be experts in our subject matter versus what's going to get consumed by the kind of AI Goliath? And one of the challenges that we have right now is even if we make efficacy claims around AI, the models are changing so rapidly that it's hard to recreate those studies. And in fact, I think there was a recent one where they used LearnLM, which was the Google learning one, got great outcomes. Then Gemini subsumed LearnLM and the outcomes actually went backwards because the generalized AI wants to give you the answer. Isn't really trained for pedagogical friction.
- [13:54]
Alex
Yeah, you're studying something that's evolving like constantly.
- [13:57]
Ben
I mean, I think the path for many entrepreneurs is likely to be digital, physical now. So we've been through a digitization revolution where we took that textbook. Maybe it was just like a doorstop holding the door open and brought it online. And that allowed greater access, it allowed for greater data capture, it allowed it to be in different languages. It could allow for personalization at scale. There's all these potential benefits, but the distraction on the machine has also shown to be a real issue, especially for younger learners. So I think the people that I know who are thinking most innovatively about this are thinking about what's the digital physical combination. Do I have printed materials that go with the digital materials? How do I connect the two? Is there a scanner or a digital camera? Or like, how do I have kids operate and learn in a physical environment as you go up to higher ed? I think this idea of real concrete projects that have performance tasks involved that then can get digitally captured and Assessed is kind of where everything's going. And the blue book is really the kind of sign that whatever you're assessing is probably not an authentic, rigorous, relevant assessment anymore and you need to start rethinking the actual assessment itself. So we're in that, like, transition period. But this is not a straight directional line to, oh, AI is transforming everything to be more personalized and more dynamic, competency based. We're actually seeing retrenchment in several vectors.
- [15:38]
Alex
And one of the big arguments for the folks who are feeling very doubtful about the edtech movement is they're saying, hey, look, textbooks were changed, people, laptops per every, you know, there was the one laptop per child initiative many years ago and now we're there, the people have Chromebooks, and yet we're not seeing results, we're not seeing the education system do better. The promise was you bring technology and you personalize, you translate, you do all the things you just named and it's going to get better. And they're not seeing it yet. Yeah, I want to focus. There's a really important headline this week that everybody in the edtech field should be knowing about. That I think is really relevant to your. Exactly your point about big tech and edtech being really coming in the same path. I know we've. Most people may have heard of it, but basically Anthropic has partnered with the Gates foundation for a $200 million partnership. It technically is health force and education, but the education piece is pretty important here. It contains K12 tutoring, career advising, college advising, literacy, numeracy, curriculum design in India and Sub Saharan Africa. It's going to focus on foundational literacy and numeracy. That's a big focus for the Gates foundation is global education in the US a lot of it is K12. But I think the thing that jumped out to me, and it's so relevant to the research piece that you just talked about, is this idea of this combination wants to fund the public benchmarks designed to test whether AI tools in these areas, K12 tutoring, college advising, whether they work before they scale quote, unquote. Right. And the problem, as you mentioned, with having research that follows on that says, okay, well let's look at this school used a AI for a year and this class used it and this class didn't. And let's see what happened. By the time you publish it, the models have changed. But if you're reversing it around and saying let's have these benchmarks that the tools have to pass before they go into schools, and they test against different types of scenarios, they test against different types of learning needs. That I think is the dream that the philanthropic community is really thinking about here. And combining the power of the finances behind Gates Foundation, $200 million in this case. And I'm sure there's more on the horizon here. We've seen other philanthropies in this kind of space with the technical power of an anthropic. It's going to be really interesting. It also gives Anthropic sort of the catbird seat in helping define what these benchmarks look like. I don't think that's meant there's anything sinister there with the Anthropic education team is totally amazing. But it just means that of all the different big frontier players, they have taken this interesting position saying they want to sort of be at the core of what these benchmarks look like and what defines AI quality. What did you make of that announcement, Ben? I know you saw that the second it came out.
- [18:12]
Ben
Yeah. And you know our friend Drew Bent at Anthropic co hosted our happy hour at ASU gsv. So just want to state our prior. Like we're really close with all of the teams at Google and OpenAI and Anthropic. One thing that's interesting about Anthropic is that they carved out a beneficial deployments team. So their go to market from the Jump was a B2B go to market path with the idea that we're going to build tooling that makes companies better and more effective. Whereas OpenAI with a consumer path with ChatGPT and Google went with a integrate AI into all the Google services approach go to market. What they've all done though in education is slightly different. Anthropic carved out a beneficial deployment team with the idea that we're going to do some things that are for good that have no commercial value to us directly whatsoever. But we do believe that our model will be the best model. And by having it in all of these beneficial deployment categories, health and education and environment, not only will we do the most good, but our models will be a backbone for social enterprise and so on. The most cynical view might be this is the Google play in education where you just get people using it from the earliest ages, they're going to become natural users. I think we're in a bigger race than that kind of framing on OpenAI education has been in their go to market team and Leah Belsky's there from you worked with her previously at Coursera. They've really been Thinking about this as a business vertical. And with Google, they've been thinking about it much more around their integrated tools for schools in higher ed. So it's actually not surprising that Anthropic's beneficial deployment team would be more nimble and less sensitive to commercial outcomes and more like, okay, how could we deploy for the benefit of humanity? So interesting that that structural piece would play out. I think the second thing to acknowledge is Mythos, which is the newest model from Anthropic. There's been a delay releasing it to the public because of the actual damage that it would do on the Internet. And we actually, in our last week in EdTech, we talked about the number of patches on sites like Wikipedia or other publicly available data. It was like a few patches per month and then boom, Mythos is released. And like a thousand X number of patches that companies had to release. Well, there is this sense of if we're going to be now in this age of releasing models that are this powerful, we need to be partnering with philanthropic organizations, almost like a B2B partnership, like they would do in legal or another vertical, to make sure that this is constrained for good. And so I think that's another element of this partnership. I will say, anybody expecting a big payday from Gates foundation through this partnership, My sense is that this money essentially funds capability building, but is probably not going to translate to like large active grants to organizations. So this fundamental like tooling layer which we've talked about with Learning Commons, we've talked about with some of the folks at Gates, I think this is supercharged for the tooling layer, but it's probably not going to lead to, you know, your local elementary school getting 2 million bucks to transform their school to AI like we saw in Gates when they did the small school movement. Last but not least, a math education is a big focus of Gates. And so I expect the tip of the spear here will be in math.
- [21:54]
Alex
Yeah, and they, they meant they call out math specifically for math tutoring and together evidence based math tutoring as a specific core of this. That's a great analysis. And I. You're really getting the 10,000 foot view about these three foundation models and how they're sort of thinking about the education space in very different ways. I say this every time. But it's also worth noting that among all the foundation models, Claude has the most restrictions for under 18s. It's not even designed to be used by teenagers or, you know, school age kids, which means that that B2B play and that sort of much More structural, infrastructural play. It's just a very different structure. And I agree with you. I don't think this is, this money is going to be designed to go out to specific companies in as much as it's going to be used to sort of fund and create an entire evaluation layer for evaluate, they call it evaluation infrastructure. So that the entire education AI ecosystem is raised in quality and privacy and security. So if you're an edtech company right now and this news comes out, I think the thing to really follow is, okay, when are they going to have these public benchmarks? How do we be on the right side of the public benchmarks? Because they might be an important thing to be able to say. Our tools do really well on these benchmarks. What aspects of the work that we do fuse with the focus areas and the sort of theses of what they're hoping to support. I don't think it's about how do I get the money fueled into my company or how do I get a pilot with a school or a district through it. There may be some of that, but it's much more about how do I position myself as a company to be on the right side, meaning the sort of the evidence based side of this. Once the evidence is actually available in this way, I mean.
- [23:31]
Ben
Alex, a question for you. How do you think this kind of thing changes the race to build the AI tutor for the next generation? Do you think that we're going to end up having what happened to lmss where there's just so much philanthropic capital that it distorts the market value and essentially it becomes commoditized and so Gates and anthropic release their AI tutor, OpenAI releases their AI tutorial. They're all relatively at token cost and that's the way it's going to go. Or do you think there's still meaningful businesses to build around an AI tutor? Or on the total other end, it could be. Every curriculum company has their own bespoke AI tutor trained on their own curricular data. Like I'm just trying to read the tea leaves because it does feel like there's some sort of holy grail of the AI tutor that people are chasing. But it's not clear whether it's a winner take all or it's a fragmented system or a distort by philanthropy outcome.
- [24:33]
Alex
It's a huge and very important question. I can tell you how I think about it, but I don't have a crystal ball on it because it's such a Complex space. I think it's as close to a sure thing as possible in this that the frontier models will continue to significantly increase their capabilities to act like different types of agents, act like different types of personalities, to constrain in different ways. We're already seeing really big leaps in their, their capabilities that started powerful and they've been leaping very high. So I think the idea that the frontier model tutors like if you go to an OpenAI, an Anthropic or a Google standard AI out of the box model and ask it for teaching support and tutoring support, it's going to do an accurate job. Will it be an extremely good tutor like a personalized assistant? Over time they've all sort of dipped their toe in that water and I don't think any of them have seen like incredible returns on it. Right. I mean we've seen Google's learning mode and by returns I don't necessarily mean financial returns. I just don't think they've seen seen enormous amounts of usage. I think what they've done tutoring modes for their commercial products. But I think many of the people who would otherwise need tutoring are choosing the tutoring modes. They're still choosing the standard Google Gemini Conversation Agent or the OpenAI ChatGPT. So I think the use case in which students self select into a tutoring mode, I don't think we've seen it yet. That said, the models will get very powerful, they'll get very, very smart about being able to support. But I do think there will be some room for as you say, two things for companies to be proprietary and get, you know, better than those models. One is proprietary data. That could be curricular data, it could be historic student behavior data. It could be because you work with an incredibly specific student population like deaf students where you know all of this additional things that about how to tutor or students with, you know, autism. There is proprietary knowledge. I don't think that proprietary knowledge will be domain knowledge. Right. I don't think it's that, you know, the person who has the most math papers is going to make the best tutor. And I think we have seen some tutoring companies that are sort of focused on that. They're like we're going to go deeper in our knowledge. I don't think you can out outpace the frontier models on that, but I do think you can outpace it with the specifics of the delivery model. Right. Who are you tutoring? Is it within a specific curricular context? Is it within a specific career context? Right. I Mean, if you're doing a career guidance tool for students in Minnesota, there is a decent chance that you can gather more specific information about what careers look like in Minnesota, what job openings are there, what skills are needed than the OpenAI would have otherwise if you put all the pieces together. So I do think there's a chance for sort of proprietary additions, but it's going to be hard because I think these frontier models are incredibly powerful. The one thing I would say though, I mean we've covered all the launches of ChatGPT's learning mode and Google's Gemini mode. None of them have quite become household names. I don't think that any of them, even though that use case of a tutor that actually acts like a teacher, that actually acts like a really good personal tutor seems like a very obvious one. We've even seen Khan and Khanmigo say, I'm not sure because I think a lot of people don't opt into working with a tutor. So unless the context is there, unless there's a structure there, it doesn't make that much sense as a product if you're a frontier model to have the tutoring model and try to sell that or promote that separately than your core model. Because I think most people are choosing the core model. We'll be right back. Innovation in pre K to grade learning is powered by exceptional people. For over 15 years, EdTech companies of all sizes and stages have trusted higher education to find the talent that drives impact. When specific skills and experiences are mission critical. Higher education is a partner that delivers offering permanent fractional and executive recruitment. Higher education knows the go to market talent you need. Learn more@higheredu.com that's h I r e e d u.com
- [28:40]
Ben
yeah, it's interesting. I think if you're sitting at one of the big models, you're thinking about what's the infrastructure that I'm building and how far do I get to the last mile delivery. And basically a lot of what you talked about, the unique contours of the use case and the population basically is applicable to everybody in every industry. And I think the scary thing for the models is if their base model goes all the way to the last mile, they don't have any customers anymore. They only have consumers who only have a finite out of pocket amount. They also have liability because now they are delivering last mile. So the efficacy or impact of what they're doing, you know, they're responsible for and I've seen a little bit more of this in healthcare and legal. They are really thoughtful about. Let's stop where our generalized model expertise ends and let's find the right companies that partner with us to translate that to specific use cases. What's weird to me about education is some people think it's an industry, some people think it's just a do good philanthropic thing for the world. So the lines are not as clear like with legal, it's like, oh, you know, you need a law degree to make that distinction. But when it's a pedagogy question, oh, do you need a teaching degree or do you not? And so that line has been constantly blurred forward and backward. And I think this makes it a really tough buyer experience because if I go and buy something that is doing last mile, but then there's a new model release and it's essentially effectively free to me versus paying for that last mile one, will I accept a B product that's free or relatively low cost versus a paid product? This is our conundrum with our buyer landscape.
- [30:36]
Alex
It's a real one. And I think there's a great segue to the next headline which is extremely relevant to exactly the question you just asked, which is, you know, OpenAI has been doing these, the OpenAI's education for countries initiative where they go to countries at a time and develop these complex. They're not complex, but they do these sort of countrywide education models where they give lots of of free access, they do classes, they do professional development. The list of companies on this is a wacky one. We know about Estonia, Greece, Italy, Slovakia, Kazakhstan, the uae, Jordan, Trinidad and Tobago. And then we just saw this week OpenAI for Singapore. Singapore is a particularly interesting one when it comes to education because as listeners to this will surely know, the Singapore education system has been very revolutionary and has actively adapted to be research based and outpace many, many other countries. It's also a country with enormous AI US usage and a lot of, you know, tech forward thinking. We also saw Malta working with Microsoft and AI to do a national AI education program that basically gives people access to tools and courses. I mean, it's a pretty strange list of countries, right? A lot of Mediterranean countries, a couple of island nations. Kazakhstan is one of is enormous country, but not very high population. UAE is an incredibly rich country. Like this is a weird group of countries, but it's also a group of countries that I think are willing to sort of be out front on things for a variety of reasons. Some because they feel like they're catching up, others because they feel like they want to leap ahead like Like Singapore and the UAE and maybe Jordan and Estonia. But like, it's a weird group. But I think it's worth thinking about in terms of the exact question you just asked. Because if OpenAI goes into a country and says this entire country is an open AI country, and every kind of system, education, health, any kind of job preparation, works through that kind of thinking, they are incentivized to go very deep in a lot of different use cases. And the country is sort of basing its entire ecosystem on one particular frontier model's offerings. What do you make of the OpenAI for country? We've covered it over time here. It's incredibly interesting to me because it's just, it feels like this, like almost like Cold war land grab. It's so wacky. How do you think about this and what do you make of, in terms of the questions you were just asking?
- [32:49]
Ben
From an economic standpoint, there's a lot to be interested in there. But there's a diplomacy story here too, which is that, you know, if US models become like anchor models in other countries, then that means Chinese models aren't becoming anchor models too. And I will say I think there's a new opportunity for edu diplomacy in the world where if you combine an AI model with educational tools, you can help accelerate leapfrogging from global south countries. And that could have really tremendous diplomatic potential. I'm thinking mainly of Venezuela, where they just kind of came online with the rest of the world. So essentially their education system is stuck in the early 90s and instead of having to rebuild or transform an existing system, they have an ability to just from a nascent standpoint, just get started with an AI native system. You know, if the US could just increase the oil exports and tax it at like 2%, they could fund an entire education revolution in Venezuela. So when you're looking at a company like OpenAI, so that's the diplomatic governmental side, let's look at the business side. What is their valuation? I mean, their valuation's so insane that any kind of normal growth does not move the needle for them. They've got to get country sized deals going. And you know, there's an open map where there's a lot of like white space and you're just trying to gobble up as much of the map as you can. And unlike the US where decision making is so decentralized, education is one of those centralized plays where you do something with the President, the Minister of Defense, Minister of Education, Minister of Health, boom, you're in full stack with that whole country. And we saw this with CURA learning with El Salvador. There's also a playbook that Ed tech companies can make globally that is essentially impossible to make in the US with our fragmented system. And I would also just remind everyone that while they're independent companies, any of the Chinese AI companies are essentially national, state run AI companies. So one question that I've been thinking about is a little bit off topic here is the Industrial Revolution had like product market fit with democracy as a governmental system. Does AI actually have product market fit with autocracy and centralized decision making as a means? Because all of the power is in the aggregation of data, of surveillance of technology. And I worry that from an education system standpoint, the top down systems are going to be far more effective at leveraging AI for student outcomes than decentralized systems like ours.
- [35:53]
Alex
I mean, I think there was a reason why there was this really pivotal inflection point moment, at least in the public narrative when Anthropic pushed back against the pentagon in the U.S. about you know, various kinds of. Because it felt like that was taking a stand about exactly the type of question you're asking.
- [36:10]
Ben
And meanwhile OpenAI just signed the damn contract and then Anthropic's already in most missile systems anyways. Like as much as people applauded the moment, I think we could also realize it's only a matter of time before like AI automated weapons systems are making decisions about human life, period.
- [36:29]
Alex
Yes. And that is definitely one of the many scary things about AI that those 42% of Gen Z is worried about. They're worried about their jobs, but they're
- [36:38]
Ben
also worried about why they're moving AI.
- [36:40]
Alex
Yeah, I mean the story's happening in Meta right now where if I'm understanding correctly, they're basically trying to force thousands of their employees to have AI surveillance on their work so that it can basically train them to do a lot of the work like they do. And the people are rebelling against this. But I mean that's exactly the kind of autocratic top down decision that could be made inside a company or it can be made inside a country. That is extremely dehumanizing to put it but lightly. I mean it's a fantastic question. I'm going to be, you know, try to be evenhanded here and say I wouldn't say that AI, it has product market fit with Autocracy directly. I would say that AI is incredibly, it basically empowers anybody who's using it enormously. That can. Yes. If you're a Centralized government that has data on everybody. That is a big deal if you're, if you're a military system that has
- [37:27]
Ben
incredible, there's a concentration advantage.
- [37:29]
Alex
But it's also a big deal for health companies. It's also a big deal for big school districts that have millions and millions of data points. It's also a big deal for, I mean for email marketers. Like everybody gets a boost and even individuals.
- [37:43]
Ben
Well, on the flip side, it does make it easier to build. Like anyone can build their own tool. So there is a democratizing element and potentially destabilizing of autocracies in that like it's really hard to put that genie back in the bottle and everyone can build when they couldn't before.
- [37:59]
Alex
I mean people have done things like built tools that inject pixels into their art so that if an AI tries to train on it, it completely confuses and bewilders the AI. Like there's a sort of counter cultural, counter revolutionary AI sabotage. Well, yeah, so we're entering a very, very strange world. I think everybody feels that it doesn't. It's hard to know what it's going to look like. But I think your AI for countries that American AI in a country like Singapore, which is much closer to China than it is to the US and has a lot of Chinese associations within Singapore, it's an interesting read on it. It makes a lot of sense. There's two more stories I want to cover but I don't want to leave this topic behind if you want to dig deeper because it's super interesting.
- [38:39]
Ben
One thing I do want to say, Alex, you know you're talking about what's the unit of change essentially. And centralized departments of education can be the unit of change. School districts could be the unit of change, a school, a parent or family. And in higher ed, you know, we're also seeing lots of different grain sizes of programs launching, including what we'll talk about with David from Masterclass. I think one thing that you and I haven't covered on week in edtech is also the unbundling of payments in the US While these other countries are going to more centralized. We're seeing incredible rise in education savings accounts. And now I don't think EdTech realizes $1,700 per child tax credit coming out from the Trump administration as part of the obba, which basically would allow parents to find educational supplemental things to subsidize their kids education. So they could still be enrolled in the public school or a private school or a home school or whatever. But the 1700 as a tax credit is available to them to purchase supplemental. And we're seeing that poll incredibly well. We're seeing a space where up to 10% of families are opting in with a growth rate that suggests this could get to like 20%. So while we're thinking about like what does transformation look like at the nation state level, I also think there's a pretty cool opportunity in the US at least to think about what does transformation look like where the family is the unit of change. And it's very rare that any industry gets an unbundling of payments like this. Like we're having where it's a government funded but a B2C acquisition motion. And I think there's a lot of business opportunity there. Back to your question about generalized AI versus last mile. I think there's a lot of opportunity for education folks to be that last mile because a student may not want that tutorial, but the parent definitely wants that child to have that tutor. And there's a real interesting triangle there with student, parent and learning product or program offering that's totally been unlocked in the last 12 to 24 months. And now with this new $1,700 tax credit. And I'm hearing that people are going to come out with like HSA type cards, like a debit card. So it's not like I have to wait till I do my taxes and it gets low adoption. It could be something where I literally am spending the money in my account and it's called a Trump credit or something so that he gets the political win. But I think there's real profound market opportunity on that motion.
- [41:22]
Alex
Yes, I agree and I think it's a really interesting one. I mean, I have heard that the infrastructure for this is still being built. Right. I mean, we have talked to Jamie Rosenberg, who is the founder of Class Wallet, which was early on this. We've talked to a number of different people in this movement.
- [41:35]
Ben
Joe from Odyssey.
- [41:36]
Alex
We've talked to Joe from Odyssey. Yeah, a long time ago. We've talked to him a few times. This is a nascent world. It's a really exciting one for exactly all the reasons you say. It's also interesting in light of the increasing pushback from public education against tech and AI. Right. If a state like Tennessee passes a law that says you can't use technology in the classroom, you very well could see a lot of parents and families that are more tech friendly saying, well, we're going to spend our supplemental money on technology because the school is pulling back from it. Or vice versa. You could see that maybe the whole state says, okay, well that's the new normal. Technology didn't work for us. We take it out of our classroom, so we're going to spend it on books or we're going to spend it on screen free devices. So it's a very strange moment and I think ed tech companies have a real opportunity to sort of help shape that narrative, help explain their value, help explain why, as you say, the unit of change. If a parent or a family is a unit of change, if they have their own expendable income on educational supplementals, people can do the marketing and do the customer acquisition cost and market. You also have opportunities for people who were previously B2C to sort of move into that space and get government funding. Right. I mean we just saw, we haven't covered this, but Duolingo is shutting down Duolingo for schools, which is really interesting. It's its whole topic to talk about. But you can see potentially companies that have been successful in a B2C context then saying, okay, well we already have the marketing, we already have the customer acquisition movement. We know how to get in front of people, we know how to get people to sign up and stay with us like a prodigy. You know, maybe now we're going to chase this, this audience and I think, I'm sure they're all thinking about it in some of those ways. It's a really crazy. I've heard people talk on both sides of this, right? I've heard some entrepreneurs be incredibly excited about that sort of unbundling and others say it should be exciting, but it's so bureaucratic. Those cards you mentioned are not fully out yet. It can go both ways. So I don't want to put my chips on one side or the other too hard there. But I agree that that unbundling and that change in the unit of change, especially away from public and districts is one of the most notable things happening in edtech right now. So this is a little off topic here, but I think it's relevant. You know, We've talked about OpenAI, we've talked about the Google I O conference just happened and number of different things were announced. But a couple that stood out to me as potentially relevant things that I think edtech companies should at least keep their eyes on because they could develop to be things that are revelations or product changes that are very relevant to the edtech world. One is the Gemini Omni video model which is considered a world model. It's basically multimodal on both you can change video to video, you can change speech to video, you can change any video to text. It's this incredibly. That's why it's called Omni. And if you haven't seen the sort of demo video of what it's doing, it is pretty crazy. And I think it's something that builds on some of the things we've talked about on this podcast about a future in which, you know, you can literally sort of create or manipulate reality in some ways or at least either in video format or in sort of live within an XRP environment. I think we're getting closer to that reality. And it just unveils some very sci fi, speculative, very wacky and exciting potential for edtech. And then I think very relevant. They're also announced their eyewear. You know, we've talked. Google Glass is now ancient tech history, but Google is giving its new AI glasses. They're taking on Meta. They're competing with Meta, but they're also competing with Snap. Who's doing that? Alibaba has been making their own AI glasses. Apple. The Apple device is unbelievable, but it's incredibly high priced. But Apple is launching AI glasses. But Google doing this and doing it on the Android system means it's another very, very big tech company sort of leaning into this XR glasses. So if you put these two together, a video model that can transform materials into anything, that can make people appear, that can do, you know, just do unbelievable things and then glasses, we're getting into some pretty weird stuff. I'm curious if you think either of those are actually relevant to edtech or they're just pure speculation.
- [45:34]
Ben
I definitely think they're relevant. One of the first principles of education reimagined, which if you haven't read it, if our listeners haven't read it, it's like from 10 years ago, 2016 and it basically said, what does the movement from the industrial model to the learner centered model need to look like? And it's learning anytime, anywhere is one of the big principles. And the idea that you could take your AI assistant with you and not only can it hear your voice, but it can see your world. Imagine walking through, you know, the streets of London with a history lesson going on as you look from building to building or imagine it, you know, you're working through, you know, motorcycle repair and it's walking through instructions and helping to
- [46:18]
Alex
teach you how to do it that exists now. We've interviewed founders who do that already, but keep going, please.
- [46:23]
Ben
Yeah, yeah. And you know, none of what I just said is like totally on the revolutionary horizon. People are already doing this. It's just an enabling technology of hardware. I think the other is engagement. We talk about engagement is not outcomes but and we've had Lawrence Holt on before, we've talked about how when there is no engagement it's very hard to get the outcomes. And I think that video as an engaging methodology of learning or like interactive with glasses has all of this potential. I think the question is just like how do we move away from talking about the tech to actually talking about the use cases where this particular tech is advantageous to what one might normally do. And that's kind of the evolution of all of these things is at first you're kind of like, oh, the tech's a cool thing, but what can it really do? And this is where I think VR failed. VR closed you out from the real world and isolated you in the solo experience. That was suboptimal. But I think XR has a lot more potential in making the actual real world more engaging or unpackable from a learning standpoint. What's your view?
- [47:38]
Alex
Yeah, no, I mean I agree with a lot of that. I think we saw Meta sort of migrate from its horizon worlds. You remember a few years ago all of Meta's big tech talks were about being in this virtual world where it's all these people without legs and they are sitting in meetings together and they're, they've migrated that strategy enormously, they spent a lot of money and they migrated that strategy to these Ray Ban glasses where it's XR and it's much more integrated to the world. And I think that xr, you know, AR is definitely replacing VR. This might just be the sci fi lover in me. Like I never want to give up on this idea even if it keeps being launched and people get so excited and then it fails. When I, my first year in San Francisco I saw, you know, I got to visit the Google campus and see all the people trying their Google Glass on before it was even out yet. And I remember people were so excited about it and it was just yet another, I mean, you know, we've had been talking about, well, you and I
- [48:30]
Ben
previewed Omni too at the Google DeepMind meeting. That was Omni there. And I remember your takeaway was wow, a kid in their living room at 14 could create a feature length motion picture like this is coming. And it's so true.
- [48:47]
Alex
You can edit by hand. I mean I recommend anybody who is just trying to figure out what we're even talking about. Here Omni is a considered a world model. Like when you talk about walking through London and seeing the history, there was this classic edtech thing called Londinium about that where you'd see the Roman history of London. But the thing is with AI, you can walk around London, look at a building and say, I want to see what it might look like inside that building 100 years ago. And it will just make it it in video. You can literally look in it and look at it like, okay, it's 1926, here's what's going on. You could talk to people inside it. I mean, it is so crazy what is possible with this that I just can't drop it. Even though in the real world we just keep trying, trying and trying and it keeps not working. I think there's going to be a
- [49:32]
Ben
moment we're going to have to get our hands on some of these classes.
- [49:35]
Alex
Oh my God. I mean, there's going to be a moment where this stuff just becomes how we live. You know, I think your training example is a really key one because we are seeing companies that are doing really interest work with sort of hands on VR and AR training. And I think that might be one way.
- [49:51]
Ben
Simulations too are really taking off as a training modality. And I talked to somebody who they do certifications for veterinarian. Like you don't want somebody training on your pet. Doing simulations was too expensive. Now it's essentially free to build a simulation to simulate, you know, a dog vet appointment or horse vet appointment. And these are professions too that once you're certified, you make a really good living. Yeah.
- [50:19]
Alex
So there's a job training use case that I think is important. But I think the VR, AR XR world and all the advocates for it still have never quite found the use case, with the potential exception of Pokemon Go. That just gets people truly on board. And they say, and regular people, not just early adopters and sort of, you know, enthusiasts actually jump into it. But I think it may be coming because I think AI just changes. You can create worlds basically with words. I mean, you could be looking at something say, I wish this was made of feathers and it will look like it's made of feathers and then you can walk through it and it's made of feathers. I mean, it's like, it's things that you just like are truly out of Alice in Wonderland.
- [50:58]
Ben
Yeah. Some of the early use cases of AI analytics were actually in sports. And so I've seen some really interesting entrepreneurial stuff around AI glasses in sports. One is when you're watching a game like the stats and interpreting what's going on and immediately I transferred that to a classroom. If I were looking out at a classroom of learners signaling and that kind of stuff and data capture and so on. I think the other thing that simulations in sports is all about visualizing the game, winning shot or whatever it is. And I think we covered somebody in the UK that was running Simu School, which was basically a new teacher program where they would simulate being a new classroom teacher. I remember the dread I had as a new teacher walking in on my first day. If there's any way you can bring that down, I think there's real opportunities. But probably just given that we're in this anti tech or tech backlash moment, it's so clear that just being really concrete with the use case is going to be the key if this kind of tech gets adopted in education, especially because if it's abstract, I think it becomes a nice to have and not the need to have. Are there any other headlines that we need to cover before we wrap up? Because I feel like we could probably do a two hour show this week.
- [52:17]
Alex
There's so much happening. I mean, there's one we should cover, which is the $70 million round from UK based Multiverse. Multiverse is a company that's been around a number of years, but they've been growing. They had their first profitable quarter recently and they had just got a 70 million, which is not a small round, especially these days. Round with, with General Catalyst, Schroeder's Capital, Lightspeed Ventures. Multiverse is really an AI and data platform. They've done apprenticeships in the past. I don't know if they consider themselves an apprenticeship platform anymore as much as a sort of skill upskilling platform. But it's a workforce platform. It's. It's run by Ewan Blair, the son of former UK Prime Minister Tony Blair, which gives it obviously a big celebrity halo in some quarters, but that's a big round. What did you make of that?
- [53:06]
Ben
I mean, reskilling, upskilling. We didn't talk too much about OpenAI today, but you know, they've launched their new foundation and that foundation owns 20% of OpenAI. So there's going to be billions in philanthropy. In our EdTech Insiders chat, we also shared a substack about like, what does this new generation of philanthropists look like? You know, the railroad barons were one type. Will we see new universities and libraries, Altman U. Brockman College, you know, et cetera, or are we going to see new methods of philanthropy. But ultimately a big concern of the AI money coming on philanthropically is around resiliency, AI resiliency and, and people's ability to pivot in such a rapidly changing marketplace. Will we be able to get people into jobs? And even if we have some sort of universal basic income, people need productive work to be doing and there will be new jobs created that it's not clear will have the matching ability. European edtech still continues to go steadily strong. I will also say AI optimism there versus pessimism there. I think they've never gotten too high on it and they've never gotten too low because one, the regulatory environment is strong. Two, labor union element is strong. But there has just been consistent meaningful growth around edtech there that I think us would probably envy now. And I feel like, like my read on India right now is it's going through a little bit of a retrenchment too because there's just been a big fall off on user use and some tech backlash there too. So we may have to be looking more and more to Europe. I just saw the CEO of Magma Math was at like, I think it was like Kensington palace or something crazy like that for like a AI and education retreat. So clearly the government's there are taking AI readiness and AI tooling seriously.
- [55:14]
Alex
Yeah, I think they feel more responsible as governments for the future workforce than the US government tends to do. I mean, we do a lot of bureaucratic programs and there is money, the Department of Labor. But I just think the Europeans have thought about it as a sort of core function of the government to keep the workforce employed and skilled for a
- [55:33]
Ben
longer time for another session. Like, I'm hearing a lot of word on the street that Department of Ed is going to be part of of Department of Labor in the next legislative cycle. So I always thought that actually could be a good thing. There's something we can all agree on, which is that schools aren't really preparing kids for the jobs of the future today and we could do so much more. But obviously there's a lot of politics in D.C. and at the state level that you wonder how is all of this stuff going to play out anyways? Well, Alex, any final words before we head out on our normal outro?
- [56:10]
Alex
I mean, it's just exciting to be thinking about these trends that feel global, they feel historic. Like, I feel like we've expanded our scope from education technology to like the future of the world in a lot of different ways, partially because of AI and partially because all These governments and giant tech companies have been really moving into the education space, but, boy, it's just fun to think about. I hope that others in EdTech feel that even if it's a little bit of a winter, there's definitely some funding issues, there's definitely some backlash. You know, there's a lot of headwinds right now. It's still pretty amazing to feel like the things you're working on are related to literally the history of humanity. And I think we're really deep in that right now. That's the only thing I'd say before we go to our amazing guests.
- [56:51]
Ben
Yeah. And as that evolves, if it happens in EdTech, you'll hear about it here on EdTech Insiders. Great to have you back, Alex. Now we're going to kick it over to our two interviews. Enjoy.
- [57:02]
Alex
We have a really exciting guest today. We're speaking to angel chung. She's a PhD candidate in operations information and decisions at the Wharton School and a Penn AI fellow. And her research develops and deploys LLM systems, machine learning algorithms and optimization methods for data driven decision making in healthcare, education, of course, and social good initiatives. She works with practitioners and policymakers to translate research into real world solutions. And her research work came on our radar because she's been studying, among other things, the results of AI and LLM based tutoring. Angel Chung, welcome to EdTech Insiders.
- [57:42]
Angel Chung
Yeah. Hi everyone, I'm Angel. Thanks for the kind introduction. I'm happy to share more about work.
- [57:49]
Alex
Absolutely. So, you know, one of the things that's really interesting about your work is it focuses on LLM systems for real world decision making. And you're really interested in proactive behavior, which is where, you know, you go out of your way in your papers to talk about proactive behavior, where instead of changing how the LLM reacts to questions, making the LLM a little more proactive in actually making the conversation happening, reaching out, guiding students for tutoring more, being more adaptive. Tell us about what proactive learning looks like and proactive LLM behavior looks like.
- [58:24]
Angel Chung
Yeah, so I think with LLM, people already think, though LLM is kind of personalized in a reactive way. Right. We have LLM give us 24, seven AI tutors that can respond almost every question you ask. It's kind of personalized. Pretty big step from before already. But the key thing is they only respond what students ask. But the problem in education usually is students often don't know what they don't know, so they cannot always diagnose them by themselves about their own gaps or figure out what to focus on next. The proactive system will help us to personalize this learning by guiding through this process. It's not just react to the question and waiting for the students to ask because that's kind of limited, making the LLM to be more proactive. You can think of it like more like one on one human tutor. So if you have a human one on one tutorial that will like a human tutor will kind of see how you are doing and give you the suitable question to practice and decide what you should practice next. So our platform is leveraging a lot of signals from students interaction with our learning platform. And the LLM generated some of real time richer students interaction with the AI tutor to quantify those things to make it all and be able to proactively guide student learning. Specifically in our study, it's more like selecting the suitable difficulty level of the practice question that students should practice next and to sustain students engagement. So if you are a faster learner, this proactive learning will guide you faster, give you more difficult question quickly so you don't feel bored. But if you are a slower learner, you might need more time to practice. So these proactive learning will diagnosis that and give you from easy to difficult in a more slower way so you don't give up easily.
- [60:33]
David
Exactly.
- [60:33]
Alex
And you know, I mean it's a really interesting combination of this adaptive type of learning where you're choosing the difficulty of the suitability and the difficulty of the questions combined with this really modern LLM. Because the signals that the LLM is using to decide the complexity and difficulty of the question isn't just whether you got the question right or wrong, but it's actually the conversation itself. It's that organic conversation back and forth helps all becomes data that helps the LLM decide how to choose a suitable question. Tell us more about how that works.
- [61:06]
Angel Chung
Yeah, so I think actually to do this kind of a personalized learning sequence or practice question sequence, it's not new, it's been studied decades. Right. But the difficulty or challenges of previous approaches, you have very limited signal like you have to have some assessment question coming up the exam and do the grading. I'm sure all the educator knows how much work is that. And then usually you are just leverage a binary signal whether student do it correctly or wrong. But the learning is way more beyond that, it's way more complicated. And with LLM, the emergence of LLM really give us a chance to get more richer signal about that. Especially when the students are struggling a Lot of them actually, especially in our context in Asia, they really don't want to ask teachers directly. They don't want to show that they don't understand. So they are worried about that, look, dump or whatever things in front of others. So with llama AI tutor they can actually ask a lot of questions that they actually have. But then they are worried to be judged by the teachers or TA or the human tutor. But those questions are actually really reflects how they are doing in their learning process. So leveraging those signal capabilities can better capture how the students their knowledge state in a more accurate way compared to the previous approach that you only leverage
- [62:32]
Alex
the binary signal 100%. Yeah. Capturing the knowledge state. I love that phrase. And you know, and you did this experiment in a really interesting context. You mentioned in the Asian context, you did it in Taiwan with I think it's 770 high school students taking an online course about AI for Python learning. So it's learning Python for AI pretty complex subject and you saw really meaningful effects which is something you know, we've been wrestling in the whole edtech and AI space to find. You know, where does AI really push the needle? Where does it actually create learning gains? And you saw some very significant learning gains. Tell us about the results of the study and some of the context.
- [63:12]
Angel Chung
Yeah, so it's very exciting to see that actually the study is pretty simple in like the experimental design is very simple. We just separate the student into two kind of treatment and control group. Which treatment will have the personalized learning things I just talked about. Your practice question is we have back end algorithm to estimate your knowledge state in the real time and dynamically adjust your practice question sequence. And the control group is just. They follow the existing common practice approach like just from easy to difficult. It's actually a pretty strong baseline as that sequence is developed by the instructor like expert designed sequence. So it's kind of amazing to see the treatment group does improve 0.15 standard deviation higher in the final exam score than the control group. The final exam. I have to say we try to make a pretty fair evaluation of the study. So the final evaluation is in person and there's no digital device allowed. There's a partner, so. So we are making sure we are evaluating the actual learning there. And this is a collaboration with Taipei City government. So it's more like a policy things that give us students a chance to use AI for learning and get some certificate for their college application. So they do have some incentives to join the study. But at the same time I have to say in this context, a lot of them are I would say I come from Taiwan. So I can say that during the high school is actually the most stressful time in your education journey to get into the college. So signing up this kind of outside certification program to learn a new thing and be able to let them stay on the platform to keep engaged. It's pretty difficult task especially with this specific context. They are very, very busy stressful for students like about their school work already. So pretty excited to see how this personalization can sustain their engagement and achieve a higher score at the end.
- [65:25]
Alex
100% and let's double click on that sustained engagement as you look at the treatment condition and the control condition and tried to figure out what was the underlying cause of some of this major. The increase in learning you said is equivalent to about six to nine months of additional schooling based on the the World bank model. That's a lot of additional schooling. So seeing this kind of result you attribute to higher quality interactions with the chatbot and more time on task, more total attempts. So it's not about just doing more problems, but it's about spending more time and conversing with the chatbot in a higher quality way. Unpack that for us. And I'm sure you can explain much better than I can.
- [66:07]
Angel Chung
So yeah, first we find we are trying to see okay then what's going on why they it can really improve this much. So one thing the chatbot quality. So for each question students will ask the AI tutor about what they are struggling and what they want to solve. But a lot of as you can see many previous study has shown that my advisor they have a paper about people might just use the AI chatbot to get the answer right. So we kind of for each of the conversation we do ask LLM to judge whether students are actually using that to learn or they are just getting answer. And then we do find it seems like the personalization because you keep them in their zone of proximal development which like it's like the difficulty level is around there. It's not too challenges but challenging enough. So they are very more engaged to really solve the question. So they will leverage the AI tutor to actually solve Instead of like control you might just feel like I just need to complete this and then just simply get an answer from AI. And for the time I have to say both the treatment and control group they complete almost exactly same amount of number of questions. Treatment group actually doesn't complete more. As I say, they are very stressful, busy high school Students but they do like for each question they are actually they do the practice, they do spend a lot more effort because they probably feel like that question difficulty more suitable for them and more challenged or enough like suitable challenge for them to do it.
- [67:40]
David
Yeah.
- [67:40]
Alex
And that's always been the dream of adaptive systems is to be choosing questions that are pushing students right into their zone of proximal development, allowing them to actually engage, sort of get into a deeper flow state and feel like, oh, it's I'm not just doing a series of questions. I'm actually doing a question that's to going, getting that's hitting me right at the right level. And I want to, I want to dive in and learn and unpack it. I mean this is like a dream result is really exciting to see. And you know, you interpret this, which I think is fair as evidence that you know, personalization and this type of adaptivity, proactive adaptivity behavior in an LLM sustains higher engagement. It gets students to be spending more time on the questions, be asking better questions themselves and then getting major learning gains. So. So what do you want to do with this result? It's a really exciting result and one we're not seeing consistently yet.
- [68:32]
Angel Chung
Yeah. So I think this is exciting results. We are also pretty happy to see that. And this is actually pretty much to test our original hypothesis that whether we can leverage the ll the signal more richer signal improve beyond a lot of literature actually R is really based on those literature to do this kind of personalization. So adding this component from our results, we just show that this hypothesis does work. So maybe we can go further into like, okay, now we can probably think about more different ways to capture all these more richer signal by leveraging this kind of advanced technology to better do this kind of proactive personalization. And one thing I have to say because our study is focusing on student learning, but we don't have capacity. But I think in the future there's a huge potential by leveraging this to empower the teachers as well. Think about like if teachers can get the real time, like real time signal or like something some tools to help them to capture what each student's unique learning progress, unique learning trajectory, then the teacher can provide a better support. And these things is like student can just use the platform and this platform our algorithm will just directly have those measure for the teacher. So teacher doesn't have to do too much. They can just log into the system to see which student might be far behind or some people, oh, maybe this student are doing so well, maybe they need the more challenging question or people who actually need more support. Right. So I think that's a one thing. We do have a teacher dashboard when we deploy the study, but we didn't optimize that, which I think there's a huge potential on that front. And I have to say this system also have automatically question practice question generation pipeline. That's kind of an agent system to generate a practice question. So because in this kind of system think about if you want to do personalization, you actually need a lot of caution bank for you to do this personalize. So a lot of people might think, oh then that's so much work. You are adding teachers workflow. But that's also taken care by the LLM as well.
- [70:53]
Alex
That's right. And the added sustained engagement and the sort of higher quality conversations that the student is having as a result of getting questions tailored to their level is something that would also be very valuable to teachers because it allows the student thinking to be articulated in a much more thorough way. Right. If they're doing just easy to hard in a set sequence and they just go at it, you know that they're not engaging as much from your study. That higher level conversation could also be a really meaningful input to teachers to further personalize or further remediate or you know, do anything that's needed.
- [71:28]
Angel Chung
Right. And we believe that teacher definitely when you have a human interaction with student, you might feel you can provide additional human signal combined with all this quantitative measure to do do a better way than us do. Right. So I think adding this teacher component would further strengthen this system.
- [71:48]
Alex
It's a great point. So one more aspect of this study that jumped out and I think is also, you know, you've mentioned so many sort of key issues in AI research right now. The idea of adaptivity, of proactive versus reactive, of academic integrity and making sure it doesn't give answers of engagement, of conversation quality. There's just so many different aspects of this. But one that people are quite worried about in the AI world is equality, is, you know, whether AI is going to exacerbate inequalities between students or shrink them. And one of the things you found in this study is that the people who were started further behind in Python, people who are beginners or were from schools that were less high status schools, actually had higher gains. So it was starting, he was shrinking the gap. That's incredibly exciting. Tell us about it.
- [72:36]
Angel Chung
So I think the key thing is, yeah, those concerts are very valid and I think think people really should Be aware of that because as many study in EdTech already show, it's double edged sword, right? If you don't implement them well, it's really hard to say if you have unintentional harm or worsen the inequality. But the thing is I think in our study we show that more gain is coming from. It's not like a very, very rigorous causal story but it's like a very discreet pretty obviously shows that beginner gain more and the high school who's like a lower admission score does also have more gain. This can kind of tell us if you do it in a more suitable way, implement it well, it's actually helping the people. I would not say it's like compared to the more higher ranked school students or better students. It's not like we don't help them. It's like what we can help them is limited because there's ceiling effect, right? If they are already good, amount of room they can improve is limited. But for the people who actually fall behind or who need more help, if you implement well, these things can actually improve the access to a more personalized one on one tutor which was very expensive and not scalable before the llc. So if you do it well, it's actually another side of a story to improve the equity issue to bridge the gap. But if you don't tell the student to use it in a proper way then it could have unintentional harm. So I think there's a very careful balance that educators should be aware of to really implement this tool in a proper way. So it goes at direction we are hoping for instead of unintentional way.
- [74:31]
Alex
100%. And just as a sort of final thought for our listeners, many things about this study jumped out and I highly recommend people look it up. This study is incredibly, incredibly interesting. We will provide a link to it in the show notes. And I actually don't have the title right in front of me, but it's all about what is the title of the paper.
- [74:49]
Angel Chung
It's an effective Personalized AI tutor via LLM Guided Reinforcement Alert. Yes, yes.
- [74:56]
Alex
Effective Personalized tutoring via LLM Guided Reinforcement Learning. Yes. That is fantastic. One of the things that really jumped out just to circle back to the very beginning is this concept. I think as soon as you say it it seems very obvious. But I think a lot of people have not really absorbed the idea that LLMs off the shelf, the sort of commercial LLMs are designed to be answer machines. They're designed to be reactive. You ask it absolutely anything and it Amazes you by having a great answer. That's sort of been the story of the last few years. But this right. There's something very very different about an AI that is proactively choosing questions for you, putting you on a learning path, adjusting difficulty, creating practice questions on the fly and actually offering them to you. That's a very different model of AI learning. And I'd love to hear you just talk to our audience about how they should think about about the differences between proactive and reactive and where this might lead us if we lean into a more proactive version of AI learning tools.
- [75:57]
Angel Chung
I have to say that it's kind of probably a little bit unfair to say like a general LLM kind of. It's not very helpful in the learning because that really just building for the general purpose. But if you are actually leveraging to learn then really need those careful design that that you just mentioned the proactive way to One thing we do is we hope to inspire a lot more people especially educator or like more experts who have more pedagogical experience. Maybe people can think about more creative mechanism to do this proactive learning. In our study we do this kind of leverage this LM signal to do a proactive personalization about the practice question sequence to sustain students engagement let them stay in the zone of proximal development things. But there are a lot of different ways you can do this to sustain or encourage the productive struggle. I would say like we have another study actually it's like using the adversarial task to like adversarial. It means that the question is generated intentionally to if you as a student. I just copy and pass this question to the AI tutor the AI tutor is highly likely to give me the wrong answer so that I will get misled. So this kind of. If the student are experienced this kind of adversarial tactics they will build awareness about oh AI could be wrong. If I really want to learn I should be more careful and I will kind of cultivate my AI literacy to how to actually solve the question. So our other experiment does shows that that's actually improved students learning a lot more than the people who didn't have this learning like adversarial text training experience. So there are a lot of I would say just reactive to proactive. There are so many different diverse mechanism way. And I think the key point is just all the design should really ground into the pedagogical experience or like educator their knowledge for decades. And the evidence from the research to really figure out what is the angle that you can help and figure out where it helps, where it doesn't and where it helps like this angle, we find it help whether we can develop either further improve from this direction or kind of see if there are more creative mechanism to the proactive learning better.
- [78:29]
Ben
I love that.
- [78:30]
Alex
Yes, I think there's lots of opportunity for both creativity and integration of learning science, as you're saying. You know, we know a lot about pedagogy and learning and we know a lot about creative tutoring and creative ways to get students to think or to be surprised or to dig deeper. And we need to integrate those into our learning systems to really get the effects of tutoring and the effects of learning that we want with LLMs. This is, this paper was fascinating and I really appreciate you being on here with us at EdTech Insiders to help explain it. Everybody here should look it up again. Angel chung is a PhD candidate at the Wharton School in operations, information and decisions and thinking, among other things, about how educational LLMs can be optimized for learning outcomes. Thank you so much for being here with us on EdTech Insiders.
- [79:18]
Angel Chung
Thank you very much.
- [79:20]
Ben
Hello EdTech Insider listeners. We have an incredible guest. David is the founder and CEO of masterclass, the streaming platform that you all know and love. You've probably gifted it to your mom. You probably thought let's watch Hillary Clinton or let's learn basketball with Steph Curry. But today we have David really transforming what learning looks like for the entire learning cycle. So we're excited to dive in. Just a little bit of background About Masterclass. Over 200 instructors, from Serena Williams and Gordon Ramsay to Martin Scorsese and Bob Iger, masterclass has built an unmatched library of Emmy nominated, Oscar nominated and James Beard Award winning content. I mean, who wouldn't want Gordon Ramsay's cooking class to be in the front of their mind at all times? The beef Wellington. Wonderful. Welcome to EdTech Insiders. David.
- [80:15]
David
Thank you, Ben. It is wonderful to be here. I'm very excited.
- [80:20]
Ben
So Masterclass really broke through by making learning feel aspirational and making it feel not confined to a standard classroom, but really learning in the world. And this was long before generative AI. How do you think AI is changing what learners expect and experience from education and what they should expect and experience over the the next couple of years?
- [80:42]
David
What they should expect, I don't know if it's going to occur because I think people are slow to actually adopt this, but what they should expect is a level of personalization of learning that Actually makes it engaging. The AI allows you to teach the same things as you would learn in the classroom in a fraction of the time. So A, you should be able to learn things much faster than you were before, but two, in a way that you actually enjoy it more and apply it more because it's very much tied to exactly what you need to learn and when you need to learn it. So I think there's tons of fear in the education world of AI. I think that's thinking about it from a viewpoint of my job might be of kind gone if you flip it and say AI now enables me to do things I could never do before. I think AI will be one of the best things for education and for the educators that actually use and adapt to it. I think it's going to be wonderful for them and they will get paid more money.
- [81:55]
Ben
You've already been a pioneer on that front with engagement with aspirational talent. In the past, universities have been kind of a group that's ring fenced talent and expertise and knowledge. And you broke the mold with bringing all the expert voices into one's living room, into my laptop where I can go anywhere, anytime and learn anything from the world's greatest. But now AI can surface some of that information instantly. What do you think higher education institutions still uniquely offer in an AI world? And where's that intersection coming from? Masterclass with institutions?
- [82:35]
David
Yeah, I think for a long time schools in higher education have really bundled three different things. They bundled one, the instruction, two, the social, interpersonal bonds and relationships and then three, the signal. I think AI allows you to unbundle though. So let me share why. We have known for a long, long time that the most effective way to learn is one on one instruction. So then the question has to be asked, why are we sitting in classrooms, if we are lucky, in elementary school of 25 people and in colleges of hundreds of people, People. There's only one reason. It's simply cost.
- [83:22]
Ben
Efficiency of delivery.
- [83:23]
David
Yeah, efficiency of delivery is too expensive to give everybody a one on one instructor. So who wasn't able to get that? Only people that were, that were rich. So what does AI allow you to do? AI allows you to create one on one instruction that's almost as good as a person and it's a hundred dollars. So that means the education that elementary schools costing a 15,000 bucks a year, year per student, undergrad 80,000 for four years, you're now able to provide that instruction for about $100. And if schools aren't adapting how they teach and what their role is. They are going to be dinosaurs now. What's the opportunity for them? The opportunity for them is stuff that AI cannot do or is not good at. For example, what we know is that the bonds you make and what you learn from those bonds are very good for a host of reasons. Number one, AI isn't good at pushing back against somebody. And we know one effective way to learn is to have to discuss it and argue it and be pushed on. What you believe and think a person is much better at that. Two, we know one of the best indicators of my attendance in school and thus how well I'm going to do in school is the bond that a schoolteacher has not only with the kid but also with the parent. That I think we should start thinking of the teachers we hire that they have to have that as a skill. I'll be honest with you, I had a lot of instructors and teachers in K12 that that was not a skill of theirs. They were actually quite mean and in fact I didn't want to go to school because of them. And then three, you learn a lot from just interacting with people, on making friends, on losing friends, on all those things. But schools aren't designed for that type of learning education. If I was going to try to optimize for the social aspect of my education, I would organize intense things that I do with peers. That means trips, that means projects I have to build, things I have to make. And so I would push schools and higher ed to redesign the on campus experience to be very much of. I interact with both my peers and with instructors. And the instructors have to be trained as to coach more than to yell or scream. The third thing I would do is I think one of the things that schools have an advantage on, or this is like the third bucket here is on the signal. They have spent hundreds of years building their brands. Those are important things. And I think a lot of them have been cheapening it by selling things that aren't worth the other brands. I would stop doing that. So those are the things I would do if I was him.
- [86:18]
Ben
What you're talking about is actually a trend we've been following over the last couple of years, which is really the great unbundling of education. And the history of technology is actually filled with bundled unbundled. Bundled unbundled. And we often in tech talk about what's the new stack. And right now everyone's talking about what the new AI stack is and where does the LLM fit, where does the harness, what's the agentic level. And if we apply that same framing or logic to education, some of the indicators outside of what you're talking about are unbundled payments. So we have education savings accounts that are unbundling the payment for K12. We have a lot more federal dollars in tuition support for career pathways, apprenticeships, micro credentialing, et cetera, which to me seems like not only a political reality, but a response to the fact that learning is going to have to be an iterative ongoing thing for all adults. So against that unbundling and rebundling backdrop, I'd love to talk with you a little bit about your masterclass OpenAI and University of Chicago Booth School of Business Partnership. Tell us a little bit about it and tell us what that new stack looks like. In this formulation, it was an attempt
- [87:34]
David
to unbundle and then like re bundle it all together. Right. So first part was the instruction, so. Oh, sorry, I should start even higher than that. What we saw, and this is why I think it's a once in a lifetime chance for any edtech entrepreneur or anybody working in edtech, the need for adults to reskill is the highest it's been in a very long time, even more than it was I would argue then with the PC because the rate of change and the adoption of AI is so fast and schools are slow at that adoption. So this is a chance to unbundle. So what we found was that employers and we went, we, we went and talked to a bunch of them, everybody from a Bain consulting to a Dick sporting goods saying the skills people are learning in school are not the skills I. And we did more research and we polled these folks and they said, hey, if I was going to hire somebody how I'd value a 12 week AI intensive course? I'd value it the same or more than an mba. Now I have an MBA from Stanford. I had a great experience from it. That was 200,000 bucks. At least a 12 week AI intensive course. You should be able to do it a fraction of that cost and a fraction of that time. And when we talk to individuals, consumers like I can't spend that much money, but also I can't take two years out of the workforce. I'm going to be left behind. So we said what can we do to create a 12 week. We think it's basically the first AI native business school experience. We did it in collab with OpenAI and the University of Chicago Booth and we did a few things that I think think rock this world and also cause people to be mad at Us number one, instruction. We actually ran tests with our advisors at Harvard and Penn. We had a group of students watch one of our classes. We had a group of students learn via AI. We then had a third group of students use what we like, our learning science, which is combining AI and videos and engagement. That third group learned just as much of the first two groups, but in over 60% less time and they enjoyed it more. So that means in a 12 week course is essentially almost a 24 week course, or, you know, something like that. Right. So I can compact how much I'd learned to be much less and the engagement is higher. So that's number one. Number two, we said, the stuff you're going to learn is stuff you actually need to know. So we are teaching things by a combination of operators, everybody from Ray dalio to the CEO of 11 Labs to folks that have won a Nobel Prize. And we said, though this is going to be applied to things that's actually going to help you on your job outcomes. And then we said, but we know the social part is really important. So we're going to host on campus experience for everybody at the University of Chicago Booth and then we're also going to do things like live zooms and AI labs. At the end of it, you're going to walk away with a certificate from Masterclass and Booth and an option for one also from OpenAI. We were told by lots of people this was never going to work. We opened it up a few weeks ago. We have 500 slots in the first group. We've had over 20,000 applications been. I don't know what people are waiting for. The signal to the market is very clear. And so when schools are rejecting this and saying, hey, I want to be cautious with AI, I mean, fine, but we are going to leave you in the dust.
- [91:18]
Ben
Yeah, I mean, I know one applicant, Alex Sarlan, my co host on the podcast Amazing, all fingers crossed, waiting for, is it going to be a fat envelope or is it going to be a thin envelope?
- [91:29]
David
I mean, this was insane. Our acceptance rate is going to be harder to get into than hbs.
- [91:35]
Ben
Well, let's just break down the elements that you laid out at the beginning, from the backwards to the forwards. That tells me signal wise, there's real value. And I want to also just be really upfront with OpenAI has signal value, Booth has signal value and Masterclass has signal value. So combined these partnerships, I think open new opportunities for universities to combine their brands with other leading brands that are front edge to really signal.
- [92:04]
David
That was Very intentional. Right, right.
- [92:06]
Ben
That's super smart. I think on the social, what we find is people are connecting in lots of different ways and the massive infrastructure of the country club of the modern university is not necessary or essential. In fact, we've had Ben Nelson from Minerva on and 10 years ago they were doing global campuses where kids would meet up in different countries around the world and do place based learning. And it was transformational. And so I think there's a natural evolution, evolution here around dosage that is really compelling and interesting. And then, you know, on the academic model, I see a lot of parallels with what Alpha school is trying to do. This idea of there's this, you know, essentially learning roadmap and if we can ramp engagement and be efficient with the content delivery, students can get through more faster with higher outcomes. And I understand the controversy for the nice thing about this is it doesn't have to work for all learners. But we know that there's a significant number of learners and professionals for whom the existing, you know, two year, four year and the dosage of in person classing and all that stuff isn't working. So really interesting to see those things come together. Can we just talk about what's one thing that people would be surprised that actually stays the same in this model? What's one element that is actually consistent with past life learning model?
- [93:31]
David
One thing that is stays consistent is that to really learn takes effort. I think the dream of everyone is can I make education something that is as easy to do as watching my favorite junk tea show on Netflix? You're able to make everything as easy from the clicking to the watching to the product experience available on all screenshots. What you can't do though is to learn at the rate that you want to learn at. You have to be engaged. So it takes effort and we try to make it as enjoyable and as useful and as practical and I'm getting as much out of it. But I think one thing that stays the same, it's not like I can just like chill, like you're gonna have to be engaged in it and the more engaging it, the faster you're going to learn and the more you're going to to learn and the more you're going to get out of it. But it's not something that's just like, you know, everybody I think wants, including me, what, you know, out of the matrix when like I want to learn to fly a chopper and it's just like, you know, beamed into my head and now I know how to fly it. Like this requires work.
- [94:37]
Ben
Yeah, I mean cognitive friction is something that AI struggles with, AI native only. And what we're finding is that when you're testing discrete knowledge, it's really hard, hard to create that cognitive friction because the tools are universally surrounding you to just get to answer. And this is why I like the business school format for learning as maybe a future model, not just for business school, but for other, you know, case study, method projects and, and so on. It implies that kind of cognitive friction. And if you've got to work on it individually, of course you've got challenges. But then if you're working on it collaboratively, that even more ups the ante around learning performance and so on.
- [95:22]
David
And there's also stuff like you're able to do. So like what we found in our research was like the number one thing you want to avoid is somebody getting bored. That's actually the worst thing. If they get bored, they're never coming back. It's better for them to be frustrated and annoyed than to be bored. So like we work very hard to make sure that you're never bored. We also try not to make you frustrated or annoyed either. Right. But there's all different things that you're able to do to make it as engaging as possible. Your point on the business school model, what also works is, is that there's a clear prize in that, like, hey, if I learn this, this is going to help me get a promotion or a raise. And so that also pushes you.
- [96:00]
Ben
So where does it masterclass go from here? Is this a one off partnership? Is this the core strategy? Are you going with a winner take all kind of of were in it with Booth and OpenAI or do you imagine future partners in the world and today your primary audience is adult Learners, you know, 18 plus. Do you imagine eventually expanding that vertically into high school or middle school? Like just help us look into the crystal ball of the masterclass future.
- [96:28]
David
I'll be honest, I didn't think we were going to make things like this. If you asked me 10 years ago, even five years ago, maybe even three years ago ago. But we are being pulled there by our students and the impact we can have is that's why I started this, right? Like to build something that people can't take away from you. Right. That help people learn. So I think based on the impact we're having and demand we're seeing, we are going to massively expand in this, in this type of education. I don't think it's just business school because I think this is a way to learn. But the reason why, like my purpose behind this isn't for us to like win. Like I want like the world to win. And so we're very open to sharing it and talking, you know, to other schools and other groups that, that are trying to apply this. And we have been to help just share what we've learned because there's no way we, we are going to do everything for everyone. And so I think there is room in the market for a few people on the K12 side. We have no plans to go anywhere around that because that's just not our expertise. It's not. I also think, for example, one of the most important things in K12 education, especially maybe to age 6 or whatever, is actual play. Play is a very effective way to learn. That's not our expertise.
- [97:51]
Ben
Can I put in a plug for dual enrollment for high school? Because I do feel like there's juniors and seniors out there there that are massively bored. They're also looking at the cost of college and they're thinking, I'm starting my junior year, I've got two years of that, I've got four years of college, massive debt after that and I could start a business now. I could do real world relevant, meaning that's learning that's going to actually translate to the things I'm passionate about. Maybe I'm going to be a veterinarian, maybe I'm going to be an astronaut and everything in between. I think we've like undercut the imaginations of 15 and 16 year olds with like a low ceiling expectation of what high school exit is. Which, you know, in California all you got to do is pass an 8th grade math test and you can graduate high school. So I'd encourage you to lean in there. You know, Sal Khan was on our podcast about a year ago talking about exactly that. That there's opportunities for kids to get early advanced credit which not only creates cost savings but allows them to do the job exploration. So I'm going to put my thumb on the. While I have you put my thumb on the scale and advocate for that.
- [99:02]
David
I will look into it.
- [99:03]
Ben
I'll look into it in terms of how people can find out more about what Masterclass is doing with these programs. What's the best way for them to, you know, learn more and explore with you?
- [99:14]
David
Go to our website, masterclass.com. you're able to look at the executive program I just mentioned. We also have been doing some really interesting stuff on the certification side. It's with Masterclass and Masterclass is always from the best in the world, but that meant for a long time the best instructors in the world. We decided to expand end of last year to also the best firms in the world. So like if you're learning how to lead, it's with Masterclass and the Navy Seals and you actually get a cert from both of us on AI, it's with Masterclass and Nvidia. And so those programs are doing exceptionally well. It's more of like kind of a crash course in it. So if you can't commit as much as 12 weeks, this is the way to do it. We also host tons of live stuff, so join any of those, learn more, and if you have any questions, I'm@davidasterclass.com wonderful.
- [100:03]
Ben
Well David CEO Masterclass, what an incredible journey you've already had with Masterclass and now the next decade is really looking transformational. Can't wait to hear all about it here on EdTech Insiders. Thanks so much for joining us.
- [100:17]
Alex
Thanks man. Thanks for listening to this episode of EdTech Insiders. If you like the podcast, remember to rate it and share it with others in the EdTech community. For those who want even more EdTech Insider, subscribe to the free EdTech Insiders newsletter on substack. Tuck Advisors was founded by entrepreneurs who built and sold their own company companies frustrated by other M and A firms, they created the one they wished they could have hired but couldn't. Find one who understands what matters to founders and whose North Star KPI is the percentage of deals closed. If you're thinking of selling your edtech company or buying one, contact Tuck Advisors now.