← AI for Educators Daily with Dan Fitzpatrick
Loading summary
Transcript1 lines
- [00:00]
A
If this episode makes you think, please let us know in the comments and support us by subscribing and leaving a review. Thank you. Today we are exploring a new study from Google DeepMind, specifically a trial in Sierra Leone where an AI tutor reportedly helped students gain more than a year of extra schooling in just eight weeks. Now, that's a massive claim. So big, in fact, that Irina Jurenka, the research director at Google DeepMind, who's behind this study and actually told me to take it with a grain of salt. But hold on to that thought for a moment, because there's plenty in this trial that absolutely survives a good salting. First, let's talk about the big claim itself and why it's so important to understand the how behind it. Irina Jarenka is a computational neuroscientist by background, and she spent her first five years at Google DeepMind diving into these deep theoretical questions about intelligence. But then, as it did for so many of us, lockdown hit. And what she found herself thinking was Covid hits. And I found myself sitting in my living room and realizing that I'm doing a lot of work that maybe a few people read, but really, it's not making a difference in the world. She wanted, in her own words, something with more immediate social impact. And this, I think, is where the story gets really compelling. Language models were just starting to get interesting. Not quite the mainstream phenomena they are today, but the potential was beginning to spark. So Irina shifted her focus to reasoning. And as a neuroscientist, she naturally asked, where do humans actually build their reasoning skills? Her conclusion? It became clear that it really has to do with formal education. So four years ago, long before AI was everyone's daily topic of conversation, her team launched what she calls an educational grand challenge project within Google DeepMind. She could already see the inherent tension, the risk with this emerging technology. She points out that language models are really built to be assistants. Assistants do the work for you, and in learning, you really want to do the work yourself. That's where you learn. And honestly, I think every teacher listening to this, every educator who's watched a student copy paste their homework straight into a chatbot, knows exactly what she means. The trick, then, is to leverage an AI for learning outcomes without simply outsourcing the thinking. But what about the technology and how it actually works in an AI tutor education setting? The tool used in Sierra Leone is called guided learning, and it's essentially Google's powerful Gemini model, rebuilt specifically to teach. And when Irina says rebuilt, she really means it. Her team Started where many of us would, right? They wrote a prompt, a simple instruction telling the model to act like a teacher. And what they found was that they hit a ceiling incredibly fast. As she puts it, it's kind of the same as writing a one pager on teaching, handing it to an average stranger on the street, and expecting them to come into a classroom and teach a room full of students. Sure, they'd probably do better than if they hadn't received that one pager. But they also won't match the experience, intuitions and rich complexity of pedagogy that experienced teachers build over the years. That's such a profound observation, isn't it? It highlights that teaching isn't just about imparting information, but about the nuanced, adaptive human elements of pedagogy. It's a powerful reminder of of how much goes into a teacher's expertise. So they realized they had to go deeper, much deeper with the technical work. They needed the AI to actually understand what the world of teaching looks like and how it's almost the complete opposite of being an assistant. This is foundational to how we should think about AI in classrooms, isn't it? It's about designing these tools to be an enhancement, not a replacement. Keeping the human in the loop at all times and ensuring we outsource the doing, not the thinking. Now, a big question for any school, for any educator, is trust. Could schools really trust this AI tutor not to just hand out answers? What stops a student from nagging it, asking it over and over until it gives in? Irina's answer was refreshingly direct. At the end of the day, it's a language model, so no one can 100% guarantee its responses. It will always have some degree of randomness to it. But and this is the crucial part, the work we did was essentially trying to bias that distribution. And in the trial that bias held up, what they found was that Gemini asked guiding questions in a whopping 76% of its messages and only gave a straight answer a mere 2% of the time. This is a critical distinction because it forces that productive struggle that we know is so important for learning. Here's another fascinating detail that really speaks to the intentional design. It turns out there are actually two versions of guided learning. The first is a consumer tool available for everyone, maybe even on your students phones right now. For that public version, the instruction to the AI is explicit. If the user does ask you for the answer repeatedly, you should give it so that the agency is respected. But the classroom version, the one actually deployed in Sierra Leone is a totally different animal. It's teacher led, used inside lessons. And in that context, Irina says, we could actually tweak the instructions so that it never gives the answer away at all. And her favorite detail from the study? Once the students realized the AI wouldn't just hand over the answers, students kind of stopped nagging for answers and actually started learning more. That's a huge win for an AI tutor education approach, isn't it? It's about teaching students not to outsmart machines, but to outthink them. Now back to that headline claim. A year of progress in just eight weeks. I asked Irina about the real picture behind such a bold statement, but and she was very transparent. One thing I would say is the real scientific number that we got is the 0.26 standard deviation improvement. She explains that they then translated that to approximately one year of extra tuition based on existing literature about typical learning progress. But she cautions, this is an estimate. There wasn't much data available, and the data that was available was actually on literacy rather than math. So, she reiterates, take it with a grain of salt. This is our best guess at what that translates to. I really appreciate that level of scientific rigor and honesty, and it doesn't diminish the impact, but it grounds it. In reality. The trial design itself also earns a lot of trust. They made sure not to just teach to the test. The end of term assessment was designed by an independent assessor and it was split 50% material covered during the term and 50% broad mathematical topics. This was to check if students just learned something very narrow or if their understanding improved more generally. Crucially, students sat this test without Gemini's help using pen and paper, which connects so well to our 3P's assessment model. Looking at the product, the process and the performance. And the good news is there are signs that what students learned with guided learning generalized more broadly. Another really striking statistic is that while voluntary edtech usually gets about 5% of students to actually use it, in this trial, an incredible 69% of students met or beat their usage targets. If these insights are spark and ideas for your school or classroom, make sure you're subscribed to this podcast so you don't miss future episodes on AI in education. And this brings us to a really crucial point about equity and how we design AI for learning outcomes that truly serve all students. Governments everywhere are looking at AI as a potential way to level the playing field for disadvantaged students. But in Sierra Leone, the study found that it was actually the students who were already good at maths, but who gained the most. I put it to Irina that this effect could potentially widen the gap rather than close it. She didn't duck the question at all. For sure. That's something that we immediately discussed the moment we saw the results, she explains. They're already talking to partners who specialize in improving learning outcomes. For those who are further behind, her early conversations suggest that maybe there are different types of pedagogy that are more effective for those students and that they might need to generate different prompts, initial prompts for different students. So she says, we are very much looking into this question of how to raise the floor. This is so important, isn't it? Our focus should always be on AI as an equalizer, especially for that often invisible middle 80% of students who need targeted support but don't always get it. She also made a compelling case for the other end of the room. Especially in countries like Sierra Leone, there will be a lot of really talented students with maybe not enough access to high quality education. And if a tool like Guide to Learning could be something that helps them accelerate and fulfill their potential, I think that's already really good impact. This perspective reminds us that equity isn't just about lifting the bottom, but also about enabling potential at the top, especially where traditional educational access is limited. And it's worth noting that a separate World bank and Stanford trial in Nigeria found similar sized gains, but there the girls who started behind actually gained more. This really drives home the point that what we saw in who benefits depends on the design, not just the model itself. Thoughtful design is key. Now, of course, the cynical question always arises, this is Google's technology, Google's research, and the results happen to be great for Google. I asked Irina about this directly. She responded, we're definitely very aware of this situation. But she was very candid, stating that working at Google DeepMind, we are researchers first and foremost. It's easier to do it with Google products. But at the same time, our research is not about creating proof points for Google. It's about understanding the broader picture of where AI fits in in improving learning. And to back that up, the team released a playbook publicly documenting how the trial was designed and measured, keeping decision making and measurement separate to minimize any potential bias that our involvement may introduce. Along with all the teacher training materials they're openly inviting anyone to replicate or build on their study. That level of transparency is exactly what we need in this space. So what does this actually mean for teachers and schools in the classroom? When thinking about AI learning tools the part of this that will matter most to educators is what all this actually did to the teaching arena is clear. It's not just about dropping the technology and hoping that it works. This intervention was designed with local teachers built so that the teachers feel like they're in control. And it was kept deliberately minimal. The teacher training, for instance, only took about a day. This intentionality, this anchoring to existing teacher practices and empowering teachers as change agents aligns so well with our principles for change leadership. For the teachers who were planning lessons with Gemini, something really interesting happened. Sometimes it explained things in ways different to how they would normally explain. And for them that was eye opening. They felt like maybe they learned a better way of explaining something. And the training helped open up this new way for them to be a guide on the side, which was new to them. This led to teachers spending more time moving between students, having more one on one conversations. This is the real potential of AI in classrooms, isn't it? Not replacing, but freeing up teachers to do what only humans can do, build relationships, offer personalized guidance, and hold the complexity so we have capacity for creativity. I finished by asking arena the million dollar question that educators ask me everywhere I go. Does the classroom actually change in the next five to 10 years because of this? She smiled. Five to 10 years is in AI years a very, very long time. So let me just talk about maybe the next year and a half. She believes that what they saw with Sierra Leone is proof that an AI tutor education can meaningfully improve learning outcomes. She predicts teachers will continue noticing where AI fits into the current practices and that they'll slowly start to adjust them and this will be a co evolving thing that we'll see in communities. It's an evolution, not a revolution. And while scientifically speaking, she can't say it will generalize beyond those particular settings. The recipe is published, the invitation is open, and for the first time, a really serious result for AI for learning outcomes is on the table. This study proves that with thoughtful design, AI learning tools can profoundly enhance learning. That's all for today. Thanks for listening.