Loading summary
Transcript1 lines
- [00:00]
A
If this episode makes you think, please let us know in the comments and support us by subscribing and leaving a review. Thank you. Today we are exploring some fascinating research from the University of Cambridge, published in May 2026, which looked at whether AI is ready to mark university essays. The core finding, and it won't surprise many of you, is that it's definitely not. The research team argues that while AI has some interesting potential uses in student assessment, relying on it for grading would ultimately lead to homogenized marks and actually underestimate brilliance. It really gets to the heart of what we mean by good assessment. So what did the researchers do? Well, a team of psychologists and AI Experts led by Dr. Deborah Ptolemy from Cambridge put some of the top generative AI models We're talking about the latest versions of Claude and Chat GPT as of April 2026 to the test, they fed these models over 750 undergraduate psychology essays from three different UK universities. These weren't just practice papers, these were actual coursework and exam answers submitted by students between 2022 and 2025. The human examiners had already marked these papers, following all the standard institutional processes. The goal was to see how well the AI could match those human awarded marks, especially when it came to degree classifications like a first 2, 1, 22 and so on. What they found was really telling. The AI models only match the human awarded degree classification about half the time, with a range of 35% to 65% accuracy across the different institutions. Now, that might sound okay, but when you dig into the details, it becomes clear that okay isn't good enough for something as crucial as a student's final grade. The big problem was that the AI routinely undervalued the work that human examiners had given top marks to, and conversely, it overvalued essays that were ranked among the lowest. It struggled significantly with the best and the worst submissions. And this is where it gets particularly insightful for us as educators. The report highlights that unlike human examiners, all the AI systems were oversensitive to linguistic features. What does that mean? They gave higher marks based on things like essay length, how varied the vocabulary was, and the complexity of the sentences. Now, on the surface, those might seem like good things, but the researchers stress that these features are often unrelated to academic standards. Think about that for a moment. An essay could be beautifully written, full of complex sentences and a wide vocabulary, but fundamentally lack deep critical thinking, original argument, or robust evidence synthesis. The AI, it seems, was rewarding on style over substance. This immediately brings to mind our core philosophy about AI in education. It's about money enhancement, not replacement. And we must always keep the Yuan Tao Lung human in the loop. This research makes it abundantly clear that when it comes to the complex, nuanced judgment required for assessing academic work, AI simply isn't there yet. It cannot replicate the human capacity for zoggy, judgment, imagination or wisdom, those things that machines just cannot do. The researchers described this as a central tendency bias. Basically, the AI assigned middling marks to almost everything. An essay that a human would mark as a solid 75 a first was on average scored several points lower by every AI system, and an essay marked 50 a low 2.2 was scored several points higher. The AI was most accurate in the upper 50s to low 60s range, right around the middle of the grade distribution. Why does this matter so much? Because, as Dr. Alexandru Marcochi, a co author, points out, human assessors judge each essay on its own argumentative and conceptual merits. While while AI marks are based on statistical predictions, he goes on to say that the AI is least accurate. Precisely where assessment decisions matter most, at the boundaries that distinguish firsts from upper seconds or passes from fails. Those critical boundary decisions which can genuinely impact a student's future are where the AI falls short. It can't distinguish genuinely exceptional or weak work with the precision and and insight that a human can. This has huge implications for how we think about assessment in the AI era. We've talked before about the ichind Frei Fran Parid 3 Ps of assessment, moving beyond just the product to look at the process and performance. And we've also discussed the idea of shoes cognitive stretch designing tasks that really demand application, unique context, perspective or judgment rather than just recall. This Cambridge study reinforces the absolute necessity of these approaches. If an AI can give a decent mark simply for linguistic complexity and length, then our assessment tasks aren't demanding enough depth, care and imagination from our students. We need to design learning that cannot be faked because it requires truly human thinking. The real value is not in what the machine produces, but in how the student responds, reflects and justifies their work. So where can AI fit in? The researchers aren't throwing the baby out with the bathwater, and neither should we. They suggest that AI could be valuable for certain aspects of student assessment. They mention things like error detection, consistency checks, and triaging feedback for students from for instance, if there's a large discrepancy between an AI's provisional mark and a human's mark, that could be a flag that the assignment needs a closer look from an assessor. This is classic outsource your doing, not your thinking. AI can handle some of those repetitive surface level checks, freeing up the educator to focus on the higher order thinking, the individual student engagement, and the nuanced judgment that makes assessment meaningful. Think about a year 10 English teacher. Instead of using an AI to grade essays, which we now know is problematic, perhaps a student could run their own draft through an AI for a basic grammar and spelling check before submission. Or, as an educator, you might use an AI to quickly scan a batch of essays for specific types of common errors, creating a preliminary list of things to discuss with the class rather than using it to assign a mark. The AI becomes a second pair of eyes, a support tool, not the primary decision maker. The study also looked at AI generated feedback. When asked to provide feedback, the AI churned out reflections that were three to eight times longer than those from human assessors. Interestingly, when the AI's responses were kept to a comparable word count, focus groups of staff and students often found it difficult to distinguish between human and AI feedback. However, once the identity of the writer was revealed, not everyone appreciated the AI generated insights. This brings us back to the crucial element of relationship in education. University staff and students involved in the study voiced a strong belief that being graded and receiving feedback from humans is fundamental to to the social contract between academics and students. Dr. Yale Benn, a collaborator on the project, noted that many students said they would feel cheated if AI marked their work, and staff warned that relying on AI risks weaken entrust motivation, professional judgment, and the human engagement at the heart of higher education. This is profound. It's not just about accuracy, it's about trust and the very nature of the learning relationship. Our students want to feel seen. They want their efforts to be acknowledged by a person who understands the context of their learning, their struggles, and their unique voice. This is where AI cannot wonder, it cannot care, and it cannot build a relationship. As educators, our role is to foster that human connection, to guide students through the one process and productive struggle of learning, and to provide feedback that is not just corrective but developmental and deeply empathetic. For school leaders, this research provides vital evidence when navigating the pressures to adopt AI for efficiency. Dr. Talmi from Cambridge specifically mentioned that universities are under huge pressure to reduce staff workload and improve efficiency, all while meeting rising student expectations, and some may start to lean on AI for assessment. This pressure is real in schools, too. The takeaway here isn't to dismiss AI entirely, but to be incredibly strategic and discerning. We need to lie start with why not how? What is the educational purpose we are trying to serve? If it's to deepen learning, free up teachers for more meaningful interactions, and and enhance human capabilities, then AI has a role. But if it's purely about automating grading to save time without considering the impact on student learning, trust, and the quality of assessment, then we're heading down a risky path. This study reinforces that we need to build from our strengths, anchoring AI to exist in friction points and teacher workflows, not novelty, but doing so with caution and a clear understanding of AI's limitations. We should be empowering teachers to experiment with AI as a supportive tool, perhaps for generating different prompt variations to scaffold student thinking, or for initial analysis of student work to spot trends, but never for the final definitive judgment. We must continuously remind ourselves that AI is helping us hold the complexity, so we have capacity for creativity, but it doesn't replace the unique, irreplaceable human elements of wonder, care, and judgment in education. This is bobo tee evolution, not revolution, and it's about carefully integrating tools to truly enhance rather than diminish the human experience of learning. That's all for today. Thanks for listening.