← New Books Network
New Books Network
Dennis Sherwood, Missing the Mark: Why So Many School Exam Grades are Wrong – and How to Get Results We Can Trust" (Canbury Press, 2022)
00:58:45
Loading summary
Transcript41 lines
- [00:01]
Home Depot Advertiser
It's time to bring on the blooms at the Home Depot with Spring Garden Deals. Find savings on hanging baskets and flowers to brighten your backyard or any space that needs instant color. Then get everything you need to plant and protect them with low prices guaranteed on soil and mulch. Dig into Spring garden deals for four days at the Home Depot now through May 10th. Exquisite supply. See homedepot.com pricematch for details.
- [00:31]
Prime Video Advertiser
This episode is brought to you by Prime Obsession is in session. And this summer, Prime Originals have everything you want. Steamy romances, irresistible love stories and the book to screen favorites you've already read twice off campus L every year after the Love Hypothesis, Sterling Point and more, slow burns, second chances, chemistry you can feel through the screen. Your next obsession is waiting. Watch only on Prime Girl.
- [01:01]
Ross Store Advertiser
Winter is so last season and now spring's got you looking at pictures of tank tops with hungry eyes. Your algorithm is feeding you cutoffs. You're thirsty for the sun on your shoulders that perfect hang on the patio sundress, those sandals you can wear all day and all night. And you've had enough of shopping from your couch. Done. Hoping it looks anything like the picture when you tear open that envelope. It's time for a little in person spring treat. It's time for a trip to Ross. Work your magic.
- [01:32]
Dennis Sherwood
Welcome to the New Books Network
- [01:36]
John (Interviewer)
and welcome to another podcast from the New Books Network and the author this week is Dennis Sherwood. And I've talked to Dennis a couple of times before and I've read his book and it's of great interest to me because I was myself involved in the education profession and indeed, indeed for a while examining as well. It's called Missing the Mark and the full title, Missing the Mark is why so many, so many school exam grades are wrong and how to get the results we trust. It's about the British exam system. Dennis is well multiply qualified to talk about these things. You're not a stranger to exams yourself, Dennis. You have a PhD from the University of California, San Diego, a physics degree from Cambridge University, another degree from Yale University, and you're an independent management consultant with Silver Bullet Machine, specialising creativity and innovation. So Dennis, welcome to the New Books Network.
- [02:32]
Dennis Sherwood
Well John, it's a pleasure to be with you and indeed to talk about a subject which is of vital importance to certainly every kid in England because their destinies depend on the grades they get in exams. Perhaps more so in England than in many other places around the world.
- [02:52]
John (Interviewer)
That is. That's a great place to start Dennis, because I, I Was a teacher and I taught in the sixth form and years. The letter years were heavily examined in the uk. Students in the UK do exams at the end of when they're 16, the end they do quite significant exams at 18. And yeah, results day was terrifying for students. I mean, the hands shaking, parents standing outside anxiously much depended on those results. Goodness me, I don't like to think about those days again, but. And Dennis, let's start. Your book is beautifully written as a kind of narrative. I was reviewing it again this morning, having read it a little while ago, and it came out in 2022 after an awful lot of things that happened which we could get into relating to exams in the uk, not least of which was the Pandemic, but the story starts. I think we could give a little bit of background to our non British audience about exams in the UK and how they work. That's where the book starts. But the story for you, in a sense, of why the books happened starts with you being commissioned by or having some sort of contact with ofqual, which in the UK is the government regulator of exams quality. The Office of Exam Quality, I think it stands for, and it's a. It's a semi independent quango in the uk, which monitors the accuracy and the reliability, two words we're going to come to. Of exams in the uk. So, Dennis, what led you to eventually to writing this book and how did you get involved with ofqual?
- [04:20]
Dennis Sherwood
Yeah, thank you. I wrote the book. Actually, I wrote it during the pandemic because obviously during the pandemic, you know, we were all using our time a bit differently, but in fact there had been some significant problems with exams in England. I'll come to that in a minute. During the 2010s, which I had studied, when I was commissioned as a consultant by the English examination exam regulator, Ofqual. But then obviously, in 2020 and 2021, when Covid struck in England, formal exams were abandoned and they had to find another way of giving students their grades so that they could progress to other things. That was a story in its own right. So there were those two stories going on. What was wrong with real exams and what was wrong when the real exams were thrown away. And I put all those together in, in 2022 really to set the scene for what is happening around now. Post Covid back to real exams with the intention of maybe exerting a bit of influence as to how to make things a bit better. But if I just backtrack a little bit to set the scene, if I may, in England. When kids go to school, they have major progression points. Age 16, at age 18, age 18, they leave school. Some go to employment, some go to colleges, some go to universities. And age 16, and there's another break between up to 16 education and 16 to 18, which is a bit of the hangover from a previous era when a lot of kids Left school at 16, they went into employment at that time, so they needed some kind of qualification certificate. So what has happened in the UK is there's really big exams at age 16 and really big exams at age 18. And since 2010 those exams have been the full story. At age 16, a 16 year old student will take what they call GCSE exams, General Certificate of Education, basically, General Certificate of Secondary Education, gcse. And those exams occur from mid May to mid June. In fact, this year's exams have already started in the UK as we speak, and the entire destiny of that young person from then on is determined by the grades on those days. Because in 2010, the government revised the exam system or the assessment system, and then any form of kind of teacher assessment or progress over the ages, say 11 to 16, was ignored. And everything depends on the exams at age 16. Likewise, everything depends on the exams at age 18. So those exams at 16 and 18, GCSE 16, A level, advanced level at 18, are critically important to the young person. The grades they get determine their destinies and that's that anything they might have done at school is just not taken into account. It all depends on that examination. Now, those exams are graded. So for GCSE A16 there are 10 grades on a numeric scale. So grade nine is the highest of the grades and at A level it's graded on an alpha scale. So A and A are the top grade and there are six A level grades and 10 GCSE grades. Now, those grades are shown on your certificate and those grades are high stakes. If you want to go to a university, the university will say, you can come and join us if you get AAB in these subjects at A level. And to do many A level courses, you have to get grades 6, 7 or 8 or whatever it might be at GCSE. So those grades are fundamentally important to students destinies. And if you get too low A grade, doors can be closed. So it's real cliff edge stuff. Now the whole basis of that is that when you sit the exam, the grade you get is right and trustworthy and everyone believes that. But the truth is much, much muddier than that. And I was commissioned by, Ofcall in 2013 to do some studies of the exam system. And I used a method known as systems thinking to draw what are called causal loop diagrams, really to get into what the exam process is doing. And as a result of that, I came across a problem. And the problem was one of measurement. How do you know that an exam grade is right? Your student, you go, as you know, John, from your students, on results day, they open an envelope or get an email, whatever it is these days, and it says GCS English, grade eight. And you think, fantastic, because that's a good grade. Other kids, as you know, say GCSE English grade three. Now, grade three is unofficially a fail. And if you get a grade three in GCSE English, you have to resit the exam and you lose a year of development and many doors are closed. So the difference between a grade three and a grade four for GCSE English language is really, really important. So the kid opens the envelope and says, GCSE Grade 3. How do you know it's right? You don't. There's no comparator. You have to trust the grade that you see. So everything is based on a belief that the grades are right. And ofcorl, the regulator is there to ensure that. Well, actually, they had no evidence that the grades was right. And indeed, when you dig around, they had plenty of evidence that the grades weren't, which is what the book is about. It's presenting the evidence that in fact, one grade in four, 25% of the grades are, are wrong. And people don't know that. So that's why I wrote the book. It's a bit of a long winded answer there, but I left a lot of clues to other things to think about and to talk about.
- [10:51]
John (Interviewer)
Yes, I think that's a brilliant introduction. Thank you for that, Dennis. And it is a narrative story. It does take us to both through the story of your discovery and your analysis. But for me, it was also deeply convincing. And I want, later on we might talk about where this has taken the people that read the book and what's happening to the exam system in this country. But so off quad, off, off qual commission you, Dennis, as it were to be involved. Was there a moment when a light bulb went off and you thought, well, it isn't. Systems are fine, but something's wrong here? And how did, how did ofqual and your, the people that had asked you to do this exercise react when you start saying things like, well, I think there's a, there's a concern here that you should you should be concerned as much as I am with the inaccuracy of these results.
- [11:40]
Dennis Sherwood
Yeah, yeah. Yes. There was a light bulb moment, and the light bulb moment was when I asked the question, how do you know how reliable your exams are? And the answer I got was the fact. And it was a fact at that time that only about 0.6 of 1%, 0.6% of grades are changed when you appeal. Now, that opens up the appeals process. If you're unhappy with the grade as a student, this is back in 2013, you, your parent, the school, could appeal to the people that do the exams and say, I'm not happy with this result. Could you please check it? And that would then give a remark by a subject, senior examiner. So there'd be, if you like, a professional second opinion. And that second opinion would either confirm the original grade or change it. Now, when I ask the question, how do you know the grade is right? Answer you don't. Well, one way to find out is to have a remark by someone that you trust, an expert second opinion, just as you get expert second opinions in, say, medicine. Now, in 2013, you could do that. And the fact is that when that was done, across the 6 million or so grades that were awarded, only about 0.6% of grades were changed, which meant that those 0.6% of grades certainly were wrong in the first instance. Now, when I had this conversation with ofqual, people were sitting around saying, well, if it's only, you know, less than 1% of grades are changed, everything must be all right, mustn't it? It was at that point. I then said, hang on a minute. Only a very few grades are challenged. About 3% of grades are challenged. So 90% of grades are not challenged, but 3% are. Now, of that 3% of grades that are challenged, yes, 0.6 of the total population was changed, but actually it's not 0.6 of the total population. It's about 0.6 of 3%. Now, 0.6%, you know, of the 3% is about 18 to 20% of the challenge grades are changed. So of the sample of challenge grades, about one in five was changed. And the 97% of grades you have no information on at all. So if you assume that only 0.3% of the total population had changed, you are, in fact assuming that every single grade in the 97% of unchallenged grades are right and that if you looked in there, you wouldn't find any mistakes at all. And I said, that's just unlikely. And in fact, it could be that the challenges, the appeals, are a representative sample of the whole, in which case it's not 0.3% of grades that are wrong, it's about 20% grading 5 that might be wrong, yet don't know. You've never looked. And that created silence because people looked at one another and said, oh, oh, oh, we hadn't thought of that. And they hadn't. So my assignment then finished. But chief regulator at that time, a lady called Glenys Stacey, now in England, Dame Glenys Stacey, had heard and listened to what I said. So that was all in 2013. And later I discovered that OFCOL then did a major research project over the next two years where they looked into the 97%. They did a study across 14 subjects where they remarked every damn script to see what. What would happen if, in fact, they were all appealed. So they weren't looking at the sample of 3%. They looked at the whole thing, and it took them two years, and they had the results of that by November 2015. And when those results were published, which was a bit later, it turned out that my estimate that 20% of grades were wrong was itself incorrect. The answer came out as incorrect, 25%. So ofQuilt's research, where they remarked entire cohorts discovered that one grade in four was wrong. And that is a problem, really is
- [16:20]
John (Interviewer)
a problem, isn't it? Because, goodness me. So basically you asked a question and there was a sort of shrug of the shoulders that said something like, well, if people don't challenge it, they must be happy with the result. And if they're happy with the result, it can't be wrong. And that's where we've been up till now. And then they look at it themselves and they. And they see that it's actually slightly worse than your extrapolation from what could have been.
- [16:42]
Dennis Sherwood
Yeah.
- [16:43]
John (Interviewer)
And just to emphasize to our audience what this means, in the uk, you can't become a nurse without a grade in English or maths above A four, at least. There's all sorts of professions you can't go into. You can't go and do A levels, the second stage of grades you. Unless your GCSes are up to it. And so not only are jobs closing down, but I've been there with students when they've had their place. Very odd, eccentric way. We apply to university. In this country, where you apply to a university, before you know your results, you've got all set to go. You may have even organized your accommodation at university, visited the place and be dead excited to go there. And on the day you pull open that brown envelope, you realize you're not going because it says B and not A. And that one difference is all. The difference is quite extraordinary how if that number are wrong now I'm using the word accurate occasionally, and I'm using the word reliable. There's a difference, isn't there, Dennis? I should be careful what I say.
- [17:41]
Dennis Sherwood
Yeah
- [17:43]
Red Bull Advertiser
Ready to soundtrack your summer with Red Bull Summer All Day Play, you choose a playlist that fits your summer vibe the best. Are you a festival fanatic, a deep end dj, a road dog, or a trail mixer? Just add a song to your chosen playlist and put your summer on track. Red Bull Summer All Day Play Red Bull gives you wings. Visit red bull.com brightsummarahead to learn more. See you this summer
- [18:09]
Plan B Advertiser
with Plan B Emergency Contraception we're in control of our future. It's backup birth control you take after unprotected sex that helps prevent pregnancy before it starts. It works by temporarily delaying ovulation and it won't impact your future fertility. Plan B is available in all 50 US states at all major retailers near you, with no ID, prescription, or age requirement needed. Together we've got this. Follow Plan B on Insta at Plan B One step to learn more Use as directed
- [18:40]
New Books Network Announcer
hey NBN listeners. We're running our 2026 New Books Network Audience Survey, and we'd love just a few minutes of your time. NBN has been bringing you in depth conversations with authors and scholars for over 15 years. We haven't done a comprehensive audience survey since 2022, and a lot has changed since then. It's time to hear from you again. Here's why we're asking. We want to understand who's listening, what subjects and podcasts you love most, and where you'd like to see us grow. Your responses help us tell NBN's story to the publishers, libraries, and institutions we partner with. When we can show that our listeners are serious readers, lifelong learners, and heavy library users, it opens doors to new partnerships, better resources, and ultimately a stronger NBN for everyone. And one more thing, if you leave your email address at the end of the survey, you'll be entered to win a $100 gift card to bookshop.org, a chance to stock up on books while supporting independent bookstores at the same time. The survey takes just five minutes, your answers are confidential and your email will never be shared. Head to newbooksnetwork.com to take the survey today. We really appreciate your Support. Now go take the survey.
- [19:53]
Dennis Sherwood
Yeah, this is. Let me just mention this concept of a wrong grade. I've got a grade three on my GCSE certificate. And if, in fact, we go back to the days before 2016 when you could get a remark by a senior examiner, John, if you were a senior examiner, you would have remarked my script, you would have either confirmed the three or said, oh, there was a problem. The grade should have been a four. So let's assume that my grade is changed to a grade four. I, as the student, would therefore infer that the original grade three was wrong and the grade four, your senior examiner's grade, is right. So grades can be wrong if they get changed. Now, why may they be wrong? There are actually three reasons why a grade might be wrong. Reason number one is what's called an administrative error, and that's something that has gone wrong in the admin system. For example, the numbers might have been added up incorrectly. When this was done by hand, that was possible. Now, that's much less likely. Other things can go wrong, like a page might not be scanned in, so it might not get marked in the first place. That's an admin error. And these are rare, but they do happen. And that can cause the mark on which the grade is supposed to be wrong. So that's category one, small and manageable because you can improve processes. Now, there's a second category of error, which is called, in England, a marking error. Now, a marking error is this. We all know that if you're writing an essay in history, for example, or writing a description in a biology exam about how the lungs work or whatever, two different expert examiners could give a slightly different mark to my answer because they're exercising academic judgment a little bit differently. Perhaps the facts are right, but because my English is perhaps better or worse or whatever, one examiner might give me one or two marks higher than another examiner. We all know that different experts can have slightly different judgment. Not way adrift, but slightly. So let's suppose that I'm writing an essay on Shakespeare or whatever, and you as an English Examiner Give me 14 out of 20, whereas, you know, someone else might have given me 15 out of 20. That's fine. And indeed, that's built into the system because every question is associated with both a mark scheme, which tells the examiners how to allocate marks for various features of an answer. And also there is what is called tolerance. And tolerance is a measure by which the mark given to a question can be different from a senior examiner's Mark, the senior examiner's mark is called definitive. It's authoritative, it's right. And if a mark of 13 out of 15 is within a mark or two of the definitive mark, that's okay, it's within tolerance. But if it's three or four marks away, that's out of tolerance. And the exam board steps in and fixes that. So this idea of academic legitimate differences in opinion is built in. One or two marks, maybe. But think about the consequences of that. I add up all the marks for an exam, it comes to 63. From one examiner, it comes to 65 for another examiner. That's totally possible, it's totally legitimate. It's just a difference in academic opinion. But when the scripts have all been marked and the marks are assigned OFQL, then draw the grade boundaries. Now, if grade 6 for GCSEnglish is all marks from, say, 60 to 66, then it doesn't matter whether it's 64, 65 or whatever, it's still a grade B. But if the grade boundary is between those two marks, one mark 65, if the grade boundary is at 64, is a grade A and a mark of 63 is at grade B. So 63 and 65, two legitimate marks, end up with two different grades. Which grade is right? They can't both be right yet both could appear on my certificate. So we come back to the senior examiner's definitive grade. If the definitive grade is mark is 65, corresponding to definitive grade A, then grade B is non definitive. It must be wrong. So the wrong grades are different from a senior examiner's definitive grades, which can arise either because of a difference in academic opinion or because someone failed the mark scheme. Now, if someone didn't comply with the marked scheme as an examiner, that can get corrected. But having that difference in legitimate academic opinion is intrinsic to all exams which are not multiple choice and which are marked by human beings that will always be there. So this wobbliness in grades is intrinsic. So that one grade in four is wrong is attributable solely to differences in academic opinion across the various subjects. And that's a real problem. Because the underlying marks are legitimate, there's nothing wrong with the marks. It's just that the grading system doesn't take account of the fact that different examiners legitimately can give different marks. And that flaw in logic penalizes young kids because young kids end up with a grade three for GCS English when the senior examiner would have given them grade four happened to 37,000 young people in summer 2025. I'm not making that number up. 37,000 young people were given a grade four for GCSE English language in summer 2025 in England, when grade three, rather, they merited at grade four. So it's a bit ticky there, but it's fundamentally important.
- [26:23]
John (Interviewer)
Absolutely. I mean, the picture that paints is one of, as you say, a sort of inherent problem with examining subjective subjects, as it were, Shakespeare or history or something, where you're going to say, well, now I have sat in an exam room, I've been sat around examiners, and my subject was politics. And one examiner will say, well, I think this is a very original answer, very interesting. And I will say, oh, no, but I don't think they've complied with. And there would be a little debate. And the senior examiner would have to say, well, I think on this occasion. And they would try to impress upon the other examiners what the definitive, as it were, view. But it was still a view of this. This. It was. And so it depended on ordinary human beings and their frailties. So the problem, in a sense, is if you say, I know this, I'm going to read an essay and you're going to think, it's great, Dennis. And I'm going to think it's not so great. The problem is the cliff edge, isn't it? It's the cliff edge of the grade boundary which says a B and an A, and the consequences that follow from that. That really is the problem.
- [27:24]
Dennis Sherwood
It is all about the cliff edge. And this has been known for Docker's years. In fact, I believe that grades were invented to solve that problem, that people recognize that giving someone 63 as opposed to 65 was fundamentally problematic. So I understand that actually at Yale University in the United states, in about 1785, they came up with a great idea. They said, let's give them grades, not marks, so that everyone between, say, 60 and 70 is a superior grade or whatever word they used. So that meant that they didn't have a problem except close to grade boundaries, where it might be that someone straddles. And they solved that problem too, because what they did was they marked all the scripts. The ones that were safely away for grade boundaries were safe as this grade or that grade. But then the academics would get together as a group, they would review the scripts that were on the grade edges, and they would take time and care to judge, is it this side of the grade boundary or that side of the grade boundary? So grading works even with edges. If there is scrutiny of the scripts of the Grade boundaries. And that's what happens and has happened for donkey's years, in particular at many universities. But when you've got 800,000 kids in England doing GCSE English, there aren't enough people or time to do that. And you've got narrow grades and nine grade boundaries. So the probability of straddling a grade boundary is much, much higher than if you had, say, three grades and two grade boundaries. So by having lots of grades and lots of grade boundaries, you're making the problem worse. So that cliff edge problem operationally is very real and affects overall one grade in every four, but with actually a pretty wide variety depending on the subject. So maths, the likelihood of having this straddling problem is much less. About 96 of maths grades are reliable in the sense we've been talking. But if you go down to history, the problem is much worse and only rather more than half of grades are reliable in history and nearly half of all history grades in England are wrong. Half too high, half too low.
- [30:01]
John (Interviewer)
It's the. It seems to me, listening to that, Dennis, is. It is part of the problem is the. Is what we imagine the grade to be. It's the sort of imaginary idea. So if it is accurate, and as you say, the historic assumption was, oh dear, I got a B, I must have missed out. That was looked at carefully. The system was very, you know, it judged me to be a B. The consequences of that follow because then the employer or the university or, and the student in their own personal sense of what they've achieved, judge it to be accurate. And therefore the consequences can be quite big. If we said to people, well, I got a B, but you know, that's not really, you know, it could have been a C, it could have been an A. That's not the way we view them, is it? We don't look at them as a B is nearly an A. No one says that, oh, I got a B, but it was nearly an A. No university says that. Well, sometimes they do. I've known students be given a bit of a break on some of their grades, but most of the time it's a fairly serious consequence because we view them as that reliable indication of something that's been looked at by people cleverer than us, by a system that is detailed and emphatic. And as you say, it might be that way, but it isn't. And things got worse, didn't they, Dennis? Because I mean, you were commissioned, I mean, that year where, 2010, where
- [31:17]
Dennis Sherwood
for
- [31:17]
John (Interviewer)
much of my early career in this country, we're moving in the direction of coursework and teacher assessment. And that was brought to a sort of shuddering halt, as judged. This was, for various reasons, not as good as the traditional exam room, the wobbly desk, the kid with their pen. Be quiet. You've got a half an hour to write something. An hour to write something. And exams made a sort of big return. So some of the problems you've described actually got worse from 2010 onwards, in my judgment.
- [31:48]
Dennis Sherwood
I certainly agree with you. The exam system has become worse in that more depends on it and the results are uncertain. They're unsafe. If someone takes a decision to admit someone to read medicine, for example, to read medicine in the UK, you need at least three 3A's in, say, chemistry, physics and biology at A level. Well, if you do the maths, one student in six with AAA on their A level certificate, these are top grades. Now, one student in six with three as chemistry, physics and biology actually truly merited at least one grade B. Now, that student should not have been admitted to med school, say they're not qualified and they have denied their place to the kid that actually had the B, but who deserved three A's. So it works all the way through the system. And the whole thing is based on a combination of belief and trust in what's going on. You know, to a certain extent, a collusion amongst those who do know a richer truth that we better keep the lid on this because it's in our interests to keep the lid on it. And if people knew that, you know, one grade in four was wrong, there would be perhaps some difficulties to the people who administer the system.
- [33:18]
John (Interviewer)
Interesting. Let's go back to the story of the. So they, they ofqual carry out this big survey. They, they, they look at, they follow up your awkward question, as it were, and they look through and they find that it's actually what's slightly worse than you'd suggested. What do they then do about this?
- [33:35]
Dennis Sherwood
Okay, this happened. There's an OFQUAL board paper dated 18 November 2015, which says the results of our work looking at Mark Inconsistency, they called it, is now complete. And that was a board paper presented on 18th November 2020. Now, three weeks later, on 10th December 2015, Ofcol went to consultation amongst the teaching community for proposed changes in the appeal system, the key one of which was to deny an appeal based on a legitimate difference in academic opinion. That consultation was put through, you know, the consultation process so that in May 2016, just before the summer 2016 exams, the appeals process was changed to block any appeal based on differences in academic opinion, which meant that if you appealed the grade, you did not get a remark by a subject senior examiner. Now, bear in mind that that one grade in four is wrong is all attributable to legitimate differences in academic opinion. How come three weeks after OFQUAL knew that, they go to consultation to change the rules for appeals to stop those grades from being discovered. And the only way you can get a script remark is by. By proof of an admin error or a marking error. And a marking error is an error which is beyond tolerance and nothing to do with legitimate differences in academic opinion, which is the basis of the whole thing. Now, that's a coincidence. And that got pushed through in 2016. So since 2016, none of these grade errors that I have been talking about have been discoverable or correctable. So if in fact I'm an English student with grade three and had a senior examiner you marked my script, I would have got a grade four. I'm stuffed. I can never find that out. It's impossible. The rules forbid it. Now, meanwhile, nothing is happening because ofcol don't actually publish, and then only partially, the the results of the study that show on Grading 4 is wrong until November of 2018. They have the results in November 2015, they changed the rules for appeals in 2016 and they don't publish until 2018 that result. So that's when I started kicking up in November 2018, after they published the results, when I discovered all of this, because between the end of my assignment in 2013 and the publication of the results in 2018, I was just, you know, not involved.
- [36:47]
John (Interviewer)
So your initial reaction was good, they're looking at it. So someone's. I've raised an issue and it's being looked at. And to your, I suppose, surprise and horror, they actually would cover up. Be fair is a word that's often used these days, isn't it? But it does seem like it's sort of swept under the carpet. We've solved the problem by hiding it.
- [37:05]
Dennis Sherwood
Am I being fulfilled? Yeah, well, in fact, no. No. In fact, when they. When Ofcoil went to consultation in 2016 to change the rules for appeals, that was in the public domain. So I knew they were changing the rules for appeals in 2016 and I thought that was a strange thing to want to do. Why on earth would you make it harder to appeal and particularly deny the discovery of the great based on legitimate differences in academic opinion between a senior examiner and the guy or Gal, who marked your script? Why on earth would a regulator who's supposed to be there to protect the interests of the students do that? So I started looking around a bit and meeting some people in 2016 and 2017. But it wasn't until November of 2018 when they published a report called Marking Consistency Metrics and Updates which contains a chart which contains the reliabilities of grades in 14 different subjects from maths at 96% to history at 54% or whatever it is with all the other subjects in between. Geography 65% for example. But since then, other than the COVID years which are obviously a bit exceptional, basically nothing has happened this summer. The kids are now starting their exams as we speak and those exams run through for the, you know, May and June and they'll get the results in August. Prediction. Six and a half million grades or whatever it is will be awarded one and a half million. 1.6 of them will be wrong without appeal. End of story. It'll happen again. Of Col and the example have been very, very coy let's say about admitting any of this.
- [39:00]
John (Interviewer)
Interesting. I mean there's all sorts of things that people who aren't involved in education and exams might be interested just in terms of kind of the psychology of a system that people don't want to challenge. And you can see that extrapolated out all sorts of. In all sorts of places a kind of group think it works. It's always. We've always done it that way and how those can be embedded in a. Almost a cultural resistance to looking at something that was going to be awkward. Maybe also a financial one. I remember the number of appeals went up in my career. So every year we would student our school would say well, anyone who's near a grade, anyone who we think put put appeals in, we'll get a percentage of them up. Because it mattered for the school. Schools became very dependent on the kind of grades for how their judgment have judgment about the schools made. So schools would blanketly put in lots more appeals and that went up a lot more. So was there a. A financial incentive to restrict the number of appeals or make appeals less. Less likely to change?
- [39:57]
Princeton University Press Advertiser
It's springtime, which means that Princeton University Press is having its annual 50% off spring sale. From May 4 through June 9, you can get 50% off nearly every single print, ebook and audiobook from Princeton University Press. Just go to press Princeton. Eduardo. To get 50% off incredible books like Disneyland and the Rise of Automation and Beyond Belief How Evidence Shows what really Works there are so many fantastic books you can get an incredible deal on. Go to press.princeton.edu and use the code spring50 that's S P R I N G50 at Press Princeton. Eduardo, the sale only lasts for a month, so go and get some books.
- [40:47]
Dennis Sherwood
Why ofcorl change the rules for appeals is an interesting question. I do not know, I wasn't around the table. It would be really interesting for a journalist or whoever, if not the Education Select Committee, to get someone who was there to answer that question. I can only speculate, but let me mention this. An ofcol, the regulator, was created by act of Parliament and the act of Parliament is called the Apprenticeships Skills, Children and Learning Act 2009. And in that act of Parliament, section 128 defines Ofqual's obligations, of which I draw attention to two. One is that OFQUAL has a statutory duty to secure a reliable indication of skills, knowledge and understanding. They have to ensure that their qualifications that they regulate do that. And the word reliable is in there. Now, reliable is not defined. But if one grade in four is wrong, is that reliable enough? I'll come to that again in a moment, if I may. There is another obligation in there which is to secure the that regulated qualifications. GCSE zaglofors and others represent value for money. Now, in the UK, schools spend more than 400£450 million a year on exam fees that they give to the people who administer and do the exams. 150 million pounds a year for a product that fails for one use. In for and OFQUAL have an obligation to deliver value for money. Now, the question is, if this is widely known, how does that comply with those two elements of the law about reliability and about value for money? Let me pick up the reliability point. I've said the evidence is that one grade in four is wrong. I believe that OFQUAL have never admitted it. In fact, they've done the reverse. They have very vigorously denied it on several occasions. But sometimes ofquall have got to hint, especially when they've been pressured to do so. So let me give you an example of what happened at the Parliamentary Education select committee on 2nd September 2020. Now, the select committee in the UK Parliament is a very important body. It's a parliamentary subcommittee, it oversees education in the UK and they hold hearings where they can bring witnesses in to give evidence and they can hold people to account. That's their job. In 2020, we had the COVID crisis and there'd been a whole disaster about exams and Covid in the summer of 2020, then chief regulator, a lady called Sally Collier got fired and there was a huge problem. So on 2 September, when the exams were over, the Education Committee held a hearing to try to find out what the hell happened then. Sally Collier, the then Chief Regulator, had stepped down and a new interim Chief Regulator had been appointed. About a week earlier, none other than Dame Glynice Stacey, the same lady that commissioned my work work in 2013, had been parachuted in. Now, at the hearing of the Select Committee, after doing the drains up on the COVID disaster, we all expected things to return to normal. We thought that Covid had been cured. We were wrong by a year, but no one knew that. So Glyce Stacey was a witness and she was there to say, well, looking ahead, you know, things are going to be fine and we're going to do this, that and the other. Now, one of the Members of Parliament asked Glynnis Stacey a question and the question he asked was all about grades being about 25% of grades were unreliable. He actually quoted that figure. We all know that grades are. Or grades are reckoned to be 25% unreliable. One in four is wrong, you know. Shouldn't you be looking at that Daimlerness? Let me read. Dane Glennis reply. Yeah. Mr. Meehans, the MP says, I am wondering therefore, whether one of the issues that you really should address, Dame Glenis, is that 25% of grades on an annual basis are regarded as being unreliable, either up or down. So Ian means is quoting 1 gradian 4 is wrong. Dane Glennis reply as in the minutes of the meeting is this. Thank you. I'll certainly keep that in mind and I look forward to speaking to you about it. Mr. Meehans. It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think my colleague mentioned earlier in this meeting, that we have faith in them, but they are reliable to one grade. Either way, we have great expectations of assessment in this country. Now, that's Den Glenis Stacey, the lady that commissioned my work and the one that got kind of concerned when I said the fact that 0.6% of grades of change on appeal is misleading and she commissioned the research, so she knows me. And Meens asks the question directly about one grading for being wrong. Now, it's very interesting what gain Gunness Stacey did. She did not push back on Ian Meehans and say, that's rubbish. How do you say that? She did not challenge the allegation at all, but she didn't answer either. She stepped around it and she said grades are reliable to one grade either way. We have great expectations, don't we? So she is saying that grades are reliable to one grade either way and if there's a three on a certificate, it means maybe it's a three, maybe it's a four, maybe, maybe it's a two. No one knows. She's admitting that in 2020. Now, that's not exactly the same as one gradient. Four is wrong, but mathematically you can show their equivalence. Now, I regard that as really quite important because the chief regulator is telling the select committee that grades are only reliable to one grade either way and you cannot take a safe decision based on the cliff edge grade Journal certificate. She said that in 2020, yet those cliff edge grades are still on certificates with no warning that they're only reliable to one grade either way to this day. And people are still taking unsafe decisions. But this has been in the public domain since 2nd September 2020.
- [47:50]
John (Interviewer)
Goodness, what a moment.
- [47:52]
Dennis Sherwood
Some bluefall.
- [47:53]
John (Interviewer)
The answer, it is unbelievable. The answer is that she gave Ms. Stacy was something along the lines of shrugging your shoulders and saying, well, it's your fault. You take too, you take these grades too seriously. We should, we all know they're not that accurate, but that's absolutely flies in the face of how they're treated in society. As we've said, they are not. They are relied upon and generally promoted as highly accurate and definitive and authoritative. That is the way they're presented. So it's an extraordinary remark.
- [48:24]
Dennis Sherwood
Well, you know, it is remarkable that the, the thing is that the, the officials Ofqual have known about this since late 2015. Ofsted also know about it because the chair at OFQUAL at the time 2015 was Lady Spielman, Amanda Spielman, who subsequently left OFQUAL to become the Chief Inspector of Schools at Ofsted. So she carried that knowledge with her. Ofsted and the Department for Education, the government department, must know, because that's what they do. And of course all the exam awards that administer the exams know it as well. And it's been known for years. But you I. The students don't know, employers don't know. And still to this day, this summer, another 37,000 English students are going to have their self confidence shattered with a grade 3 on GCS English language, not because they don't know their grammar, but because ofcol cannot meet its statutory obligation to secure a reliable indication. And I come back to that. OFQUAL have a duty that regulated qualifications should show a reliable indication and the chief regulator is saying they are reliable to one grade either way. The law does not define what reliable means. So is reliable to one grade either way compliant with that legal obligation? Yes or no? Supreme Court of England, please decide.
- [50:03]
John (Interviewer)
Well, we will get onto in a second what's happening right now. I know the Labour government, which we have now have carried out a major review of the curriculum in schools and, and I think one of the things they concluded we have rather too many exams in this country. But in 2020, you wrote this book missing the mark and what was the. It was, in a way, it was a Covid project for you. It's a. It's a wonderfully detailed analysis, as I say, takes you through the history of the things we. You've been describing today and the statistics, highly convincing statistics that you give. It's. How did you go about writing it? Did you. Because you, because you do other things. I mean, you're a management consultant with Silver Bullet and so on. Did you. The actual process of writing it? Did you think, well, I've got to. I've got to do this and it's going to take up a lot of my time, but it's going to involve a lot of research. How difficult was it to write this book?
- [50:58]
Dennis Sherwood
Yeah, writing a book is. Requires a degree of effort and concentration, that's for sure. But as it happened, I had written a fair bit about it over the preceding few years, which had been published on various platforms like hepi, Higher Education, Polish Institute, Times Educational Supplement, TES and so on. So I'd written a number of blogs highlighting various features of it. So I wasn't starting from scratch. I had a lot of good material there, so it was really bringing it all together. And with COVID and the lockdown and all of that, I wasn't traveling to run creativity workshops with clients as I had been in obviously the normal working day. So I actually sat down and wrote it and thought that the reintroduction of exams in the summer of 2022 was a good opportunity to give this matter publicity because after the COVID interruption, 2021, things had gone wrong in a different way and there was an opportunity there with exams coming back and a new chief regulator to say, well, let's sort out the mess. So I was hoping that that might have some influence either directly on off call or by pressure that people reading it would put on at that time. In fact, that pressure has not had any effect at all. So we're in exactly the same position now as we were in 2019. But the book is still there on the record and the book is still very valid. The book talks, as you know, about the history. There's some important stuff there on what happened in Covid that's in the past now. But anyone that wants to know about the mutant algorithm or teacher assessed grade or whatever happened, that's there. But importantly, the last part of the book, the last couple of chapters are on solutions to the problem, how to give reliable grades even when we're doing sit down exams with human markers, with intrinsic variability between experts, but still have results that people can trust. And the last section of the book is all sorts of ideas to do that.
- [53:19]
John (Interviewer)
Brilliant, Dennis. So there's a great way, as we come towards the last few minutes of this, of our discussion this morning of your book, would you like to pick out a couple of ways? Because it can't be beyond the wit of man to, to devise if we believe that students do need the assessment in society is quite good. We, we respect that when people do a driving test it's quite handy if we know that people are good drivers, not incompetent drivers. So we're going to have some forms of assessment in this world is what's, what are some of the suggestions if you'd like to pick out a couple of that we could do this better.
- [53:53]
Dennis Sherwood
Hey, the easiest one is literally a stroke of a pen or rather a line of printing. The simplest thing to do is to be honest and tell the truth. And that way at least everyone knows what's going on. And people are not penalized for this because the students are penalized so literally today OFQUAL could instruct the exam awards to print on every certificate in big letters ofqual. Warning. The grades on this certificate are reliable at best only to one grade. Either way, that's the truth. That could be done literally this afternoon. It would cost absolutely nothing, but has a major impact because it means that anyone taking a decision on the single grade on the certificate knows they might be getting it wrong. They might be disadvantaging the student. Now obviously that's got knock on implications because it means that basically the entire population of young people would actually say, well that is an X on my certificate, my true grade is an X plus one. And that could be the truth. And that means we're giving young people the benefit of the doubt. Now to my mind, that's a better place to be than penalizing kids on the basis of what happens now and getting it profoundly wrong. Because if a kid has Got, say, grade three on the certificate, but they're not forced to reset. They don't lose the year development. They've got an opportunity. Good for them. They might actually do something with it. And the downside is very small. And I put that in contrast to being a brain surgeon. You can't give a brain surgeon the benefit of the doubt that maybe you would have got a higher grade had your brain surgery been assessed by a different surgeon. The risk of that going wrong is too high. So you don't do that for brain surgery, you don't do it for the driving test. But it doesn't matter for GCSE history. So that's the simplest thing you can do. Tell the truth, reliable to one crate. Either way, there are a whole load of other things, but to my mind, the one that is also pretty honest is to get rid of those grade boundaries. There's a kind of mindset that says all A is good, all B is bad. Now, the truth is the difference in ability between the bottom person on the A grade and the top person on the B grade is tiny and much bigger than the difference between the top person at B and the bottom person at B. So these cliff edges are really misleading. All B's are not the same. All as are not the same. So throw the grade away and say your raw mark was 65. And for geography, the uncertainty in that is about 5 marks either way. So it shows 65 plus or minus 5. It might say maths 72 plus or minus 1, whatever. So the actual certificate is different, which shows a mark and an uncertainty. Now, that is true. And that means that if you've got a kid with 64 plus or minus 5 in a subject and another kid 66 plus or minus 5, those two young people are indistinguishable. You can't say good, bad if there were a great boundary there. They're basically the same. So you need to work a bit harder if you need to distinguish between them, you talk to them, they. You take a more mature approach if you need to decide between them for a job application or whatever. Now, once again, of course, know all of these numbers. And although it's a little bit more work to get the plus or minus bit, it's not a lot more work. So it's technically easy to do. The issue to manage with that solution is actually just the education of us, all that we can accomplish, accommodate what that really means, because A, B, C, all A is good, all B is bad is a lot easier to kind of, you know, process in your head, but it's flawed. So 65 plus or minus 3 might be a bit harder to process, but it's much, much fairer. So those are two solutions. Tell the truth or be rather more realistic in how you actually print the assessment. There are lots of other solutions, too, but those are two.
- [58:25]
John (Interviewer)
And thank you. And once again, those solutions are described very well in your book and this morning. Thank you so much for that, Dennis. That is brilliant. As always. I always come away from the hour discussions feeling sort of a little bit outraged and appalled by my fellow human beings and hopeful that someone like yourself is out there drawing attention to this. And we've been talking this morning on the New Books Network with Dennis Sherwood, author of Missing the Mark, why so Many School Exam Grades are Wrong and how to get the Results We Can Trust. And it's as relevant and as powerful a book. If you're interested in psychology of systems, if you're interested in teaching education or wherever you're teaching in the world, I recommend Missing the Mark by Dennis Sherwood. So thank you so much for joining me this morning.
- [59:11]
Dennis Sherwood
Thank you, John, for a really lively conversation and giving me time to talk and listening to and thank you all the listeners around the world as well.