← New Books Network
New Books Network
James Elwick, "Making a Grade: Victorian Examinations and the Rise of Standardized Testing" (U Toronto Press, 2025)
01:03:06
Loading summary
Transcript23 lines
- [00:00]
James Elwick
Hello, everybody.
- [00:01]
Marshall Po
This is Marshall Po. I'm the founder and editor of the New Books Network. And if you're listening to this, you know that the NBN is the largest academic podcast network in the world. We reach a worldwide audience of 2 million people. You may have a podcast or you may be thinking about starting a podcast. As you probably know, there are challenges basically of two kinds. One is technical. There are things you have to know in order to get your podcast produced and distributed. And the second is, and this is the biggest problem, you need to get an audience. Building an audience in podcasting is the hardest thing to do today. With this in mind, we at the NBM have started a service called NBN Productions. What we do is help you create a podcast, produce your podcast, distribute your podcast, and we host your podcast. Most importantly, what we do is we distribute your podcast to the NBN audience. We've done this many times with many academic podcasts, and we would like to help you. If you would be interested in talking to us about how we can help you with your podcast, please contact us. Just go to the front page of the New Books Network and you will see a link to NBN Productions. Click that, fill out the form, and we can talk. Welcome to the New Books Network.
- [01:07]
Jacob Ward
Hello, everyone, and welcome to this episode of New Books Network, where I, Jacob Ward, will be interviewing James Elwick about his book Making a Victorian Examinations and the Rise of Standardized Testing. So, Jamie, thank you very much for joining us. Delighted to have you on the podcast. And before we start talking about the book, I'd like to just ask if you could tell me and tell our listeners a little bit more about yourself.
- [01:36]
James Elwick
Well, I thank you so much for having me. And so this. I'm. I teach in a Science, Technology and Society department at York University here in Canada. It's actually a little older than the University of York in York, so I like to call our colleagues there, our younger. Our younger institutional brothers, and they get very surprised by this. But no, I mean, I've been so. My first book was a history of biology just before Darwin, essentially, looking at. It's quite a technical history of biology that looks at biological individuality as the kind of master narrative. It was reissued with Pittsburgh, I think, in 2020 or 21. It came out in 2007. And it identified. It essentially started out by asking, well, why does Herbert Spencer read this way? Why is it so strange? And I sort of identified the problem as that what was considered a biological individual was odd. And so skipping along past this, essentially the end of the book ends, so to speak, with concluding that his competitor in this, who's T.H. huxley, effectively wins by redefining what a biological individual is, not by winning technically, but by training the new generation of biologists to use his terms. And the way he did that was through examinations. And so around 2007 I got quite interested in this and started to look at, well, what does it mean to actually be certified as an expert in something? And so out of this and my knowledge of the UK sort of system, and it's extremely convoluted 1800s educational system, how did that result? And so how does somebody get certified as an expert? And so I thought I'd build on work that was done especially by Andy War and William Clark. And so, yeah, and. And then also combined with something that I think many ex, many academics feel, which is the unwritten part of their entire lives, which is what is it exactly that my students have actually learned? Did they actually learn anything at all? Um, and when you mark their exams and you read what they pour out, does that actually. Did that actually. Did they learn anything at all? And, and so I have often a very deep skepticism about this, but also the best part about it is how funny the whole thing is. And again, the, the whole point is just from often a very blackly humorous sort of topic, sort of approach. And so I think that looking at students in this way, and you read the same comments, so Alfred Russell Wallace doing exactly what I and other grad students did is writing out the, the, the, the funniest answers that one could find, the most ignorant answers. And so this has really not changed that much. So that's a very long winded way of sort of my, my journey here.
- [04:46]
Jacob Ward
Well, that's. No, it's lovely to hear and some of the people we've heard about it, especially Huxley, I think will come up later in the, in the interview. But we'll get there when we get there, I should say. But first of all, I'd like to ask, just to kind of really continue where you were going and tell us more about the. Kind of. Give us an overview of the book, the arguments you make in a kind of high level overview.
- [05:11]
James Elwick
So the book essentially starts out with this question and it says, well, people complain about standardized tests, but nobody ever asks how they get standardized. And so what I do is I take an approach that's used in sts, particularly by Jeff Bowker and Susan Lee Star about infrastructure. So I do essentially what's called an infrastructural inversion and look at, well, what are the tools and routines that are used to actually standardize these exams? And effectively, the other thing that I do is to look. I'm always puzzled by the confusion that's often used. So here in my province of Ontario, it's not called the department, the Ministry of Schooling, it's called the Ministry of Education, which is really interesting because there's this confusion. When people mean education, they usually mean schooling. And they're not. They're not identical things. Schooling, above all, the reason that so many people attend schools often because they just do out of a matter of habit. But schools are not just places where we learn things, but where we're certified for things. Otherwise, why wouldn't people just go to the Khan Academy or, you know, pick up things on YouTube? And so this is where I sort of take this, what might call a learner focused, a highly skeptical. Well, how do I know they know what they're talking about? And just look at the how are. How are nowhere certified. And so effectively I take that infrastructural kind of approach and look at mass examinations. And so the book takes three kind of levels. There's a wonderful article that came out and the term that's used is called gaze scaling, in which somebody looks at different levels of organization from the sort of very, very broad and abstract. So the first chapter is after the introduction of second chapter is all about sort of, hey, what happens in 50 years. And then I. And then the next sort of gaze is more, much more focused on what are these remarkable examinations that are called the Department of Science and Art exams that ended up training millions and millions, excuse me, examining millions and millions of people in science and art, what we now call STEM exams. And then the third is to actually recreate an actual examination that was run and marked by TH Huxley and going down to the level of forensically recreating what a student, excuse me, an examinee, would write out and how Huxley set the exam. And then that leads us then into areas such as thinking about not only what the examiners did, but also what the examinees did. So that's the next part is that I sort of coming out of history of biology, it's very easy to see arms races everywhere coming to see, or you might call it a dialectic or a pattern of reactivity in which on the one side you have examiners setting out rules, and then on the other side you see people trying to either live up to those rules or in the more interesting way, find loopholes or find ways around the rules, and then the examiners react to this and try and plug up these loopholes. And so this kind of pressure on both sides, which I find is, is fascinating. And then the final bit is trying to look at how we, how exams became for so many people, this key document of what we now call merit and meritocracy. The author Michael Sandel wrote a, wrote an interesting book about meritocracy and thinking about merit. And it's taken on an interesting tone, especially in the US but he never once questions the tools by which we define what merit means, but also how you get there in the first place. And so I find it kind of funny that someone who sort of came out of elite high schools and then went to Harvard and is at the top now all of a sudden starts complaining about merit and the role that it takes in US society. And again, I'm Canadian, so we sort of have this sort of slightly slanted view on the US where such exams are not nearly as important as one's sort of the schools one goes to, or things like that. And, and that leads me to the final bit which is focusing. And this is something that history of science has not been great at in my, in my opinion anyway, which is the history of education and in particular tying this to questions about epistemology and credentialism. What is it to know who is certified, but also how do large groups of certified people interact in ways to make it possible? And so that fits in with themes in history, philosophy, sociology of science, about abstraction. So what is it to take this unfathomably complex set of ideas or concepts or skills and to, you receive a piece of paper that says, oh, you've graduated from this four year honors degree in, I don't know, biology or in political science or things like this. And credentials, again are such an important area of research. Economists are fascinated by them, by what are called their, their signaling mechanisms for the labor market again in the US and, and then in the uk I was in the UK where all of a sudden they started to impose tuition fees. And so you have students paying extraordinary amounts of money for this now in this, in this, what seems to be an inflationary spiral. And so I actually look at this and go, well, what about, what does this mean about currency and that inflationary role? And so credentials are seen as a kind of a currency. So I know I'm not supposed to give more than two minutes for an answer, but that's, that's about five. So I will finish with that.
- [11:28]
Jacob Ward
That's very nice. Thank you. I mean it's an incredibly rich book and I think at a top level, I think where you started your answer was talking about how this infrastructure of examination is built is for me one of the most kind of compelling narratives in the book. I think you mentioned your own practice as an educator and certainly I was thinking a lot about mine as I was reading the book and why we test in the ways we test. But I'd actually like to ask my next question or next couple questions about where you've ended your answer with credentialism. And one of the things that really struck me about several of the figures that you talk about in this, in this book is they, they seem to have career trajectories where they are taking lots of exams, huge amounts of exams at places like the department of Science and Art to get credited in subjects like animal physiology so that they can then make a career tutoring other people to pass those exams. And so there is this kind of testing, standardized testing in this stuck almost seems like a kind of the birth of a self reproducing system. And I wanted to ask specifically about two people who have slightly different roles in this. But the first is somebody who many listeners will probably know of, H.G. wells, who more famous for other reasons but with a tutor and as you say, as you write about, wrote a textbook as an examiner defeating mechanism to help his students pass these tests themselves. The other person I'd like to ask about is Robert Goffin who was embroiled in a national scandal, the Goffin Affair. And he was also a tutor but found maybe slightly more illegitimate ways of defeating examiners for his students. So let's start with Wells and Goffin, please.
- [13:24]
James Elwick
So effectively part of. And again we can maybe talk later about my new interest is in what's the origins of, of academic integrity. Cheating in other words. But effectively the definition I came up with at the time is what is cheating if not to sort of follow the rule, to sort of subvert the standardized rules. And so H.G. wells is one side of the ethical of the. His feet are on the, on the inside of the line of the rules that are set out. Like he doesn't actually wish to cheat and just completely ignore the rules. And Goffin does. So H.G. wells is, is a fascinating character. He effective he and the word, the phrase examiner defeating mechanism is actually his. It comes out of his experiment in autobiography. But he writes a number of books his love and Mr. Lewisham and others. And so I sort of pulled that as well as a lot of his life and letters and some biographies, as well as his. His very first book, which is his textbook of biology. So effectively, when he was very young, it was discovered that he was very good at just learning things by reading about them. And then he would write these Department of Science and Art exams, which, and this is a very important point to bring up, I haven't brought this up yet, is that teachers were actually paid according to how well their students did. So if you had a student, I can't remember the figures right now, if you had a very, very good student, you could win up to four pounds, three to four pounds, I believe. So this was this incredible incentive to have good students. And effectively all you did was you were measured by how well your students did on an exam. Which leads to these fascinating questions about. Again, we talk about feedback mechanisms. This helps to define what education is, which is success on examinations, which in some ways is still what we. We see today. So anyway, H.G. wells was first discovered by. So he became the sort of profit center for at least one school when he was young, and then, I believe, a second one. And then he discovered, well, I'm just going to start doing this myself. And so his teachers effectively use him as a kind of. He writes all sorts of exams. His very first exam is writing a bookkeeping exam at age 12. And then he goes back, and then he goes on to start teaching other students how to write these exams. And so he learns, for instance, and the best part is that contrary to what TH Huxley says, there's no practical examinations, because at the time, at least there are. But no, they're not. They're not practical exams. They can't do it because they can't standardize. How can you do this, at least on the DSA exams? So it's all about book learning. So, well, how do you show off your knowledge? Well, you have really big words and you follow exactly the correct sequence of operations that are prescribed in a textbook and these kinds of things. And so he records how he would teach his students. As a science educator, he would not do any lab experiments at all. It was all about writing words on blackboards. And they would repeat that and then they would do well. And then he uses that in his own way to negotiate better and better deals for himself. But my favorite part is when he gets a job. This is before the Time Machine, which is his first book. He gets a job with what's called the University Correspondence College, which is set up by this. By this chemistry tutor who essentially sees all chemistry, all Chemistry knowledge, exam knowledge as essentially a series of recombined questions. And so you go back and you collect all of the answers that you can know about that have been, excuse me, all the questions that have been collected over the last 20 years of exams. And you tabulate and calculate what's the probability of a question about carbon that's going to be asked, or when are they going to ask about what? What do you do when you dissolve something with sulfuric acid, et cetera, et cetera. And then you predict that. So Wells's role is to develop that for zoology. And so he does that and makes quite a bit of money about this. But he also develops this, a fascinating thing, which is another theme of the book, which is this, we could call this in the spirit of chemistry, bivalence, sort of a positive and a negative attitude. And he's al. Always known as, oh, this is terrible. And it's very cynical and he uses a lot of metaphors that we may talk about. My favorite is food as, excuse me, knowledge is food. And so what he was selling was something that was fake, fake nutrition. So counterfeit food or counterfeit currency. That's another lecture. It's another theme that is that. That is recurring, but at the same time he's also quite aware that it works. So in the 1920s, he actually talks with someone from China. And in the 1920s, the then famous Imperial exams had been discarded and I, I believe they've been discarded at this time. He said, oh, I actually think they're very positive and could be valuable if we update them for modern things. So again, there's that bivalence that he always has. So Wells is on the cynical but still ethical side. And Robert Goffin is the more interesting one because like Wells, he's also a gifted teacher. But Goffin effectively demonstrates a cheating principle that we see again and again and again, which is effectively the simplest way to do well on exams, is to intercept them in advance and then drill your students on the answers right before. And then they go and they write them and they do very well. And so I call this the Goffin technique. And so there is a case where he's discovered in 1879, he's a. He's one of the most successful science teachers in, In England at the time, just based on how many, quote, gold medalists, unquote, that he's turned out. And they, the Department of Science and Art, the people that run it, know there's something wrong with him. They know there's problems. Some of them, the Main person who runs it, one of th. Huxley's friends, a guy named Donnelly. But at the same time, there are other people in exactly the same department who write letters of recommendation for him and say how wonderful an educator is. So they don't even quite know what to do with him. But then it's finally definitively established that he has been opening these exams in advance. There's this whole. It's very, very funny. What do you call his. His. One of his main apprentices, a guy named Ledger, Charles Ledger, has. He gets his sister pregnant. And so. And so Goffin. And so there's this estrangement and then Ledger starts. Leaves and then starts to write anonymous threatening letters and. And everything. And. And so. But Goffin is fascinating for the sense that on the one hand, he systematically engages in fraud and he. He's never actually fully caught. He gets into a lot of trouble. But he manages to get his way out of this by creating a. A counter there. There's essentially. There's two inquiries, there's an inquiry in the House of Commons, and then there's an inquiry at his own school, the United Westminster Schools, that he has as a. He basically suborns and they come up with a completely alternative story. And so here is this again, something that. Is this pattern in this book where you have kind of the. Again, a dialectic or an arms race. You have two. You have the official one and then you have the unofficial one that kind of keeps coming up. And to cut my answer short, again, because I'm running on effectively, the Goffin technique is something that we see again and again and again, we saw this in 2010 in Atlanta where there was a. This then very quite famous educator who was credited with turning around Atlanta area schools. And it was discovered that, well, no, her secret was just getting the exams open in advance and then getting teachers to drill them in it. And so I call that the Goffin technique. But again, here you have somebody who. He's a gifted teacher, but he doesn't buy into the inherent idea that exams test for morality. So. Yeah, so that's. That's my answer there.
- [21:57]
Jacob Ward
Thank you. And I think one of the things that you've shown in that answer, the ways that Goffin prepared his students, the way that Wells prepared his students, and indeed with this metaphor of knowledge as food that can be regurgitated at a later moment is this idea, in a sense, that you. As you call it, that these answers that students give show a form of thin knowledge, that it's knowledge that is flat it's compartmentalized, it's compressed, it's presented perfectly for the purposes of a standardized test that it doesn't actually show a form of rich learning. But did you argue, as you say, in this dialectical sense in which exams and examiners and examinees kind of march in dialect with one another? This is in part a product of the fact that these certificates themselves, to go back to credentialism, are what you call thin descriptions. And there's a very nice description where you. A not so thin description where you describe examiners as anti anthropologists in a sense. But that thinness of description I suppose is one of the reasons that standardized testing became such a powerful infrastructure. You talk about it, for example, as central to the rise of the University of London as a federated system that enabled long distance education for perhaps one of the kind of first in the Victorian era. So I'd love if you could talk about the other side of thin knowledge exams as thin description.
- [23:31]
James Elwick
So this term thin description is Ted Porter's and it's from a paper he gave in 2009 and of course he's responding to Clifford Geertz's idea of thick description. The purpose of the ethnographer is to describe a situation relative to the context that gives its activity meaning to the participants. And of course that's very important for the ethnographer. And so the term anti anthropologist is Simon Schaffer's. I was talking with him about this and he said oh no, that sounds like they're doing the opposite there, that they're, there are people are being forced into boxes. And so I thought oh that's, that's a great one. And so I, I used that and effectively again in order for us to participate in large social groups, to be able to know what other people know. There's this problem of, of how, how do I know? Let's say that I'm hiring a, I'm, I'm interviewing somebody who wants to become a postdoc at my, in my laboratory. I'll, I'll stick with that even though I'm not myself a scientist. So I wish to find out if. Well, do they know I'm a. I don't know. I'm, I'm working in molecular biology. How do I know they know the basics? How do I know they know how to, how to run a centrifuge or whatever it is that they do? And part of that is this trust that they have. Well, they've got a PhD. Well, how do I know that this PhD they learned anything. And again, there is this, this gigantic system of trust. And this comes from this thing that the sociologist Ellie Gerson really got me interested into which he talks about. I'll go to another problem of what, what we call common knowledge. So when we're driving in traffic, on whatever side of the road you drive you, there's this enormous amount of trust. You trust that every other driver, not only that the green light means that you should drive through, but you trust that everybody else on the road knows that a red light means to stop. So there's this gigantic kind of problem. And so I kind of thought about this in terms of thinking about knowledge in general. How do I. And again, this gets back to that problem of the, the, the. The molecular biologist I'm hiring. So that's in some ways what credentials are. And they are forms of. They're, they're, they're thin descriptions. They sort of say, yep, this person is certified as knowledgeable in a particular area. And it strikes me that this in many ways is what so much schooling is about. Not so much education, but schooling. That schooling isn't just about teaching us things, but it's actually about devising a. Taking a certification that will stand for something that is trustworthy. And this is not to cynically say, oh, we should do away with all credentials, because I don't see any other way of it working. Just as we are talking now and abstracting, simplifying an unfathomably complex world into something that can make sense. And I think that in a lot of education scholarship that I've noticed coming from outside education, sociology, history of education, there is this interesting sense that everybody thinks that testing or credentials are bad, that they're evil, that they should not be done, that they're somehow inauthentic. And my rejoinder is, well, how could it be otherwise in an unfathomably complex social system that requires expertise? And so thin description in that way is a. Is something that we can't do anything other than do this. Just as we, we must use things like maps, so maps as simplifications of, of some unfathomably complex territory are necessary for this. So this brings me into areas such as Bruno Latour's actor network theory stuff, his Visualization and cognition in 1986, which is sort of my favorite article of his, about how so much of thinking about knowledge and science is about simplifying things and how else could we do it? So I try and bring that to that and Ted Porter's description again, and it's Again, the point is that bivalence, of course it's going to leave out things. Of course it is a. It is going to overly simplify things, but how else could we function? So that's. That's sort of my question there. I does that. I hope that answers the question about thin description.
- [28:24]
Jacob Ward
No, absolutely. And I think what you've just mentioned about not only how could it be otherwise, but that so many of the discussions that I think those of us who work in education or perhaps more accurately in schooling, these conversations we have about examination often do come from, I think quite a point of frustration, not wanting to reduce our students to these thin descriptions. But what I really appreciate about your book and what I'd like to ask about next is how you show that standardized testing isn't it bivalent and that it hadn't necessarily been bad for all people in all places at all times, because you show how women in the late 19th century turned standardised testing to their advantage in their fight to join places like University of Cambridge and so on, that they kind of turn it against the authorities who built this infrastructure to stake a claim for the equality of women in schools in universities. So it is more complicated story and I wonder if you could maybe elaborate more on people like Emily Davis and Philippa Fawcett and so on who see these thin descriptions and see them as a wedge. I think you say to leave her open the door for women and more.
- [29:48]
James Elwick
Yeah. So this is again part of this is so one of the most interesting things about exams when they're first proposed as a widespread area in the 1850s, you often get this query, well, how is that? I mean, why? What kinds of moral qualities are these testing for? And so you get this famous quote that I found going all the way to. By Robert Caro's extraordinary biography of Robert Moses. Robert Moses, the guy who basically built much of New York City, had in this very beginning tried to reform New York's civil service service. And he goes all the way back to Macaulay and he actually quotes the exact same thing that Macaulay said some 60 years before, which is that exams don't just test cognitive ability, literary ability, as they say, they test morality. They test your ability to work hard or to be self denying. It could be masculine virtues, as Andy Warrick has talked so much about, but it's. It could be seen as just something that's less gendered and, and that gets to. We could talk about commensuration at great length. And so someone like Robert Goffin, I say, is Somebody who does not buy into this, into this moral economy just as well. It's just, it's an empty ritual that I'm going to game. But the, the other case are the women and this is a case of where women. So stepping back a little more, I'll talk about commensuration a little bit. Something that the sociologists Wendy Nelson Esplund and Michael Souder point out. One of the. It was something that was just extraordinary for me to read. I, I encountered it through Ted Porter's work and they've since done work on ranking, ranking and reactivity in their discussions about law school ranks. But effectively they, they talk about the role of commensuration. What is it to actually decide that one quality should be assessed across different groups? And when do we consider different groups to actually be worthy of commensuration? So I'll give the example of everybody accepts that we should just test all 14 year olds. This is, this is just a. And in effect deciding that that quality of age 14 is something that is worthy to group all that group together. However, in the 1850s you could not say, well, women and men to that quality of gender, sex or whatever you want to call it was equally commensurable. So there was a fight about this. So it was unproblematic to group, to measure, to assess a bunch of 14 year old boys. But it was really a horrible problem for many people to commensurate women and men or boys and girls together. But you had the reformers who we now completely agree with, again, as a historian, you can't just say well this one group was right, you have to actually encounter what was it that were the, what were the terms of the debate at the time, the actors categories. And so effectively one of the sort of reformers, Emily Davies, essentially insisted for over a 30 year period that girls, female candidates should be tested in exactly the same standards as males. And so this led to. This led to over 30 years to give away the story. This led eventually in 1890 to this student, Philippa Fawcett, who ironically wasn't at the, at Davies's College but at the other one at Newnham and scores 13% above the senior angler, in effect on exactly the same examination that all the males were taking and to show in this very, very assist in a system that people had spent hundreds of years to ensure was as standardized and as uniform as possible. And she won on this. And this was used later on to demonstrate that women had the right to where were as cognitively were cognitively Equal to males. And we all find that completely unproblematic. At the time this was seen as revolutionary. And again this. So in that the hero of this story, so to speak, if one can speak of heroes, is Emily Davies at recognizing that what mattered was the standardization. That in order to claim moral equivalence the commensuration had to be as exact. Excuse me, the, the. The. The system had to be as equal as possible. And this led to. A lot of historians of education have looked at what Davies was actually insisting they be studied on and saying well this is all very antiquated. And it was using looking at Euclid or Classics and why should they be studying Greek? And that ridiculous. And to an educator's perspective, a modern day educator's expect perspective, this is absolutely quite correct. It is quite ridiculous to be doing that. But as an historian sympathetically recreating this, Davies insisted on this equality because it would give, it would give grounds for claiming the worthiness. This is Charles Tilly's term, the worthiness of the under represented group at being treated equally legally. And of course this is. I got in trouble when I was first arguing this because of course the, this didn't actually do anything at Cambridge for another 50 years. Cambridge didn't actually admit female students until the early 1940s. So it was really shameful. But the whole reason that this was shameful is because it had been demonstrated that women were commensurate with males. So in order to ha. In order to establish that fact and to get morally outreached about that, you actually had to have that equal treatment in the first place.
- [35:48]
Jacob Ward
And I think what you nicely show is that, is that demonstrating this, I think you say it kind of de energizes those men and examiners who would otherwise find it quite problematic to see women and men as commensurate, that they, they kind of think oh it's been proven by this point. So even if, you know, institutional changes like admitting women to Cambridge haven't happened yet, the male examiners themselves feel de energized because these trials have proven that they just have to accept the wheel to change.
- [36:26]
James Elwick
So my favorite letter in the University of London archives is by a obstetrician, professor of obstetrics, who writes this letter, this complaining letter to another obstetrician who says to think that we gave the gold medal to this to this woman. Oh, but it was the best, it was the best paper so it can't be helped. And so they're not going to. And the doctors were just awful about this generally they would write these petitions, oh, we can't allow women to do this. But gradually they're kind of peeled away and de. Energized. Well, why fight? Why fight history? Why, why go against this? And so gradually they're kind of whittled away. And, and, and again, obviously marginalization of women is always going to be, is always going to be a factor and it does for the next 50 years. But there, there's something else that's really interesting that I, that I identified, which is, and this was a, a common theme in a lot of history of science scholarship, studying, or just history of studying exams is the separate place that's given to women. So Marsha Richmond has won studying a sort of subcultures that are created by women studying biology at Cambridge and other places. But what I find in the, in the archives was, and, and so what would often happen is when women were first admitted to actually study and actually take exams, it was insisted upon, well, they can take separate exams, they should take separate exams. And we're going to rate them differently and give them slightly different material and sort of special. So separate and equal. And Emily Davies fought back against this by going, well, they're going to define this as not just separate, but as inferior. And this is why it has to be identical. But what happens in the background is that you get examiners complaining that, well, why do I have to write? Why do I have to set up a separate exam? Why do I have to do this? Oh, can I have more money? Because this is taking me twice as much time to actually set and mark these exams. Because an underlying kind of organizational logic of exams is that it's very easy to just add a new exam, a new copy of the exam, add a new student to writing exactly the same one. Your marginal cost, as it were, would just be added. Whereas to set up an entirely separate new exam is going to take more time. So you get these letters effectively saying, can I have more money please? And no, you can't. And so they'll say, well, fine, I'm just going to give them the same exam. So you get this underlying thing that nobody sees about. And that's part of the infrastructure that I like to look at. Follow the money, which is very, very important. You're a guy who just wants to look nice, the kind of nice where you might get a nice compliment on the niceness of your nice new outfit. Good thing. Men's Wearhouse has everything from polos to jeans and yes, suits, plus a team to help you find the perfect fit. To make sure you look nice nice. Love the way you look Menswear Warehouse.
- [39:22]
Marshall Po
It'S okay not to be perfect with finances. Experian is your big financial friend and here to help. Did you know you can get matched with credit cards on the app? Some cards are labeled no Ding decline, which means if you're not approved, they won't hurt your credit scores. Download the Experian app for free today. Applying for no Ding Decline cards won't hurt your credit scores. If you aren't initially approved. Initial approval will result in a hard increase inquiry which may impact your credit scores experience.
- [39:52]
Jacob Ward
Well, I'm also really glad that you've mentioned the administrative side of these examinations, that one reason men and women ended up being examined together was because examiners wanted to avoid the administrative workload of running separate exams. And in general, the administrative workload and the sheer scale of this examination infrastructure I think really took me by surprise when reading the book to return to TH Huxley I promised we would come back to at a later point to return to T.H. huxley. You talk about this animal physiology exam that he ran in 1873, I think it was, and that he had in a matter of weeks, almost 7,000 papers to grade with a team of six assistant examiners. And the sheer numbers involved in that kind of boggle the mind. And this is somebody who is best known for his kind of role, I suppose, as a public scientist, a public intellectual, involvement in big debates, big scientific debates. And yet, as you point out, the sheer scale of scientific work that he's doing just in administering and grading exams surely lends him a place as an important man of science, not just in these public debates, but also in building this examination infrastructure. So I'd love to hear more about that side of things.
- [41:11]
James Elwick
So this chapter I saw Hans Jor Greinberger this summer at Ishkabib Instit, the International Society for the History, Philosophy and Social Studies of Biology. And so I said I named one of my chapters after sort of as a play on your book. So he wrote a book called An Epistemology of the Concrete. And so I wrote sort of following Andy War's sort of example, sort of a sort of honoring. So I call this chapter An Epistemology of the Mundane. And effectively it's just a forensic recreation of T.H. huxley's 1873 examination, his animal physiology exam. And there's 6,834 of them. And of course it's important to note he didn't have to mark every single one of them he farmed that out. But what I find interesting is that this sort of activity, which is. So it took three weeks, I, I estimate about three weeks maybe. And that's just to mark everything. None of this is ever mentioned in any of th. The biographies of TH H. Oxy. So Adrian Desmond, who otherwise just knows every single thing, really doesn't mention this as much because it's seen as such an ordinary, taken for granted part of our life. But it's, it's actually, it's quite. Insane is not quite the right word for it. But it's this extraordinary amount of work that we have to do. And yet TH Huxley says he actually writes letters in support of the DSA exams that support his friend. Especially after there's a lot of complaints about it. Oh, people are getting over examined. He's actually writing letters that his friends at Nature magazine that he helped establish Nature, they think, oh, this is a little too. In support of the Department of Science and Art. Like they think it's, it's, it's too, it's, it's a bit too propagandistic. He, so he, he does this because he believes that he is really changing the grounds on how science in general is to be learned. And he's only one part. There's John Tyndall and I worked on this wonderful project, kind of transcribed the letters of John Tyndall and so saw a lot of the background. John Tyndall comes in on this hooker, comes in on all the X Club members, the famous X Club members. And of course there's much attention paid to, well, what were they writing in popular articles for the Times or founding Nature things. And if you want an actual larger change about how people think about science and what's called scientific naturalism, to actually get 6,834 students to actually have to go through your exam and, and the better part is they actually prescribe their own textbooks as, as, as the textbooks to be written, which is a bit. So this is very interesting is that I'm actually researching Canadian cheating. So cheating. And I'm going in the archives now. And in exactly the same time Canadian mathematics examiners are prescribing their own textbooks on exams and they get in complete trouble for this. There is this scandal, these newspapers are saying you can't do this. This is a complete conflict of interest. From What I've seen, T.H. huxley and the X Club, there is no mention of this. So it's quite interesting in Canada, this is seen as a horrible conflict of interest. Joy That's a side note. But the larger issue is for historians of science, we often focus on these public pronouncements. And there has been, since Jim Secord's look, since his Victorian sensation, how's it actually read? Forget about what's actually written, because people can take out so many different messages, different meanings of a textbook. So when I look at Herbert Spencer, Herbert Spencer is, oh, he's often seen as a free market social Darwinist, but you could, there, there were Italian communist Marxists who thought Spencer was wonderful. And so people can take very, very different meanings from a textbook. And so here's this fascinating case where you can, the authors of their own books can actually enforce the correct, in their view, reading of their own textbooks. And again, this is to get away from my more orthodox science colleagues saying, well, they're testing what's right. And it's like, yeah, you, yes, but you can't say that as a, as a good historian of science, you have to actually use the categories that are discussed at the time. So, so, yeah, so I, I think that this is just such a, a wonderful. And it's. Part of this, again, is you sort of, you read about this and part of this is also where you have that laborious, repetitive. Somebody says, oh, this is almost like being sentenced to jail. To be marking us to look over a thousand sets of answers to the same paper by people you don't care about as next door to penal servitude is what one observer says. And you think, yeah, that's, that's pretty right. But, but part of this also is, is Huxley is being paid for this, but underlying this is this idea that he's shaping society in a, in a very, very new way. And, and I want to really, forensically take that apart.
- [46:21]
Jacob Ward
Yeah, absolutely. I think it was, for me, a very new way to look at how, I guess scientific consensus is formed in this period of time. And that is, as you say, through eminent scientists being appointed to administer exams and then getting their own textbooks and then marking thousands of people based on how well they could reproduce their own knowledge. But you mentioned something just now about Huxley being paid for grading these exams. And payment is actually where I'd like to kind of ask my last question, really about, about the book before moving on to hear about your current project. You mentioned payment by results earlier that teachers in schools would get paid according to how many students passed or did so well on their exams. And you refer to this as part of a broader examination of what you call the political economy of Examination and I'd like to ask a question especially about a man called Robert Lowe who you call perhaps the most important figure in this book. So yeah, why is Robert Lowe so important to this book and how does that fit into the. Yeah. The economy of examinations.
- [47:33]
James Elwick
So, so Robert Lowe is this fascinating character. He is a. He is, he is, he is albino and has very, very near sighted so he can't really see very much but he's, he's a son of a reverend who championed workhouses. So very, very. If you want a Victorian who champions self improvement and industry, he, as a bit of a backstory, he is a, he goes to Oxford, he's a class, he does well in classes. I can't remember what college. Oh, modeling is at Modeling College and he is a tutor there and his rule is to get Oxford students through the class the dreaded Classics exams and he doesn't work with the really gifted students that Andy Warrick talks about in his sort of math tripos. These are students who really. How do you get them through? And he manages and he does a very, very good job at churning these people out. He then goes off to Australia for a bit. He becomes a lawyer, comes back from Australia and becomes elected with the Liberals. So he's effectively Gladstone's enforcer, so to speak. He has a sideline in writing leaders, so editorials in the Times as well. And everybody knows about this. And so they're, they're, they're quite afraid of him. And he has, in the 1850s he gets charge of this Department of Science and art. In the mid-191850s it's through the board of. It's. I forget the exact details now, it's so hard to remember but he, he effectively the, his boss, Lord Salisbury says this Department of Science and Art doesn't teach any science. You better do something about that, we're going to get rid of this. So he is asked and he works with Cole, Henry Cole, the old king. Cole, the bureaucratic reformer says how are we going to do this? And so through a series of efforts that Henry Cole comes up with to train and certify fine arts instructors, rewarding them for how well they do on art. Because Cole, when he first took over the Department of Science and Art would test, he himself would test artists and find that, well, they can't actually do any art. They can't actually, they can't actually, they're, they're terrible at this. He devises a system of rewarding people for their success on exams and then he realizes, let's transfer that to rewarding the teachers for how well their students do on exams. And then Cole, and then Cole is given these instructions by Low, we better do this for science. And so by 1859 this new system of what is called payment on results is developed effectively. So this, this tiny little system is expanded out and there's only 500 students in 1858 or nine that do this. Huxley and Tyndall are in the beginning in on the ground floor. And this works quite well. But above all the numbers explode. So it just grows and grows and grows. I don't have the numbers in front of me, but it grows from 500 to several thousand to. By the end of the century they're working with hundreds of thousands of, of, of students, excuse me, examinees. But above all, the whole point here is that if you take an exam and this changes, there's some. But to keep things simple, if you take an exam and let's say the elementary paper and, and animal physiology. So this is Huxley's paper about trachea. Tell me about the trachea. If you pass that exam, you are certified to teach other people, to coach other people in that exam. And if your students also do well on the exam, you will get money for their successes. So it's almost like Amway or multi level marketing or something like this. It's this wonderful kind of thing. And so as a result, someone like Goffin sees this and says, oh, here's this way that I can start making lots and lots of money. And so of course people are going to be rational and figure out this. So part of the irony in this that I love is that the people that devise the system decided how do we make teachers work harder and be more efficient? Well, we will reward them for exam results, but they never think about, well, what about students or what about people trying to game the system. And that's exactly what happens. And so they have to then work in this reactive way. And so most famously the system in Department of Science and Art 1862 then is taken up by Robert Lowe, who is at this point the Education Minister, Secretary or whatever it is and rules this out for effectively schooling of poor kids. National education across England. Yeah, it's just England. So that every child who goes through a government funded school has to be certified or not certified. They can, they're being tested in reading, writing and arithmetic. It's the three Rs. And so this system becomes known as payment by results. And this is what is so famous. And essentially it is there was a letter from Edward Gray to, to Cole that actually. And I couldn't find it. I was, it was driving me crazy where Cole, where, where Gray says to Cole, if you can find a way to make education funding reactive to the number of students that go through the system, you will be celebrated as the Newton of. Of government reactive funding. And I couldn't find it. It's out there, I know it's there. But effectively this is what Cole was trying to have is something that is like the invisible hand that effectively just, it's elastic. I don't want to use that term. It really automatic. It's automatic and it reacts to how many students and it just is very simple and it just works because something that low is trying to do and low is also important in a, in a side digression. He is one of the people that comes up with limited liability corporations in the 1850s. And so talk about someone who's. Yeah, that's a. He's a fairly important person. But he champions this as. And of course here is somebody who has already been a successful Oxford tutor who is judged by. What are you successful in? Well, it's churning out students who really don't care about Classics but who are certified in this. So it's a very natural step for him to just champion this payment by results for all school kids at. Across England and, and for those who are saying, well that's just England. Well, this system also then gets rolled out to Ontario, so here in Canada. So what's Canada West? No, it's still. It's Ontario by this point, Victoria, the state of Victoria and Australia, whereas anyway, Victoria so around Melbourne, but most importantly India. So this whole idea of paying students for their successes on exams is then exported to much of the British Empire. And then that larger sense of well, what do we mean by education? Well, it's just, it's doing well on exams and that makes this intuitive, extremely simple approach that we can definitely see at work even to this day. How do we, how do we judge whether a student has been successful? And to be sure, there is something, there's something to that, but it oversimplifies.
- [54:57]
Jacob Ward
Well, thank you. I mean, I think that's a nice, somewhat depressing point to finish talking about the book, but before we finish the interview as a whole, you mentioned earlier that when a system like this arises, you have people who inevitably try to gain the system. And that seems to me like a nice segue to your, your current project. So perhaps you could tell our listeners a little bit More about your current projects before we finish.
- [55:28]
James Elwick
So I, I got interested in, in just cheating. And again, you read the literature about academic integrity and it's historically shallow, shall we say. It doesn't. And it, it makes these assumptions that just don't really seem to. They seem to assume a lot of things about the individualistic epistemology. And it's, it's just. And, and so I thought, well, I. Surely we can do a little better. And so of course we can talk about AI quote unquote, large language models and, and its role in how we think about assessment now. And this is roiling the academy. I myself am going entirely to handwritten in class assessments while also allowing them to use ChatGPT and NotebookLM. I think it's a tool that's interesting, but how do we develop that ability to appreciate it in the first place? And so my larger question is, well, where does academic. Where does this idea come from? And so I said, let's look in the historical archives and let's look at cases of cheating. And so the Gothin affair I recreate in 1879. And so I am looking at. I've sort of shifted across the Atlantic to look at Canada. It's difficult as a Canadian to go to UK archives. And so wonderfully here in Ontario, at my home university, the archives of Ontario, and I look at cheating. And the best part is I'm working on a paper now where I've uncovered so effectively my role is a history of. A history of exam cheating, assessment cheating more broadly, not just mostly at a university level, but also at sort of higher levels. And then through that and thinking about what do we mean by academic integrity. And so I found this wonderful series of cases here in Ontario of examiners, excuse me, of teachers exams, so qualifying teachers exams in the 1870s, that there is these rash of cheating that starts up. So it's so teachers cheating on their exams, but it's also examiners themselves being censored for activity. So I mentioned earlier about how the examiners are putting their own textbooks on and then marking the students. And these are arithmetic papers. They're marking them on how well they do their. How well they create the. The examiner's work. But it's also. So it's again partly how teachers are being socialized into the role of writing exams. The expectation is, well, what is edu. What's good education? Well, it's doing well on exams, but you have to get people to actually be able to write the exam in the first place and get better at it but then socialize everybody else into what it is to write exams and then devise of course something that I haven't talked as much about about is about statistics and metrics. So part of this whole book is also how do you make schools places to collect statistics and how do you pre shape that data? You know, the presuming shape putting people in specialized classes writing exams under exactly the same conditions so that you can devise metrics that can then describe whether you think students are actively learning things or not. And also using exams not just as to this is to use Donald McKenzie's phrase that exams are not just cameras sort of taking a picture and a snapshot of how people learn, but they're also engines being used to drive reform and to stimulate people to work harder on their, on their exams. So anyway this is so studying exams from this perspective you see exactly the same logic occurring in Ontario which is not just relying on what they're learning, this is actually earlier than the cheating in the UK but in fact and, but they're also using payment by results but they're using it not from the, from the British but from the Australian context. So there's a lot of interesting things but again the notes are very, very similar. So that's a very long winded way of saying I want to understand cheating and to better historically interrogate that as well. What is it that we mean by someone knowing something especially as things like AI so called rise rises and why is it that we always come back to this idea that the only legitimate knowledge is that which is held in the sort of container like brain and the substance of knowledge is, is a kind of a food that we digest.
- [60:27]
Jacob Ward
Well, thank you. And I mean that sounds incredibly relevant for the current context of crisis and panic and height and anti height about AI and what it's going to do or undo when it comes to human learning. So I think we'll finish there. Jamie, thank you so much again for talking with me about your book Making a Victorian examinations and the rise of standardized testing.
- [60:51]
James Elwick
It's been a pleasure Jacob, and thank you. Thank you so much. I'm just so delighted to be doing this.