← Cautionary Tales with Tim Harford
Loading summary
Transcript3 lines
- [00:00]
A
This is an iHeart podcast.
- [00:04]
B
Run a business and not thinking about podcasting? Think again. More Americans listen to podcasts than ad supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. Learn how podcasting can help your business. Call 844-844-IHeart.
- [00:26]
A
Pushkin Nobody really wants to be deliberately bombarded with radiation, but if you have cancer, radiation therapy might just save your life. That must have been what a young man called voin Ray Cox hoped as he made visit after visit to the East Texas cancer center. Just 33 years old, Ray to his friends, was young to be a cancer survivor. He'd had a tumour cut out of his shoulder, and now, March 1986, he was there for his ninth session of radiation therapy, designed to ensure that no traces of the cancer remained. Despite his bad luck, Ray was a cheerful, resilient man. He knew the drill. Press his bare chest and stomach onto the cold metal treatment table, chat to the operator while she manoeuvred him into position underneath the looming bulk of the Ferac 25 radiation therapy machine. The operator knew the drill, too. In his account of the case, the ergonomics expert Stephen Casey called, calls her Mary Beth, although that's not her real name. We'll do the same. Marybeth cheerfully caught up with Ray as she used a console control to precisely position him under the Therac 25's radiation beam gun. Then she walked down the corridor to the control room, which was at a safe distance. Ordinarily, the control room and the treatment room would be linked up by CCTV and microphones, but neither the cameras nor the audio link were connected that day, and that didn't seem to matter. Normally they'd be useful for some reassuring chat or to give the patient a word of instruction. And Ray, being an old hand, didn't need any of that. Mary Beth typed the treatment instructions into the computer, a series of letters. She pressed fast X to choose the mode, then straight away realized her mistake. Ray needed the other mode. She deleted the X and pressed E. She checked the instructions. They were all correct. Beam ready, the computer told her. She pressed B to administer the treatment. Down the hall on the treatment table, Ray Cox heard a sizzling sound and saw a blue flash and then agony. It was like someone had thrust a hot skewer through his shoulder. This wasn't right. He knew it couldn't be right. The last eight treatments had been nothing like this. Back in the control room, Mary Beth couldn't hear Ray's cry of pain and she couldn't see his body contorting on the treatment table. All she saw was a bland Little notification malfunction 54. It wasn't clear what that meant. The machine would often pause and produce an unexplained error code, sometimes 30, 40, 50 times a day. As one operator later commented, I can't remember all the reasons it would stop, but there were a lot of them. The machine indicated that Ray had received only a tiny fraction of the intended dose. And Marybeth had been assured that the Therac 25 had so many safeguards it was almost impossible to overdose a patient. It only took a single key press for her to reset the machine and try again. I'm Tim Harford and you're listening to Cautionary Tales. Mary Beth was an experienced operator of the Therac 25. She must have seen it crash and pop up an error message 10,000 times or more. Although she didn't know what those error messages meant. How could she? The machine's manual didn't explain them. It didn't even mention them. And because she'd been in that position so many times, it took her mere seconds to reset the machine for another try at giving Ray his treatment. That wasn't enough time for Ray to get off the table. He'd rolled onto his side. But Marybeth, of course, couldn't see that, and she couldn't hear his agonised yelling. The machine fired again. Another flash of blue light, another sizzle. And this time a hot skewer went through Ray's neck. He was in too much pain even to scream. Then the agony started to fade. He gulped in some air and blew it out again, tried to calm himself. Hey, are you pushing the wrong button? But Mary Beth couldn't hear him. Something had gone wrong with the Therac 25. But what loyal listeners to Cautionary Tales will be familiar with the Swiss cheese model of accidents made famous by the psychologist James Reason. Imagine slices of Swiss cheese with those distinctive holes in them. Each slice represents some kind of safeguard against an accident. Maybe it's a failsafe in the hardware of a system. So it simply won't work if the right pieces aren't in position. Maybe it's a subroutine in the software, monitoring what the system's doing and shutting it down if another part of the software happens to glitch. But no safeguard is perfect. Every slice of cheese has holes in it. In James Reason's model, an accident becomes possible when all the holes line up. That means that every safeguard in the system becomes simultaneously vulnerable to the same kind of problem. To prevent accidents, then get extra lines of defence and try to strengthen the defences you already have. To put it another way, get more slices of cheese with fewer holes in them. So did the therac 25 need better hardware or better software to prevent the accident that happened to Ray Cox? As it turns out, yes. But that's also the wrong question. We should be looking at a different kind of cheese slice altogether. Nine months before Ray Cox's excruciating experience, in July 1985, 40 year old Frances Hill arrived at the Ontario Cancer foundation clinic for her 24th round of radiation treatment for cervical cancer. The clinic was using a Therac 25 machine, but there was a problem. The machine didn't seem to be working. Every time the operator tried to fire the radiation beam, the machine paused, produced an error message and reported that no dose had been given. The operator hit the P key to proceed and the same thing happened. The operator hit P again and it happened again. We've all been there. Clicking an icon on a screen, finding that nothing seems to happen, and then clicking it again. After four attempts, the operator called a technician who couldn't find anything Wrong with a Therac 25. Francis Hill left and the machine was used successfully on half a dozen other patients that afternoon. That sort of thing wasn't particularly strange, the operator reflected. The Therac 25 would often seem to glitch like that, producing mysterious error messages and then suddenly working again for no particular reason. But while glitches didn't seem strange to the operator, something seemed strange to Frances. She could feel a kind of burning, tingling, electric shock kind of sensation in her hip, near where the therapeutic beam had been aimed. When she came back for another round of paper treatment three days later, her doctors immediately diagnosed a radiation burn in her hip, which was painfully swollen. They called the machine's manufacturers to report a suspected radiation overdose. The manufacturers were Atomic Energy of Canada Ltd. AECL. A radiation overdose. AECL had never heard of anything like that before. Strange. They sent an engineer along to investigate. The Therac 25 could be used in two different modes. The electron mode attacked cancer near the surface of the patient's body. The machine emitted a beam of electrons spread out by an array of magnets. The X ray mode attacked cancer deep inside a patient's body. The magnet array would be moved aside and replaced by a device called a flattener, which focused the X ray beam precisely on the cancer. The flattener absorbed a lot of Energy, which meant the X ray beam had to be very powerful. The components could which diffused the electron beams or focused the X rays on the Therac 25 were positioned on a turntable. As the machine was programmed to fire either electrons or X rays at the patient's tumor, the turntable would rotate automatically to fix the right component into position. At least that was the idea. Two types of radiation beam, then. One that needs diffusing, one that needs focusing. If this sounds like an accident waiting to happen, well, it was. But machines like the Therac 25 are expensive. And this dual purpose design meant that hospitals got more bang for the buck. As long as the right component was in place for the right beam, there would be no problem. The Therac 25 was fully controlled by a computer, unremarkable these days, but radical for the mid-1980s. Its predecessors, the Therac 6 and the Therac 20, allowed a human operator to physically position the magnets or the flattener on the Therac 25. This manual positioning was replaced by servomotors, computer controlled to quickly and precisely put everything in position. When AECL investigated the incident with Francis Hill, they weren't actually able to reproduce the error, but they suspected that the turntable system hadn't worked properly. The turntable had three tiny switches designed to measure when it was in position. But it emerged that a single bit of error, the computer glitching and mistaking a 0 for a 1, could produce a faulty reading of the turntable's position. So AECL told the clinics that used the Therac 25 to visually confirm before each procedure that the turntable was in the correct position. Just as a precaution until further notice, they tightened up the software, making it more robust to a small error like that. Then they got back in touch with the clinics. No need for those visual checks anymore. We've just made the machine five orders of magnitude safer, which in plain English means it's about 100,000 times safer than before. And it was safe already. But one thing AECL don't seem to have done is to have notified the clinics that an accident had happened and that a patient had been injured by the machine. The clinics at least say they weren't told of any injuries. When AECL announced that they'd fixed the problem, it was September 1985. A month later, they were sued by a woman named Katie Yarborough. AECL had never heard of Katie Yarborough. Who was Katie Yarborough? Cautionary tales will return after the break. Katie Yarborough's injury happened seven weeks before Frances Hills, early in the summer of 1985. She was being treated at the Keniston Oncology center in Marietta, Georgia. Katie was 61. She'd had a malignant tumor removed from her breast and now she needed follow up treatment to destroy any secondary tumours which might have spread to the lymph nodes under her collarbone. That treatment, of course, would be provided by a Therac 25 machine. But when the technician fired up the Therac 25, Katie felt an agonizing pain, a tremendous force of heat, this red hot sensation. You burned me. Katie exclaimed to the technician, who was puzzled. That shouldn't be possible. The technician said the Therac 25 was safe and there wasn't any sign of a burn. Perhaps Katie's clavicle was a little warm to the touch, but otherwise nothing seemed to be amiss. The physicist at the Keniston Centre was a man called Tim Still. When he was informed, he was just as puzzled. Tim knew about the two treatment modes. The powerful X ray fired through the flattener and the gentler electron beam fired through the magnet array. Katie Yarborough had been treated in the electron mode. But Tim still wondered if something had gone wrong with the array. He called the manufacturers AECL with a question. Was there any way the electron beam could be fired directly at a patient without the magnet array in position? After three days, AECL replied no. They explained, that was simply impossible. But somewhere along the line, AECL didn't seem to get the message that a patient had been injured. Maybe Tim still didn't tell them, or maybe he did. But the message didn't get through to the key decision makers. It's quite possible that still didn't even realise that Katie Yarborough was injured after all. At first, she seemed fine. But Katie Yarborough wasn't recovering from that mysterious burn. In fact, her symptoms were getting worse. The skin above her left breast had reddened, her shoulder would freeze up and she suffered excruciating spasms. Her doctors were baffled. They continued sending her for THERAC treatment. After all, with malignant breast cancer, you can't afford to take risks. But when her skin started to fall off, Katie refused to continue lying on the treatment table underneath that machine. And when Tim still later examined Katie, he noticed something strange. Not only did Katie seem to have a severe burn to her upper chest, but her upper back was starting to redden too. It was as though whatever had burned her had passed right through her body and caused an exit wound. When Frances Hill was Injured her clinicians didn't know about the injury to Katie Yarborough. And as hospitals across North America continued to use the Therac 25, they didn't know about Frances Hill or Katie Yarborough. They only knew that the machine was already safe and had just become 100,000 times safer. And that makes what happened next almost inevitable. In December 1985, Dora Moss, a patient at the Yakima Valley Memorial Hospital in Washington state, complained that her right hip seemed red and inflamed in a distinctive striped pattern. Dora's doctors were puzzled. It wasn't clear what could have caused the inflammation, although possibly it was a perfectly normal reaction to the course of radiation therapy she was having on her hip. Which device was being used? Funny you should ask. It was a Therac 25. But because the hospital staff at Yakima weren't aware of the history of accidents and they were baffled, some of them wondered whether a slotted component on the Therac 25 might explain the striped pattern. Others suspected that it was a burn caused by Dora's habit of sleeping with an electric heating pad. Maybe those heated wires had slowly burned her skin. Although on closer inspection, the arrangement of wires in the heating pad was. Didn't actually match the sore stripes on Dora's hip. So they contacted aecl, who responded after careful consideration. We are of the opinion that this damage could not have been produced by any malfunction of the therac 25 or by any operator error. So that was it then. Officially, the cause of that stripy burn was cause unknown. But if the cause was unclear, the consequences were stark. Dora Moss needed surgery and skin grafts to patch up her ulcerated skin and treat her chronic pain. Maybe it wasn't the Therac 25. It certainly wasn't a burn from a heating pad. The Yakima hospital staff were even told that there'd been no other incidents with the Therac 25. Was anyone putting all these incidents together and spotting a pattern? It seems not, although it's hard to be sure. Nancy Levison, software safety expert and the author of a definitive account of the affair, explains that because there was never an official investigation, it's often unclear who exactly knew what and when. They knew it. At the East Texas cancer center in March 1986, three months after the injury to Dora Moss, Mary Beth was puzzled. She'd tried twice to administer the trouble treatment to Ray Cox, apparently without success. Third time lucky, she hit p again. Ray Cox had been trying to ease himself off the table, but when that searing skewer feeling hit him for a third time. Jabbing through his neck and shoulder. He leapt for safety, barged open the door and ran to the nurses station. When Marybeth emerged, Ray was obviously shaken by what had happened. He told her that he felt like he'd been given three separate powerful electric shocks. How strange. Marybeth reassured him that the machine had automatically shut down and according to the computer's display panel, Ray had only received one tenth of the intended dose. Marybeth informed Ray's doctor and the Centre's physicist, Fritz Hager, about the electric shocks. They came to examine the machine and Ray. There seemed to be nothing wrong with either of them, but that's the nature of a radiation overdose. It's invisible. And at first the injuries it causes are invisible too. Ray looked fine, but he really wasn't. Hager called the manufacturer of the machine, aecl, to report the incident. Then he ran through some tests and since everything seemed to be in order, pronounced the machine good to go for the afternoon. Patients who were waiting for treatment and everything went smoothly. That's what happened. Remember, the Therac 25 often produced mysterious error messages and then suddenly started to work again for no obvious reason. Three weeks after the strange incident with Ray Cox, Marybeth had a new patient, 66 year old Vernon Kidd, who had a tumour on his ear. As soon as the treatment beam was activated, Vernon cried out and started moaning for help. This time the audio link was working. What happened? Asked Mary Beth. Fire, he replied. Fire on the side of his face. When the physicist Fritz Hager arrived on the scene, Vernon elaborated. He'd heard a sound like frying eggs and a flash of light and then pain. He was confused and upset. What happened to me? We'll find out what happened to Vernon Kidd after the break. September 9, 1947. Computer scientists at Harvard University get to the bottom of why their fancy computer, the Mark 2, is malfunctioning. It's a bug, a literal bug. A moth in fact, which has crawled into the mass of electrical relays in the room sized computing machine and caused a short circuit. The logbook tells the tale. Handwritten on blue gridded paper are the words relay 70 panel F moth in relay. Next to those words, the bug itself is preserved under a short length of yellowing sticky tape underneath the dry remark. First actual case of bug being found. Software bugs are the bane of programmers. Although as that punchline implies, this wasn't the first time the word bug bug had been used to describe a device malfunctioning. It was just the first time that an insect had been obliging enough to turn the metaphor into reality. In fact, software bugs are older than computers. The first computer program is widely thought to have been written by Ada Lovelace, an English mathematician and friend of the engineer Charles Babbage. In 1843, Lovelace published an algorithm that would enable Babbage's protocomputer, the analytical engine, to calculate a particular sequence of numbers. What's so striking about Lovelace's algorithm is that the analytical engine had not been built, nor would it ever be. Babbage's designs were just too ambitious and nobody would be able to construct a general purpose computer for for another century. But modern analysis concludes that if Lovelace's program ever had been run on an analytical engine, it wouldn't have worked. Not first time anyway, because there was a typo. Not only had Lovelace published the first software, she had also published the first software bug. Anyone who's had the experience of writing computer code, or even for Gen Xers like me, of typing in a computer program printed in the pages of a magazine, will know what it's like to have a bug. In some cases, the bugs are easily fixed. You run the program and it doesn't work. Maybe the computer even tells you where things went off the rails, exactly where the error was. Or maybe not, because some bugs are more like an unsuspected hole in one of James Reason's slices of cheese. They sit there hidden by other slices, causing no trouble until the holes in the slices align and the computer crashes. You may not know why it crashed. You may be able to restart the program and find it runs with no trouble. But somewhere under the surface, the bug is still there. Two of Fritz Hager's patients had apparently been electrocuted. And while the Therac 25 seemed to be working perfectly well, he wasn't going to risk a third. He shut the machine down, notified aecl, and then tried to figure out what had occurred. It wasn't easy. After each incident, the machine seemed to be working just fine. Hager and Mary Beth worked together, trying to figure out what had triggered the malfunction 54 message. Eventually, they succeeded. Mary Beth recalled how she'd originally typed X to use the Therac 25 in X ray mode, then realized she should have typed E quickly. Using the cursor keys to move to the correct box, she corrected the entry to E, repeatedly hit return to accept all the other treatment variables which were correct, and awaited the message beam. Ready. What she hadn't known, what nobody had known Is that a quick edit like that confused the computer's subroutines, which checked the setup only at certain moments. The result? The beam was set to full X ray power. But the flattener that would absorb most of the radiation wasn't in position. To make matters even worse, the Therax software was confused by this dangerous setup and didn't correctly monitor the dose administered. It was a particular sequence of keystrokes that caused the problem. An unlikely sequence, but not an inconceivable one. It should. It shouldn't be particularly surprising that experienced operators such as Mary Beth might type the wrong mode, notice the error, then swiftly correct it. It was the swiftness of that correction that bewildered the software. Fritz Hager explained to AECL that he could now reproduce the error on demand. The AECL engineer couldn't until Heger explained that all the keystrokes had to be entered in less than eight seconds. The next day, the AECL engineer called back. Yes, he could now replicate the error. And he had bad news. If the beam was fired in such conditions, the patient would receive a dose of 25,000 rads, more than 100 times more than intended, which potentially could be fatal. Over the course of three weeks, 66 year old Vernon Kidd moved from disorientation to a coma to death. The autopsy revealed that the section of his brain running from under his right temple to behind his right ear had been withered by a high dose of radiation. It's natural to describe the problem with a Therac 25 as a software bug. And while that's true, it doesn't really help us understand the problem or prevent similar problems in future. As Nancy Levison writes, virtually all complex software can be made to behave in an unexpected fashion under some conditions. Demanding software with no bugs is like demanding a slice of Swiss cheese with no holes. It's in the nature of Swiss cheese that there will always be holes, and it's in the nature of complex software that there will always be bugs. The question is, what happens when a bug appears? Perhaps another part of the software is able to spot the problem. A separate slice of software cheese. That didn't happen. With a therac 25. The computer would tell the operator what dose the patient had received, but there was no direct measurement of that dose. The bug that led to the overdose also led to the software failing to report the overdose. Or perhaps there are fail safes in the hardware. Again, not with the Therac 25. The machine relied on the software being perfect. The shielding components were put in place by the software. The decision to allow the electron gun to fire or not was made by the software and the dose the patient received was reported by the software. If the software was wrong about one of these things, it could easily be wrong about the others. There is a different approach. The Therac 20, the predecessor of the Therac 25 was designed to operate with or without a computer and it was built with mechanical interlocks. If you tried to fire the beam without the right component in place, the machine just wouldn't do. Was only after the Therac 25's problems became widely known that something began to dawn on Therac20 users. Sometimes the machine's fuse would blow after quick edits and was annoying and a bit mysterious. The machine would have to be switched off, the fuse replaced and everything restarted. The Therac 20 software had been built on the same code base as the Therac 25. And on closer examination it became clear that the Therac 20 was had the same software bug. Because of the mechanical interlocks which would physically prevent the machine from working unless it was correctly set up. The bug was never anything more than an annoyance. The most precious thing that was damaged was a fuse. The Therac 25 needed better software and it needed better hardware. But as I said earlier, that's also the wrong place to focus. Another kind of cheese slice was missing. There was no proper process for noting anomalous incidents, suspected malfunctions or possible injuries. Hospitals should have been reporting both the strange incidents and the later mysterious injuries. Someone, either AECL or a regulator, should have been collecting and analyzing the reports because that didn't happen. Every new incident caused a new wave of muddle and bewilderment. Of course we should try hard to eliminate bugs, especially when lives are at stake. But the real lesson here is that safety is not a function of good software alone. It's the function of the whole system. And the system goes beyond the software. In fact, it goes beyond the machine. The system includes the network of people who make the machine, use the machine and regulate the machine. In this case, they should have been keeping each other closely informed. They weren't. And as a result, it took months for anyone to assemble the pieces of this awful puzzle. Voin Ray Cox was young and strong, but he'd taken a triple blast of high dose radiation to his back, shoulder and neck. Before long, he was starting to spit up blood. The awful radiation burns on his back and neck turned into yawning lesions as the skin and flesh started to peel off his body. While over the weeks that followed the damage to his spinal column paralyzed his left arm, both legs and his vocal cords. Ray tried to keep his sense of humour before he lost his ability to speak. He would joke to friends and family. Captain Kirk forgot to put the machine on stun. Ray cox died in August 1986, a few months after the accident and five months after that. Glenn Dodd, a 65 year old cancer sufferer, was given radiation therapy at Yakima Valley Memorial Hospital, where Dora Moss had acquired those mysterious striped radiation burns after treatment from a Therac 25. Glenn Dodd was being treated by a Therac 25. He received a fatal overdose. Dodd was terminally ill, but doctors concluded that the injuries he'd suffered from the malfunctioning machine were had killed him before his own cancer could. Why hadn't AECL fixed the software fault? Well, as it happens, Glenn Dodd wasn't killed by the glitch that had killed Ray Cox and Vernon Kidd, but by a different software error. You see, there's always another bug lying in wait for the moment to strike. This isn't the first cautionary tale we've done about radiation overdoses. Two of my very favourite episodes of the podcast were Glowing Peril, the Magical Glitter that Poisoned a City, and how the Radium Girls Fought Back, two unforgettable episodes which came out as a pair late in 2023. You may enjoy listening to them. I first heard about Ray Cox's case from Stephen M. Casey's book Set Phasers on Stun, while the definitive Authority on the Therac 25 case is Nancy Levison's investigation, written with Clark Turner. For a full list of our sources, see the show notes@timharford.com Cautionary Tales is written by me, Tim Harford with Andrew Wright. The show is produced by Alice Fiennes with Marilyn Rust. The sound design and original music are the work of Paul Pascal Wise. Sarah Nix edited the script. Cautionary Tales features the voice talents of Ben Crow, Melanie Gutteridge, Stella Harford, Gemma Saunders and Rufus Wright. The show wouldn't have been possible without the work of Jacob Weisberg, Ryan Dilly, Greta Cohn, Eric Sandler, Carrie Brodie, Christina Sullivan, Keira Posey and Owen Miller. Cautionary Tales is a production of Pushkin Industries. It's recorded at Wardour Studios in London by Tom Berry. If you like the show, please remember to share, rate and review. It does really make a difference to us. And if you want to hear the show ad free, sign up to Pushkin plus on the show page on Apple Podcasts or at Pushkin fm. Slash plus, this is an I heart podcast.