← Podcast – Cory Doctorow's craphound.com
Podcast – Cory Doctorow's craphound.com
Code is a liability (not an asset)
This week on my podcast, I read “Code is a liability (not an asset),” a recent post from my Pluralistic.net blog, about the bad ideas behind the drive to replace programmers with chatbots. Code is a liability. Code’s capabilities are assets. The goal of a tech shop is to have code whose capabilities generate more... more
Loading summary
Transcript1 lines
- [00:12]
A
Well, hey, and welcome back to the Cory Doctorow Podcast. Here's some places I'm gonna be if you're in Denver, catch me at the tattered cover on January 22nd. They are currently full up, but I believe they have a waiting list. And you can generally, if these things are full, you can kind of just turn up. Usually maybe 25% of people who get tickets don't bother showing, so there's often seats at the Last minutes worth coming I will be coming to Colorado Springs January 23rd through 25th to be the guest of honor at the Cosine Science Fiction Convention. Please come to that. I will be in Ottawa on Jan. 28 to give a talk at Perfect Books and again, that one's sold out, but I bet if you show up there will be seats. And then Tim Wu and I are giving a talk at the Tiff Theatre in Toronto on January 30th. I'll be in Victoria BC March 3rd through 5th for the 28th annual Victoria International Privacy and Security Summit. And I'll be in Berlin for Republica May 18th through 20th. I'm also doing a talk at the Other Land Bookstore while I'm in town that is still being finally confirmed. And I will be at Hay on Wye on the Welsh English border for the how the Light gets in festival May 22. 25. Lots of other talks coming. These are the ones where they've actually gotten me URLs and put up web pages. So what's going on in my life? Well, I continue to receive extremely benign treatment for my extremely treatable cancer. I'm in the midst of some immunotherapy and this week I got to see a very cool new medical gadget. When I went in for my treatment, I had a big bruise on my arm from the place where they put in the catheter last week because I had to run out of the clinic right after they pulled the needle out in order to make my flight to Las Vegas and hang out with Ed Zitron and co and make fun of CES's dumb AI products. And I didn't have time to apply the compression that stops the bruising. So anyway, I wanted it in the same spot on my left forearm so I could continue to work. And I was like, are you guys going to be able to find a vein? And they said, absolutely, we have a vein finder. So a vein finder, if you've never seen one, I had never seen one. Projects a rectangle of very bright like acid green light on your skin and that light is in a wavelength that is absorbed by something in blood. The nurse wasn't really clear. I think she said it might be plasma. And so if you can imagine a green square or green rectangle on your skin, but wherever there's a vein immediately beneath your skin that is black, there's no light. It's all being absorbed rather than being reflected. And so you have a map of the veins on your skin and when you move your arm around, like, it shows you where the veins are. Holy moly. That was cool. Go look them up. They're really interesting. So what else has gone on this week? Well, it has been a week of many deadlines. I had to write the introduction to the paperback of Insidification. I got that done. I had to write a giant speech on Canada and the Post American Internet for a conference I'm going to in Ottawa. That's the reason I'm actually going to be in Ottawa, is not to speak at that lovely store, but because the CIO of Canada has invited me to her annual summit where I'll be speaking for the CIOs and CTOs of every government ministry federally and provincially in Canada and also all the major companies in Canada. It's a very exciting opportunity. So I've been working my brains out on that. I've been writing the post American Internet. I wrote an afterword for the graphic novel of Unauthorized Bread that'll be out in early 2027. And gosh, there must have been other stuff I did on this deadline as well. It was a really intense week, I'll just say. But I got it all done. And so I am going into the next week, which admittedly has some travel in it, but with all of my recurring projects, except for my blog. And I did get five blog posts out last week, except for my blog and the Post American Internet. The thousand words. I'm doing a day on that. With all of that done, it's very exciting. And I also booked in to record the audiobook for the Reverse Centaur's Guide to Life after AI. This is my how to be a Good AI Critic book I've got coming out in the spring. You can read a long essay of mine on that subject today in the Guardian, both in the print edition and in the Sunday online edition. They adapted that from a speech I gave at the University of Washington. And so that's very cool if you want to go have a look and see how that went. The other travel I've got coming up that I thought I would mention to you that I'm quite excited about, I have to say, is my parents are coming to town and we are going to go to Santa Cruz where my daughter Posey is a freshman because she's turning 18 and I'm gonna be the father of an 18 year old, which is super cool. I'm very excited about it. It did happen faster than I expected, but I am very happy about it. I have to say. There's a lot of weird logistics we have to do, like I help her manage her bank account and now because she's 18 she has to do it. And so we're transferring some knowledge to her. But yeah, super cool. Anyway, onto today's podcast. I'm gonna read to you today a short essay I wrote. It was inspired by something I came up with when I was writing my speech for the Chaos Communications Congress in Hamburg that I gave over Christmas. There were a lot of little throwaway ideas that I was like I need to turn this into an essay later that I have been knocking off one after another in my blog in Pluralistic and so this is one of them from the January 6, 2026 edition of Pluralistic.net, this is code is a Liability and not an Asset. Code is a liability, not an asset Tech bosses don't understand this. They think AI is great because it produces 10,000 times more code than a programmer, but that just means it's producing 10,000 times more liabilities. AI is the asbestos we are shoveling into the walls of our high tech society. Code is a liability. Code's capabilities are assets. The goal of a tech shop is to have code whose capabilities generate more revenue than the costs associated with keeping that code running. For a long time, firms have nurtured a false belief that code costs less to run over time. After an initial shakedown period in which the bugs in the code are found and addressed, code ceases to need meaningful maintenance. After all, code is a machine without moving parts. It does not wear out. It doesn't even wear down. This is the thesis of Paul Mason's 2015 book Post Capitalism, a book that has aged remarkably poorly, though not perhaps as poorly as Mason's own political credibility. Code is not an infinitely reproducible machine that requires no labor inputs to operate. Rather, it is a brittle machine that requires increasingly heroic measures to keep it in good working order, and which eventually does wear out in the sense of needing a top to bottom refactoring to understand why code is a liability, you have to understand the difference between writing code and software engineering Writing code is an incredibly useful, fun and engrossing pastime. It involves breaking down complex tasks into discrete steps that are so precisely described that a computer can reliably perform them, and optimizing that performance by finding clever ways of minimizing the demands the code puts on the computer's resources, such as RAM and processor cycles. Meanwhile, software engineering is a discipline that subsumes writing code, but with a focus on the long term operations of the system the code is part of. Software engineering concerns itself with the upstream processes that generate the data the system receives. It concerns itself with the downstream processes that the system emits processed information to. It concerns itself with the adjacent systems that are receiving data from the same upstream processes and and or emitting data to the same downstream processes the system is emitting to. Writing code is about making code that runs well. Software engineering is about making code that fails well. It's about making code that is legible, whose functions can be understood by third parties who might be asked to maintain it, or might be asked to adapt the processes downstream, upstream, or adjacent to the system to keep the system from breaking. It's about making code that can be adapted, for example, when the underlying computer architecture it runs on is retired and has to be replaced either with a new kind of computer or with an emulated version of the old computer. Because that's the thing. Any non trivial code has to interact with the outside world. And the outside world isn't static, it's dynamic. The outside world busts through the assumptions made by software authors all the time. And every time it does, the software needs to be fixed. Remember Y2K? That was a day when perfectly functional code running on perfectly functional hardware would stop functioning. Not because the code changed, but because time marched on. We are 12 years away from the why 2038 problem, when 32 bit flavors of unix will all cease to work, because they too will have run out of computable seconds. These computers haven't changed, their software hasn't changed. But the world, by dint of ticking over a second at a time for 68 years, will wear through their seams and they will rupture. The existence of the world is an inescapable factor that wears out software and requires it to be rebuilt, often at enormous expense. The longer the code is in operation, the more likely it is that it will encounter the world. Take the code that devices use to report on their physical location. Originally this was used for things like billing, determining which carrier or provider's network you are using and whether you were roaming Then our mobile devices used this code to help determine your location in order to give you turn by turn directions in navigation apps. Then this code was repurposed again to help us find lost devices. This in turn became a way to locate stolen devices, a use case that sharply diverges from finding lost devices in important ways. For example, when lost locating a lost device, you don't have to contend with the possibility that a malicious actor has disabled the Find my Lost Device facility. These additional uses upstream, downstream and adjacent expose bugs in the original code that never surfaced in the earlier applications. For example, all location services must have some kind of default behavior in the very common event that they're not really sure where they are. Maybe they have a general fix. For example, they know which cellular mass they're connected to, or they know where they were the last time they got an accurate location fix. Or maybe they're totally lost. It turns out that in many cases, location apps draw a circle around all the places they could be and then set their location to the middle of that circle. That's fine if the circle is only a few feet in diameter, or if the app quickly replaces this approximation with a precise location. But what if the location is miles and miles across and the location fix never improves? What if the location for any IP address without a defined location is given as the center of the continental USA and any app that doesn't know where it is reports that it is in a house in Kansas, sending dozens of furious, occasionally armed strangers to that house, insisting that the owners are in possession of their stolen phones and tablets. You don't have to just fix this bug once, you have to fix it over and over again. In Georgia, in Texas, and in my town of Burbank, where Google's location sharing service once told us that our then 11 year old daughter, whose phone we couldn't reach, was 12 miles away on a freeway ramp in an unincorporated area of LA County. She was at a nearby park but out of range, and the app estimated her location as the center of the region it had last fixed her in. It was a rough couple of hours. The underlying code, the code that uses some once harmless default to fudge unknown locations, needs to be updated constantly because the upstream, downstream and adjacent processes connected to it are changing constantly. The longer that code sits there, the more superannuated its original behaviors become, and the more baroque, crufty and obfuscated the patches layered atop it become. Code is not an asset, it's a liability. The longer a computer system has been running, the more technical debt it represents. The more important the system is, the harder it is to bring down and completely redo. Instead, new layers of code are slathered atop it. And wherever these layers of codes meet, there are fissures in which these systems behave in ways that don't exactly match up. Worse still, when two companies are merged, their seamed, fissured IT systems are smashed together so that now there are adjacent sources of tech debt as well as upstream and downstream cracks. That's why giant companies are so susceptible to ransomware attacks. They're full of incompatible systems that have been coaxed into a facsimile of compatibility with various forms of digital silly putty string and bailing wire. They are not watertight, and they cannot be made watertight. Even if they're not taken down by hackers, they sometimes just fall over and can't be stood back up again, like when Southwest Airlines computers crashed for all of Christmas week 2022, stranding millions of travelers. Airlines are especially bad because they computerized early and can't ever shut down the old computers to replace them with new ones. This is why their apps are such dog shit, and why it's so awful that they fired their customer service personnel and require flyers to use the apps for everything. Even though apps do not work, these apps won't ever work. The reason the British Airways app displays an unknown error has occurred 40 to 80% of the time isn't just that they fired all their IT staff and outsourced to low bidders overseas. It's that sure. But also that BA's first computers ran on electromechanical valves and everything since has to be backwards compatible with a system that one of Alan Turing's protege's not out of a whole log with his very own front teeth. Code is a liability, not an asset. Bas new app is years behind schedule. Code is a liability. The servers for the Bloomberg terminals that turn Michael Bloomberg into a billionaire run on risk chips, meaning that the company is locked into using a dwindling number of specialist hardware and data center vendors, paying specialized programmers, and building brittle chains of code to connect these risk systems to their less exotic equivalents in the world. Code isn't an asset. AI can write code, but AI can't do software engineering. Software engineering is all about thinking through context. What will come before the system? What will come after it? What will sit alongside of it? How will the world change? Software engineering requires a very wide context window, the thing that AI does not and cannot have. AI has a very narrow and shallow context window and linear expansions to AI's context window require geometric expansions in the amount of computational resources the AI consumes. Writing code that works without consideration of how it will fail is a recipe for catastrophe. It is a way to create technical debt at scale. It is shoveling asbestos into the walls of our technological soc. Bosses do not know that code is a liability, not an asset. That's why they won't shut the fuck up about chatbots that shit out 10,000 times more code than any human programmer. They think they found a machine that produces assets at 10,000 times the rate of a human programmer. They haven't. They found a machine that produces liability at 10,000 times the rate of any human programmer. Maintainability isn't just a matter of hard won experience teaching you where the pitfalls lie. It also requires the cultivation of fingerspitzengefel, that fingertip feeling that lets you make reasonable guesses about where never before seen pitfalls might emerge. It's a form of process knowledge. It is ineluctable. It is not latent even in the largest corpus of code that you could use as training data. Boy, do tech bosses not get this. Take Microsoft. Their big bet right now is on agentic AI. They think that if they install spyware on your computer that captures every keystroke, every communication, every screen you see and sends it to Microsoft's cloud and give a menagerie of chatbots access to it, that you'll be able to tell your computer, book me a train to Cardiff and find that hotel that Corey mentioned last year and book me a room there, and it will do it. This is an incredibly unworkable idea. No chatbot is remotely capable of doing all these things. Something that Microsoft freely stipulates. Rather than doing this with one chatbot, Microsoft proposes to break this down among dozens of chatbots, each of which Microsoft hopes to bring up to 95% reliability. That's an utterly implausible chatbot standard in and of itself. But consider this. Probabilities are multiplicative. A system containing two processes that operate at 95% reliability has a net reliability of 90.25% or decimal 95 times. Decimal 95. Break a task down among a couple dozen 95% accurate bots, and the chance that this task will be accomplished correctly rounds to zero. Worse, Microsoft is on record as saying that they will grant the Trump administration secret access to all the data in its cloud. So as signals Meredith Whitaker and Ub Dav Tiwari put it in their incredible 39C3 talk last year in Hamburg, Microsoft is about to abolish the very idea of privacy for any data on personal and corporate computers in order to ship AI agents that cannot ever work. Meanwhile, a Microsoft exec got into trouble last December when he posted to LinkedIn announcing his intention to have AI rewrite all all of Microsoft's code. Refactoring Microsoft's code base makes lots of sense. Microsoft, like British Airways and other legacy firms, has lots of very old code that represents unsustainable tech debt. But using AI to rewrite that code is a way to start with tech debt that will only accumulate as time goes by. Now, some of you hearing this have heard software engineers extolling the incredible value of using a chatbot to write code for them. Some of you are software engineers who have found chatbots incredibly useful in writing code for you. This is a common AI paradox. Why do some people who use AI find it really helpful while others loathe it? Is it that the people who don't like AI are quote, bad at AI? Is it that the AI fans are lazy and don't care about the quality of their work? There's doubtless some of both going on. But even if you teach everyone to be an AI expert and call everyone who doesn't take pride in their work out of the sample, the paradox would still remain. The true solution to the AI paradox lays an automation theory and the concept of centaurs and reverse centaurs. In automation theory, a centaur is a person who is assisted by a machine. A reverse centaur is someone who has been conscripted into assisting a machine. If you're a software engineer who uses AI to write routine code that you have the time and experience to validate. Deploying your of fingerspitzingefuhl and process knowledge to ensure that it's fit for purpose. It's easy to see why you might find using AI when you choose to in ways you choose to at a pace you choose to go at to be useful. But if you're a software engineer who's been ordered to produce code at 10x or 100x or 10,000x your previous rate, and the only way to do that is via AI, and there is no human way that you could possibly review that code and ensure that it will not break on first contact with the world, you'll hate it. You'll hate it even more if you've been turned into the AI's accountability sink personally on the hook for the AI's mistakes. There's another way in which software engineers find AI generated code to be Incredibly helpful when that code is isolated. If you're doing a single project, say, converting one batch of files to another format just once, you don't have to worry about downstream, upstream, or adjacent processes. There aren't any. You're writing code to do something once without interacting with any other systems. A lot of coding is this kind of utility project. It's tedious, thankless, and ripe for automation. Lots of personal projects fall into this bucket. And of course, by definition, a personal project is a centaur project. No one forces you to use AI in a personal project. It's always your choice how and when you make personal use of any tool. But the fact that software engineers can sometimes make their work better with AI doesn't invalidate the fact that code is a liability, not an asset. And that AI code represents liability. Production at scale. In the story of technological unemployment, there's an idea that new technology creates new jobs, even as it makes old ones obsolete. For every blacksmith put out of work by the automobile, there's a job waiting as an auto mechanic. In the years since the AI bubble began inflating, we've heard lots of versions of this. AI would create jobs for prompt engineers, or even create jobs that we can't imagine because they won't exist until AI has changed the world beyond recognition. I wouldn't bank on getting work in a fanciful trade that literally can't be imagined, because our consciousnesses haven't been so altered by AI that they've acquired the capacity to conceptualize if these new modes of work. But if you are looking for a job that AI will definitely create by the millions, I have a suggestion. Digital asbestos removal. For if AI code, written at 10,000 times the speed of any human coder, designed to work well but not to fail gracefully, is the digital asbestos we're filling our walls with, then our descendants will spend generations digging that asbestos out of the walls. There will be plenty of work fixing the things that we broke, thanks to the most dangerous AI psychosis of all, the hallucinatory belief that writing code is the same thing as software engineering. At the rate we're going, we'll have full employment for generations of asbestos removers. All right, that is it for this week. I hope to see you in Denver and Colorado Springs. It'll be great to hang out and talk. And again, if the tatter cover says it's full, I mean, don't quote me on this, but I think if you show up, they'll probably just see you because there's always so many no shows, so if you're in Denver, don't despair. If it says that there's no tickets available, I think you can probably get in. Anyway, it's lovely to talk to you again this week. I'm gonna go put my speech notes for the speech I'm going to give in Ottawa next week in my bag so that they're safe and ready to go and bask in the warm glow of knowing that I wrote something I'm pretty happy with. Talk to you later. That was the Cory Doctorow Podcast Licensed Creative Commons Attribution Non commercial share alike 4.0 or as woody Guthrie put it in another context, this song is copyrighted in the US under seal of copyright 154085 for a period of 28 years. And anyone caught singing it without our permission will be a mighty good friend of ours cause we don't give a dern. Publish it, write it, sing it, swing to it, yodel it. We wrote it and that's all we wanted to do. Many thanks to John Taylor Williams of Ryneck Studio. That's W R Y N E C K for engineering and mastering. John Taylor Williams is a broadcast technology specialist, an audio engineer, and a musician. In his spare time he likes to carve useful objects out of wood, antler and steel.