When Schiphol Airport shredded 400 squirrels (and why no one was to blame)

Summary

---

Podcast Summary: Nudge – "When Schiphol Airport Shredded 400 Squirrels (and Why No One Was to Blame)"

Host: Phill Agnew
Guest: Dan Davies (Author, "The Unaccountability Machine")

---

Episode Overview

In this episode of Nudge, Phill Agnew and Dan Davies explore the real-life tragedy of 400 pet ground squirrels that were destroyed at Schiphol Airport in 1999 due to missing paperwork. Instead of focusing solely on this shocking event, the episode delves into the broader theme of how large organizations create "unaccountability machines": systems that allow decisions to be made without any individual bearing responsibility. The discussion unpacks why these systems emerge, how they operate, and the unintended, sometimes disastrous, outcomes they create across industries—from aviation to marketing, media, and education.

---

Key Discussion Points & Insights

1. The Schiphol Squirrel Incident: A Case Study in Unaccountability

• [00:00–08:22]
  • Background: In 1999, 400 pet ground squirrels were imported into Europe, but missing paperwork meant they couldn’t travel onward or be returned. Schiphol Airport authorities disposed of them using an industrial shredder.
  • The case caused international outrage, but ultimately, no one was held accountable.
  • Phil Agnew: “KLM absolved itself from responsibility while still managing to show a bit of ethical remorse. They seemed saddened, but not responsible.” [07:49]
  • Dan Davies: “No one really seems to have done anything wrong… Everyone was just following rules. Everyone was doing what they were told to do.” [08:22]
  • The KLM press release became a business school example of effective crisis PR that acknowledges ethical regret without accepting culpability.

2. How Large Organizations Create “Unaccountability Machines”

• [08:23–13:34]

  • Dan Davies defines "unaccountability machines" as systems where responsibility is so diffused that no one can be blamed when things go wrong.
  • Employees act as cogs, bound by protocols, unable to deviate even when outcomes are clearly wrong or harmful.
  • Davies: “You can create a system in which nobody is really making the decisions… Decisions have been made a very long way away, often just coming out of an algorithm or a computer.” [09:49]

• Accountability Sinks:

  • Illustrated through travel/airline experiences where frontline staff lack both power and agency to address customer issues, leading to frustration and a sense that one is battling a ‘faceless system.’
  • Davies: “There is no individual human being who is accountable for it… That’s something you really need to analyze...” [12:14]

3. Systemic Failure Across Industries

• [13:35–19:15]
  • Fox News & Dominion Lawsuit:
    • Fox News lost a record libel lawsuit for airing baseless conspiracy theories about voting machines.
    • Organizational incentives and the desire to maintain audience ratings perpetuated a system where staff acted against their better judgment.
    • Davies: “Bad ratings make good journalists do bad things.” [15:58]
    • Davies: "They’d created this group of people who were angry...and then at times, time came when Fox found itself having to tell them something they didn’t want to. And they’d created this system that they couldn't control anymore." [16:26]
  • Old Systems in a New World:
    • Many damaging companies are merely outdated systems operating in a changed reality (e.g., oil companies struggling with climate change, old management paradigms in the face of shifting evidence).

4. When System Logic Defeats Purpose

• [19:16–23:44]
  • Boeing Case Study:
    • Post-merger, Boeing’s culture shifted from engineering-led to finance-led, sacrificing safety and quality for short-term profit.
    • Led to the 737 Max crisis—two crashes, 346 deaths, and ongoing reputational and financial damage.
    • Davies: “Each one of those decisions might have looked great on a spreadsheet, but eventually...resulted in Boeing putting into production an aircraft with a fundamental design flaw...Boeing still hasn’t recovered.” [21:19]
  • The pattern: financial logic overrides core mission, resulting in self-destruction.

5. Marketing & Brand Missteps from Following Broken Systems

• [22:24–25:29]
  • Burberry’s Check Pattern:
    • Overexposed their luxury check design through mass-market items, undermining their high-end brand positioning for short-term gains.
    • Davies: “You’ve done serious damage to your ability to position that as a high-end luxury brand.” [22:55]
  • Agnew relates this to exclusive clubs (e.g., The Ivy, Groucho) suffering when system incentives undermine original appeal.

6. Other Systemic Incentive Problems: Football, Education & Ratings

• [25:30–31:48]
  • Football Clubs:
    • Chasing glory at all costs leads to unsustainable debt cycles and collapse (chasing winning, not financial health).
  • UK Education Appeals System:
    • Designed to be unbiased, the appeals process inadvertently amplified social inequality, benefiting those with more resources and savvy.
    • Davies: “The combination of an unbiased exam system and an unbiased appeal system creates an extremely biased system...” [28:35]
  • Online Reviews & Goodhart’s Law:
    • Incentivizing good reviews distorts the original measure's purpose.
    • Davies: “If you’re using that measure as part of someone else’s reward system, then you have to take into account the feedback...” [30:59]

7. Artificial Intelligence: Promise, Peril, and Systemic Risk

• [31:48–34:34]
  • AI is a tool: it can increase the efficiency of both “problem solvers” and “problem creators.”
  • Davies: “It improves the productivity of people who create problems as well as ...solutions.” [32:10]
  • Drawing from Stafford Beer, Dan worries that AI (like computers before) will be wasted on optimizing trivial processes (e.g., sending more emails), rather than fundamentally rethinking and improving organizational systems.

---

Notable Quotes & Memorable Moments

• On the human tendency to oversimplify:
  • Dan Davies: "The only people in the world who try to pay attention to every single thing that's happening around them are babies....Being an adult and functioning in the world is always a matter of trying to simplify things.” [05:33]
• On accountability in faceless systems:
  • Davies: “There’s no individual human being who’s done that...when you see that kind of social institutional arrangement in place, it’s often interesting.” [12:14]
• On organizations outliving their logic:
  • Davies: "Many of the most damaging companies today are simply old companies with systems that made perfect sense 50, 20 or 100 years ago, but now function in a very different world." [17:59]
• On Goodhart’s Law and measurement:
  • Davies: “If you’re using that measure as part of someone else’s reward system, then you have to take into account the feedback from that to what you’re measuring.” [30:59]
• On AI and systemic risk:
  • Davies: “With AI, we’ve got a technology that…could be used to help us process more information, to help us make better decisions. It’s also possible for it to create chaos…What worries me is we’re using it to send more emails.” [32:10–34:09]

---

Important Timestamps

• 00:00–08:22: Schiphol squirrels story, PR, and organizational absolution
• 08:23–13:34: How “unaccountability machines” work in practice and travel
• 13:35–19:15: Fox News, Dominion lawsuit, consequences of system incentives
• 19:16–25:29: Boeing, cultural shifts, and failures of system-first logic
• 25:30–31:48: Marketing/brand blunders, football club debt, education appeals bias, review systems and measurement distortion
• 31:48–34:34: AI’s dual potential; historical context of organizational underuse of technology

---

Conclusion

Phill Agnew and Dan Davies draw powerful parallels between infamous organizational failures—whether the tragic destruction of squirrels, TV network scandals, or the downfall of trusted brands and companies. They show that the root cause is rarely individual malice, but instead the rise of complex, self-perpetuating systems that absorb accountability, incentivize the wrong actions, and respond poorly to unexpected events.

The episode ends with a sober view on technology like AI: its danger is not that it will take over, but rather that organizations will once again squander its potential by using it as a crutch to entrench old, broken systems.

---

For more insights, Dan Davies’ "The Unaccountability Machine" is recommended—find the link in the show notes.

Transcript

[Phil Agnew]

In 1999, a consignment of 440 ground squirrels left Beijing and China, bound for Athens in Greece. En route, their KLM flight stopped at Schiphol Airport in Amsterdam. Here, the 400 or so pet squirrels faced a problem. They didn't have the right paperwork. Usually missing paperwork results on a slap on the wrist for the sender or, in the worst case scenario, sending the package back. But here, something much worse happened.

[Dan Davies]

It doesn't end well for the squirrels. I want everyone to be prepared for that. All 400 of these ground squirrels, apart from one or two who had escaped, got thrown into an industrial shredder.

[Phil Agnew]

The story sparked international outrage. Mieke Holstag from the squirrel refuge was outraged, saying it made him nauseous. The case was even debated in Dutch parliament. And yet, strangely, no one was to blame. The airline avoided any penalty. The employees were suspended, but almost immediately returned to work on full pay. The squirrel sender was never reprimanded and the customers were compensated. Today's episode of Nudge covers how cases like this can happen, how large organisations create unaccountability machines that absolve them from wrongdoing and allow corporations to get away with far more than individuals ever could. This is happening around the world in thousands of companies, and it's happening more than it ever has done before. Hear all of that on today's episode of Nudge. HubSpot makes impossible growth seem easy for some of their customers. And there is a perfect example. It is Morehouse College. This is a college in Atlanta in America, and like most organizations that have been around for, you know, decades, they had a huge amount of content on their website. 900 different pages and even the tiniest of updates took 30 minutes for them to publish. And yet they needed to reach new students with fresh, engaging content. So they used Breeze, HubSpot's collection of AI tools. This helped them write new content, optimize their content in a fraction of time, and essentially create results that really worked. They got 30% more page views and their visitors now spend 27% more time on their site because they are creating content that people really care about. So if you feel like growth is impossible, it might be worth reaching out to HubSpot. Go to HubSpot.com Today's guest wrote the Unaccountability Machine. The book was long listed for the Financial Times Book of the Year and it won the New Statesman's Book of the Year as well.

[Dan Davies]

Hi, I'm Dan Davis. I am a former economist and current writer about systems that I find interesting basically in the bits of the world and the bits of the economy that conventional economics leaves out.

[Phil Agnew]

During his time at the bank of England, Dan noticed something odd about his fellow economists.

[Dan Davies]

A lot of economists just work on this assumption of either perfect information or perfect rationality. And they create a world in which only prices and quantities exist.

[Phil Agnew]

This isn't the real world. The real world is full of unknowns and unquantifiable outcomes and what Dan calls partial information.

[Dan Davies]

But actually a whole lot of the rest of the world just really runs on gaps in information, partial information and the way that people try to deal with their partial information and simplify the world to make it comprehensible and manageable for themselves. And one of those things, of course, is, you know, the whole art and science of marketing, because it is, and this is something that really always dissatisfied me when I was doing economics, that you've got huge industries in marketing, advertising and media. And economics has basically nothing to say about any of them because they're all operating in that world of informing and persuading the consumer. And you can't really talk about that if you're. Science is set up on the basic assumption that the consumer already has perfect information and only kind of needs to be given an offer of a price in a quantity.

[Phil Agnew]

Take Red Bull. If consumers were entirely rational, there would be no need for Red Bull's tagline, Red Bull gives you wings. Because of course it doesn't. It's irrational to believe that it would. Red Bull wouldn't need to sponsor extreme sports or football clubs. Neither have anything to do with what energy drink you should buy. And yet everybody listening that these sponsorships and ad slogans add some maybe intangible value, but genuine value to the drink. Red Bull reveals that we will never be able to quantify everything, that we'll always have partial information. And that's a problem for businesses who try to quantify everything.

[Dan Davies]

In a lot of my work, I talk about systems which oversimplify the information around them. A lot of the time when you kind of reach something that's gone very wrong or gone very badly, it' because someone wasn't paying attention to a particular piece of information. They were working with a model of the world that was wrong or partial or basically oversimplified.

[Phil Agnew]

And oversimplification is human nature.

[Dan Davies]

The only people in the world who try to pay attention to every single thing that's happening around them are babies. And they tend to get pretty tired and cranky pretty quickly from doing that and just being an Adult and functioning in the world is always a matter of trying to simplify things, trying to work on the basis of partial information and a subset of all the information that's out there.

[Phil Agnew]

So let's bring this Back to the 400 squirrels. How did partial information and oversimplification lead to this tragedy?

[Dan Davies]

These poor little squirrels. It doesn't end well for the squirrels. I want everyone to be prepared for that. Around the turn of the century, there was a brief craze for keeping ground squirrels as pets. And so they were kind of being imported from breeders in Asia and sold as pets in Europe. In, I think, around the late 1990s, a consignment of ground squirrels arrived at the cargo centre in Schiphol Airports in Amsterdam. And they didn't have the right paperwork, didn't have the kind of biosecurity, safety and veterinary forms that would have enabled them to be sent on to the eu. But there was also no paperwork explaining where you could send them back. So you couldn't send them back to Asia, you couldn't send them on to the final customer. What do you do with them? You can't keep them in Schiphol Airport indefinitely. It's a real problem. It has to be said that there was probably had to be a better solution than the one that KLM Cargo actually found, which was that all 400 of these ground squirrels, apart from one or two who had escaped, got thrown into an industrial shredder that they had hanging around in the cargo department for shredding things. This was bad. The news got out that they'd done this to the squirrels. Questions were asked in the Dutch Parliament. The reason I found out about this is that when I was at business school, the apology press release that KLM put out apologizing for what they had done is still studied in business schools as a really great example of crisis pr.

[Phil Agnew]

It's studied because it absolved KLM from any responsibility. A BBC News article published back in 1999 Quotes the PR response. The article states how a spokesperson for the airline said, though on formal grounds the action taken was correct, KLM admits to having made a grave mistake on ethical grounds. The spokesperson goes on to say the shredder was the most humane way to destroy animals. KLM absolved itself from responsibility while still managing to show a bit of ethical remorse. They seemed saddened, but not responsible.

[Dan Davies]

If you actually look into how this disaster happened, no one really seems to have done anything wrong is the strange fact the European Union had passed a set of sensible legislation on the Importing of rodents from other parts of the world, you know, which is something that has to be controlled. You know, you really can't be importing non native animals without plenty of controls over doing that. The Dutch Ministry of Agriculture had turned those into legislation. And part of that legislation said that imported animals which could not be set on would have to be destroyed or euthanized at the expense of the importer. KLM had implemented that Dutch legislation into its own policy. And all the way down, everyone was just following rules. Everyone was doing what they were told to do. The only protection anyone had against something like 400 lovable, cuddly little rodents meeting a horrible death was if someone right at the bottom of the chain decided, hang on, this is horrible, we can't be doing that. And that's just, it's just implausible that that could happen. At the end of the day, if you work in a shed, you are not usually allowed to question orders that come directly from the government.

[Phil Agnew]

Dan calls these systems unaccountability machines.

[Dan Davies]

You can create a system in which nobody is really making the decisions. You have kind of decisions taken that don't seem to have any identifiable human being as their owner.

[Phil Agnew]

These unaccountability machines are far more common than you might expect. In fact, many of you have faced an unaccountability machine probably while going on holiday this summer.

[Dan Davies]

This was an arrangement that is designed to create a decision with no human owner. And you have this weird, weird kind of relationship with someone who's giving you bad news, like at a train station or an airport departure desk, because you have to talk to them like they're a human being like yourself. You know, you're a good person. You can't kind of get angry at them. You have to kind of respect them as a human being. But they're talking back to you with something that is not the voice of a human being. Because they're not actually making the decisions about what they're going to say to you. What they're telling you is coming from a script or from a set of policies and information on the screen in front of them. And the decisions have been made a very long way away by people you will never met. And often those decisions are actually just coming out of an algorithm and just coming out of a computer.

[Phil Agnew]

In 2019, my partner and I boarded a flight from Lima to Barranquilla on the northern coast of Colombia. Our flights transited through Bogota. Now, we'd paid for check in luggage and we were told that the luggage would be transferred onto Our second flight. Both flights were of the same airline and the transit time was less than an hour. So there was really no way that we would even do the transfer the luggage ourself. Then in Bogota, by chance, we saw our bags on the luggage carousel. We frantically picked them up, ran to check in to say there's been a mistake, our bags are here, they shouldn't be, they should be at our final destination. And we were told check in's closed so they can't do anything. And we'll have to pay $100 to change flights, even though we were already checked in and our luggage tags clearly said the items shouldn't have been here, they should have been transferred onto the next flight. This is what Dan calls an accountability sync. An interaction where a customer is not negotiating with a human that has any agency, but instead with a faceless system where decisions are pre made.

[Dan Davies]

You have the accountability sink because you can't expect the person who's talking to you and telling you you've been bumped off a flight. You know, they can say they're sorry, but they can't really feel sorry about it because it's not their decision. And there's no individual human being who's done that. And therefore there is no individual human being who's accountable for it. And when you see that kind of social institutional arrangements in place, it's often interesting. That's something that you really need to analyze because when you're seeing that accountability sink, you're seeing a decision that hasn't been made by a human being. And that means you have to look at those things because lots of those time those institutions haven't been particularly well designed, usually because the people who did them were either consciously trying to produce accountability for themselves, which is usually something that people do when they're doing something they shouldn't, or they just fell into it because it was an easier way to operate. And they've not really necessarily paid attention to how that mechanism works. And particularly that mechanism works when it's presented with an unpredictable situation that they weren't thinking about when they set up the policy.

[Phil Agnew]

Organisations that become unaccountability machines or contain accountability sinks can start to damage themselves when the system makes poor decisions. This doesn't just happen with airlines, but with all sorts of companies like Fox News.

[Dan Davies]

In 2020, Fox News did lose the second biggest libel lawsuit in the history of the American courts against a manufacturer of voting machines because they have, you know, the, the single most destructive thing you can say about a company that makes voting machines is that it puts some fraud into the voting machines to fiddle the elections. You know, that's completely survival level libel against a company.

[Phil Agnew]

And Fox News said this a lot.

[Fox News Reporter]

Talking about the election. Touchscreen voting machines are common in polling places all across the country. But some people complain that the devices can flip votes, registering the wrong candidate when voters push the button on screen. Well, what happens if you go vote for Donald Trump and Hillary Clinton's name is checked? Well, voters have said in past elections their choices were swapped. When Jim Moynihan ran for Illinois state representative, he tried to vote for himself. I went and touched the box next to my name and the check mark magically appeared at my opponent's name.

[Dan Davies]

And you know, it was not true. Everyone understood that Dominion Voting Machines did not do that. And it was a crazy conspiracy theory. And so the question is, you know, why did Fox News broadcast this conspiracy theory when you have plenty of internal memos from that time showing that almost everyone there knew that it was false, it was crazy, and that constantly broadcasting it was stirring up rage in a whole load of people who really shouldn't be incited at that time. And the picture that comes out of those court documents is that every individual knew that they were doing something wrong. Nobody felt able to stop themselves because every time they started saying, you know, actually that 2020 election results were pretty legitimate and they shouldn't be questioned in this way, they lost viewers. And there's a great quote in those court documents from a senior executive saying, bad ratings make good journalists do bad things. The really crazy thing, of course, is why were the viewers so fired up? Why were they so inclined to. To believe in conspiracy theories? Why were they kind of so outraged and disrespected anytime anyone told them something they didn't want to hear? And the answer is that Fox News had spent the previous two decades making them that way. You know, they'd created this group of people who were angry, who were used to living in a reality of their own political convenience, and who were used to being told that they should react really extremely whenever anyone contradicted them. And then at times, time came when Fox found itself having to tell them something they didn't want to. And they'd created this system that they couldn't control anymore.

[Phil Agnew]

Dan points out that Fox News's libel case and KLM squirrels, they have a lot in common. They're both examples of systems breaking down after receiving an input that the system didn't expect.

[Dan Davies]

What seems to happen, or what is that you set a system up which functions in the way that you want it to. When it gets the kind of input that you're expecting it to get. You set up a system and you kind of go, I'm tired of making all these decisions one by one. I'm just going to hire an employee and give them a set of written instructions and tell them that they've got to follow the process, follow the algorithm for a while that looks like it's working really well. Then the system gets some output that you weren't anticipating because, you know, the world is just a weird and chaotic place and it's always changing and the system gets an input that it wasn't really set up to deal with. And then how that system is going to respond to that is anybody's guess.

[Phil Agnew]

Dan makes the point that many of the most damaging companies today are simply old companies with systems that made perfect sense 50, 20 or 100 years ago, but now function in a very different world.

[Dan Davies]

The obvious one, you know, to try and bring this back to a more concrete example, because I realized that sounded a bit abstract. You might set up a company to extract oil from the earth's crust and burn it for energy, which would be a good thing to do. And you might decide to run that process on the basis of profit maximization. And that works really great for several hundred years. At some point you start getting evidence that the amount of carbon dioxide in the atmosphere is, could have really bad consequences for all life on Earth. It certainly, you know, as we see now, it's very difficult to get that information to change the huge system that you've set up to extract hydrocarbons and burn them for profit.

[Phil Agnew]

This happens with all sorts of companies.

[Dan Davies]

And that's how you get corporations doing things that are really damaging to them in the long term for the benefit of accounting profits or their estimation of shareholder value.

[Phil Agnew]

I asked Dan for an example of a modern day company that has imploded by simply following a system.

[Dan Davies]

Well, the one that really is quite serious, which I've ended up speaking to a lot of people about, was Boeing. Boeing, particularly after its merger with McDonnell Douglas, in which the corporate culture of McDonnell Douglas absolutely took over because Boeing was an engineering driven company. You know, it was always a company in which the top management were aeronautic engineers. All of the values were values of aeronautic engineering. They wanted to make a profit, but finance was very much a service center to the purpose of that company, which was to make great aeroplanes. It was, we want to build the 747. Finance has to Think of a way in which we can do that while raising the huge amounts of capital needed to make that airliner. Post McDonnell Douglas Finance was in charge. It was now a thing where all engineering decisions were subject to financial constraints and all the engineers had to do was justify themselves to finance all the time. And so they made some decisions which in retrospect were really incredibly bad. They designed the 737 Max according to non engineering constraints, particularly it had to be one that existing 737 pilots did not have to retrain on. The 737 Max was grounded worldwide in March 2019 following two crashes that killed 346 people. They outsourced some of their key production knowledge to other providers and started taking cheapest quotes. We spoke to a whistleblower and he said he often found up to 200 defects on parts being ready for shipping to Boeing. And each one of those decisions might have looked great on a spreadsheet, but eventually, you know, they resulted in Boeing putting into production an aircraft with a fundamental design flawless, killing nearly 1,000 people and damn nearly destroying the company. And Boeing still hasn't recovered from the consequences of those decisions, even in financial terms. Well, problems continue to mount for US aerospace company Boeing after another string of international safety incidents. What's happened there is simply that the company went. If the company was a human being, you'd say it went mad. You know, it developed an obsession with something that wasn't really relevant to its purpose and it started pursuing that obsession to the detriment of what it actually wanted to do. Boeing is a really horrible example, in my view, of the way that these things can go bad. And a lot of the time when companies go bad, it's because finance, which ought to be a support centre, has become the highest level of the company.

[Phil Agnew]

Boeing, KLM and Fox are examples of organisations failing due to diligently following poor systems. But I wondered if Dan had any examples of marketing functions, marketing teams, marketing departments that made this same mistake.

[Dan Davies]

The one that comes to my mind is the great example of the Burberry check.

[Phil Agnew]

The Burberry check, for those who don't know, is that beige tartan pattern with black, white and red intersecting stripes. It is instantly recognisable as Bourbon's iconic design.

[Dan Davies]

You have this iconic brand, it's an upmarket brand, you start selling a few baseball caps, you think, oh, actually we're making some decent money out of those baseball caps. Five years later you've made a ton of money out of selling all of those goods. But You've done serious damage to your ability to position that as a high end luxury brand. The story of Burberry seems to be that they spent the next 20 years buying to dig themselves out of the hole that they dug themselves into by not realizing that if you start selling something piled high and sell cheap in mass market fashion retailers owned by Mike Ashley, that's going to affect your ability to shift the same product out of boutiques on Bond Street.

[Phil Agnew]

I mentioned to Dan how this reminded me of the Ivy, a once revered exclusive celebrity dining spot that is now a mass market change.

[Dan Davies]

Every member's club in Soho has always had that problem. The issue that a club like the Groucho has, and the Groucho is the only one that has managed to that managed to solve this problem for more than a short while, is that in order to make the club work, you need people who spend a lot of money, buy memberships, but are too busy to really come in there. Which means that your Soho members club really wants to recruit a lot of investment bankers because they've got loads of money and they're always too busy to come in, so they're not there much. So you can sell loads of memberships, which is great, except if you do that too much, then your club is now not a fun media haunt. It's full of assholes in suits crowding up the bar and braying about their latest Porsche. So it's not fun anymore. And all of the fashion people and all of the media people leave. And then after a while, the bankers start noticing that they're in a club full of bankers, which is the last thing they wanted to do. They wanted to be in a club full of models and they leave too. And that is the life circle of a members club in Soho.

[Phil Agnew]

That summarises Dan's point. Organizations run with only a financial or economic goal are bound to fall into these traps of chasing profit at the cost of their core offering. But chasing a financial goal isn't the only trap, because Dan says football clubs often fail because they chase success and actually forget about the finances.

[Dan Davies]

For a lot of football clubs, they actually end up getting it wrong the other way around. The absolute story of many Premier League clubs is you want footballing success. So you think, well, the only way to get footballing success is to have good players. And that means we need to buy good players, and that means we need to take on a bunch of debt and they take on more debt and the performance on the field maybe improves a bit, but it doesn't necessarily improve quite enough to generate extra revenue to service the debt. So you know, when they need some more players the next season, they take on a bit more and then suddenly two or three seasons down the line, you might have taken your club from the top of the championship to halfway up the Premier League, but you've now got an absolute mountain of debt and suddenly everything comes due. You have to cut everything back within a couple of seasons. After that you're right down in, you know, in the championship or in league, League two, playing against Exeter City or.

[Phil Agnew]

My local club or whatever Dan says. All of these examples, Burberry, Soho clubs, football clubs, coal companies, Fox News and klm, they all fall foul of the same problem.

[Dan Davies]

It's just a matter of a management system which is meant to be an information processing system that doesn't have all of the relevant information going into it in a way that it can be converted into decisions.

[Phil Agnew]

It is not just private organisations who struggle with this. After the break, Dan shares an example of the UK education system.

[Dan Davies]

Who did Everyone was trying to do the right thing. The results were terrible.

[Phil Agnew]

Hear all of that after this quick break. The podcast I'd like to recommend today is Creators are Brands. It is hosted by Tom Boyd and is brought to you by the HubSpot Podcast Network, the audio destination for business professionals. Creators Our Brands explores how storytellers are building brands online, from the mindsets to the tactics to the business side. They break down what's working so you can apply that to your own work. One of the recent episodes I listened to tackled how some creators are being paid hundreds of thousands of pounds to promote brands, which I think is a kind of incredible thing that happens in this day and age. So if you want to listen to that episode or any of the brilliant creators of brands episodes, go and listen to Creators Are Brands wherever you get your podcasts. Hello and welcome back. You are listening to Nudge with me, Phil Agnew. Dan in his book talks about the British education system, calling it a great example of how good people doing the right thing produce things they didn't intend. I asked him to explain this point.

[Dan Davies]

If we consider the British A level grades, A level grades are the gold standard of the British education system.

[Phil Agnew]

For those of you Outside of the UK, the A levels are the exams 17 to 18 year olds take before they go to university. Universities typically require quite good A level results to guarantee entry.

[Dan Davies]

They are, broadly speaking, unbiased. There is really no socio economic kind of skew. A lot of work is spent on making sure that they're not culturally specific to kind of rich people things, broadly speaking, that is not a biased exam system. If your grades are kind of at the borderline between like a B and an A or a C and a B, then you can appeal against those grades. The appeals system is also extremely fair. There is no kind of bias in that. It's been really designed with a lot of thought to make sure that it's not culturally specific, it doesn't favour rich kids or whatever. It's an unbiased system. However, the combination of an unbiased exam system and an unbiased appeal system creates an extremely biased system because the system is now biased in favour of people who are willing and able to appeal. What it means is that for grades which are on the borderline for university acceptances, which are the only grades that really matter in the A level system, you have a system that is very biased in favour of the middle classes. You have a system that is very biased in favour of private schools because they know and how and have the incentive to work the appeals system. And you have a system which is ludicrously, comically biased towards the children of teachers. Teachers are the ones who absolutely know which grades are worth appealing and how to appeal them.

[Phil Agnew]

No one set out to create a biased system, but because of insufficient resources and the nature of systems, it ended up becoming extremely biased.

[Dan Davies]

Everyone does their best, the results are often terrible and nobody seems to know how to do anything about that.

[Phil Agnew]

I mentioned to Dan that this reminds me a bit of Google Reviews. Google Reviews should have really helped create a system that diners and cafe goers and whoever else wants to find a good cafe or restaurant could use these reviews to find a restaurant that they actually would like. But instead, what this system has done is incentivised businesses to game the system, creating offers for those who leave reviews, and leading to a system that promotes brands not purely based on their food and service, but actually on how they incentivise reviews.

[Dan Davies]

Well, no, that's. That's a great example, of course, of something I'm sure people will talk about on the podcast before, which is Goodhart's Law. Charlie Goodhouse, another former bank of England economist like myself, considerably more successful at it. When you're trying to measure something, you need to understand that if you're just measuring something, you can measure it. If you're using that measure as part of someone else's reward system, then you have to take into account the feedback from that to what you're measuring. Trustpilot ratings in principle, a Great idea. Companies tell their staff that they've got to get a trustpilot rating above so far, or they've got to get so many reviews per month, you're now measuring something which is partly people's opinions and partly the effects of the incentive system that you've put in there.

[Phil Agnew]

As I neared the end of my chat with Dan, I wondered what he thought about AI. I wasn't sure if he'd think AI would exacerbate all of these properties, problems, make systems even less accountable and create a world where responsibility no longer lies with humans, but with AI. Or I wondered if he maybe thought that AI could be used for the complete reverse. So I asked him what he thought.

[Dan Davies]

I think it's one more technology that we've got. The whole story of organisations and management has always been the world getting complicated at an incredibly increasing, accelerating rate, and us, the poor managers, trying to do our best to keep up with it. And every now and then someone invents a new technology that improves our ability to process information. You know, even the humble filing cabinet was a massive step forward in its day. It really helped make the American railroads possible. With AI, we've got a technology that can certainly be used to help us process more information, to help us take in more information and make up better decisions. It's also possible for it to create chaos. One of the things that kind of worries me about the effect of AI on the economy is that it improves the productivity of people who create problems as well as the productivity of people who create solutions. When you're trying to design a system, you have to design it consciously and think about what kind of information it's going to cope with and what it's going to do when something unpredictable happens. I kind of come back to a great quote from a guy called Stafford Beer, who was around in the first computer revolution when companies started installing mainframes. And he said, you know, we have a technology here that really could have changed the way that we use and process information. But doing that would have meant redesigning our organizations, upsetting hierarchies, and it was kind of. That sounded difficult and unpopular. So what we did was automate all of our existing payroll and accounting processes. And he said it was as if American business had got the opportunity to hire Shakespeare, Leonardo and Mozart and then put them to work memorizing the phone book so that they could look up telephone numbers a little bit more quickly. That's what worries me about AI. We've got this incredible flexible system for organizing knowledge and coming up with connections in ways that, you know, that just so much more efficient than any previous kind of information organisation system and we're using it to send more emails.

[Phil Agnew]

Stafford Beer, who inspired much of Dan's work, said companies kind of wasted computer power. He said that a 1970s supermarket could have used computers for things like real time sales data to instantly flag shortages, reorder stock and highlight unusual demand pattern patterns like umbrellas during rain, for example. And yet in reality, in the 80s and 70s, computers were essentially just used for payroll and accounting. Maybe the fear with AI isn't that it'll remove all accountability for companies, but it's that we won't use it to anywhere near its potential. Rather than using it to cure cancer or create genuine climate solutions, we'll instead use it for automating LinkedIn comments or sending cold emails. That's all we have time for today. Thank you so much for joining me, Phil Agnew and the brilliant Dan Davies. Dan is an incredible thinker. His points are eye opening and his books are fantastic. I have left a link to his latest book, the Unaccountability Machine in the show notes and I've also left a link to his fantastic Nudge Stock talk where he goes into much more detail about Stafford Beer. So do check both of those out. If you've enjoyed Dan today, you will love his book. Next week on Nudge, I'll share my final episode with Phil Graves. We'll cover social norms and the power of the herd. That's a really interesting episode. And then after that I've invited the neuroeconomist Matt Johnson back on the show to talk about why people are put off by AI art. This is a really interesting chat. He shares an amazing study. I think it's from Stanford, where people are shown AI art and human art and they prefer the AI version until they find out that it's created with AI. It's fascinating and a must listen for anyone who uses AI in their work. Make sure you don't miss those two podcasts by subscribing for the Nudge newsletter. If you do, you get an email announcement every single time a new episode goes live. You get instant access to my bonus episodes and I have created a bonus episode with Matt Johnson as well, so you'll get that straight in your inbox. You'll also get my free Friday weekly newsletter which shares the best behavioral science tip I've come across that week. And it takes two seconds to sign up. Just go to nudgepodcast.com, click Newsletter Enter your email address. That's it. And you'll be instantly subscribed and you can unsubscribe at any time. That is all for me this week. I'm actually away for this week. I'm running the West Highland way in Scotland, or at least I'll try and run it. We'll see how I get on. So if you do email me or contact me during that time, I might be a little bit slow to respond, but I will be back next Monday for another episode of Nudge. Thank you so much for listening. Cheers.