The internet’s memory problem

Summary

Business Daily – "The Internet’s Memory Problem"

Date: March 12, 2025
Host: Frey Lindsay
Guests/Contributors: Danny, Cory Doctorow, Aaron Smith, Anusha Hossain, Zynga, Onur, Christina

---

Overview

This episode of Business Daily delves into “the internet’s memory problem”: the widespread and accelerating loss of digital memories, documents, and cultural artifacts. Host Frey Lindsay, alongside guests ranging from tech researchers to journalists and everyday internet users, investigates why so much of our personal and shared history is vanishing, focusing on what’s known as “link rot” (dead or disappearing links) and the challenges of digital archiving.

---

Key Discussion Points & Insights

1. Personal Stories – The Loss of Digital Keepsakes

• Lost Mixtapes & Personal Archives
  • Frey and his friend Danny recount losing a unique mixtape CD, using it as a metaphor for personal digital losses which, once online, can be easily lost forever.
  • Frey Lindsay: “We've all got a story like this, right? …These things are like little markers of who we used to be and they lead us to who we are now.” [02:23]
• A Spanish Social Network Gone
  • Olivia (pseudonym for Anusha Hossain) tells how she lost years of photos and writings when the social network Twenti shut down.
  • Zynga: “But it wasn't my Gmail that I was actually checking. It was a Hotmail account. I lost several years of photos and writings and friendships.” [03:49]

2. The Scale and Reality of Link Rot

• Research on Disappearing Web Content
  • Aaron Smith (Pew Research Center) explains “link rot” and shares alarming statistics:
    • 25% of almost 1 million URLs from 2013–2023 are now inaccessible.
    • For URLs from 2013 alone, 38% are gone.
    • 1 in 5 government/news websites have at least one broken link; 5% have all links broken.
    • 1 in 5 tweets disappear just months after posting.
    • Aaron Smith: “A quarter of those pages no longer resolve to a functioning website.” [04:47]
• Wikipedia & Broad Impact
  • Frey discovers that about 30% of links in Wikipedia reference sections are dead, even on major topics like Fela Kuti, Angela Merkel, happiness, or the Boeing 747.
  • Some of these can be found only via the Internet Archive's Wayback Machine.

3. The Challenge of Preservation

• Physical vs. Digital Storage
  • Digital materials are stored on servers requiring maintenance. When companies go bust, change strategy, or do migrations poorly, digital materials disappear.
• The Role of the Internet Archive
  • The Internet Archive, via its Wayback Machine, has saved over 835 billion web pages, but still cannot keep pace with web decay—especially as newer content moves behind social media “walled gardens.”
  • Frey Lindsay: “But even with astonishing numbers like that, we're still seeing linkrot encroaching at a regular pace, taking away large parts of our cultural and even political memory with it.” [07:22]
• Obstacles for Researchers & Regional Content
  • Historian Anusha Hossain describes difficulty accessing historical conversations and debates about South Asian writing systems, as much of that content is now lost:
    • Anusha Hossain: “The Silicon Valley stuff is really well preserved. But when I try to get the other … piece of the transnational debate, it's much harder to find pieces of the Bengali Internet.” [10:02]
  • She and other historians often rely heavily on the Wayback Machine for any hope of recovering lost material.

4. Journalism and the Vanishing Record

• Journalists’ Work Disappearing

  • Zynga, a French journalist, recounts shock at losing access to published digital work when outlets closed or stopped maintaining their websites:
    • Zynga: “I was genuinely surprised. It seemed directly counter to the Internet's biggest promise.” [12:14]
  • Even broadcast programs like this episode may eventually disappear, with only the current listener experiencing them.

• Grantland: An Exception

  • Katie Baker (Danny’s pseudonymous interview) describes her relief when Grantland (archived by ESPN) remained accessible, though she still feels anxious it may vanish:
    • Danny/Katie Baker: “I always have a moment of nerves where I think that maybe this time I'm going to go and it's going to be gone. And I'm always so relieved when it's still there.” [13:54]

5. Legal and Commercial Obstacles to Archiving

• Copyright Lawsuits Threaten Archives
  • Cory Doctorow explains how organizations like the Internet Archive risk lawsuits from publishers and record labels that equate preservation with piracy:
    • Cory Doctorow: “...publishers and record labels have brought an existentially threatening lawsuit against the Internet Archive … two lawsuits that could erase that archive at the stroke of a pen and … salt the earth so that nobody ever tries to do what they've done before.” [14:45]

6. Bigger Picture: The Role of Capitalism and Monopolies

• From Diversity to Control
  • Doctorow traces a shift from the internet’s early pluralism to today’s monopolistic power:
    • Cory Doctorow: “What one was not capitalism, but monopolism, where we created a system that more or less explicitly encouraged the formation of monopolies who … earn their money by owning things that other people needed… What happened on the internet is what happened everywhere. We let the landlords win.” [16:33]

7. Does Anyone Care About the ‘Trivial’ Internet?

• Why Everyday Content Matters
  • Doctorow and Hossain argue that the tiny details and everyday conversations of the internet are irreplaceable pieces of social history:
    • Cory Doctorow: “The trivial is where it's at. …The Rosetta Stone is not a work of high art, right? It's an invoice … the ability to preserve the mundane and the trivial pays dividends that we will never understand until, in retrospect, we recover them.” [17:34]
    • Anusha Hossain: “It's very important to also understand how different generations were communicating with each other. So I do kind of fear that we'll lose that…” [18:17]

8. Human Impact: Grieving Lost Connections

• Final Listener Story
  • Christina, from Germany, laments the loss of a teenage chat website holding memories of a now-deceased friend:
    • Christina: “…a friend of mine died in high school and there are loads and loads of messages and pictures and it would just be really nice to go back to them. And obviously as teenagers, we didn't … kind of understand the Internet that much and we thought it's on our computers, so it's there forever.” [18:47]

---

Notable Quotes & Memorable Moments

• “These things are like little markers of who we used to be and they lead us to who we are now.” — Frey Lindsay [02:23]
• “A quarter of those pages no longer resolve to a functioning website.” — Aaron Smith [04:47]
• “The trivial is where it’s at… The Rosetta Stone is not a work of high art, right? It’s an invoice.” — Cory Doctorow [17:34]
• “It seemed directly counter to the Internet’s biggest promise.” — Zynga [12:14]
• “We let the landlords win.” — Cory Doctorow [16:33]
• “…we thought it's on our computers, so it's there forever.” — Christina [19:06]

---

Timestamps for Key Segments

• Intro & Lost Mixtape Story: 01:08–02:23
• Technical Explanation of Link Rot (Aaron Smith): 04:29–05:50
• Wikipedia Link Rot Demo: 05:50–07:22
• Internet Archive’s Limitations: 07:22–08:46
• Censorship & Disappearing News Archives in Turkey (Onur): 08:53–09:46
• Challenges for Internet Historians (Anusha Hossain): 09:54–11:35
• From Open Web to “Walled Gardens”: 11:35–12:14
• Journalists Losing Access to Their Work: 12:14–12:58
• Rarity of Preserved Archives (Grantland/Katie Baker): 13:30–14:20
• Threats from Copyright Lawsuits (Cory Doctorow): 14:45–16:01
• Shifts from Open Web to Monopolies: 16:01–17:10
• Importance of Preserving the “Trivial”: 17:34–18:17
• Listener’s Lost Memories (Christina): 18:47–19:29

---

Conclusion

This episode highlights the fragility of digital memory in an online world dominated by profit-driven companies and legal battles over ownership. As link rot accelerates, large swathes of culture—from everyday conversations to journalism—risk vanishing forever. While initiatives like the Internet Archive provide a lifeline, they remain embattled and incomplete. The loss isn’t just technical—it is deeply personal, historical, and cultural, underlining that trivial digital traces may prove, in the long run, most valuable of all.

Transcript

[BBC Announcer]

This BBC podcast is supported by ads outside the uk.

[Frey Lindsay]

I love ravioli.

[SAP Concur Representative]

Otanta fame.

[Danny]

Since when do you speak Italian?

[SAP Concur Representative]

Since we partnered with SAP. Concur. Their integrated travel and expense platform and breakthrough solutions with AI gave me time back to dive into our financial future. We expand into Europe in 2027, so I'm getting ready.

[Aaron Smith]

Well, you can predict the future.

[SAP Concur Representative]

I can predict. You'll like that message.

[Cory Doctorow]

What message?

[Frey Lindsay]

Oh, hey, we all got bonuses.

[SAP Concur Representative]

You can save for college now.

[Danny]

I don't have kids.

[BBC Announcer]

You don't say SAP conc helps your business move forward faster. Learn [e-mail] at thebc we go further so you see clearer. With a subscription to BBC.com you get unlimited articles and videos, hundreds of ad free podcasts and the BBC News Channel streaming live 24. 7 from less than a dollar a week for your first year. Read, watch and listen to trusted independent journalism and storytelling. It all starts with a subscription to BBC.com. find out [e-mail] unlimited.

[Frey Lindsay]

Welcome to Business Daily on the BBC World Service. I'm Frey. Lindsay.

[Danny]

Hello.

[Frey Lindsay]

Hi.

[Danny]

Hey, how you going?

[Frey Lindsay]

Yeah, fine. This is my friend Danny. He's just gotten back from a long holiday in Japan. About 10 years ago, Danny and I were going around record stores in Tokyo. In one of these stores, we picked up a free mixtape CD that they were just kind of giving out.

[Danny]

I think you were given the mixtape and yeah, it was just this really carefully crafted selection of 60s, folky sort of pop music. All of the tracks were just these really rare finds. A lot of those tracks ended up playing quite an important part in our lives.

[Frey Lindsay]

Eventually we lost the physical CD and I've long since lost the digital copies that we made.

[Danny]

Do you remember last time we were in Japan? I was convinced that I had found that record store and we went back there And I said, 10 years ago, we came and we got this CD, can we have another one? And they said, you've got the wrong store. We weren't open 10 years ago. That was pretty devastating.

[Frey Lindsay]

We've all got a story like this, right? Some piece of media, some childhood drawing or a letter that you sent when you were young. Something that's one of a kind and that you'd give anything to get back. These things are like little markers of who we used to be and they lead us to who we are now. They're a big part of our history. And that's what this episode of Business Daily is about. But writ large, because we're talking today about how, for many of us, so Much of those memories are stored online.

[Cory Doctorow]

So I've been blogging for a quarter of a century and that means that I've got tens of thousands of old blog posts going back to the early.

[Frey Lindsay]

2000S, and we're going to talk about how often because of commercial reasons, much of that cultural, personal and even societal and political history is being lost every day.

[Aaron Smith]

If we focus on pages from 2013, 38% of all of those pages that did at one point exist are no longer accessible.

[Frey Lindsay]

The Internet has a memory problem and few are willing to pay for an upgrade. That's all here on Business Daily.

[Anusha Hossain]

Hi, I'm Olivia and I grew up in the south of Spain. There was the Spanish thing called Twenti.

[Zynga]

I refused to get Facebook for a couple of years.

[Anusha Hossain]

So everything of me as a preteen and teenager, first loves, first crushes, songs I wrote, were going on there.

[Zynga]

And then when I went to university, I guess that they sent an email.

[Frey Lindsay]

Round saying that 20th was about to.

[Zynga]

Close and that you had to download everything. But it wasn't my Gmail that I was actually checking. It was a Hotmail account. I lost several years of photos and writings and friendships.

[Frey Lindsay]

We think of the Internet as a kind of a place, right? A place where you can go and see things that are stored there. But of course, the material in that place is kept on servers on physical hard drives all around the world. That media requires upkeep. Someone has to make sure the servers stay online and remain accessible. Most of the time that's businesses taking care of that Web service, companies, online hosting, etc. But companies go out of business or they shut down a particular service or division, or they just migrate to a new website and they don't do it very well. And all that leads to what's called linkrot.

[Aaron Smith]

In the simple explain, like I'm 5 version. Linkrot is I want to access something on the Internet. I know it existed at some point in time because there's a link to it on the page that I'm reading. I click on the link and it takes me to a 404 message or a file not found.

[Frey Lindsay]

Aaron Smith is director of Data Labs at Pew Research Center.

[Aaron Smith]

What we found is from all of the almost 1 million URLs that we looked at from 2013 to 2023, a quarter of those pages no longer resolve to a functioning website. If you go back in time, as you might imagine, this problem gets worse. So if we focus just for instance, on pages from 2013, 38% of all of those Pages are no long accessible at the URL that they were originally created at.

[Frey Lindsay]

Aaron told me that some corners of the Internet are more prone to this phenomenon than others.

[Aaron Smith]

Around 1 in 5 government websites and news web pages have at least one broken link on them. Around 5% of government websites and news websites have all of the links on the page broken. We found that about half of reference sections on Wikipedia pages had at least one link that was no longer functional. And when we looked at this in the context of social media, we found found that about 1 in 5 tweets are no longer available even a few months after they're posted.

[Frey Lindsay]

Aaron and his team focused on the bluntest type of link rot. Click a link, it's not there. But there are also other more subtle forms, like the material on a site being changed. Imagine say, the official website of a country where they've just had an election and a new government's come in. Or the SoundCloud account of an artist who's taken down old songs and put up new ones. Think even of the daily changing advice on public health sites during COVID So let's take a look at this effect in action. Okay, so Aaron mentioned Wikipedia, so let's go to the page for legendary Nigerian musician Fila Kuti. I'm gonna go down to the references section and I'm gonna click every link there to see to see what we get. One of the 73 individual URL links in the reference section, 18 were dead outright. They went to 404 page not found. Two other links went to something totally different. One is a cryptocurrency site now. So we're talking about just under 30% of that page succumbed to link rot. I found similar results on the Wikipedia pages of Angela Merkel, Jody Foster, Enron, the Concept of Happiness, and the Boeing 747. The thing is though, for many of those dead links, there's still a copy of the original page preserved on the Internet Archive and a few other similar sites. The Internet Archive is a non profit that since 1996 has been archiving snapshots of the Internet with its Wayback Machine program. According to its website, the Internet Archive now hosts over 835 billion web page snapshots. But even with astonishing numbers like that, we're still seeing linkrot encroaching at a regular pace, taking away large parts of our cultural and even political memory with it. You're listening to Business Daily from the BBC World Service.

[SAP Concur Representative]

Hey, let's talk about your expense report.

[Aaron Smith]

I didn't submit an expense report.

[SAP Concur Representative]

You will Custom saddles and dog training services are not within policy.

[Aaron Smith]

What are you talking about?

[SAP Concur Representative]

SAP Concur uses advanced AI to audit and automatically detect out of policy expenses. It's the breakthrough I needed to focus more on our future.

[Cory Doctorow]

These are my future expenses.

[SAP Concur Representative]

Yes. And self defense classes are out of policy.

[Frey Lindsay]

I'll need self defense classes.

[SAP Concur Representative]

You will?

[Frey Lindsay]

For what?

[SAP Concur Representative]

It's a big dog.

[BBC Announcer]

SAP Concur helps your business move forward faster. Learn [e-mail] at thebc we go further so you see clearer With a subscription to BBC.com, you get unlimited articles and videos, hundreds of ad free podcasts and the BBC News Channel streaming live 24. 7 from less than a dollar a week for your first year. Read, watch and listen to trusted independent journalism and storytelling. It all starts with a subscription to BBC.com find out [e-mail] unlimited.

[Frey Lindsay]

I'm Frey Lindsay and today we're exploring Linkrot what happens when content just disappears from the Internet?

[Onur]

My name is Onur and I'm from Turkey. Turkey's story is quite long as you may imagine, in terms of the censorship in the media. Like back in the days the newspaper name was radical and they were the first ones, one of the first ones who straightforward went to digital. So what happened after the coup d' etat attempt? We lost the the archives that all of a sudden we start to not touch, have an access to those archives and also we have our own sort of Reddit and all of a sudden we also lost a lot of archives of that which was a huge issue. Like details in the history, like what happened this year, that day, what was that occasion, what was that event, and so on.

[Frey Lindsay]

Given how much websites change, disappear, or are censored as Ona described there, it can be incredibly hard to track how our culture has developed in this ethereal space.

[Anusha Hossain]

So my name is Anusha Hossain and I'm a historian of computing and the Internet. I'm currently a researcher at UC Berkeley.

[Frey Lindsay]

Anousha basically researches the way communication and Internet standards have translated to different languages and regions of the world.

[Anusha Hossain]

I've been writing about these debates about how South Asian writing systems are represented in the Unicode standard, and it's kind of necessarily a transnational discussion. There are Silicon Valley technologists involved in and there were also kind of software communities in India and Bangladesh involved. The Silicon Valley stuff is really well preserved. But when I try to get the other kind of piece of the transnational debate, it's much harder to find pieces of the Bengali Internet. I know there is a really active blogging culture in the 2000s and people will reference debates they had and issues that came up that they were grappling with, but I just can't find the sources for it.

[Frey Lindsay]

As servers go offline or URLs point to a different site or the content of a site is altered, it can be incredibly hard to track the day to day developments of social phenomena like this.

[Anusha Hossain]

One challenge for Internet researchers is kind of just making sense of this big mass of content and kind of figuring out what to keep and not to keep. I think link grad is something that we all deal with on a daily basis. It's very common for links to, to godad or even in just a news article, you know, from a couple years back, it might not go where you want it to. So most of the links that I am trying to find, they aren't live. I access them through the Internet archives Wayback Machine. So that's a big resource for me.

[Frey Lindsay]

What we're talking about here, linkrot is in part a consequence of the coalescence of the Internet from many small sites to a few large ones.

[Anusha Hossain]

In the 2000s, everyone was producing content that was saved on webpages and those could be snapped up by the Wayback Machine. And now I think more user generated content is happening behind walled gardens or on social media platforms that aren't as easy to access for that content. You really can't depend on the Wayback Machine because these sites and platforms, pretty much all of them ban it. You've seen in the last 5, 10 years these platforms becoming slightly more antagonistic to academics. So I feel worried about that that will happen to all of that content from the last five, 10 years and moving forward.

[Zynga]

I'm in Zynga. I'm in Marseille, France. When I wrote regularly for publications, I always assumed, even if they had a print edition, that I'd always be able to find my work online. So I was shocked to find that if those publications were no longer in print or no longer maintained an Internet pres, I had no access whatsoever to material I'd assumed would be easy to find. Over time I've been able to gather hard copies and scan them in and have them to share as PDFs in that way. But I was genuinely surprised. It seemed directly counter to the Internet's biggest promise.

[Frey Lindsay]

Journalism is a job that over the last few decades has become ever more online. You might be hearing this program live on air or you're listening back from the BBC World Service website. But who knows how long this program stays online. Surely not forever, right? You listening right now might be the last person to ever hear this. And that's the story of a lot of now defunct online news sites such as Gorka, MTV News, the All. And for a lot of the journalists that work there, big parts of their careers are now just gone. It doesn't have to be this way though.

[Danny]

My name is Katie Baker. I am currently a senior staff writer at the Ringer and was previously a staff writer at the Ringer's predecessor which was called Grantland.

[Frey Lindsay]

Grantland was a well respected US long form sports and culture site for the four years that it existed before the owner, ESPN pulled the plug. But the thing is, ESPN are still paying to keep Grantland's archive online.

[Danny]

I have the experience every now and then where I go back to find an old article that I've written and in many cases it's, you know, it's a long feature. You know, there's one recently where I went to Russia in, you know, in 2014 and talked to a hockey coach there and I went to find the link and I always have a moment of nerves where I think that maybe this time I'm going to go and it's going to be gone. And I'm always so relieved when it's still there.

[Frey Lindsay]

ESPN have chosen to pay that small amount to keep Grantland up, but it's not normally that way. Businesses typically operate for profit and there's little incentive to pay to maintain archives of these sites, even if the owners are still in business. But at the same time, companies really don't like it when someone else makes a copy of their intellectual property. And that's where the Internet Archive and its Wayback Machine come back into the picture.

[Cory Doctorow]

When you look at the actual efforts to preserve things where people have taken risks with ip, most notably the Internet Archive, you see relatively low resource organizations being quite clever and quite conscientious and literally making a copy of the web every couple of days, which is a remarkable accomplishment. Of course, today publishers and record labels have brought an existentially threatening lawsuit against the Internet Archive or two lawsuits that could erase that archive at the stroke of a pen and moreover, salt the earth so that nobody ever tries to do what they've done before.

[Frey Lindsay]

That's writer and activist Cory Doctorow. He's a visiting professor of computer science at the Open University and co founder of the Open Rights Group. He's also an author. His latest book is called the Bezel. The two lawsuits that he mentioned are from Book Publisher Hachette, as well as Universal Music Group and Sony Music Entertainment. They're basically objecting to the way the Internet Archive preserves material they own the copyright for. They argue that by making such material available, even in a limited form, the archive is taking income away from them and the creators. But without that archiving, how much of our history might be lost if nobody's willing to pay for it? Here's Cory Doctorow again.

[Cory Doctorow]

You know, in the early days of the Internet, I was very excited about it. The Internet was weird and it was diverse and it was pluralistic. And everyone who tried to enclose it and capture it failed because they had this vision for kind of monolithic Internet where stuff would be crammed into your eyeballs by people who knew better than you, did what you wanted to see. And every time they tried it, the Internet said no.

[Frey Lindsay]

What Corey is objecting to isn't the mere presence of money on the Internet, but how that money has been able to shape its structure.

[Cory Doctorow]

The people who were, you know, setting up early websites and creating online services, they were capitalists. What one was not capitalism, but monopolism, where we created a system that more or less explicitly encouraged the formation of monopolies who, rather than earning their money by making things that people liked, instead earn their money by owning things that other people needed to make things that people liked. And you can see the growth of the last 40 years as the triumph of owning things over doing things. What happened on the Internet is what happened everywhere. We let the landlords win.

[Frey Lindsay]

And this state of affairs has implications for the trillions, gazillions, even of gigabytes worth of online material that, without any incentive to preserve it, might just disappear forever. But does that matter? Most of the Internet is pretty meaningless, right? Old business websites, pointless tweets, the scores of long forgotten football matches. Isn't all of this just kind of trivial?

[Cory Doctorow]

You know, just from the perspective of history and culture, the trivial is where it's at. You learn virtually nothing about how medieval people lived by looking at tapestries that kings commissioned. We find out about how people live by digging up the middens. And the Rosetta Stone is not a work of high art, right? It's an invoice. And so, you know, on the one hand, we're just losing the kind of the texture of our history by getting rid of all this stuff. The ability to preserve the mundane and the trivial pays dividends that we will never understand until, in retrospect, we recover them.

[Anusha Hossain]

I think it's important for historians to also be writing about things that happen in everyday life. You know, we have the high political events that do get covered in news stories, but it's very important to also understand how different generations were communicating with each other. So I do kind of fear that we'll lose that kind of very important aspect of society and culture.

[Frey Lindsay]

That was author and activist Cory Doctorow and Internet historian Anusha Hussain. Now, before we go, we've got time for one more message.

[Christina]

My name is Christina and I'm from Germany. And a website I would like to see come back to life would be be our local chat website that we had when we were teenagers. You and all your classmates were in this chat room and we're speaking after school. And the main part, why I would like to have it back is because a friend of mine died in high school and there are loads and loads of messages and pictures and it would just be really nice to go back to them. And obviously as teenagers, we didn't have. Didn't kind of understand the Internet that much and we thought it's on our computers, so it's there forever.

[Frey Lindsay]

And that's it for this episode of Business Daily on the BBC World Service. I'm Frey Lindsay. Thank you very much for listening.