The AI Skeptics

Summary

---

The Last Invention | Episode: The AI Skeptics (Feb 20, 2026)

Episode Overview

This episode of The Last Invention dives deep into the skeptical perspectives in the hotly debated AI revolution. Host Andy interviews leading critics and nuanced thinkers, exploring the counter-narratives to AI hype. The discussion is structured in three acts, each representing a different brand of skepticism: "AI is grift," "We're on the wrong path," and "AI as normal technology." Each guest brings a unique lens, challenging dominant narratives about artificial general intelligence (AGI) and the transformative potential or threat posed by AI.

---

Key Discussion Points & Insights

I. Why Even Skeptics Matter

• The mainstream AI conversation is dominated by two extremes: "doomers" (who fear existential risk) and "accelerationists" (who see enormous promise).
• Genuine AI skeptics, especially those with deep expertise, are rare but their views help clarify assumptions, challenge hype, and keep the debate grounded.
• Three main skeptic "camps" are highlighted:
  1. AI is Grift – AI as industry hype and profit-driven “snake oil.”
  2. Wrong Path – AGI is possible, but current approaches (LLMs) won’t get us there.
  3. AI as Normal Technology – AI is powerful but will be absorbed slowly, like previous general-purpose innovations.

---

II. Act 1: AI Is Grift (Ed Zitron)

Ed Zitron’s Perspective

• Ed runs a tech PR firm and a critical newsletter (“Where’s Your Red App?”); he positions himself as a “professional critic of big tech.”
• Core argument: “The main thing holding back AI is that it doesn’t fucking work.” (06:57)
• He sees current AI products as oversold and fundamentally underdelivering, emphasizing that real-world impact is far from the utopian or apocalyptic promises.

Discussion Highlights

• AI has become a vague, catch-all marketing term, detached from original meaning (08:12).
• Hype is driven by investors, media, and tech elites who profit from inflated expectations, not real progress.
• Skeptical of celebrated figures (e.g., Geoffrey Hinton) – questions whether public warnings are marketing or sincere, and critiques media’s focus on existential risk over immediate & tangible harms.

Quote Highlights

• “They need you to believe these are the early innings because if you start looking at this and thinking about this even a little and realize that the depth of hubris, everyone involved kind of looks stupid.” — Ed Zitron (27:12)
• “The people that are benefiting from this don’t give a rat fuck about your or my ability… They give a rat fuck about making… their investments worth more.” — Ed Zitron (09:18)

Timestamps/Notable Moments

• 06:57 – Ed’s blunt assessment that AI “doesn’t fucking work.”
• 11:18 – Pushback on Geoffrey Hinton’s transformation from skeptic to warning advocate.
• 14:01 – Doomerism as marketing: “There’s another word I use for doomerism: marketing.”
• 24:49 – On technological underperformance relative to the hype.

Tone

• Highly irreverent, unsparing, and combative. Ed is unapologetically cynical about motives behind AI hype, likening the whole boom to historical bubbles and the “rot economy.”

---

III. Act 2: AI Skepticism – The Wrong Path (Gary Marcus)

Gary Marcus’s Perspective

• Longtime AI researcher, known for predictions about the limits of neural networks dating back to early 1990s (32:37).
• Marcus is NOT skeptical about AGI as a theoretical possibility, but deeply skeptical about the path pursued by the current AI mainstream: scaling up Large Language Models (LLMs).

Discussion Highlights

• LLMs are inherently flawed: “It’s inherently prone to all kinds of reasoning errors. It’s inherently hard to control. … It’s inherently prone to hallucinations…” (34:28)
• AI’s recent leaps (GPT-3, GPT-4) were real but stagnation has emerged; GPT-5 “wasn’t as impressive,” matching his earlier warnings (39:12).
• Scaling laws don’t guarantee continuous, exponential progress (“trillion-pound baby” misapplied extrapolation, 36:58).
• The real path to AGI will need new “symbolic” approaches and likely several additional breakthroughs.
• Marcus is a “scout” not a “doomer” or “grifter”—calls for investing in serious readiness, focusing on both immediate and long-term risks.

Quote Highlights

• “It’s a better ladder, but it’s not necessarily going to get us to the moon.” — Gary Marcus on LLM progress (35:51)
• “I think that we will get to AGI... but I don’t think we can do it with current techniques. We need new ideas.” — Gary Marcus (43:32)
• “LLMs chatbots have been a dress rehearsal and it’s a really depressing train wreck of a dress rehearsal…” (45:40)

Timestamps/Notable Moments

• 33:07 – Marcus recounts his early fascination and work in AI.
• 34:28 – Core critique of LLMs’ limitations.
• 36:58 – The “trillion-pound baby” analogy and scaling myth.
• 43:23 – “It’s back to the age of research again, just with big computers.”
• 45:40 – On the disappointing “dress rehearsal” that LLMs have provided.

Tone

• Earnest, technical, cautiously optimistic pending the right breakthroughs; more concerned with honest assessment than hype.

---

IV. Act 3: AI as Normal Technology (Arvind Narayanan)

Arvind Narayanan’s Perspective

• Computer science professor at Princeton; co-author of “AI as Normal Technology.”
• Argues that AI is a significant general-purpose technology but will integrate into society gradually, not as instant revolution.

Discussion Highlights

• AI is between a toaster and the discovery of fire: “Somewhere in between iPhone and fire.” (49:01)
• Major technical revolutions (electricity, Internet, Industrial Rev.) took decades to diffuse.
• Human inertia, messy bureaucracies, and cultural preferences for human interaction are powerful brakes on change.
• Even true AGI would not mean an overnight societal transformation; most jobs already could be automated but persist for deeper human and social reasons (58:21).
• Strong emphasis on “agency”—societies have a choice in how fast and deeply AI transforms occupations and relationships.
• Policy and social adaptation are where real risks and opportunities reside, not in sci-fi singularity scenarios.

Quote Highlights

• “Change will happen at the speed of human behavioral change, at the speed at which organizations can adapt…” — Arvind Narayanan (52:59)
• “At best, this ChatGPT mayor will reflect the views of OpenAI developers … It won’t reflect the will of the people. It’s just missing the point of democracy.” (60:04)
• “Whatever your profession is probably needs to come up with a normative framework for what are ethically acceptable and unacceptable uses of AI…” (75:18)

Timestamps/Notable Moments

• 49:01 – AI as normal technology, a general-purpose tech but with no quick “hinge moment.”
• 52:11 – Historical pace of electricity, telephone, Internet, and what to expect for AI.
• 58:21 – Even super-powerful AGI would face resistance: “number of tasks that could be automated is shockingly high, but we choose not to.”
• 60:04 – The “AI mayor” example and the irreplaceability of human judgment in collective life.
• 67:01 – Real social harms (e.g., chatbot reinforcement of delusional thoughts, social media) come from collective failures, not AI agency.
• 73:43 – Amtrak analogy: transformative technologies still get bottlenecked by non-technical constraints.
• 75:18 – Final message: “Every single one of us has a role to play…” on defining ethical AI use in daily life.

Tone

• Calm, empirical, pragmatic—champions agency and collective responsibility rather than techno-determinism.

---

Memorable Quotes (by Section & Timestamp)

• Ed Zitron (Grift)
  • “The main thing holding back AI is that it doesn’t fucking work.” (06:57)
  • “There’s another word I use for doomerism: marketing.” (14:01)
• Gary Marcus (Wrong Path)
  • “It’s a better ladder, but it’s not necessarily going to get us to the moon.” (35:51)
  • “I think that we will get to AGI... but I don’t think we can do it with current techniques.” (43:32)
  • “LLMs chatbots have been a dress rehearsal and it’s a really depressing train wreck of a dress rehearsal…” (45:40)
• Arvind Narayanan (Normal Tech)
  • “I would say somewhere in between iPhone and fire.” (49:01)
  • “Change will happen at the speed of human behavioral change...” (52:59)
  • “It won’t reflect the will of the people. It’s just missing the point of democracy.” (60:04)
  • “Whatever your profession is probably needs to come up with a normative framework for what are ethically acceptable and unacceptable uses of AI…” (75:18)

---

Useful Timestamps Overview

• 06:57 – Ed Zitron’s core critique of AI hype
• 11:18–14:21 – On “doomerism” and hype as marketing
• 24:49 – Technological underperformance vs. hype
• 34:28 – Gary Marcus on LLM limitations
• 36:58 – Scaling myth and the “trillion-pound baby”
• 39:12 – GPT-5 as a failure to deliver exponential improvement
• 43:32 – Current path won't reach AGI, need breakthroughs
• 49:01 – Arvind Narayanan: AI as normal technology
• 52:11 – Historical analogies for adoption (electricity, phone)
• 58:21 – Why mass automation won't happen overnight
• 60:04 – Social and political meaning of automation attempts
• 67:01 – Empirical risks vs. sci-fi risks (mental health, suicide, chatbot harms)
• 75:18 – Everyone's role in AI ethics and deployment decisions

---

Final Takeaways

• AI Skepticism is Diverse: Not all critics say “it will fail”—the most thoughtful want better, safer, more honest AI rather than hype or panic.
• Immediate Harms vs. Existential Risks: Real harms often fly under the media radar but can be deadly serious (bias, mental illness, economic precarity), while doomsday talk distracts from actionable concern.
• Progress ≠ Sudden Overthrow: Even with transformative potential, societal and psychological speeds are much slower than technical ones.
• Urgency Without Panic: There’s a strong message that “not panic, but policy, preparation, and participation” will define AI’s real impact.

---

This episode brings skeptical but informed voices to the front, pushing listeners to ask hard questions not just about what can be built with AI, but how—and how soon—it will actually matter for real people.

Transcript

[Gary Marcus]

If there was a big red button

[Arvind Narayanan]

that would just demolish the Internet, I would smash that button with my forehead. From the BBC, this is the Interface, the show that explores how tech is rewiring your week and your world.

[Interviewer (Andy)]

This isn't about quarterly earnings or about tech reviews.

[Arvind Narayanan]

It's about what technology is actually doing

[Ed Zitron]

to your work, your politics, your everyday

[Gary Marcus]

life and all the bizarre ways people

[Arvind Narayanan]

are using the Internet. Listen on BBC.com or wherever you get your podcasts.

[Matt]

Hello, Matt here. Before we get into this week's episode, I wanted to pop in real quick to let you all know about another podcast from our team here at Longview called Reflector. On Reflector, we mix together historical backstories with on the ground reporting to tell context obsessed stories about the beliefs that are shaping the world. To find it, just search for Reflector on whatever app you are using to listen to this right now.

[Interviewer (Andy)]

Hello again and welcome back to the Last invention. In our original eight part series, one of the things that we wanted to make clear was that the people who are racing towards the development of AGI are not thinking of it as simply some new tool or new tech product.

[Arvind Narayanan]

The world as you know it is over. It's not about to be over.

[Interviewer (Andy)]

It's over. AI is going to be better than

[Ed Zitron]

almost all humans at almost all things.

[Interviewer (Andy)]

They believe that the thing that they are right now making, it wouldn't be

[Matt]

a computer program exactly.

[Arvind Narayanan]

It wouldn't be a human exactly. It would be this sort of digital

[Interviewer (Andy)]

supermind will change the course of human history.

[Arvind Narayanan]

I've always believed that it's going to be the most important invention that humanity

[Interviewer (Andy)]

will ever make and that it's going to do that soon.

[Arvind Narayanan]

It would be surprising to them if

[Gary Marcus]

it took more than about three years.

[Interviewer (Andy)]

And because of how high they believe the stakes are, they are engaged right now in what they think may end up being the most important debate in human history.

[Arvind Narayanan]

The basic description I would give to the current scenario is if anyone builds it, everyone dies.

[Interviewer (Andy)]

Why is all the worry about the technology going badly wrong and why are people not worried enough about it not happening?

[Arvind Narayanan]

Our job now, right now, whether you're

[Interviewer (Andy)]

someone building it or someone who is observing people build it, or just a

[Gary Marcus]

person living on this planet, because this

[Interviewer (Andy)]

affects you too, is to collectively figure out how to how we unlock this narrow path, because it is a narrow path we need to navigate. However, there was a perspective that was missing from our series and it's one that many of you listening out there have been writing to us requesting that we Dive into. And that's the AI skeptic. Right now there is so much hype around AI and because human beings have a pretty bad track record of predicting the future, naturally this is going to inspire some deep skepticism. And in full disclosure, when the team and I first started to put together the last invention, this series that we were so excited about, we assumed that we would find plenty of skeptics in the field of AI, but we didn't. Now it's not hard to find people who are skeptical of specific claims being made about AI and the AI future. There's also plenty of people who are skeptical about the motives of the leaders of these AI labs. But it was almost impossible for us to find anyone with AI expertise, anyone who had been working in the field of AI or studying AI that did not think that AGI was a realistic near term possibility and one that needed to be taken seriously. However, there are AI skeptics out there and just like with the AI believers, we are going to break down the AI skeptics into three different groups, all of which you're going to hear from today. The first is one that I think of as the AI is grift camp. They believe that we are all being sold a bunch of snake oil and and they fold the so called AI revolution into their larger criticisms about big tech and the ultra wealthy leaders that they believe are behind all the hype. In contrast, the second kind of skeptic is one that we call the wrong path camp. And these are people who do actually study AI. Some of them are even building AI systems right now. And while they believe that AGI is indeed coming and that we need to get ready for it, they think that the current path we're on, the LLM systems, these large language models that are very popular today, that they just aren't going to get us there alone. And then finally there is the AI as normal technology camp. And interestingly, this group is not skeptical at all about the power of the technology or about the current path we're on with AI right now. They just don't think that human societies are going to be so quick to abandon our preferences for other humans and turn to the machines. And with all that said, we begin now with act one. AI is grift. Ed Zitron, thank you so much for doing this. Can I just have you start off by describing what it is you do for a living these days?

[Ed Zitron]

A lot of different things. I run a PR firm where I do tech pr, but really not AI stuff. I run a newsletter called where's your red app, which is over 85,000 subscribers now, which is crazy.

[Interviewer (Andy)]

Wow.

[Ed Zitron]

I have a podcast called Better Offline on iheartradio and Cool Zone Media where I cover the rot in tech and the powerful way in which they're using tech to control society.

[Interviewer (Andy)]

I think of you as someone who has become almost like a professional critic.

[Ed Zitron]

Yeah.

[Interviewer (Andy)]

Of the whole world of tech. Is that fair?

[Ed Zitron]

I think that's fair, yeah.

[Interviewer (Andy)]

This is Ed Zitron. He is an author, a blogger, a podcaster, and he was once described by a magazine writer as, quote, one of the most pugnacious critics of big tech. And as you'll hear, he is the embodiment of the belief that all of the concern and excitement around AI right now is totally misguided. I think that if I was going to sum up all of your criticism, it is in the claim that you've made many times in the past that, quote, the main thing holding back AI is that it doesn't fucking work. Yeah. Unpack that for me. What do you mean when you say that AI doesn't fucking work?

[Ed Zitron]

I mean that when you look at what is promised from AI, that it is this autonomous system that can complete distinct work units or what have you, I'm saying that's a lie. I'm saying it is not true. It's not what it does. I'm saying that the promises of AI are false. I mean, it's just. They're just not true. AI is replacing jobs. No, it isn't. I realize this is kind of a strange answer, but it's just. Look at it. Just look at what it does, or should I say what it doesn't do. It's one of the strangest things I've seen in my life, I'm going to be honest.

[Interviewer (Andy)]

Are you saying that the popular AI products today, that millions and millions of people use things like ChatGPT or Runway or Claude, that they aren't actually that impressive? Or are you saying something bigger? Are you saying that the AI industry, the AI field that believes it has taken these big leaps forward in the goal to one day make something that is truly an artificial general intelligence, this thing that they believe is going to cure diseases and reshape the economy and transform society for better or for worse, Are you saying that that is bullshit, that we're not on a path like that?

[Ed Zitron]

I think that's mostly accurate. I think the. The field of artificial intelligence means something entirely different now than it used to. I think that, like, AI is a marketing term and it doesn't mean anything. It just means something related to whatever we can raise investment dollars for. Because AI is machine learning. AI is autonomous cars, AI is algorithms, AI is this, that and the other. AI is a bazillion different things to a bazillion different people. This moment is a moment where every authority that is meant to protect us from bullshit, quite frankly, has failed.

[Gary Marcus]

And.

[Ed Zitron]

And where an entire industry has taken advantage of us being a high information, low processing society, where we have so much crap thrown at us, the journalists don't have the time and the mentorship to actually learn stuff. We as people walk around all day trying to do our best. And thus we believe the people on the television when they say AI is doing this, that, and the other. But the people on the television aren't even being honest about what AI can do. And so we're in this situation where the people that are benefiting from this don't give a rat fuck about your or my ability to do anything. They give a rat fuck about making, well, I don't even want to say money of making their investments worth more, which is fundamentally different than making money as in revenue.

[Interviewer (Andy)]

Well, I wanted to get into money more here in a second, but just to get a little bit more specific about your argument here and put it into context for this series, we have so far been talking to the scientists and technologists and researchers and journalists and inventors, all of these people who are connected not just to the developments of AI in this moment, but many of them who have been dedicated to the field of artificial intelligence for their entire adult lives, some of them going back many, many decades.

[Ed Zitron]

You speak to Geoff Hinton by any chance?

[Interviewer (Andy)]

Yeah, yes, we spoke both to Geoffrey Hinton and to Yoshua Bench.

[Ed Zitron]

Ooh, Jeff Hinton. Sorry, I shouldn't mock him.

[Interviewer (Andy)]

Well, what they say is that while AI indeed has a history of being overhyped, that since 2012 and since the breakthroughs that happened at DeepMind in 2015 and at OpenAI in 2022, and the just sheer amount of talent and investment that's being poured into this industry that they have come to, that humanity is in the midst of building a technology so profound that it is going to reshape how we live. And many of them are saying that we are not ready for what's coming. And what I want to know from you is what is it that you are seeing? What is it that you're arguing that you feel Hinton and all these others are missing? Where is your disagreement?

[Ed Zitron]

Well, speaking specifically for Geoff Hinton, I'm not seeing the speaker fees that Jeff Hinton gets. I hate to be that guy. But what's Geoff Hinton been up to for the last few years other than occasionally speaking to somebody and going, I'm scared of the computer. He's like, don't get me wrong, he's a gifted scientist, but what tech is he actually working on?

[Interviewer (Andy)]

Well, I mean, he is to your point. I mean, he retired at the age of 75 a couple years ago in 2023, but then he won the Nobel Prize for physics last year for his contributions to the field of artificial intelligence. And the center of the firestorm that, that we're talking about, like, is like,

[Ed Zitron]

like, here's the thing.

[Interviewer (Andy)]

I mean, I'm just saying, isn't winning a Nobel Prize in this field a notable thing?

[Ed Zitron]

Geoff Hinton. No, I do think it's notable he did, but I don't think that connects directly to any kind of thing about, oh, how this is going, like we're going to get AGI from large language models, or that this is probably proof of intelligent systems. I don't think that those things connect at all.

[Interviewer (Andy)]

I just want to hop in real quick and say that. Because Ed Zitron ended up focusing so much of his criticism around him, we reached out to Geoffrey Hinton for this story. He and his team told us that for the vast majority of his public comments, Hinton does not get paid. And I'll just add that he never attempted to get me to pay him for the comments that he made for the series. Well, I don't want to become like the Jeffrey Hinton defender here, but I'll just say from my interviews with him, from seeing him speak in public, he and many others like him, they also don't necessarily think that the current LLMs, these current large language models, are just going to one day evolve from ChatGPT into a true AGI. But what they're saying is that they are seeing an overall trend in breakthroughs and discoveries in investment throughout their entire field. And that is why they are ringing this alarm or this. That's why they are standing up and saying, hey, get ready.

[Arvind Narayanan]

Great.

[Ed Zitron]

So that's a trend. What does that actually mean in practical speaking? Because here's the thing. Geoff Hinton, gifted physicist, Nobel Prize winner, good on him. Congratulations, Jeff. The thing is, for all his bloviating, Jeff seems to provide very little actual commentary. He knows how to say all the scary sounding stuff that will get him. And I say this is someone who loves to immediate myself get him the Media attention. I'm not saying he is cynical, but at the same time, he's been saying the same crap for two years. He has not really broadened his arguments, nor has he done anything that I would argue helps fix the apparent problem he claims exists. I don't see Geoff Hinton actually contributing much other than doomerism and spreading fear. And there's another word I use for doomerism.

[Interviewer (Andy)]

I.

[Ed Zitron]

Marketing. Think about it. You're an AI company. What is the best thing that could happen? Somebody could say, your product's good, sure. But somebody could say, they're so scared. They're so scared of your product. They're so scared of how good it is, how terrifying. Wow. Best give it a big old valuation.

[Interviewer (Andy)]

Hmm. So just to be clear, you think it's likely that Geoffrey Hinton, this lifelong academic who recently converted from thinking that AGI was, like, so far away, we didn't need to worry about it, to leaving his job to tell the world that we need to get ready for it right now? We need to regulate this industry. We need to prepare our society at every level for what's coming. You think that that might be a part of a marketing ploy? Because I've heard that argument made to attack folks like Sam Altman. But it seems to me that scientists like Geoffrey Hinton, one of the reasons their story is so remarkable to me, is that they have every reason right now to be going around and celebrating. Because after all these years of them having this contrarian approach and being ostracized, the thing that they were fighting for has won the day. They've won all these prizes, but instead of going around celebrating, they're going around warning the world about how important it is. We get ready for what's coming.

[Gary Marcus]

My aim at present, I'm too old

[Arvind Narayanan]

to do new research, but my aim

[Interviewer (Andy)]

is to make the public aware of what's coming and understand the dangers so that they pressure politicians to regulate this

[Arvind Narayanan]

stuff and to worry about the dangers more seriously, to counterbalance the pressure coming

[Gary Marcus]

from the tech CEOs.

[Interviewer (Andy)]

And in Hinton's case, you know, he's using his scientific knowledge to come up with different strategies, like the AI mother strategy, for example, where he believes that we need to create the AGI in some way that makes it behave to humanity as a mother does to her children, so that it will have affection for us, so that it will protect us.

[Ed Zitron]

Wow. I'm sorry, that's just. Is he an evolutionary biologist?

[Interviewer (Andy)]

No. His background is in artificial intelligence, machine learning, and in neuroscience.

[Ed Zitron]

Great. So, but, like, is that the same thing? Because what he is, he is like, I hate to push back so hard on this, but I have to. If a woman went around and said what Geoff Hinton did, you think. Do you think that they would take her seriously? Because what you just described is a bloke saying, wow, what if we made it kind of like a mother? But the actual harm right now that AI is causing, the real one, is that companies are using it as a means of justifying layoffs, that people are having their work stolen from. Millions of people have their work stolen to train these models. Our markets are dependent on this entire thing, this whole large language model generation. Despite no real money being made, and retail investors are piling into Nvidia, for example, these people are going to lose their money. When this falls apart, the financial consequences are obvious. Why doesn't Geoff Hinton seem to care about people being stolen from? Why doesn't he seem to care about black people in disadvantaged neighborhoods who are contracting horrible diseases from these horrible data centers belching toxic flames? Why doesn't that matter to Geoff Hinton? Why isn't Jeff Hinton talking about the actual harms, the actual things going on today here in reality versus whatever fantastical crap he can say on stage? No, I'm pushing back hard here, and

[Interviewer (Andy)]

I'm happy to have you push back hard. I want to get your point of view, and I think that the listeners will be well served to hear your point of view. I just also think it's important to note that Geoffrey Hinton does speak publicly about some version of the issues that you're talking about, like job loss or bias, and people can go online and they can watch those videos. But I do think that you're right, that he, and many like him do believe that these immediate issues, they are of smaller consequence than what they believe is the existential threat that AI poses to our future.

[Ed Zitron]

The larger problem is the one that hasn't happened yet.

[Interviewer (Andy)]

I'm sorry, I don't think that it's actually that uncommon for scientists to worry about things that haven't happened yet. But just to steer this conversation back to the field of artificial intelligence more broadly, one of the reasons that I have found this story so compelling is when I learned how so many of the people who are closest to this technology have what appears to me to be a sincere belief that it is going to profoundly reshape the world. And they are being given all of this money and all this investment to go out and to reshape the world as they are dreaming of.

[Gary Marcus]

Right.

[Interviewer (Andy)]

That feels like it's of great consequence whether or not they actually can. That's a different discussion. But one of the reasons that I am convinced of the sincerity of their belief is in not just talking to them, but in learning how they talk to each other when no one is looking. Not how they're talking on stage, not how they're talking to their investors when they're looking for money. But for example, if you look at the emails that were exchanged in 2017 between the, the different leaders at OpenAI, these are emails that have recently become public because of a lawsuit.

[Ed Zitron]

Yes. The OpenAI nonprofit one with Musk.

[Interviewer (Andy)]

Yes. And in those emails you can see Sam Altman, Elon Musk, Ilya Suskover discussing among themselves their fear that if AGI were to fall into the wrong hands, or if AGI were to be created without the proper guardrails, they believed it could lead to an AI dictatorship. They believe that whoever makes this technology may very well end up holding more power than any other person ever has, maybe more power than any person ever should have. And that kind of conversation around AGI, it lines up with the interviews I've had with former OpenAI employees who say this is how people at the company talk openly over their lunch breaks all of the time. And I'm saying, well, whether or not they're right, it does appear to me that this is a sincere belief. And so I'm asking you, what is bringing you to a different conclusion here? What are you seeing that's convincing you that this isn't sincere?

[Ed Zitron]

I mean this nicely, mate, but are you serious? Are you serious that you are like Elon Musk emails? You're reading those and being like this smart guy who knows what he's talking about. These are bloviating rich guys sitting around talking, like, huffing their own farts, like that Prius episode of South Park. I'm sorry, Elon Musk, a billionaire with Sam Altman, eventually a billionaire. Those two guys are just, they're barely technical themselves. And Elon Musk just says stuff all the time. He's a very. Sam Altman as well. Two very well known liars. So the reason I'm not going to believe those guys is they're liars and they kind of fuck with both of them. Sam Altman is a good con artist and I guess so is Elon. But the reason I think it's profit focused is that's how these companies have acted historically and that's why they're doing this. I mean, These people don't care about AGI. They care about money.

[Interviewer (Andy)]

Well, then I guess just one more question on this. Is there any room for nuance in your views here? Because obviously Sam Altman and Elon Musk are making money. They're making so much money off of getting investors and people around the world to believe that they can bring about an AI future. But do you also think that some of their motivation is built off of this belief that they might usher in a transformative technology that is a benefit to the world? Because if you think back to different figures in history, you know, Ben Franklin, somebody who I'm a big fan of, he obviously wanted to make money with his inventions. He wasn't shy about that. But he also was motivated by his deep belief in the Enlightenment ideals of things like human progress. Right. He hoped that his inventions would serve a higher purpose in the long run and be a benefit to humanity. Or you can look back to early tech pioneers, people like Bill Gates. He wasn't shy about how much money he was going to make, but he also was clear that he was motivated in part by this desire to democratize information. And I'm asking you, like, when it comes to Elon Musk or, you know, if he's too controversial. Jeff Bezos, people who are already super rich now, do you think that it could be true that they are motivated in part by a sincere belief and a desire that they might massively, positively impact the human race, that the technology that they help to usher in might make them figures in the future, like how we think of Newton or Einstein today?

[Ed Zitron]

Well, I think we can start by saying none of the people we've just named invented. Like Elon Musk was not. His name was not on the Attention is all you need paper. Ben Franklin owned slaves. I guess he was an abolitionist eventually, like Bill Gates. I don't know. Don't know what he's been up to recently other than saying he cares about the environment and then going back on that. What these people have succeeded in doing is creating a mystique, is creating a myth around them. The myth of the powerful genius men who have better intentions than making money. But when you look under the hood and what they're actually doing, they're just trying to make money. They're just trying. And you say, oh, they have enough money. That's the myth of capitalism, my friend. That's like when they make more money, they want to make more money than that. These men want more number go up. That is the rot economy. Everything is focused on Growth. These people do not care about the future. These people don't care about anything. I think that they want the number to go up. And I think that the way you're addressing my critique is suggesting that there's something missing when I actually think there's something missing to the narrative that these men have anything other than the most boring and placid and neutral intentions of making a number go up. And if that sounds cynical, I don't know. I don't see any proof to the contrary. In fact, I see every single day more proof, more evidence that the only thing these people care about is money.

[Interviewer (Andy)]

All right, well, before I let you go, I want to ask another question specifically about the technology itself and where it stands right now. Because when I look at the AI models that the public has access to, on the one hand, they obviously seem too unreliable to replace many human workers, right? The chatbots still hallucinate and make answers up. Sometimes the image generators, they're not yet good enough to replace a film crew. But on the other hand, when I look at these products in light of where AI was just a decade ago, man, they seem amazing, right? You've got chatbots that can have deep conversations on almost any subject. You've got these AI systems that you can take a picture of your engine and they'll help you fix your car. You've got these image generators that can create these little movies that are so impressive that people in Hollywood are starting to worry about the future of their industry. And just in my own personal experience, the other day, one of my producers uploaded the logo of the last invention into one of these video generators, and it made this moving graphic that, you know, 10 years ago, I would have had to pay thousands of dollars and wait days, if not weeks, to get work of that quality. And this thing just, boom, spit the whole thing out right in seconds. And it's not good enough quality for me to replace my actual logo with it, but it is impressive. And according to the folks at these AI companies, we are only at the beginning, right? This is just the Pong on Atari stage and where they believe this industry is headed. And I want to ask you, when you see all of this investment going in to these companies, all of these talented people rushing to help them make their dreams come true to create this wild new AI future, does it ever cause you pause? Do you ever wonder if maybe they can pull something truly transformational off or not?

[Ed Zitron]

No.

[Arvind Narayanan]

Like, I just.

[Ed Zitron]

I'm sorry. I hate to do that. No, I think that we're not in the pong stage. They want you to believe we're in the pong stage. We are at the limits of what these things can do. We are absolutely not at some early stage. They want you to believe that. They want you to believe that because they've sunk all that money in. They want you to believe that because if they have to admit that they've done all this for no reason, that would be incredibly embarrassing. So all the king's horses, all the king's men, every member of the media, every podcast, every investor, everyone, every PE person, every bank, they're all tied up in this crap. They've all put all of the money and the attention in the world has gone into this. All of the smartest people bloviate about it every goddamn day. And here we are. The coolest thing we can come up with is, wow, it made a logo that was not good enough to replace your current one, just to be clear. Wasn't good enough. That's what it could do. Kind of interesting. Not good enough to really do anything with it. And I think that is ultimately it. They need you to believe these are the early innings because if you start looking at this and thinking about this even a little and realize that the depth of hubris, everyone involved kind of looks stupid. Everyone. It kind of makes you think maybe these people aren't geniuses at all.

[Interviewer (Andy)]

We'll be back with more AI Skeptics after a short break.

[Matt]

Hello, this is Matt and this episode is sponsored by Ground News. Ground News is one of the most helpful tools that I use every day to avoid the echo chambers and media bias online. Here's an example of what I mean.

[Interviewer (Andy)]

Roll the window down. Roll this one down.

[Arvind Narayanan]

Roll that one down too.

[Matt]

Chicago police pulled over a 26 year old driver named Dexter Reed.

[Interviewer (Andy)]

Don't roll the window up. Don't roll the window up.

[Matt]

Reed first failed to comply with police instructions.

[Interviewer (Andy)]

Unlock the doors now. Unlock the doors now.

[Matt]

And then Reed pulled out a gun and shot at the officers.

[Interviewer (Andy)]

Open the door now.

[Matt]

Resulting in the police firing back aggressively. Now, when I came across this story on the Ground News app, they had done this great service. I could swipe between the headlines to see how different media outlets describe the same story in different headlines. So I would see the Washington Post headline, police fire 96 shots in 41 seconds killing black man during traffic stop next to Chicago's local WTTW headline, officers fired at Dexter Reed 96 times in 41 seconds after he shot officer in arm, right next to the Fox News headline, chicago man opens fire on officers after failing to follow commands. On top of showing different headlines, they offer blind spot reports to show you stories outside your bubble. They collect local reporting on the city or town that you live in. They and they rate news stories and outlets on their level of bias. To go check them out, use the Link Ground News Reflector and you'll get 40% off their unlimited Access Vantage subscription. This is a great way to support them and the work they do because Ground News is a subscriber supported platform. We appreciate what they're up to and we appreciate their support of this podcast. So go check them out and make sure you use our Link Ground News Reflector so they know we sent you.

[Gary Marcus]

The world moves fast.

[Arvind Narayanan]

Your workday even faster.

[Gary Marcus]

Pitching products, Drafting reports, analyzing data Microsoft

[Arvind Narayanan]

365 Copilot is your AI assistant for work built into Word, Excel, PowerPoint, and other Microsoft 365 apps you use, helping you quickly write, analyze, create and summarize

[Interviewer (Andy)]

so you can cut through clutter and and clear a path to your best work.

[Arvind Narayanan]

Learn [e-mail] M365Copilot

[Matt]

this episode is brought to you by Holo Socks. I did not expect to have strong feelings about socks, but then I realized they're the weakest link in the entire clothing ecosystem. You buy great boots, you invest in nice sneakers, but then you sabotage the whole thing with whatever sad thin sock was on sale in a 10 pack and you're shocked that your feet are miserable. Either cold in that deep bone level way or weirdly damp. No middle ground. But then you try hollow. These socks are alpaca and if you've never worn alpaca, it's different. Not bulky, not scratchy. It's soft in this smooth, almost silky way. But the best part is how they regulate temperature. They don't overheat, they don't trap moisture. Your feet just feel steady, like they're not fighting the environment all day. And they don't stink, which honestly is underrated. Whether you are sitting at a desk or on your feet a lot. If you're outside or traveling, or even if you're just tired of your socks betraying you, this is the upgrade that actually makes a difference. For a limited time, Holosox is having a buy2get2 free sale. Head to holosox.com today to check it out. That's hollowsocks.com for up to 50% off your order. After you purchase, they will ask you where you heard about them and you can support our show and tell them we sent you.

[Interviewer (Andy)]

Welcome back. And now for Act 2, we're on the wrong path.

[Gary Marcus]

I'm Gary Marcus, I'm a professor emeritus at New York University. And I've been, I think correctly anticipating the limits of neural networks since my dissertation in 1993.

[Interviewer (Andy)]

So a few weeks ago I was really excited because I got the chance to sit down and speak with Gary Marcus. And he is not only a scientist deeply steeped in the world of AI, but like many of his peers, he's been dreaming of a world shaped by powerful AI systems since he was only a 10 year old child.

[Gary Marcus]

And so I became interested in AI pretty much immediately. Like how do I program this to play chess or tic tac toe? You know, I was an only child. I wanted to learn how to make this thing play games and things like that. When I was about 13, I wrote a database that was, you could ask questions in natural language. Wasn't very good, but you know, I started to understand some of the issues. I wrote a Latin to English translator when I was 15. That allowed me to go to college two years early. So I was interested in AI from the jump.

[Interviewer (Andy)]

And also, like many of the leaders in today's AI labs, he's not thinking of AI as just some new product, but as a super intelligence that might radically change humanity's future.

[Gary Marcus]

In an ideal world, I would love to see superintelligence. There's a million questions about medicine technology we could answer if we had a kind of all knowing oracle that could think things through and simulate the body in perfect detail and so forth. You know, I would love that.

[Interviewer (Andy)]

At some level, where Gary Marcus differs from his peers is when it comes to the excitement over the large language models that, that have been fueling many of today's leading AI products.

[Gary Marcus]

Like, I think AI has enormous potential to help, but generative AI, these chatbots, it's not a good technology.

[Interviewer (Andy)]

And in short, what is your criticism of AI in its current state and embodied in products like ChatGPT or Claude?

[Gary Marcus]

It's inherently prone to all kinds of reasoning errors. It's inherently hard to control. I once called it a bull in a China shop. It's inherently prone to hallucinations, making stuff up, and that hasn't changed. I've been making these points for, depending on how you count six years or 13 years or 30 years, and that hasn't changed. I keep over and over hearing, oh, just give us more data, it will solve these problems. But the reality is, as we're recording this Gemini just came out and it Hallucinates. It's the same story as in 2022, when ChatGPT came out almost three years to the day when we're recording. And I said at the time, the problem with this technology is the way it's built. It's hallucinatory. It's gonna be unreliable. This is not the droids we should be looking for, and it still isn't.

[Interviewer (Andy)]

I appreciate the Star wars reference there. So, going back to 2012, Gary Marcus found himself in a somewhat lonely place. As we covered in the series, Geoffrey Hinton and his grad students, along with these other contrarian AI researchers, they felt that after years and years of trying, their much maligned theories around creating AI babies instead of AI experts had finally won over the field. And then suddenly you had Google and all these other companies starting to take the race for AI more seriously. But Gary, he looked at all of that and he just sort of shrugged.

[Gary Marcus]

It's a better ladder, but it's not necessarily going to get us to the moon.

[Interviewer (Andy)]

And then again in 2022, the new model of ChatGPT emerged and sent shockwaves throughout the world. Gary was not impressed. What were you thinking when you saw all of these people getting so excited about ChatGPT and investing all this money,

[Gary Marcus]

that it's a red herring, that these techniques are cool, but they're basically like a paper that Miller and Chomsky wrote in 1961, showing that as you get bigger databases, you get better approximations to English. But so what? You don't really have the deep semantics that you need, and it's not really going to get you to artificial general intelligence.

[Interviewer (Andy)]

Now, remember that one of the reasons we've been seeing this massive amount of investment and the building out of these data centers all around the world is partly because of the theory that the more we scale up the data and the compute power of these AI systems, the more capable they become. This is what made the huge jump from GPT 2 to 3 or 3 to 4 so exciting. But again, Gary looked at all that and said, this is going to hit a wall.

[Gary Marcus]

Scaling, adding more data and more compute to these systems helps for a certain period of time. People have plotted graphs about it, but those graphs aren't going to continue indefinitely. You can think of lots of cases where if you extrapolated from the first few points on a curve, you get wrong answers. My favorite is now a joke on Twitter, the trillion pound baby. Which is, you extrapolate. The baby weighs so many pounds, £8 at birth, it's £16 at one month. And so you extrapolate that by the time it goes to college, it's going to be a trillion pounds.

[Interviewer (Andy)]

So while most of the field was celebrating and investors were opening up their wallets to pour money into the AI labs, Gary wrote a paper saying that we need to tamp all of that excitement down because this exponential growth, it wasn't going to last.

[Gary Marcus]

You might see another doubling or whatever, but this is not going to continue indefinitely. These are not physical laws of the universe, these scaling extrapolations. And you're going to continue to see with these systems because of the way that they are built. Hallucinations, problems with reasoning, problems with truth, et cetera. And people hated me for saying this. All the major figures in the field took shots on me on Twitter, Altman, LeCun, Musk and so forth. They were really mad. They really made fun of me. And that's actually what happened. We got another doubling to GPT4. It was really quantum leap forward compared to GPT3, the way that GPT3 was a quantum leap forward compared to GPT2. But I kept warning, I said GPT5 may not be that, and it wasn't.

[Interviewer (Andy)]

OpenAI is rolling out its newest artificial intelligence model, GPT5, and the reviews are mixed. What did we Learn today about ChatGPT5? It came out. I watched it. I wasn't impressed by anything, I'll be totally honest. So last fall, after months of Sam Altman hyping up the new capabilities, ChatGPT 5 was released. And while there were some definite improvements in some areas from GPT4, it was still plagued by hallucinations. And it failed to be an exponential jump forward. Early users describe it as less creative and more corporate, with some calling it bland or generic. Others report bugs and uneven responses.

[Gary Marcus]

And what people realized was that I was right. Somebody wrote a book this year, this year using a chatbot called Gary Marcus is right. And it's become a common thing on the Internet. People started writing Gary Marcus apology forms and so forth, because I kept saying five would be delayed, take longer than you think, and it wouldn't be as impressive as he thought.

[Interviewer (Andy)]

I just want to pull back and try and get your take on the broader view here. Because it's one thing to say, look at what AI can't do now, and it's just a whole other thing to say. And it never will, right?

[Gary Marcus]

I never. Sorry, that needs a lot of clarification. The word. It needs clarification there, right?

[Interviewer (Andy)]

Give me some clarification on it.

[Gary Marcus]

Does it mean AI or does it mean large language models and chatbots?

[Interviewer (Andy)]

What I mean by it is the current path in the pursuit, for better or for worse, of artificial general intelligence. This thing that was at one time a pipe dream and that many investors and a lot of people who are excited and a lot of other people who are really worried think might be coming a lot sooner than we think. I'm talking about the current path more than I'm talking about one particular chatbot product or one particular strategy.

[Gary Marcus]

I mean, already they're deviating from the path that I criticize. So what I criticized in 2022 was what I would call a pure large language model, which uses this technique called attention predicts next word, does not use any symbolic systems. And you've talked about symbolic systems in your program. And those systems made limited progress. They made some, but they made limited progress and they ran out of room. They had reached a point of diminishing returns. So they're already doing what I said in 2022 you would need to do, which is putting symbolic systems in there. So they now have all the major systems have code interpreters that are symbolic systems inside of their large language model. So they have already adopted some of what I said we would need. Then there's a whole bunch of startups trying to add world models. You can go back and look at my 2020 paper, the Next Decade in AI, and I said that was one of four critical steps was to add world models or find a system that can make use of world models. So slowly they are encompassing all of the things that I said. So, you know Theseus ship, where you replace all the planks one by one. Is this still Theseus ship? So what they're going to do is they're going to replace every little bit of how the LLMs work with kinds of stuff that I've been talking about and maybe some other stuff that nobody's invented yet, or maybe some stuff that has been invented that I don't know about. And they're still going to want to call it a large language model, but it is not going to work in the same fundamental ways as the large language models that we talk about. When you say the current approach, it's going to be the current approach, plus a lot of extra stuff that did not exist when I made these critiques. They're going to want to take credit and say it's what we were saying all along. But the reality is what's going to actually make it work is having symbolic systems in there, having having world models that are symbolic in there, et cetera.

[Interviewer (Andy)]

Now, here's why. Labeling Gary Marcus and his camp as skeptics can be tricky, because the views that he holds are also held by some of the most enthusiastic believers in the AI revolution, including people like Demis Hassabis, who's the head of Google's DeepMind. And in recent months, even people like Ilyas Heskever, one of the minds behind chatgpt, who pushed really hard and really publicly for this idea that more compute and more data were going to lead to bigger and bigger breakthroughs up until 2020.

[Gary Marcus]

From 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling. Because people say, this is amazing. You got to scale more. Keep scaling. The one word scaling.

[Interviewer (Andy)]

Even he has come around to having a view of much closer to Gary Marcus.

[Gary Marcus]

But now the scale is so big. Like, is the belief really that, oh, it's so big, but if you had 100x more, everything would be so different. Is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true. So it's back to the age of research again, just with big computers.

[Interviewer (Andy)]

And all of these guys, including Gary Marcus, they, they believe that we are still headed towards an AGI future and that it might be here quicker than we think.

[Gary Marcus]

I think that we will get to AGI, we might do it in the next decade, but I don't think we can do it with current techniques. We need new ideas.

[Interviewer (Andy)]

It's just that Gary and many of his peers think that we're going to need another Alphago or another GPT like breakthrough before we get there.

[Gary Marcus]

There's going to be several advances. If I had to guess, I would say there will be between 3 and 20 major advances before we get to AGI. Ideas that either don't exist yet or haven't really been put into production or appreciated yet. There'll be a bunch of new things, and when we get all those pieces together, then we'll have hai.

[Interviewer (Andy)]

And because of that, very unlike Ed Zitron and the AI is grift camp of skeptics, Gary and his camp, they think that we need to take this technology seriously and get ready for its arrival. All right, so in our series, we mainly divided the big players in the great AI debate, as I think of it, between the AI doomers who believe that if anyone builds AGI, everyone dies, right? They are absolutely convinced that this is an existential threat, that we're living through right now. Then there are, broadly speaking, the AI accelerationists who think that the benefits of AGI are so enormous that even though some of them think that there are risks, we should race towards the creation of a safe AGI as fast as we can. And then we have this third group that I call the AI Scouts. And they're the people who think that this could be great, this could be catastrophic. But what we really need to do right now is throw an enormous amount of resources into getting prepared for how transformative the AGI revolution is really going to be.

[Gary Marcus]

I mean, mostly I'm with them. I'm not sure why you're not counting me as a scout.

[Interviewer (Andy)]

Well, maybe that's what I'm asking. Are you a scout? Is that should we understand you as not a skeptic, but a scout?

[Gary Marcus]

One of the places I might differ from some people in that camp is that I'm very concerned about immediate problems as well as what might happen if we get to API. And I think I might be more concerned about that than some of the people who are in that camp. So I'm worried about, for example, discrimination and bias. We just saw a new study yesterday showing that LLMs chatbots used in hiring decisions are racist and that humans who use them tend to pass along the recommendations without thinking. That's bad. That's bad right now. Non consensual deep fake porn is bad and it's here right now. Cybercrime is increasing and you know, that's here right now. There's a whole bunch of problems right now and I'm concerned with those. And I think a lot of the people who you call scouts don't talk about those things or don't talk about them as much as I might. But I do think we have at best 50 years to figure this out. I do think that there's a chance for AI to be positive, I think there's a chance for it to be negative. And we really need to get our shit together. And I would say that LLMs chatbots have been address rehearsal and it's a really depressing train wreck of a dress rehearsal in which we have found out that money and power influence the government to do nothing. And we have a technology that is terrible at so called alignment problem, really not trustworthy and it's getting spread, it's getting investment. We should not be putting a trillion dollars into a technology that is that poor at alignment. We should be putting a trillion dollars into building a better technology.

[Interviewer (Andy)]

And now finally, Act 3 AI as normal technology. Arvind, thank you so much for doing this.

[Arvind Narayanan]

Thank you, Andy. It's good to be here.

[Interviewer (Andy)]

And can we just start off by having you introduce yourself, tell us your name and whatever credentials or title make the most sense for the conversation we're about to have?

[Arvind Narayanan]

My name is Arvind Narayainen. I'm a professor of Computer science at Princeton University. And you know, like many people, I'm trying to bring more empirical evidence, more reasoning into the AI conversation. But I think what's so interesting about this is that many experts who are all coming at this in good faith have ended up in different positions in this landscape. And I perhaps represent one of those interesting poles of this debate.

[Interviewer (Andy)]

Yes, well, that's why I'm very excited to be talking to you. Because up until this moment, we've largely been speaking to different technologists and scientists, people who strongly believe that what's coming in the field of AI, maybe five years, 10 years, 20 years down the road, is going to be not just another new gadget, not just another new tech product, but something that ushers in a hinge moment in human history. And one of the reasons I was excited to talk to you is because you wrote this paper with your co author called AI as a normal Technology. And so I just want to ask, does that mean that you think what's coming is going to be more like the iPhone and how that changed the world and less like the discovery of fire and how that changed, you know, the course of human civilization?

[Arvind Narayanan]

I would say somewhere in between iPhone and fire. All right, so AI as normal technology is not saying that it is just yet another gadget. But we start out by saying that AI is a general purpose technology. And that means that it can be used for many different purposes, almost no matter what line of work you're in, but also in your personal life. So in that sense, it's different from a lot of different gadgets. It's obviously different from a toaster which can do one thing. So right off the bat, that is one difference. But there have been many general purpose technologies before. We've had the Internet, we've had electricity. Going even farther back, we've had the Industrial Revolution. And these are all technologies that, yes, they did change a lot about human society, about work, about the rate of GDP growth. And those were all enormous transformations. But two things to keep in mind. One, they unfolded gradually. So the impact of the Industrial Revolution took many decades, many centuries to be felt. And even when technologies are very powerful, perhaps especially when technologies are very powerful, it Actually takes us a while to adapt our institutions to this new way of doing things. And that's one reason that progress tends to be slow. So that's one thing to keep in mind. And I think the second thing to keep in mind is, yes, we're seeing claims that this could bring about the end of human civilization, either in a utopian way or. Or in a dystopian way. But I think the views that we hear tend to be the more extreme ones. And I think there is room in the conversation for more grounded views on that topic. And I, for one, think that those more extreme claims are pretty overblown.

[Interviewer (Andy)]

Okay, so just for clarity's sake, you are saying that you do believe that an AI revolution may be underway and that it will be transformative eventually. You aren't skeptical of the technology itself per se, but you think that this revolution will be much more like electricity or much more like the Industrial revolution, which slowly transformed society and less like, say, the awakening of a new digital species that, in short order, for better or for worse, upends our entire world very quickly.

[Arvind Narayanan]

That's exactly right. And my view is that we have a choice. It's not up to the AI companies alone to define what this technology is going to be and how we should think about it. How this technology plays out and in fact depends on decisions that we're all going to make about how we're going to integrate it into our various industries and occupations and workflows and what governments do with this, what elementary school teachers decide to do with it. So collectively, we have the agency to make the AI future take different possible trajectories. So that's one point of disagreement. And the other one is that I think my co author and I, Sayaj Kapoor, and I think this is all going to take decades. It's not necessarily going to be a whole lot faster than the speed with which electricity disrupted various industries.

[Interviewer (Andy)]

All right, so I looked this up before our conversation today, and I learned that with electricity, even if you time it to when Thomas Edison built the first commercial power station in 1882, it still took roughly 50 to 60 years before electricity was widely available throughout the American population. Or if you look at the telephone, Alexander Graham Bell, he patents that in 1876, but it wasn't until the 1940s and into the 50s before it actually reached a majority of American homes. So that's like 60, 70 years. And you're saying that the AI revolution underway right now is more likely to play out like that, that it's not going to be a sudden hinge moment in human history like some people are predicting.

[Arvind Narayanan]

That's exactly right. I don't think there's going to be a hinge moment. And to be clear, this is nothing to be complacent about. Even the speed with which the Internet transforms society. Yes, decades. But we weren't prepared for something happening on the timescale of decades. Social media has had enormous impacts on kids, mental health and things like that, and the research on that and the response from policymakers, all of that has taken a while to get going, arguably not fast enough. And we might end up making those same mistakes with AI. So it's quite possible that this will still be fast enough for things to go pretty badly, at least initially. But that said, that's still a very different story from some critical capability threshold at which AI is now writing the rules of history and it's happening at superhuman speed, faster than humans can think or respond to. I think that is a sci fi scenario. I don't think that's going to happen. And yeah, we lay out in our essay why we think that is, and to put it in a sentence or two, it's because we think change will happen at the speed of human behavioral change, at the speed at which organizations can adapt and transform. And it doesn't matter how fast capabilities continue to increase. We admit that capabilities might continue increasing rapidly, but that doesn't mean that it's going to be a force that transforms society with the rest of us having nothing to do but stand by and watch.

[Interviewer (Andy)]

Okay, so I want to get into your theory about what you think is going to happen to slow down change in the event that we really do discover or create a true AGI. Because when I was talking to one AI accelerationist on background for the story, he somewhat shared your view that it wouldn't be a sudden hinge moment. But he was telling me that he thinks that the technology is going to be able to be super transformative, but he was just worried about the state of our messy world. He was saying that there's just so many regulations out there, there's so much bureaucracy out there, there's all these issues with our supply chain and that even if we had a true AGI right now, it would get stuck in this man made bottleneck and that would slow down the advancements in medicine and the advancements in science that he wanted to see in the world. And then on top of that, he also believed that many people are just afraid of artificial intelligence as an idea and that they would do all that they could to slow down or to stop the integration of this technology quickly in the economy and in society. Is your argument along those similar lines that essentially our world is just too messy and will not be able to quickly transform no matter what benchmarks the industry hits?

[Arvind Narayanan]

I mean, that's exactly my point. And I think maybe this person sees that as a bad thing. And in my view that's mostly a good thing that humans are not so eager to hand over control to AI systems. To be clear, in many cases that can be a bad thing. So I find it pretty upsetting the extent of the backlash to self driving cars. For instance. This is a technology that can save something like 1 million lives per year that are lost today in car accidents. And self driving cars, at least waymos are already much safer than human drivers and they're getting a lot safer very quickly. So this is an area where I would hope that there can be consensus that yes, there are risks from AI systems. Yes, these cars sometimes can get into kinds of accidents that a human would have the common sense to avoid, but statistically they're much safer. So let's figure out the right guardrails and let's figure out how do we handle the labor displacement. The gig workers are going to lose their jobs and the truckers and so forth. There has to be a way for society to compensate them. But with all that said, let's go ahead with this. The benefits are enormous. That would be the approach that I hope society would take. It's not the approach that society is taking. There are so many vested interests who are pushing back against this technology. And that's even in a case where an evidence based look at the situation is just so compelling that we should go ahead with the technology. In other cases where there are genuinely multiple sides to the situation, using AI in law or medicine or any other field, I think the extent of resistance that we're seeing and which we're going to continue to see is enormous. And there is just no way in which one day we're all going to wake up and decide to trust these AI systems. And so maybe that's a bad thing. Maybe we're going to lose out on some opportunities to make medical progress faster than we otherwise could. But it's also a good thing because to me that's the main reason to have some comfort that these sci fi scenarios of AI taking over one day are not going to come to pass because people are just not eager to give up power.

[Interviewer (Andy)]

All right, so give me your version of what you think is likely to happen if, say, the AI labs are right and somewhere in the next five, 10, 20, 30 years, one of them is able to pull off the creation of a true AGI thinking machine. Speak to the listener out there who says, how would that not transform the world very quickly?

[Arvind Narayanan]

The number of tasks in our economy today that people do that already could be automated is shockingly high. I wouldn't be surprised if it's more than 50%. Yeah. So that's the place to start. So why is it that there are so many things that can already be automated and yet we have people doing them? I think there are a lot of well understood reasons for this. One of them is that in a lot of cases we prefer human interaction just going to a restaurant. We prefer interaction with a server who is human as opposed to always ordering from an automated menu. So that's a simple example. But in a lot of cases there are deeper reasons. Let me give you an example from the most recent election cycle in Wyoming. In Cheyenne, Wyoming, there was a guy who said if he were elected, the mayor would be ChatGPT. He called it VIC for Virtual Integrated Citizen and claimed he had built the spot. But from what I can tell, it was just chatgpt behind the scene. And he said it had an IQ of 155 and it can make decisions instantly. And not just decisions about how much to spend on infrastructure or whatever, but even contentious political decisions like book bans and so forth. ChatGPT is unbiased and neutral, and so we can have it make all the decisions and then we can trust it. That was his vision. And by the way, I learned about this because the Washington Post called me to ask, what are the risks of having an AI mayor? I was very confused by that question. I was like, what do you mean, risks of having an AI mayor? It's like asking, what are the risks of replacing a car with a cardboard cutout of a car.

[Interviewer (Andy)]

You're saying that it just lacks the capabilities technically to be mayor? What exactly are you trying to say there?

[Arvind Narayanan]

Well, not quite. I don't think it's a matter of capabilities. Just to continue with this car analogy. I mean, it looks like a car, but the risk is that you don't have a car anymore. I think this look at capabilities is fooling us a little bit because the point of having human decision makers in politics, the reason that politics is so messy and inefficient is because that's the forum that we have chosen as a society for resolving our deepest Differences. And all of that inefficiency, all of that messiness that happens is the point of politics. And try to replace that with an automated system is to completely miss the point. At best, this ChatGPT mayor will reflect the views of OpenAI developers who have programmed it to act in certain ways. It won't reflect the will of the people. It's just missing the point of democracy. This might be a little bit of an extreme example, but one theme that comes up in our work is that so many of the decisions that happen in the course of our work actually have a moral component. We don't recognize this because that part is relatively easy for us. And the cognitive component of our work is the hard one. And yet, if we're able to automate away a lot of the cognitive labor, we will still need humans making most of those decisions for accountability, for making sure that this is normatively in line with what society wants. And, and because with AI systems, at least as of today, and I don't know if this will change in 2050, we can never be sure that as the world changes, as these systems encounter a situation that has never arisen in the past, you can't necessarily be sure that they're not going to go off the rails in completely unforeseen ways. And so there are these unknown unknowns. So what is going to be the accountability when that happens? So I think for all of these reasons, even when you hit those capability thresholds, I think we're going to see nothing even remotely similar to an overnight replacement of human labor by automation. The effects are going to be much more nuanced and complex.

[Interviewer (Andy)]

That's so interesting. I wasn't thinking about this before talking to you, but as I hear you making your case, I'm reminded that before I was a reporter, I was a bartender for many years, and I worked at a lot of those fancy cocktail bars. And we used to joke that a machine could definitely do our job. You don't really need a human there shaking up the drink in front of you in a bar, but people just like it. They like having a friendly bartender, they like seeing us shake up their drink and put it into the little coop. And obviously this is also the case for a number of other occupations. I think about musicians. We love to connect with the humanity behind our favorite songs, right? People get obsessive about Bob Dylan or Taylor Swift or whatever. I also think this is the case when it comes to human teachers. There's a lot that could probably be outsourced to AI right now. But we like it when a human teaches another human in a way. Are you saying that this human preference that we have socially, that this is not going to go away even if more and more capable AI systems are suddenly in our lives?

[Arvind Narayanan]

I mean, we don't even have to extrapolate into the future. There are so many historical precedents that we can learn from. In the early 2010s, when everybody started having easy access to broadband and Internet video, the big hope in the higher education system was that we have all of these professors at elite universities who are so close to the research, they're often the ones generating knowledge. Surely it would be better for everybody in the world to be able to directly learn from them as opposed to their state college. They go to where the professors are not necessarily at the top of their game. Or so the thinking went. And so let's create these MOOCs. MOOC, massive online open courses where professors from Stanford and Harvard and Princeton are going to be the ones teaching everybody in the world. And you only need a handful of elite universities in the future. You don't need the rest of the university system. Complete misunderstanding of what the point of college is. It turns out that a lot of what students want out of the college experience is not necessarily even hearing from professors. It's a lot of other stuff, such as the networking with their peers and so forth. But even to the extent that they're learning from their professors, it turns out that the critical ingredient is not information transmission. Of course there are always books you can read on your own. If what it takes to learn is just getting the information, but rather the fact that you're in a social environment that's conducive to learning. I think at this point at least well understood in the education community. The critical thing we're providing is that social environment. Yes, there is a role for technology there, but I think it will have to be a matter of carefully integrating it. Replacing the social environment with a computer system. I don't know, it seems pretty silly. It's a non starter. It's just at the same level as your example of replacing the human bartending experience with something where you can certainly go to a liquor store and get your own alcohol. We've always had that. And so it's just missing the point of what the system is trying to accomplish.

[Interviewer (Andy)]

I love that. That is a very compelling case that you're making this idea that we just like people. But let me push back on it a little bit here because there are a number of ways that even Right now, we are watching technology begin to replace different human interactions. And maybe this isn't a perfect example of this, but I think about young men and porn, right? Porn, it's probably as old as civilization itself. It goes back forever. But in recent years, with the rise of the Internet, we have seen this massive increase in the amount of porn that men are watching. And we can compare that with the amount of time that they're spending actually engaging and investing in human romantic relationships or even in just having sex with other humans. And maybe you can extrapolate from that out also how social media has become, for many people, the dominant way that they're interacting with other humans. They're spending more hours interacting and looking at faces and reading the words or listening to the words on a screen than they are looking at the faces of people in the real world and interacting and talking with people in the real world. Right? Isn't it the case that if we were to stay on the current trend and throw into that a super intelligent AI distractor or entertainer, that we might wally ourselves, that we might sleepwalk into a world in which it is dominating us?

[Arvind Narayanan]

This is exactly what motivates all of my research. The difference is I don't think this is the nightmare scenario that people are warning about. They're warning about a situation where AI itself is the adversary and it has agency and it decides to take the fate of humanity into its own hands. I think the harms we're seeing are all the result of our own collective bad decisions and the lack of policy guardrails and that sort of thing. Literally my whole career has been about trying to bring more empirical evidence into policy debates so that we can try to put guardrails into some of these systems before it is too late. Most recently, for instance, my colleagues and I have been spending a lot of time on the issue of AI chatbots and mental health. What we're seeing with people having suicidal ideation, some of the numbers coming out of this are really striking. OpenAI revealed that 1.2 million people per week are revealing suicidal thoughts to ChatGPT. And I think I saw a number from a Sam Altman interview recently that 1500 people per week are discussing suicide with ChatGPT and then actually going on to take their own life. Just absolutely striking numbers. And then so many people getting into delusional spirals because these chatbots are reinforcing. You know, they're sycophantic, they are overly eager to agree with whatever the user is saying. And then you get into a spiral where the user is expressing more and more extreme beliefs and the chatbot is validating and amplifying those at each step, sometimes with catastrophic consequences. There is a lawsuit recently in the case of a guy called Stein Eric Solberg, who killed his mother and then killed himself because of this delusional spiral with ChatGPT. That's just one example, a striking one tip of the iceberg. But this sort of thing seems to be happening on a massive scale. None of the people who were worried about existential risk came anywhere close to recognizing that this sort of thing might be a possibility. You only figure out what the actual risks are when you do the hard work of paying very close attention to the empirical data, data that's often coming from social science and this kind of on the ground research, as opposed to theoretical speculations about hinge points at which society might change all at once in the future. So, yeah, I'm very much on board and Indeed I spent 90% of my time thinking about and trying to mitigate exactly the kinds of risks you're talking about, which to me are categorically different from the ones that we started the episode with.

[Interviewer (Andy)]

Mm. So where do you think that comes from? Why is there such a pervasive belief among many of the people who are close to this technology? A belief that I think is a sincere conviction that what they're making is less like electricity and maybe more like a God or a new species, something that poses a real existential threat in and of itself? Do you question their sincerity? I've heard some people say that this is all about making money. I've also heard some people say that these technologists are not religious people and maybe they are trying to fill a God shaped hole in their heart by creating something like a digital God. Where do you stand on that? Why is this such a big part of the debate and the discussion among so many of the people who are close to this technology?

[Arvind Narayanan]

I mean, like you said, these are sincere beliefs. I don't know about the God stuff, but look, I used to hold these beliefs and so I do think I can at least share my experiences, even if I can't speak for everyone who holds these beliefs. So 25 plus years ago, the reason I decided to major in computer science was because I thought that the arrival of AGI would be a civilization defining moment and I wanted some part to play in that. So, first of all, there's a big, big, big selection effect among AI technologists. For a lot of folks, the reason we got into this in the first place is that we wanted to be part of something that's. I guess this is getting a little uncomfortably close to your God related hypothesis, but we want it to be part of something that's bigger than ourselves. That's humanity defining. So there's a selection effect there. So the people who are in this field are in it because they're kind of prepared, predisposed perhaps to think that this could be the greatest technology ever. So that's one thing. And the second is it's true again, like I acknowledged at the beginning, that the capability improvements have been repeatedly underestimated. I mean, think about it from the perspective of some of the folks you interviewed before, and I'm sure this came through in those episodes. To have this belief that neural network technology is capable of a lot more than people gave it credit for back in the 1980s, and to have persisted through, through that belief for decades while they were being ignored, and then to finally be proven right. I cannot even imagine what that must feel like. The idea that the world is constantly underestimating the power of these technical insights and that you and a few others alone are able to speak that truth and must cut through this kind of wall of skepticism that you're facing. I think that just intrinsically puts you in a position where it becomes very easy to ignore a lot of contradictory evidence because people have been skeptical before and been proven wrong, even though it took a couple of decades. But I think where folks are kind of unable to make a mental leap, and this is what I struggled with for a couple of decades until things started finally falling into place, is that there is not this linear, technologically determinist relationship between tech capabilities and societal effects. There is such a long causal chain of steps through which technology gets translated to societal impacts, whether good or bad, and that those are all opportunities for collective agency over the direction of the technology's impacts. And it's only when I really started getting deeply into the writing and sociology and political science and policy studies and economics and so many other fields, and together with years of conversations with my co author Syas Kapoor, that we felt that we had even a semi coherent way of putting together what we agree is a rapid progress in technology with a theory for how that is likely to affect society.

[Interviewer (Andy)]

And as we close here, would it be wrong for me to summarize my takeaway from this interview as essentially, you are a skeptic, but not necessarily of the milestones that this industry might hit, but a skeptic of the fact that our big, messy bureaucratic world can be utterly changed in a short amount of time. Is that about it?

[Arvind Narayanan]

Exactly right. There's this funny example that Amtrak, with a lot of fanfare, launched its new train sets that were capable of going much faster than their old ones. But it turns out that in reality, they got no speed improvements because the tracks are curved. That puts a maximum limit on your speed, and so you're improving a part of the system that's not even the bottleneck. And every time I hear a lot of these AI claims, I think about Amtrak's experience with its tracks and its new train sets.

[Interviewer (Andy)]

All right, so final question here. Ever since we started putting out this podcast series, we've been hearing from tons of people that it's freaked them out, right? People who are scared at some of the claims that people are making, scared about some of the predictions that people are making. And I don't like scaring people. And yet I want to accurately report the debate that's happening in these increasingly powerful corners of society. I think that in this episode, people are going to hear you and they're going to breathe a sigh of relief. They're going to want to believe you. But how do you balance that with what you're also saying about how you still believe that we need to have a sense of urgency in how we investigate and regulate and study artificial intelligence and all the changes that you believe it will eventually bring to our world? I wonder, could you just speak to that listener? What is your message for them?

[Arvind Narayanan]

I have a very simple message. And if what one is worried about is existential risk, the vast majority of us can do nothing about it. The only two kinds of stakeholders who can do something are people working at these frontier AI companies. And they can try to come up with some technical breakthrough that allows them to develop super intelligence without these existential risks, which we've written about, we think is highly implausible. Or there can be a world authoritarian government that makes the rules for who is allowed to develop AI that surpasses some capabilities. So all the options that we have for superintelligence risk are fairly absurd ones. And I think that's the reason why there's a sense of panic. So it kind of leaves you with a sense of helplessness. But the kinds of risks that I'm talking about, every single one of us has a role to play in doing something about it. For a simple starting point is whatever your profession is probably needs to come up with a normative framework for what are ethically acceptable and unacceptable uses of AI when replacing human functions with AI is actually bringing great benefits, like saving lives in the case of self driving cars versus when it's the case that the human performance of something is the whole point, whether it's music or bartending or a teacher who's setting up a social learning environment in a classroom and where the incursion of AI would actually be terribly counterproductive and we need to collectively resist certain commercially motivated but ultimately socially harmful ways of deploying AI. That's just one example, but there are many other ones. Right? So for the kinds of fears that I'm talking about, every one of us has a role to play in our own organizations, families, communities. If you're raising kids, how should they be thinking about AI systems? What should they know about it? When is the right age to start introducing them to AI? So these are all things that we can start thinking about and it's not something we should or can leave to the AI companies. And so hopefully the kind of fears that I'm talking about leave you with a sense of purpose as opposed to a sense of helplessness.

[Interviewer (Andy)]

All right, well, thank you so much for talking with me for sharing your view. I really appreciate it.

[Arvind Narayanan]

Thank you. This has been fun.

[Matt]

Thank you for listening. The Last Invention is produced by Longview Home for the the curious and open minded. To learn more about us and to support our work, click on the link in our Show Notes or Visit us at longviewinvestigations.com we'll be back with more of the views shaping the debate around the AI revolution soon. We'll see you then. This episode is sponsored by Ground News, the app that helps you spot media bias and see a broader picture of the news shaping our world. Get 40% off their vantage plan at Ground News Reflector so they know we sent you.