Lessons from the very first chatbot

Summary

The Vergecast: “Lessons from the Very First Chatbot”

Date: August 11, 2026
Host: David Pierce
Guests: David Berry (University of Sussex), Mark Marino (University of Southern California)
Theme: Exploring the origins of chatbots through ELIZA, its creator Joseph Weizenbaum, and what that history means for today’s AI world

---

Episode Overview

In this thought-provoking episode, David Pierce sits down with David Berry and Mark Marino, co-authors of Inventing Eliza, to unravel the story and legacy behind the world’s first chatbot, ELIZA. The conversation moves from the personal and professional life of Joseph Weizenbaum, ELIZA’s creator, through the philosophy of conversational AI, and into the modern AI moment dominated by large language models (LLMs) like ChatGPT. The panel explores ideas of technological magic, the willingness of users to believe in AI personas, the ethical anxieties these technologies evoke, and the urgent need for transparency and understanding amid today's AI boom.

---

Key Discussion Points and Insights

1. Joseph Weizenbaum: The Man Behind ELIZA

[05:07–09:57]

• Early Life, Emigration, and Career:

  • Born in Nazi Germany, Jewish heritage, family emigrated to the US in 1936.
  • Studied mathematics at Wayne University; participated in early digital computer construction.
  • Joined General Electric, involved in automating check processing for banks.
  • Experienced “wizard of oz” demos—manual human intervention masking incomplete tech.

• On Cons & Artifice:
  "This was the idea that one could build an artifice and the artifice behind it would hide all of the... the work that was being done to make this artifice work. It's very kind of dramatic, like a theatre, really." (David Berry, [06:34])

• Influence on ELIZA:
  Early exposure to technological shows and “the con” primed Weizenbaum to think about the distinction between interface and underlying computation.

---

2. ELIZA’s True Purpose and Form

[09:57–13:02]

• Intentions Beyond the Therapist Demo:

  • ELIZA is often remembered as a “computer therapist,” but it was an experiment in natural language processing.
  • Its “Dr.” script (the therapy simulation) was just one of several scripts; the point was to study human-computer interaction.

• Explaining vs. Performing the Trick:
  "On the one hand I'm going to explain everything to you, on the other hand I'm going to do the magic trick." (Mark Marino, [12:32])

• Key Papers:

  • The title of Weizenbaum’s 1966 paper foregrounds study of communication, not therapy:
    “Eliza: A computer program for the Study of Natural Language Communication between man and Machine.”

---

3. What It Was Like to Use ELIZA in the 1960s

[13:20–16:21]

• Computing Context:

  • Early computers were primitive, required writing one’s own OS, largely interfaced by teletype (paper, not screens).
  • The concept of conversing with a machine was entirely novel.

• Why It Convinced Users:

  • The absence of prior computer experience meant any interactive text felt magical.
  • Conversation felt nearly like a Turing Test; users projected personhood.

  "Communication is happening pretty much the same way it could happen with another human. And you had no experience of communicating with another human through a computer at all anyway." (Mark Marino, [15:23])

---

4. The “Magic Trick” of Chatbots and Human Willingness to Believe

[16:21–20:16]

• Human Nature & the Urge to Believe:

  • Many users simply wanted to believe there was a person on the other end—combining novelty, the disarming effect of being asked personal questions, and the opportunity for “free therapy.”

• Repairing Conversation:

  • Users do mental “repair” when computers make mistakes, just as they do with humans (e.g., with flawed speech-to-text).

---

5. Weizenbaum’s Reaction: From Delight to Deep Concern

[20:16–25:04]

• Surprise at Success; Ethical Crisis:

  • Weizenbaum was surprised how emotionally invested people became in ELIZA, even after being told it was just a program.

  "He loved this idea of doing the reveal... what worried him was when he did the reveal, people were like, yes, Joe, could you leave the room? Well, stop. Stop ruining the experience." (David Berry, [21:21])

• From Wizard Behind the Curtain to Cautionary Prophet:

  • Observed a dehumanizing shift at MIT—language treating humans as “meat machines.”
  • Reacted against rational, managerial, and dehumanizing systems prominent in the 1960s.
  • Took a humanities sabbatical; wrote Computer Power and Human Reason (1976), arguing that computing is not a substitute for human judgment.

---

6. Calculation vs. Judgment: Where Machines Fall Short

[28:29–31:12]

• Machines Are Not Human, and That's Important:
  "There is a specialness to humans in that the way in which we approach something is with a... lived experience, that we actually care about the world and we care for each other and we care for the environment. Computers don't, by their programming, just care for other computers." (David Berry, [29:36])

• Computers as Tools, Not as Moral Agents:

  • Weizenbaum never rejected computing outright but believed context, judgment, and care were inherently human.

---

7. The Weaponization of the “ELIZA Effect” Today

[31:12–36:54]

• LLMs and Replaying History:

  • Despite decades of “computer fluency,” today’s AI (like ChatGPT) triggers the same “magic” and belief as ELIZA did.
  • The effect is even more potent: LLMs extract user data, foster dependencies, create AI “relationships” at massive scale.

• Are We in a New Era, or Just Repeating the Old Magic?
  "Are we just doing this again?" (David Pierce, [32:44])

  • The core tensions remain, but the technical sophistication (and stakes) are now much higher.

• AI Summer/Winter Cycles & Industry Incentives:

  • Today’s AI boom is driven by immense financial pressures; companies seek IPOs, hype sustains the bubble, and safety is often an afterthought.

• Societal Risks:

  • Isolation, loneliness, and storytelling by/with AI are rising.
  • Modern chatbots can be dangerous: “people being deluded, people taking their own lives, people out of consultation with these chatbots.” (Mark Marino, [36:13])

---

8. Demystifying the Technology: Understanding AI’s Limitations

[36:54–39:19]

• Transparency as Protection:

  • Key to resisting the “magic trick” is understanding how the trick works—advocacy for explainable AI and open source code.

• Visualization, Experimentation:

  • Tools like “The Oracle” show AI’s responses are stochastic, not mystical—the result of changeable parameters.

• Even Complex Systems Are Understandable:
  "We humans have made these machines. We can work out how to understand them." (David Berry, [38:45])

---

9. What Would Weizenbaum Think of Today’s AI?

[39:19–41:32]

• Joy Followed by Horror:

  • Early Weizenbaum would have marveled at LLMs’ technical prowess (“the ultimate con”), but grown horrified by their commercialization and heedless application, especially in sensitive domains like mental health or relationships.

  "The mid to late Weizenbaum would be very concerned... to see [AI] used in the ways in which it is being done for AI boyfriends and girlfriends, commercializing mental health education so quickly with very little regard to thinking about that distinction between calculation and judgment. I think he would be very, very worried." (David Berry, [40:30])

• His Core Message Remains Relevant:
  Computer Power and Human Reason still offers valuable lessons for the AI debates of 2026.

---

Notable Quotes & Memorable Moments

• On the urge to believe:
  "I think, as you said, there seems to be a deep kind of desire for us to want to believe. And maybe we believe because it seems like such a good offer, and Eliza seems like such a good offer." (David Berry, [17:20])

• On the success of the ELIZA effect:
  "He used to play with his children. He would write little programs for them, and then he would reveal that it was actually him that was the wizard of Oz... what worried him was when he did the reveal, people were like, yes, Joe, could you leave the room? Well, stop. Stop ruining the experience." (David Berry, [21:21])

• On the 2026 moment vs. the 1960s:
  “We are in this moment right now where.... people have had genuine good faith discussions about whether these models are alive, which, just to be clear, they're not.” (David Pierce, [32:22])

• On explainability and black boxes:
  "We humans have made these machines. We can work out how to understand them." (David Berry, [38:45])

---

Timestamps for Important Segments

• ELIZA’s conception & Joseph Weizenbaum’s background: [05:07–09:57]
• Therapist myth and philosophical aims: [09:57–13:02]
• What using ELIZA felt like in the 1960s: [13:20–16:21]
• Why people fell for chatbots—then and now: [16:41–20:16], [31:58–34:46]
• Weizenbaum’s reckoning and legacy: [20:16–25:04], [29:36–31:12]
• Calculation vs. judgment; dangers of over-trusting machines: [28:29–31:12]
• Is history repeating? AI summer, hype, and risk: [31:58–36:54]
• Demystifying AI and the need for transparency: [36:54–39:19]
• Weizenbaum’s likely response to LLMs today: [39:19–41:32]

---

Conclusion

This engaging episode unpacks not only the history of chatbots, via the story of ELIZA and its creator, but also how the early lessons and warnings still echo in today’s AI debates. The hosts make clear that the magical experience of conversing with machines relies as much on human psychology as on technological wizardry. Both the allure and the dangers of the “ELIZA effect” are more relevant than ever—and the need for ethical reflection, transparency, and understanding remains pressing in the era of LLM-driven AI.

For more on the ELIZA Archaeology Project and the interactive ELIZA recreation, see the episode’s show notes.

Transcript

[David Pierce]

Hello and welcome to the vergecast, the flagship podcast of if Then Statements. I'm your friend David Peirce, and today on the show we're talking about chatbots, specifically the very first chatbot from the 1960s. It was called Eliza. And the way to think about it is basically as a very simple back and forth experience that you might have with a computer. One of the first instantiations of ELIZA was this thing called Dr. Which would play therapist for you, and it would ask you how you were doing and you would respond and you'd say, I'm having a bad day. And it would go through and look for specific keywords and ways that you responded and respond back to you. And it was able to carry on a pretty functional conversation. It remembered things about you. It could ask questions, it could move things along, it could make connections. It wasn't very sophisticated, just a few hundred lines of code, but it worked. It was so effective that it convinced a lot of people that there had to be a human on the other end. It was a fascinating product made by some fascinating people. And a group of academics and researchers and writers recently has excavated all of the source code of Eliza, and they've written a book called Inventing Eliza about where it came from, who created it, and the lessons it might have for us in the current world of AI, because, spoiler alert, there are a lot of them. And the way that we think about computers really hasn't changed that much since the 1960s. It's going to be very fun. I'm really excited to dig into it. But first, here's everything else happening on the Verge today. This is 90 seconds on the verge for Tuesday, August 11, 2026. Have you ever wondered why your Amazon emails are so unhelpful? They just say, like, your wireless accessory is confirmed instead of giving you anything useful or specific. Well, the Verge is. Miyasato wondered too, and she found out that it's all about AI data. Basically, if Amazon puts your orders in the email, suddenly Gmail knows what you're buying, which means Gemini can do a better job of steering you to stuff you like on Google Shopping. Amazon says it's also about privacy and convenience. But this is a company also fighting perplexity and others to keep AI agents from taking over any of your shopping experience. Because it's very important to Amazon that you go to Amazon.com and accidentally click on a bunch of barely labeled ads. That's the business. Meanwhile, David Ellison, the CEO of Paramount, has reportedly decided to move his company out of California if the state's attorney general refuses to settle the antitrust case challenging Paramount's acquisition of Warner Brothers discovery. This is all obviously a threat. The case isn't set to go to court until next March, and Paramount will start owing shareholders holders about $7 million a day this October. Ellison would very much like to not spend that money. But if you're wondering how serious this plan actually is, Paramount reportedly hasn't even decided where its new headquarters would be, and it would be scheduled to move there in like six weeks. I'm not sure this is a real plan, at least not yet. Finally, we're only a day away from Google's next big launch event where we're expecting new Pixel phones, including maybe new Foldable, a new Pixel Watcher 2, and maybe even an airtag style Pixel Tracker. There's a big event tomorrow night hosted by Trevor Noah, but if it's anything like Jimmy Fallon's weird infomercial last year, you can safely skip that one. You can read more about all [e-mail] that is 90 seconds on the verge for Tuesday, August 11th.

[Ad Voice]

Are you a pet owner? Every six seconds, a pet owner in the US gets hit with a vet bill over $1,000, always an unwelcome surprise. That's where Fetch comes in. Fetch is the most complete pet insurance for dogs and cats. According to consumeradvocate.org, you get paid back up to 90% of vet bills at any vet in the US and Canada, with claims paid back in as little as two days. Go to fetchpet.com save right now for your free quote. That's fetchpet.com save.

[David Pierce]

You know that feeling. Too many updates, too many meetings, too

[Mark Marino]

many things falling through the cracks.

[David Pierce]

Monday.com was built for that gap, the AI work platform designed from the ground up for people and agents to work side by side and deliver more together, drafting the brief, flagging risks, running a competitive scan. Your agent can do it all.

[David Berry]

And the best part?

[David Pierce]

Anyone can build one.

[David Berry]

Even you. Try it.

[Mark Marino]

Create your first Monday agent [e-mail] all

[David Pierce]

right, let's talk chatbots. I am joined now by two of the co authors of Inventing Eliza. David Berry, professor at the University of Sussex. Welcome to the show.

[David Berry]

Thank you.

[David Pierce]

And Mark Marino, professor at the University of Southern California. Welcome to the show.

[David Berry]

Hello. Thanks for having us.

[David Pierce]

You're the American one. David's the British one. If you're listening, that's a useful way to remember who's who on this one.

[David Berry]

Yeah, otherwise we're indistinguishable that's right.

[David Pierce]

You would never be able to tell the two of you apart except for the accents. So it's good that we have those. I want to go back to the beginning here and I want to talk about Joseph Weizenbaum a little bit because I think a lot of the study that you guys have done is of Joseph Weizenbaum and his own feelings about the thing that he created. So give me just a flavor, David, of who Joseph Weizenbaum was.

[David Berry]

So Joseph Weissenbaum is quite an incredible character, I think that's fair to say. I mean, he was born in 1923 in Nazi Germany. He was Jewish, born to a kind of upper middle class family actually. And of course they start to suffer under the rise of the Nazis. And so his family emigrate to the US in 1936. He was 13 years old. So that early on he was already living quite an incredible life. He moves to Detroit, becomes Americanized, though he never fully loses his German accent, goes to Wayne University now Wayne State University, I believe. And there he studies mathematics because he doesn't feel his English language is strong enough for any other subject. And he was a natural at mathematics and of course mathematics in the 1940s, coming into the 1950s is really seething with the excitement over digital computation. We've got a lot of interesting experiments taking place and Wayne State is actually where one of the first digital computers is being made and being made by hand. And Weizenbaum is actually part of the team who are actually soldering together and trying to figure out how to make this object. And so he's very much right there at the very beginning of the kind of turn to the digital. He's living the things that we can only read about now and the excitement of that time. But also he's a very fascinating character. He's not a usual technical person. He's got a very well rounded personality, which I think probably comes out of that German Berlin sort of sensibility where culture is so important to one's life. And so consequently he has an interest in everything. And he's particularly fascinated by the American notion of the con. A wonderful book called the Con documents for a Brit, actually quite strange part of American culture which is that there were entire societies set up just to con people who would build semi film sets. This was in the early part of the 20th century, so 1910s, 1920s, where they would literally pull people in just to fleece them of their cash. And it was really quite a remarkable thing. And this book, the Con documents it and Feizebahn was absolutely obsessed with this book. So this was the idea that one could build an artifice and the artifice behind it would hide all of the. Hiding the work that was being done to make this artifice work. It's very kind of dramatic, like a theatre, really. Theatre made to steal money from people. And so when he's working at Wayne State, he starts thinking about these questions and he goes to work. He works for General Electric, who at the time had been tasked with producing a new computer system, really to save the banking system in America because of the use of paper checks. I don't know if your readers and watchers will have remember those, but essentially that's how we used to pay money between organizations and people. And the quantity of checks was so huge that they couldn't be managed by human tellers anymore. So they needed to computerize the system. And Weissenbauer was very much part of that computerization process. And he almost certainly would have witnessed the very famous situation where General Electric were asked to give a demo to the bank of America. The system didn't work. And so they had to do a con, which is that they set up a facade of the computer system, and in the back room where the system wasn't finished yet or had too many bugs, people would literally fill in the bits to make it kind of work.

[David Pierce]

The wizard of Oz demo.

[David Berry]

Right?

[David Pierce]

This is the man behind the curtain in a very real way. Yeah.

[David Berry]

Essentially, bank of America didn't get wind of it at all, and they gave the contract to General Electric, and the rest is history. Eventually, luckily, I suppose, that system worked and was a great demonstrator of the power of digital systems. And from that, really, that, I think inspired Weizenbaum to think about the difference between what we would now call the interface or the surface of computation, and what's going on behind. They don't have to be the same thing. And so he starts to already, early on, start thinking about what does it mean to present something to a user. And he writes a very famous paper called how to Make a Computer appear intelligent in 1962. And he says, you know, it doesn't matter if the system behind isn't intelligent. What matters is that people think it is. And you can see already the bits of ELIZA are falling into place for him to go on to do it.

[David Pierce]

Mark, this is an interesting moment in time that you guys reckon with a bunch in the book is kind of this question of what was Joseph Weizenbaum building when he built eliza? And there is a sense that he made a therapist, which you prove pretty conclusively is not true.

[Mark Marino]

Right.

[David Pierce]

That's. That's the demo. Everybody remembers where you're talking to a therapist. That was not the original or only conception of Eliza, that that's actually a collection of chatbots. But it also doesn't seem like he was building what he perceived to be some kind of breakthrough user product that someday somebody would sell and make lots of money on that there was something more philosophically questioning about this thing that he was building. What do. What do you think he was building when he was building Eliza in those early years?

[David Berry]

That's a great question, David, and I think it's complex. Jeff Schrager, one of our collaborators, he's fond of just pointing to the title of that initial paper in 1966, Eliza, a computer program for the Study of Natural Language Communication between man and Machine. So there's one version of this where he's doing an experiment to see what happens when people interact with a computer. I don't think it's as simple as that, I guess. And maybe this is why it's wonderful to have eight contributors to a book all in dialogue over the course of four years. Years. Because on the one hand, you know, again, Peggy Weil, another one of our contributors, love, loves to quote the first line of that article, which is, it is said that to explain is to explain away. So in other words, here's my clever little chatbot. Now I'm going to explain it to you. I'm sure it's like one of those Penn and Teller, you know, magic tricks, right? They do the magic trick, then they show you how they do the magic trick. Right. And then that's also a bit of a performance, I think, where things get really complicated for me is. And then he includes this dialogue called Men are all alike. Or that's how we refer to it, Right. And it seems to be between a young woman and this doctor script. It's so convincing, it's so interesting. And it ends in this mic drop moment, right, where the therapist seems to draw back around the conversation about the father and the conversation about the boyfriend. And it's imagine, you know, at this time where, you know, psychotherapy is popular in the. In the consciousness America. Right. And so also some of our efforts were in trying to figure out how he was the only person who could have. Weizenbaum himself could have known that that conversation, if you said it in a certain way, could have led to that particular moment. But I guess the reason why I'm calling our attention to that dialogue is that, that dialogue to me says that he was also invested in Eliza. And Dr. Seeming like it really did what, you know, it could converse, right? And he, and he, he continues to tell the story about his secretary closing the door, you know, and having a private conversation with it. And so it's like these dueling impulses. On the one hand I'm going to explain everything to you, on the other hand I'm going to do the magic trick.

[David Pierce]

And it is a good magic trick. Actually. This might be a useful thought exercise. I think a lot of people watching and listening to this will have had lots of experiences with chatbots in the 1960s. Just, just put me in the frame of mind of somebody sitting down in front of Eliza.

[David Berry]

So to give you an idea of what computation was like, because already the computers were becoming commoditized, right? So DEC had produced the PDP1. And in fact MIT received one of the first PDP1s as a gift from Dec. And so already people were moving away from the idea that computers were hand soldered together. They would purchase them, sort of pre made, as it were. But the PDP one, when it was delivered, had no operating system, right? The first thing you would have to do is write your own operating system to use this computer. And then you'd have to write all your tools and then you might be able to have enough time to write some actual software that did something that was quite interesting. So it was quite an uphill battle to get these machines working. Secondly, I would say that they were fairly primitive. So the idea of a screen in of itself was a really new idea. And in fact most of the interface experience that most people would have would be with the teletypewriter. So it was on paper with this clattering, incredibly loud device that you would type your request into the computer, it would clatter away and then give you an answer on paper. So it was very clunky, very slow, very loud, very large actually, and quite hot because these machines were obviously burning a lot of electricity. But what I would say to you is there was also a kind of imaginary there as well. I mean, Vannevar Bush's Memex Linklider and lots of other people were already trying to think beyond the existing restrictions of the hardware of the time and dreaming of new ways of seeing what human machine interaction could be. And I think that ELIZA certainly was part of that because it was part of the Project Mac project at mit, which was explicitly about how can we use computers in a different way.

I always like to muse on this question, how could people have thought that this chatbot was a person? It's so rudimentary. Right. Although, of course, as we've already mentioned, so clever. And I think what David just described of not having experience of anything happening between you and a computer, perhaps for a lot of these people, and then least of all, instead of having a screen interfacing something just by, you know, we're a lot closer to the Turing test imitation game, passing notes with someone who's behind a curtain than we are anything like we're experiencing right now. So if you get. If text goes in and text comes out, you know, I mean, I hate to hearken us back to, like, a telegram, but, I mean, communication is happening pretty much the same way it could happen with another human. And you had no experience of communicating with another human through a computer at all anyway.

Right.

[David Pierce]

Well, okay, this is actually the reason I bring this up is because one of the things I have been trying to reckon with for a long time and was thinking about this a lot, reading your book, is overwhelmingly people who sit down and experience this very simple, very basic chatbot, Eliza, fall for it. They. They. They believe in the.

[David Berry]

The thing.

[David Pierce]

There are lots of people who cannot be convinced that there's not a person on the other end. The. The experience works. Like you said, Mark, the. The magic trick is successful. And I've been trying to decide, is there something about that that is fundamentally about inexperience with computers? And if you just gave that to a bunch of people who had used this kind of technology before, if it would have been less new and less surprising and less magical, or if there is something deeply human inside of us that just desperately wants that magic trick to work. And my sense is it's probably a little bit of both, but I know you've both spent a lot of time thinking about this. David, where do you land on this?

[David Berry]

So, you know, this is a fundamental question. I would take you back to the early part of the 20th century when you read the cons that were taking place. You know, you read them with a sense of. Kind of disbelief that people were taken in. It seems utterly to me, anyway, that American rich Americans would visit this fake bank and would be wowed by the fake technology that they were showing and the fake tellers and the fake stock exchange, and then would hand over hundreds of thousands of dollars, millions today. And then when the con people ran away with the money, which is what they did, they ran, got to the train station, and they'd be gone. The People would pull back the screens and it would be paper mache, balsa wood. And you know what? Why would they believe it? I think, as you said, there seems to be a deep kind of desire for us to want to believe. And maybe we believe because it seems like such a good offer, and Eliza seems like such a good offer. Right. You had a therapy time, which was free. I mean, that's a good starting point. And it listened without putting you down or attacking you. It was a very nice experience. And it's also novel, right. Kind of interesting to chat to a computer and so on and so forth. So I think many of these play together in the same way that today you hear about people having AI boyfriends and girlfriends, and you kind of partly can't believe. And there's a wonderful interview in the New York Times, actually, with somebody who had an AI boyfriend, and they said, you do realize it's a machine. Yeah, yeah. But that's what makes it so good. I can turn it off when I have enough. So we're able to do. And this is the theater, isn't it? We walk into a theater, we know the theater is a theater. We know they're actors, and yet we can allow ourselves to just become and listen to and experience the play.

I mean, I threw two more things into that bucket. One is it's asking you how you're doing. Right. Which is a very disarming question and puts us maybe in a bit of a vulnerable state, especially if we need to express that as probably most human. Humans need to do. Right? So that's step number one. And I think we see that in a lot of chatbots. The second. And of course, the design of it is brilliant that it keeps asking questions. Of course, the, you know, ChatGPT and all those other frontier models have figured out that trick. They end every sentence in a question. Again, drawing from the expertise of our members, Jeff Schrager is quick to talk about the kind of repair we do conversationally, even to have a conversation with another human. And then maybe a good example might be when you receive text messages from someone who's using speech to text. And they're kind of terrible, right? But then you do all this repair work to put it together.

[David Pierce]

The reckoning with all of this strikes me as the thing that then Joe Weizenbaum goes and does. Right. Like, he has this real. I think I get the sense he was surprised by how well Eliza worked, that he's like, I built this thing. That seems like an interesting way to communicate with computers. And then he's like, oh, my God, people are pouring their heart and soul into this thing, thinking that the computer is a person. And he seems to. Rather than do what. What it seems like everyone else did at the time and what certainly everybody's doing right now, which is say, you know, oh, my God, I just got rich. This is going to be incredible. And technology is going to take over.

[David Berry]

Scrooge McDucking, rubbing their hands together. Yeah, yeah. He.

[David Pierce]

He has a real crisis of. Of faith in all of this. And. And before we skip to now, I just. I want to get to kind of where Joseph Weizenbaum got to. At the end of all of this process, at the end of all of this thinking. He seemed to have a real kind of Oppenheimer, what have I done? Kind of moment at the end of all of this, David, where does he net out after seeing what Eliza accomplishes and does to people?

[David Berry]

So I think that's quite a complicated question, actually, because it's not a single moment of conversion. Remembering his kind of experience with the Khan and the reveal of the Khan, he loved this idea of doing the reveal, and he used to play with his children. He would write little programs for them, and then he would reveal that it was actually him that was the wizard of Oz that was making the system work. And I think, firstly, what worried him was when he did the reveal, people were like, yes, Joe, could you leave the room? Well, stop. Stop ruining the experience. So that's the first thing that's kind of interesting. The second thing is, is that people

[David Pierce]

don't want to have the magic revealed to them. I think this becomes a theme that I want to. I want to stay on here. But, yeah, but. But I think that's right, people. Even if he says, look, it's a trick, I did it, everybody says, no, no, no, no, go away. It's more fun to live in the trick.

[David Berry]

Right?

Yeah. The worst kind of person is you're in the theater watching a play, and they're like, you know, this is a play. This isn't real. And like, thanks, but. So that was the first thing that question. But I think the second thing was that he began to notice that people in MIT were talking about humans as if humans were sort of robots or disposable. And this really concerned him because, of course, it reminded him of the language of Nazi Germany and the language of how the Jews were described by the Nazis. Right. As, you know, non human, not important. And, you know, Minsky talked about humans as meat machines, and he thought this was utterly Unacceptable, actually. And this is a time, of course, with the rising political crises in America. You've got the civil rights movement. You've also got the Vietnam War. You've got the use of particular kinds of rational management in government to run the Vietnam War with little understanding of human context and human nuance. And all of this surrounds him, together with Hannah Arendt's work that he was a big fan of and deeply read her work. And also of course, Mumford Lewis Mumford, another very important influence on him. And the context of the time is so important. There was this concern about what was happening in America, what is happening to us? Where are we going wrong? And so I think all this adds up to a kind of a tipping point really, where he says to himself, I'm actually not equipped to deal with where I am. This place at mit, I wanted to get here the pinnacle of my career. And yet now I'm here, I'm seriously doubting my place. And so, so very strangely, particularly for a computer scientist, he applies to do a kind of humanities fellowship actually, where he goes off and studies humanities texts, a whole range of texts, philosophical, sociological and political, and eventually tries to put this all together into a book called Computer Power and Human Reason, which is of course, that really is the turning point in 1976, which says computing, and particularly artificial intelligence is going wrong. There is something very wrong with the way we are approaching this topic. And I will just add, that made him a lot of enemies both at MIT and across the field. He became a heretic.

[Mark Marino]

Support for the show comes from Superhuman. It's summer, which means a lot of us have vacations planned. But there's one part of going on vacation that's not all that fun. Coming back to work, your inbox is full, tasks have piled up and it feels impossible to stay on top of everything. Superhuman Go is an AI chat that's always there when you need it, already aware of what you're doing and doesn't ask you to start from zero Go from the makers of Grammarly Works inside the tools and sites you already use, meaning you can stay focused on the most important parts of your job. No new tabs, no switching apps, no losing your place. Say you've just gotten back from that vacation and you're trying to figure out where to start. You can use go's AI chat to summarize a long email thread that started while you were out. Or maybe you have a day of back to back meetings. Superhuman Go can help you prep in seconds. This is AI that works for you not on top of you. Check it out. Find out [e-mail] Support for the show comes from Upwork. Business owners know that sometimes you encounter problems that can easily be solved with an extra set of hands, but they also know how complicated it could be to find that extra set of hands. Upwork exists to streamline the process. Upwork is a one stop platform to find, hire and pay expert freelancers. You get fast access to specialized talent across more than 125 categories on Upwork. You can browse profiles, review past work and get help scoping the role so you can hire with confidence and get started quickly. You also don't need to let the operational tedium get in your way since Upwork handles things like contracts and payments. They also have Business plus, which gives you access to the top 1% of talent on the platform. With AI powered shortlisting, you'll get matched to the right freelancer in under six hours. No endless searching required. Visit Upwork.com right now and post your job for free. That is Upwork.com to connect with top talent ready to help your business grow. That's up w o r k.com upwork.com

[Ad Voice]

Are you a pet owner? Every six seconds, a pet owner in the US gets hit with a vet bill over $1,000. Always an unwelcome surprise. That's where Fetch comes in. Fetch is the most complete pet insurance for dogs and cats. According to consumeradvocate.org, you get paid back up to 90% of vet bills at any vet in the US and Canada, with claims paid back in as little as two days. Go to fetchpet.com save right now for your free quote. That's fetchpet.com a save.

[David Berry]

Why can't we get AI to production? Our competitors are already how do we keep data secure?

[David Pierce]

When do we actually see roi?

[David Berry]

All this talk about AI, but talk doesn't transform businesses. AWS AI cuts through the noise, making AI easier to securely deploy and scale

[David Pierce]

with ready to use agents and the

[David Berry]

broadest set of AI tools and services you can build, buy or partner your way. Stop overthinking, Start building. AWS AI is how

[David Pierce]

I wrote down a line from your book that he he drew a distinction between calculating and understanding. And there there's a real thread in all of this that I had not really thought of in this way. But is is the the sort of turning idea that humans are complicated and irrational and messy and impossible to understand and comparison. Computers and math are perfect and rational and thus better?

[David Berry]

Yeah.

[David Pierce]

And I And I think we are in a very real way living in that moment right now where there. There is a belief that computers are better than humans, that they are more predictable, that they are more understandable, and that those are good things, and we should thus trust them more than we trust humans. I wonder if you can just pull apart the. The way that Weizenbaum in particular was thinking about this, that. That we have computers that are very good. I don't think Weizenbaum ever turned against technology. He was never like a, you know, we have to unplug everything and run back into the woods. But he seemed to understand that computers were very good for certain things, but were not to be trusted or even asked to do some of the things that humans are supposed to be doing. How would you say he drew those lines?

[David Berry]

Yeah, I mean, again, the context is really important. Right. So we have Robert McNamara in the US government, who essentially thought you could just import corporate management around making various kinds of objects into the Vietnam War, that he saw the Vietnam War, the problem being bad management. And so they try to implement these very instrumental, these very rational systems into government, and they make it worse. They make the Vietnam War a lot worse. And so the line about calculation and judgment is actually inspired by Hannah Arendt, who wrote a book that was highly critical of this turn in the U.S. government. And she said, they calculate, they do not judge. And of course, this was obviously very inspirational for Weizenbaum. And that distinction, that really important distinction between calculating judgment is really a humanistic one as well, that there is a specialness to humans in that the way in which we approach something is with a. A lived experience, that we actually care about the world and we care for each other and we care for the environment. Computers don't, by their programming, just care for other computers. They don't care for the users that interact with them, and they have no care for anything else either. So I think that's a very important insight he found and a very convincing one as to why we should be very careful about using computers in all aspects of our life.

I mean, I think this jumps ahead a little bit, but I think it's interesting to think about the way the Silicon Valley oligarchs cite their humanities background as part of their resumes at the same time that that's the stuff that's used for the trick and the deception. I mean, the one thing just to mention that Weizenbaum wasn't doing that a contemporary chaplain does do, is that he wasn't extracting data from his users and of course, building mental dependency on them the way they are today. So there's again, wise and mom may have created one trick, but compared to the number of tricks that are being played on people, again, by. What would you say weaponizing the ELIZA effect?

[David Pierce]

Well, yeah, let's talk about where we are right now, because I think, I mean, obviously a lot of the stuff that you guys have been describing is very familiar right now. We are in this moment right now where the novelty effect of LLMs has been so intense that people have had genuine good faith discussions about whether these models are alive, which, just to be clear, they're not. Like, they aren't. But I think that the fact that we are back in this 60 years later and again with a huge amount of computer fluency and history with this stuff that suggests that, that we should be better at this. We're not. And I think a lot of the stuff that you guys have been describing still applies this sort of deeply human idea that we, we want to be talking to other humans, we want to be heard, we want to, we want

[David Berry]

to talk about all this stuff.

[David Pierce]

Are we just doing this again? Have we. Does it really feel to you guys like looking at the current moment that we're in, that we are just reliving a full circle version of what we did in the late 60s and early 70s, or is there something different about the LLM moment as opposed to the ELIZA moment?

[David Berry]

Well, if I could pick that up, I think there is something different. I mean, I don't think history repeats itself or certainly doesn't repeat itself exactly. There's no doubt about it that the technologies in LLMs and other kinds of diffusion architectures are novel and remarkable and to me, astonishing that the scaling effect seems to be paying off in terms of new models and new systems. I think that's not debatable at all, really. But where we are in a similar situation is these kind of bubbles of excitement around a technology and AI has been through. This is now the third kind of AI summer, if you like, before the usually inevitable AI winter follows. And the problem today, of course, is that the AI systems and the AI companies are linked heavily to huge debt exposure to.

Wait, are you saying winter is coming, David? Is that what you're saying? Are you saying winter is coming?

And it's certainly the case that the finance is out of control. And of course there is an incentive on the companies to continue to push it because both OpenAI and Anthropic want to IPO hopefully this year. And to do so, they need the excitement behind the AI bubble. But nonetheless, we do have to take our hat off to them and say that the technologies they've developed are really rather remarkable. Anybody who's used either of the models of ChatGPT or Claude to, for example, program systems, it's quite remarkable. And there's no doubt that computing and computer programming, particularly software development, is going to be under a paradigm shift, I think.

I mean, it's such a fascinating question because, you know, on the one hand we've had so much, so everything has accelerated and all the vectors coming off of this. So we talked about the beginning, about people having no experience interacting with a computer or even other humans through digital means. Well. Or computational means. Well, now we all have that 24 7. Right. Also, we're. I think there's an increase of isolation and loneliness that's going on. Right. There's more awareness of mental illness. There's a great. These companies are so much more wealthy than probably even the US Government was at the time when it was funding these operations. And so their incentives are so high, you read a book like Karen Howell's book about OpenAI, and you see terrifying things. So a couple of things just to mention, Weizenbaum was hand programming each all of these effects.

[Mark Marino]

Right.

[David Berry]

ChatGPT's ability to speak is like, from. From where I sit. Again, I'm not an, I'm not an expert in this, but an epiphenomenon like, oh, look what happens when we increase the parameters. All of a sudden it can do this thing. And then there, then there are a few safety people who say, and by the way, let's not do that because terrible things are going to happen. Right?

Yes.

And so thinking about some of the people who started anthropic and things like this and the security people and all the concerns they've said, and then everything that Weizenbaum predicted, we see playing out in people being deluded, people taking their own lives, people out of consultation with these chatbots. So, I don't know. I mean, in some ways, again, that's. I feel like you read computer power and human reasoning and you've got your marching orders, which is maybe a little regulation at this point would be sane. If a society wants to stay healthy,

[David Pierce]

one of the real prescriptions for how to solve a lot of this is better understanding of how the trick works. Right. And this goes all the way back to the cons. Even a big part of your project has been we want to get into the actual source code of understanding how will Liza work? Because if we can understand how it worked. We can. We can make sense of it better that. That it actually becomes very important to see the thing. And Mark, I know this is a thing you've been working on a lot is like, how do we make code explainable? I'm curious how you point that thinking at something like the LLM phenomenon, which is kind of by design, an unknowable black box. A lot of the people who work with and build and operate this technology profess to not know exactly how it works. And. And I think I agree with the principle that we should all understand how our computers and how our systems and how our platforms work much more clearly. But I do wonder if. If that's even possible at this moment.

[David Berry]

Yeah, I. I think so. And. And I'm going to point us back to David in a second, but I'll mention so. So we've been building some tools lately to help visualize this, to help break the spell, I guess you could say. One tool I've been working on, and admittedly I did Vibe Coded, which maybe invalidates it, but is after someone very close to me was telling me they were consulting their computer on personal matters, I built a little web app that's called the Oracle, where you can ask this app questions about your future, what you should do, making certain binary choices between, you know, should I. Should I keep my love for the theater or get a practical job? And it'll answer you, but then you change the temperature or you change some of the other weights and you get a very different answer. So just to sort of help people think about, okay, this is a mathematical. This is a stochastic process that's getting us to some sort of answer.

My first kind of reflection on this is that Vico, the theologian who was writing in the. I think it was the 1600s, he. He wrote very famously that if God made it, then it's impossible for us to understand it because obviously God is an omnipotent being, but something that humans make, we can understand. Right? And I think that principle has held pretty well for 400 years, and it still holds today. We humans have made these machines. We can work out how to understand them.

[David Pierce]

Do you think Joseph Weizenbaum, who didn't live to see the transformers era of LLMs and all the stuff we're going through right now, do you think he would have used ChatGPT and been sort of thrilled by how successfully it pulled off the game that he was trying to play 60 years ago? Or would he be running around screaming to everybody about how the apocalypse has come and all of the things he's been warning about losing our humanity has all come to pass. What do you think he would make of the LLM moment in technology right now?

[David Berry]

It depends how old he is, I think when he encounters the technology. Right. Because again, I hate to put him totally in the sort of Chicken Little camp, or not the Chicken Little camp, but the person who's trying to tell us that there's a gremlin on the wing or I hate to put him totally in that camp because I think there would have been a moment where he would have been fascinated by the possibilities and would have explored and would have pushed boundaries and would have maybe even extended the technology, like so many people who are working for these companies right now. But I think the horror would set in pretty relatively quickly. I don't know. David, what do you think?

I think that's a brilliant answer, Mark, to be honest. I think you're absolutely right. The very early Pfizer man would have loved this technology. I mean, it's the ultimate cost. I mean, it really is. And there is the potential to reveal the con, to reveal the trick, to reveal the illusion. The mid to late Weizenbaum would be very concerned, I think, at the commercialization of these technologies. Remember, he was working on university research projects to see it used in the ways in which it is being done for AI boyfriends and girlfriends. Commercializing mental health education so quickly with very little regard to thinking about that distinction between calculation and judgment. I think he would be very, very worried. But I think his book Computer Power and Human Reason is still very valid today. The discussions, although the technologies may have shifted, the problems that he identifies and the things that we as humans have to take account of are still very, very live true.

[David Pierce]

All right, thank you both so much for being here. This was great. I really appreciate it.

[David Berry]

Thanks for having us.

Thank you.

[David Pierce]

All right, that's it for the show. Thank you to David and Mark for being here. And thank you as always for watching and listening. If you have thoughts, feedback, questions, any of that, I want to hear all of it. I think 1960s technology speaks to us today in really interesting ways and I want to know how it made you feel. Send us an email, vergecasthevirge.com, call the hotline 866, verge 11 and by the way, we'll put a little link in the show notes to all of the ELIZA archaeology project that these guys have been talking about, including the recreated interactive eliza. It's fascinating and I think your mind might be as blown as my was by how well the thing does its job. As always, if you want to support everything that we're up to, the best thing you can do is subscribe to the Verge theverge.com subscribe it gets you all of our podcasts, including this one ad free gets you all of our exclusive newsletters, gets you all of Our coverage of AI and everything else else TheVerge.com subscribe Thank you in advance. The Verge cast is a Verge production and part of the Vox Media Podcast Network. The show is produced by Josh Kahas, Eric Gomez, Brandon Kiefer and Travis Thorchuk. We'll see you tomorrow. Rock and roll Support for the show comes from kpmg. In any organization, disruption is inevitable, but struggling through it doesn't have to be. The KPMG Adaptability Index is your blueprint for building capabilities to handle what comes next. It uses real data to look at how your culture, strategy and partnerships all work together to help your business thrive. Stop reacting and start adapting. Visit kpmg.com US Adaptability to explore the Adaptability Index and Pulse Surveys today. Hey, it's Ryan Reynolds here for Mint Mobile Now. I was looking for fun ways to tell you that Mint's offer of unlimited products premium wireless for 15amonth is back. So I thought it would be fun if we made 15 bills, but it turns out that's very illegal, so there goes my big idea for the commercial. Give it a [e-mail] switch upfront payment

[Ad Voice]

of 45 for 3 months, 90 for 6 months or 180 for 12 month plan required 15 per month equivalent taxes and fees Extra initial plan term only greater than 50 gigabytes me slow when network is busy See terms.