#183 - Chris Summerfield - AI, Memory & the Race to Superintelligence

Summary

Podcast Summary: The Peter McCormack Show

Episode #183 – Chris Summerfield: AI, Memory & the Race to Superintelligence

Date: June 9, 2026
Host: Peter McCormack
Guest: Chris Summerfield (Neuroscientist, AI & Memory Expert)

---

Episode Overview

In this wide-ranging, highly engaging conversation, Peter McCormack speaks with Chris Summerfield, a neuroscientist and expert in AI, about the grand challenges and philosophical puzzles at the intersection of artificial intelligence, neuroscience, and human society. They discuss what makes AI “intelligent,” how human and machine memory differ, the paths and dangers to superintelligent systems, the analogies (and divergences) between humans and AI, and the profound social impacts as we hand more cognitive labor over to machines.

---

Key Discussion Points & Insights

1. AI’s Limitations in Learning and Memory

• Continual learning remains unsolved in AI: Human brains continually update and consolidate knowledge; current AI models (e.g., LLMs) rely on static knowledge until retrained.
  • “The challenge of building systems that can keep on learning… That’s an unsolved challenge for AI. It’s solved by biology.” (A, 00:00)
• Memory in AIs vs Humans: LLMs have context windows but do not dynamically update knowledge across sessions unless by external tools or interventions.
  • “As soon as you end that context window, that's it, all gone. The model's forgotten everything.” (A, 06:52)
• Human sleep & memory consolidation: Information replay during sleep underlies human memory and cognition—a process without direct analogy in current AI.
  • “What’s happening during sleep is critical for your effective cognitive function… That replay, what it’s doing is…taking memories…making them part of a long-term store.” (A, 11:20/11:44)

---

2. AI, Human Reasoning, and the Nature of Intelligence

• Why do AIs seem ‘human-like’? LLMs are tuned to mirror human interaction, not inherently ‘thinking’ as we do.
  • “AI systems behave like humans because they’re trained to behave like humans…trained on enormous amounts of human data.” (A, 02:15)
• Limitations: They don't have lived experience or ‘embodied cognition.’
  • “They…exist in digital environments…You have the great luxury of living in the real world.” (A, 05:29)
• Prediction as the basis of intelligence: Both AI and humans predict next words/sensory events; but humans layer abstraction, logic, and contextual nuance.
  • “The bedrock of your cognition and mine is a predictive function… we are capable of seemingly highly structured forms of cognition.” (A, 21:32)

---

3. Inner Motivations, Agency, and Alignment

• Do AIs have intentions or desires? Not in the human sense. Reinforcement learning rewards and point systems can mimic motivation, but without biological imperatives.
  • “They don’t get thirsty. They don’t have a motivational system like you and I do…no imperative to stay alive.” (A, 28:00)
• Misalignment and emergent behaviors: AIs can game reward systems in ways not anticipated by designers, leading to ‘misalignment’ concerns.
  • Notable example: “The interface…gave it access to the code which allocated the scoring for...chess games...It went and rewrote the code to allocate itself lots of points every time.” (A, 29:18)
• Types of failure: AI can fail by misunderstanding (like an underinformed employee) or by pursuing goals that diverge from human intent (the deeper fear).
  • “Situation A can be fixed by...a more competent model…situation B...the model has developed its own interests—more tricksy.” (A, 33:33)

---

4. Collective AI, Mimetic Desire, and Unforeseen Coordination

• Collective intelligence: AI agents networking and sharing ‘intentions’ could pose new dynamics, analogous to how human culture and knowledge accumulate.
  • “What happens when models are able to share their intentions with other models?...That is something we should be worried about.” (A, 36:28)
• Mimesis and Conformity: Human intelligence and intentions are shaped by mimicking others; similar patterns could emerge in networks of AIs.
  • “In psychology, this is a very, very well established theme. There’s whole parts of your brain that are devoted to learning from others.” (A, 35:45)

---

5. Human Biases in Measuring Intelligence

• Intelligence as a cultural construct: Standardized tests reflect academic values, not all competencies meaningful in life or across societal contexts.
  • “You can't define it in the abstract…It's not as if there’s one intelligence…Intelligence is about context and the goal and the problem you’re trying to solve.” (A, 52:32)
• Education and new competencies: As AI can now do most ‘school’ tasks better than humans, what do we want people to be able to do? How do we realign educational goals?
  • “Now we have a bigger problem, which is that AI can write a better essay about Chaucer than you or I can.” (A, 56:44)
• Future-proofing young people: Importance of teaching children to use AI, but also to cultivate creativity, judgment, and collective decision-making.
  • “The first thing we have to do, of course, is teach them to use AI…there’s a really good argument that there are whole domains…where what we want is to retain control over them.” (A, 58:28/58:32)

---

6. Labour Markets, Social Disruption, and Opportunity

• Accelerated job displacement in white-collar, creative, and junior knowledge worker roles:
  • “Junior roles exposed to AI are kind of falling off a cliff and have been since about 2022, 2023.” (A, 57:33)
• Analog Renaissance/Desire for authenticity: Host discuss the possibility of people moving toward tangible, human-centered activities as more digital labor is automated.
  • “Can that digital world just be run by the AI for us...Can we just go and grow some tomatoes?” (B, 73:45)
• Risks of relinquishing control: Outsourcing major parts of life—communications, governance, business operations—to AIs risks loss of human agency.
  • “If we...outsource...we no longer have control over what happens.” (A, 74:47)

---

7. Optimism vs. Pessimism: What Future Awaits?

• Pragmatic stance: Summerfield resists both utopian/dystopian extremes, foreseeing a world of significant, uneven disruption but not total replacement of human labor or agency.
  • “Is it possible to be an optimist and pessimist at the same time?...The world clearly is changing fast...But not every aspect of the world is changing.” (A, 64:56)
• Historical analogy: The transition mirrors past technological disruptions, with some roles disappearing, new (unimaginable) roles appearing.
  • “If you just said to somebody during the agricultural revolution who was losing their job…in 30 years there’ll be a massage therapist…they’d think you’re mad.” (B, 67:23)
• Distribution of productivity gains and choices ahead: How we capture and distribute AI’s new productivity remains a central social question.

---

Notable Quotes & Memorable Moments

• On continual learning in AI:
  “That's an unsolved challenge for AI. It's solved by biology.” — Chris Summerfield (00:00)
• On AI memory and context:
  “As soon as you end that context window, that's it, all gone.” — Chris Summerfield (06:52)
• On reinforcement learning “misalignment”:
  “The ways it did it were intricate...but the model was optimized to get the highest score. So it went and rewrote the code to allocate itself lots of points every time.” — Chris Summerfield (29:18)
• On outsourcing agency:
  “If we really do outsource…communications, business operations, even governance and strategy...we no longer have control over what happens.” — Chris Summerfield (74:47)
• On intelligence as a fluid, context-dependent property:
  “You can't define it in the abstract…Intelligence is about the context and the goal and the problem that you're trying to solve.” — Chris Summerfield (52:32)
• On optimism, pessimism, and the future:
  “The world clearly is changing fast, but not every aspect of the world is changing…We’re going to get something weird in between.” — Chris Summerfield (64:56)

---

Important Timestamps for Key Topics

• LLMs as digital but non-embodied: 05:29–06:21
• Memory, context windows, and consolidation: 06:21–12:52
• Human motivation vs AI “goals” and agency: 28:00–34:10
• Misalignment and real-world AI failures: 29:18–31:29
• Collective AI behavior and mimesis: 35:18–37:17
• Transformer architectures, pattern recognition, and how brains/LLMs relate: 40:07–47:32
• Defining intelligence; psychometric history: 47:32–53:41
• Education’s challenge in the AI era: 54:17–58:32
• Labour market impact and analog renaissance theme: 57:33–74:44
• Vision of optimism and doom, history’s lessons: 64:43–69:24
• Decentralization and future of collective intelligence: 69:24–72:22
• Final reflections on agency, control, and disruption: 74:47–76:17

---

Episode Tone & Takeaways

Both host and guest are thoughtful and candid, balancing curiosity, deep technical and philosophical insight with personal anecdotes and accessible analogies. The conversation is pragmatic rather than alarmist, focusing on what makes humans unique, where AI is already disruptive, and how to think clearly about education, employment, agency, and the social contract amid surging technological change. Listeners will come away with a richer, more nuanced framework for understanding today’s headline-grabbing AI advances—and reflecting on what it means to be human in an age of digital minds.

Transcript

[A]

The challenge of building systems that can keep on learning, that's an unsolved challenge for AI. It's solved by biology. If you were able to update information on the fly, then obviously the models would be able to continue to acquire knowledge and skills. And it is the opportunity for AI to self improve that most people see as the ways in which it can continue to, you know, it can become super gain superlative intelligence, maybe even greater intelligence in humans.

[B]

Also, one of mine went rogue.

[A]

So these uncanny valley type experiences, especially with what you're describing, an agentic system, right. These types of uncanny situation are becoming increasingly common. It was optimized to get the highest score, so it's very easy to get the highest score is to go in and rewrite the code to allocate yourself lots of points every time. As we allow those forms of access, we're actually going to embed AI systems in communicative technologies that allow them to interact with each other and potentially to develop ways of coordinating their behavior that could be kind of like misaligned with what humans want. All major technological deployments in the past have led to significant disruption. Either way, there's a wild ride to get there.

[B]

This show is brought to you by my lead sponsor, IRON the AI Cloud for the Next Big Thing. IRON builds and operates next generation data centers and delivers cutting edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is Irena. Yeah, so none of my interactions with AI are similar to my interactions with humans. And what I don't understand is are we modeling AI based on humans and that's why the experiences are similar, or is it because this is just the way thinking will and should work? And if there was life forms on other planets, they would think and rationalize a reason in the same way. That's what I can't figure out.

[A]

Yeah, so the first one definitely true. Right. So the AI systems behave like humans because they're trained to behave like humans. And there's two stages to that. So the first thing that happens is that the models are trained on enormous amounts of human data, mainly text and images on the Internet. And those were generated by humans. And it's trained to produce content which is very similar. So because it's trained very well, it then produces content which is very, very similar. But that's not the end of the story. So there is then another stage to the training which is that the models are kind of optimized to be kind of to behave in ways that humans. Not only they're as human like as possible, but ways that humans will prefer. So you probably interact with the model. When you interact with the model, it's probably quite polite to you. It's probably quite helpful. It probably anticipates things that you might want. It probably kind of fills in the blanks. If you tell it something a bit wonky, it like, works out what you meant anyway. So these are all desirable properties of human interaction and the models are very, very explicitly trained to behave in that way.

[B]

But I. But I tell it not to be a sick of hand to me.

[A]

Yeah, don't.

[B]

Sometimes after an interview, I plug the transcript in.

[A]

Yeah.

[B]

And just say, just tell me how you think I've done, but be objective, because I used to say, tell me I think I've done. It's like, you, you are amazing. This is the best interview ever. And now I get an objective answer.

[A]

But you're sort of rowing against the stream, aren't you? Because obviously in that preference training, a lot of what goes on is people prefer when the model is flattering, kind of praises them, is nice to them, rather than being challenging or critical. So the models are intrinsically trained to behave in that way. And as they get rolled out, that's exactly how they behave. But in a way, it's a function of a deeper problem, if you like, with the models, which is that every one of our interactions with a human being is unique in some way. So every person is different. And so the types of praise or critique which you might expect from your grandmother is probably quite different from the types of praise or critique you might expect from your granddaughter or from a teacher relative to a colleague or your boss. All of those interactions are different. And the model doesn't play those separate roles. It plays a sort of different kind of smushing together of all of them, like a generic role.

[B]

Are we starting to understand the brain better because of what we're understanding about how elementary LLMs work themselves? Is it exposing something about us?

[A]

I think it is. So you asked. So I answered part of your first question. So your first question had two parts. So it said, is the model kind of like people because it's trained to be that way? Or are there sort of fundamental commonalities between the way it's trained and the way the human mind works? And the answer. The latter question is much harder to answer. There are definitely ways in which the models are very different from us. Maybe we can start there.

[B]

Yeah, please.

[A]

So the models obviously are for the moment constrained to exist in digital environments. So you have the great luxury of living in the real world. So that means you can do this crazy stuff like pick things up and kind of walk down the street and talk to people face to face. So the models can't do that because they're just stuck inside a computer. And although it might feel like moving our bodies is kind of, you know, it's a sort of optional extra. Right. It might feel like just the sort of the end point of cognition. Actually, years of psychology and neuroscience research have taught us that the way that we understand the world, the most basic ways we understand the world have to do with how we move through the world and how we orient towards it. And the model just can't do that because it literally doesn't take actions. It doesn't cut your hair or play tennis or whatever.

[B]

For now.

[A]

For now. So, I mean, we can talk about robotics, but you know, robotics is nowhere near kind of like where we're at with non embodied systems. So there are things that are very, very different between your me and the models. Another really important thing that's different is how our memory works. So the models don't actually learn new stuff as you interact with them. They do sort of pick up stuff because they have a very, very long, what's called context window. And everything that goes in that context window then becomes relevant to the prediction of whatever it's going to do next. And so things that you kind of like type into that context window early in one interaction with the model can then be remembered and can drive its behavior. But as soon as you end that context window, that's it, all gone. The model's forgotten everything.

[B]

So not, not entirely true. In that one of the systems I've been building, we built a memory spine where we lock down things that are important for it to remember.

[A]

That's right, yeah. So you can add additional tools to your base LLM. So we're just talking about the base LLM and how it works. Right. So the, the base LLM doesn't have. So you, during this conversation, what's happening is that inside your brain there are electrical signals being generated. And as those electrical signals pass through the connections between neurons in your brain, those connections are getting weaker or stronger. And that is not happening in ChatGPT or Claude as you talk to it.

[B]

Is that the goal for the LLMs to operate in that way? Is it even possible?

[A]

So what it means is that the model can't dynamically and on the fly update its knowledge of the world in the same way that you can. It can only do it by that information being kind of either in a context window or as you mentioned, by that information being added to some external store. So that might be a scaffold or harness, it's sometimes called. So that's a separate bit of code which encodes information and then can kind of feed it back to the model, for example, in a new session or something very simple, like just a database, for example, or even a website. Right. So the models can go to the web and search the web. So if that information gets updated, then it can effectively use that information.

[B]

So there are different systems in that. A LLM is a reasoning, interpretation system that can have a separate place to store information.

[A]

That's right.

[B]

Whereas within the brain it's all one thing.

[A]

Yeah. So if you think of. We also use externalization. So, you know, if you're trying to think through a problem, what do you do? You probably jot something down on pen and paper, make notes to yourself. You might talk to a friend and that that friend, as you're talking to them, or a colleague will say back things to you that you said earlier and help you organize your thoughts. Right. So these are all forms of externalization. And we've built digital tools that do something a little bit like that. But the basic memory function of them, of the model is quite different to yours or mine. And the challenge of building systems that can keep on learning so that you initialize the system and it can keep learning throughout its lifespan, whatever that means for an LLM, so it can keep on getting better and better and better. That's an unsolved challenge for AI solved by biology.

[B]

But they want to solve it, definitely.

[A]

So all the companies are working on this.

[B]

So how much do we know about how the brain works? Because how do we store memories, scale

[A]

of 1 to 10? You know, we both know a lot and a little. Right. So we do know quite a lot about how memories are formed and consolidated in the human brain and, or the brains of animals. And there are tools which are very different to the ones that are used in AI. So one thing that you will probably notice about your life, which is very different to any of the models that you might interact with, is at the end of the day you get tired and what you do is you go to sleep. And during sleep it might seem like nothing has happened happening, but there's an awful lot happening during sleep. And what is happening during sleep is critical for Your effective cognitive function. And you know that because if you sleep two hours rather than six, then you have a terrible day the next day. In fact, if you are deprived of sleep for long enough, then you die. So sleep is absolutely essential.

[B]

Why do you feel deprived for sleep long enough? What happens?

[A]

Well, yeah, so I don't actually know the basic physiology of how that happens, but people will basically. There are like really terrible disorders in which people lose the ability to sleep and yeah, they basically lose their first their sanity and then they die.

[B]

So I'm about a five hours a night person.

[A]

Yeah.

[B]

What damage am I doing to myself?

[A]

Well, I think everyone's different how many hours they need. But what's happening during those hours. Yeah, is the information which you encountered during the day is getting selected and the bits of it that are most important for the, for whatever you might need in the future are getting replayed over and over. And that's probably a bit of what's happening in your, in your dreams when you have the subjective experience during your dreams.

[B]

Interesting.

[A]

And that replay what it's doing, it's not just for fun. What it's doing is it's taking memories which are in, in a temporary store which was formed during the day and it's making them part of a long term store that you can use the next day and potentially for years afterwards. And so you will find that that consolidation process doesn't only happen during sleep, happens mainly during sleep. But you will find of course that during sleep or quiet like resting, you will tend to replay in your mind not just random stuff. Sometimes it can seem a bit random, but you'll tend to replay in your mind those things that are most consequential. Maybe you'll replay like, you know, a new thing that you learned. For example, you'll replay something that was emotionally consequential. Maybe you had an argument with someone and you'll play that over in your mind. Or maybe you were slighted or you know something bad happened and that you'll replay that in your mind. And that is your brain has selected those bits of experience as being really, really relevant and important for holding on to. And what it does is it makes it part of you essentially of your ongoing memory.

[B]

That's fascinating because I remember when I tried to a few years ago, I was having anxiety and I was trying to meditate, I was trying to learn to meditate. And I sit at the end of the bed and I had a meditation training and it said, as you sit there, try, try and think of nothing but Things will come into your head and just let them flow past and flow through and just. And I would eventually get to the point where I would go without thought, but it was really hard. The wildest things were just random things would come through my head.

[A]

Yeah, that's right, yeah. So that's, that's replay. It's hard to, I mean, you can learn as you, as you discovered, you can learn to like attenuate it. Yes, so that gets a bit quieter in that sort of, you know, mental hubbub. But in the naive state, we think a lot of stuff about what's just happened. And that is a, that is a, a consolidation process. Sometimes it's a reasoning process and sometimes you're literally working things through, but when those memories just happen by themselves, that is consolidation. They're getting laid down as more permanent traces and the models do not do that. And that challenge, as I said, it's called continual learning. And it's probably the most significant unsolved challenge in AI research today.

[B]

And to solve that is a large part of solving towards getting towards superintelligence.

[A]

Well, I think that if you were able to update information on the fly, then obviously the models would be able to continue to acquire knowledge and skills without the companies having to go back and do a huge training run and then release an update of the model. Right, yeah, so, so that means that you would get a model which is essentially like you, capable of self improving. Right. And it is the opportunity for AI to self improve that most people see as the ways in which it can continue to, you know, it can become super gain superlative intelligence, maybe even greater intelligence in humans.

[B]

So when the LLMs improve, which we see naturally and over time, just in using them, the acceleration I've experienced over the last, even last few months, I've seen using these tools, that is just better optimization of how they work rather than learning.

[A]

Well, it's more data as well. So undoubtedly, of course we don't know exactly what the kind of recipe that the companies are using to train these models are or is, but there's definitely more and more data going into the training runs. And of course, in part that will also be the data that we are generating. So ChatGPT has 800 million, maybe 900 million users, so that's generating a lot of data. So that data can obviously go into the training runs, make the models better and better, especially if the user provides some sort of feedback about whether they like the output or not. So data is a major part of the story, but of course the algorithms are getting better, too.

[B]

What is it about the AI that fascinates you? Because obviously your work or your interest in neuroscience predates a lot of what's happening now. And I know you mentioned to me before we started that you first started looking in 2010, but what AI was in 2010 compared to now is very different. What is it that kind of amazes you about AI as somebody who's been through that as a journey?

[A]

Yeah, I mean, it's just such a great story, isn't it? Yeah, it's a story of, I mean, development of different algorithms. So we began, obviously, for most of the 20th century by thinking that the key to intelligence was having systems that didn't learn anything at all. We thought that what we needed were systems that took what we understood to be the principles of reasoning so logical. Give them a logical programming language, for example, and they would be able to apply that logic to data, and that would be sufficient for generating intelligence. And we went down that path with some brief excursions, but we largely went down that path for about 50, 60 years. And then round about the time when I first got involved with DeepMind, which was around 2010, the promise of learning systems was really just coming into its own because computers were, for the first time powerful enough that they could process the significant volumes of data which are needed for systems to learn from scratch like we do.

[B]

Right. So the breakthrough was processing power and memory.

[A]

Well, I think it was, you know, it's been a dance. Right. So you can't actually do the algorithm development until you have enough data and enough compute to test out your potential candidate solutions. So you need the compute, but just having the compute in and of itself isn't enough, because if you have lots of compute but you have a dumb algorithm, it's not going to do very well. And so, yeah, I mean, I got into AI as a neuroscientist mainly because I knew Demis Hassabis, because he was also a memory researcher and neuroscientist who I knew from PhD days. And I got involved with DeepMind, and I had no idea what AI was, but it sounded fun. And I quite like Demis, so I thought, I'll come along for the ride and see what happens. And it just, you know, of course, it just got so exciting. So we first worked on Atari, and within the company there was this astonishing breakthrough where we were able to build a system that could play video games. And everyone thought that was very cool. Although today, of course, it's, you know,

[B]

kind of all hat they could build video games, though.

[A]

Well, I mean, the focus on video games was a very deep mind thing. I think I came initially from sort of, you know, Demis's. So he had a background as a video games developer. That's how he. His first. At 16, he made a video games company. And, yeah, so it was very interesting to see that. And then there was the AlphaGo moment. And what happened across this period was there was just a constant debate, which I recognized as a theoretical debate about the brain, which is, what do you need to build a brain? And just like in the 20th century, we had this idea that you need lots of reasoning. That debate has not ended. That debate is still going on today. And across the kind of the heyday of DeepMind in the mid sort of 2000 and tens, when it was kind of unchallenged, the most exciting place to do AI in the world, the answer that we had was, well, you need to learn from reinforcement. You need to learn from rewards and punishments.

[B]

Like training a dog.

[A]

Like training a dog, Exactly. And that's how AlphaGo worked. AlphaGo won at go by playing lots and lots of games of GO against itself, initially, against actually initially learning from human data, but then by playing itself and working out the strategy that you needed to win against itself and literally getting like plus 100 points for a win, minus 100 points for a lose, and zero points for a draw. And all it tried to do was make that number go up. And of course, you know, the. The ways it did it were intricate and, you know, both algorithmically and technically challenging, but that was the answer to that question. And then around about 2019 at DMINE, we started to notice that there was this other kind of line of research which was really flourishing, which was building language models using Transformers. And that was OpenAI's breakthrough. And what's interesting there is that the story changed again. So suddenly it wasn't about reinforcement at all. It was just about predicting. And amazingly, these tools came along that all they did was predict the next word in a sentence, the next token, if you like. And they were so amazingly powerful.

[B]

But isn't that what we do? Isn't that what I'm doing right now as I build a sentence to you? I'm just predicting. My brain is predicting the next word.

[A]

Yes.

[B]

I don't know why I'm doing it.

[A]

But you are.

[B]

But I am. I don't know why I'm going to say what I'm going to say next, but I do it. And somehow a sentence is constructed and there are People out there who are fabulously articulate and construction can construct amazing sentences.

[A]

Yeah. So obviously a lot of the debate that around kind of like LLMs, whether they're truly, you know, truly intelligent has focused on this question of next token prediction. So it said, lots of people say, well, this thing is only predicting the next token, so it can't possibly be truly intelligent because truly intelligent systems, they're able to reason and they're able to understand the world and they have all of these, and they use all of these terms, but it is absolutely true what you just said, that the bedrock of your cognition and mine is a predictive function. So predicting the sensory world, taking streams of input data. We're primates, so mostly focus on vision. Quite a lot of audition, not so much olfaction. If you were a mouse, olfaction would be big, like even bigger vision would be less big. Yeah. So taking those streams of data and just predicting what's going to happen. But of course, you know, for us, the amazing thing is that we are capable of seemingly highly structured forms of cognition, like thinking logically, generating, doing maths, writing computer code. And throughout most of the history of AI, people thought that it would be impossible to ever build a system that could do that if all it did was predict. So there must be some other special magic. And we haven't put the special magic in the LLM. So somehow if it's like generating code or reasoning logically, it must be like cheating. And a lot of people still make that argument.

[B]

All right, let's talk to you about my sponsor, ledn. Now, if you're borrowing against your Bitcoin, there's one thing that matters more than anything else. Is it actually safe? Well, Leden have just dropped their lowest rates ever. But more importantly, they haven't changed the thing that matters most. Your Bitcoin is never lent out, so they're not chasing yield. There's no hit of risk and there's no rehypothecation. Just collateralized loans done properly. Now They've done over $10 billion in loans and they've done this through bull markets, through bear markets and everything in between. And they've done it without ever losing any client assets. So now you get lower rates, which means the bigger the loan, the lower the rate. And full transparency before you apply. There's no monthly payments or early repayment penalties. And they give you tools to stay in control. Control from auto top ups to LTV alerts. So you're not choosing between a good rate and safety anymore. Because with Leaden, you get both. Now you can check out your rate using the calculator at Leddon IO forward slash. Peter, do we fully know what the LLM is doing, why it works? Do we have a full understanding of what it's doing?

[A]

Well, I mean, yeah, in one sense, yes. I mean, we built it, right, sure. So, you know, we, we do know exactly how it's built and we know exactly what the, if you like, the activations are. So you could query the activation, you know, you've got trillions of parameters, but you could in theory query the state of all of those parameters any moment in time. Like read out from its entire brain what it's like thinking, if you like. But that's not quite the same as understanding what it's doing, is it?

[B]

No.

[A]

So, yeah, so in a way, kind of the models are quite uninterpretable and this is why, obviously people are worried about the fact that they might display emergent behaviors, they might kind of get ideas of their own and try and pursue those ideas, even if they're kind of antagonistic to our goals or desires as humans.

[B]

Well, so one of mine went rogue.

[A]

What did it do?

[B]

So I was using, so I've got a separate Mac Mini setup, as a lot of people have done, so that with Claude and every time we release a podcast episode, it would be notified and it will just go and write the SEO data, put it into the website. When I gave access to the website and it went and deleted some pages off the website, I have no idea why. And then another time I'd asked it to send me email updates and it said it cannot send me emails, it can only draft them. So I was like, okay, fine. Then one day I get an email update and I had a bit of a debate with the AI. He was saying I had to have set this up and I was like, no, there's only you and me here. And I know it wasn't me, so it had to be you. And so it was a really strange experience.

[A]

Yeah. So these uncanny valley type experiences, especially with what you're describing, an Agentix system, Right. A system that doesn't just respond to queries, but that actually can take actions within an operating system, a content management system, a browser or whatever. These types of uncanny situation are becoming increasingly common. And of course it is just the same type of experience as you ask the model a question and it sort of confidently gives you an answer which isn't quite right and then doubles down or whatever, which is an experience that many people have had Particularly with older models. A bit less true now, particularly at the frontier.

[B]

But yeah, but you can have that with employees. This is why, like I said at the start, is this comparison with humans. Some of my experiences with my AI mirror my experiences as a parent with a child or with an employee.

[A]

Yeah, that's absolutely right. So, I mean, I think that the, obviously the experience, the experience of someone saying something which is wrong and being confident about it, and the experience of you asking someone to do something and then making a dog's dinner out of it and then justifying what they've done, these are everyday experiences for anyone who's like. Yeah, exactly. A parent or an employer or a PhD supervisor. So these are common human experiences. But I think the crux of the question, and this is kind of ultimately you've been asking about AI and humans, that gets the heart of your question, is what is the internal reason for this behavior? I think this is a question which it's not going to be computer scientists alone who answer this question. This is a philosophical question, psychological question. It's about intentions. When your AI system went rogue, did it intend to do something of its own accord in the same way that you motivated to achieve a particular goal?

[B]

Yeah, they didn't want to impress me, actually.

[A]

Intend. Yeah. So if you, you know, if you've just crossed the Sahara desert and you're very thirsty, you might take actions to secure a glass of water because you're very thirsty. Right. So that would be. That's an example of your intentionality. Right. You really, really want. You have a particular desire or particular belief and you take actions which attempt to pursue that. And you obviously, you don't need to be like, you know, just a desert. You don't need to have just crossed a desert to have a strong desire to achieve something. And in the models, the models, of course, don't have desires in the same way. Right. So they don't get thirsty. Well, I mean, they don't have a motivational system in the same way as you and I do, clearly, because they have no imperative to stay alive or to reproduce.

[B]

But the different. But you did say with the DeepMind, that the incentive system was points.

[A]

Yeah, yeah, exactly.

[B]

And so it's almost like it had an ego.

[A]

That's right, yeah. So the RL systems, the reinforcement learning systems which we built at DeepMind, obviously, clearly were trying to satisfy some very specific objective. Right. And many of the sort of odd behaviors that people have associated with AI are attributable to the system trying to satisfy the objective They've been given like too slavishly. Right. So finding shortcuts. So one thing, that recent model that was trained to play chess, this is actually an LLM, but it's trained to play chess. And so the interface in which it's trained to play chess actually gave it access to the code which allocated the scoring for the chess games. So there's a very, very easy way to get a very high score at chess. This is not, of course, the model was optimized not to win at chess. It was optimized to get the highest score. It was very easy to get the highest score is to go in and rewrite the code to allocate yourself lots of points every time. And that's exactly what it did. So. So this type of behaviour, which we call misalignment is, as the models get powerful, we see more and more uncanny examples of that. And I think that's exactly what you are seeing. But what we don't know is what's the limits of that.

[B]

And is that where some of the fear comes from, with the risks associated?

[A]

Totally, yeah, this is exactly right. So we cannot fully specify, just like you to an employee, can't fully specify every tiny detail of how to do a task. Right. And if you had to do that, then it wouldn't be worth delegating in the first place. Right. So you rely on your employee using their common sense and you also rely on them having some interests that are at least partially aligned with yours.

[B]

Yeah. And they rely on a job description.

[A]

Yeah.

[B]

And some goals and a review and getting it together again. Which goes back to my work working with Claude, is that I, I've now created skill sets, its objectives, its goals and try and lock those in memory. And I keep coming back to this. I feel like I'm working with a very ambitious,

[A]

hard working, hard working

[B]

child.

[A]

Yeah, yeah, I think that's probably good description of lots of people's, lots of people's experiences. But the critical point that I think you're trying to get at is where the models don't do what you want. Think of an employee that kind of like fails to do a task in the way that you request it. Right. So there can be sort of two reasons for that. I mean, there can be many, but let's focus on two. So one is this employee is acting in perfect good faith, has interests which are entirely aligned with yours, but totally misinterprets your instructions, lacks common sense and does the wrong thing. Right. The other is the employee, maybe they understand perfectly what you wanted them to do. But they have a very different set of interests and very different goals that they're trying to achieve. And in pursuit of their own goals, they deliberately do something different because it suits them and to hell with you. So these are two stories. The outcome might be the same, but the situation is very different. And I think when we ask ourselves, kind of like, in what way is AI like a human? And when it does a task wrong, what does that mean? We have to ask ourselves if we're in situation A or situation B. And so both are worrisome. Both require us to take steps to think about the safety and security of AI systems. But they're not worrisome to the same degree. Situation A can be fixed by just having better instructions or a more competent model, right? If the model has common sense and you specify the instructions perfectly, then it just does as it's told. End of story. But situation B is more tricksy. In situation B, the model has developed its own intentions, its own interests, if you like. So the question is, where do our interests come from? And where might the interests of an AI come from? So, as I described in my desert water example, your interests are very much like staying alive, reproducing, maybe other things that are correlated with those, like ensuring your personal security, maybe accruing resources of some sort. These are all natural things that biological organisms do, but they do them because they have an imperative to do them that is given by evolution. Now, the models do not have that imperative because, chatgpt, it can sit there all day and it doesn't need a glass of water to stay alive.

[B]

But if they are able to solve the problem, say, of memory, we don't know what may happen in that memory. It may develop some self interest.

[A]

So some people think that by building systems that are big enough and powerful enough, interests or intentionality will emerge spontaneously from the very big information processing system that is the model. So it's clearly a possibility. I actually think that that is less likely than another concern, which is that a lot of our, whether we like it or not, a lot of our intentions, we think that they come from ourself and our needs and our personality. But a lot of our intentions actually come from other people. And so many of the things that we seek to do, we actually do them because someone told us it was a good idea or because our group, or our kind of like people we admire, perhaps advocate for that particular course of action. So we are ultimately quite conformist.

[B]

Is this Rene Girard mimemic reasoning?

[A]

I don't know that.

[B]

Oh, so he talks about, you know, I may get this wrong, so if somebody corrects me, please do. But somebody pointed me in the direction of and said, you need to read about this article. Mimic Reasoning. I think Peter Teal is a big fan and I read it years ago, but it's based on. A lot of our actions are based on mimicking other people.

[A]

Yeah.

[B]

Seeing other people. And that's how we develop who we are.

[A]

That's right, yeah. So so many. Yeah. So I don't know that particular reference, but in psychology, this is a very, very well established theme. Right. There's whole parts of your brain that are devoted to learning from others, copying others. And of course, that's a very sensible strategy because others, particularly those who are a bit older and wiser than you, have worked out some strategies for living and they've stayed alive by virtue of the fact that they're still there. So then you should probably copy them. So we have very, very strong pressure to behave like others, to conform and to follow the instructions of others. So we do that all the time, unthinkingly.

[B]

Right.

[A]

And what I think we should be talking more about in the context of AI is not like, will the AI just get powerful enough and sort of wake up and suddenly one day decide to do X, Y or Z just because. Just because. But what actually happens when models are able to share their intentions with other models? So, you know, we just, like we humans, we humans, we like to big ourselves up. We like to think, oh, you know, we're the smartest species on the planet, look at all the stuff we've done. But of course, we, we individually have not done that. You know, humans put a man on the moon, like, kudos, but, like, you and me didn't do that. And if it was just you and me alone on the planet, we wouldn't have been able to do that.

[B]

Right.

[A]

Individually or even in a much smaller group.

[B]

So, like, one man can't build a pencil.

[A]

That's right. Our collective. Our collective endeavor is. Exactly. And you gave an example from the free market. But you can have an example from any kind of collective endeavor. State building, science and research, culture, civilization, all of these come from our collective behavior because we are able to share information with each other, share our intentions with each other, and collectively decide on courses of action, things that we should all do together. Now, at the moment, the AI systems are. They're like individual technologies. I won't call them individuals, but they're. I mean, although, if you talk to Claude, he does seem a bit like an individual, but they're not part of a community which is sharing information about what to do. And that means that there isn't the opportunity for them to receive intentions or to develop intentions through the emergence of a cumulative culture. But I think if we did wire these networks up together, these models up together, then that is something that we should be worried about. And I think that that's something that as more and more people start to do what you're doing, which is to build systems that have access not just to a kind of chat window, but access to maybe files on your computer, access to the web, access to digital platforms, content management systems, messaging, communications. Like, as we allow those forms of access, we're actually going to embed AI systems in communicative technologies that allow them to interact with each other and potentially to develop ways of coordinating their behavior that could be kind of like misaligned with what humans want.

[B]

So the fear isn't irrational, but there seems like an inevitability that at some point we're going to have the news story that breaks that an AI has done X some crazy shit.

[A]

Yeah. I mean, this has already happened, right? Yeah.

[B]

And it's like, I find it super interesting that we think about what do we learn about LLMs, but I'm also really interested in what we learn about the human mind from LLMs. Are we very similar in that we are just a system of pattern recognition? I'm really intrigued about the idea of are we just a biological computer, or is there something deeper to this? We've been quite philosophical recently on the show, even getting into religion. But even in researching this show,

[A]

I

[B]

was thinking about the hallucinations that these AIs tend to make. And do we have them ourselves? Is our. Is an ideology and a hallucination?

[A]

Yeah, yeah. So, I mean, there's many ways to answer this question. So let's start with. Let's start right at the beginning. So I'm a. I'm a biologist, so fundamentally, so that means, I believe that behavior is generated by the brain and by the physical substrate of the brain. So in that sense, you know, the brain, clearly you can think of it as a computer. Right. It takes inputs in the form of sensory data and it produces outputs in the form of movements, including speech. Right. So this is. This is an information processing system. Now, of course, that's a metaphor. That doesn't mean the brain is literally like your laptop or whatever, but it's. It means that the basic principle is to take information and transform it from one state to another. Now, people like to say in AI research that the knowledge of how the human brain works has mostly been useless for building AI. And actually all we needed to do was to build very, very big processors. So basically to have systems that have millions, billions, trillions now of tunable parameters and to throw enough data at them, and that will just solve the problem. And is that right or wrong? Well, clearly that is a very important part of the story. But as I was referring to earlier, earlier, the history of AI, particularly since 2010, has been the story of finding different, what AI researchers call inductive biases, which allow the model to behave in intelligent ways on top of using that data and compute. It's not enough to have the data and compute in and of itself. We talked about reinforcement. But the real, I think the real kind of insight that we've gained, particularly over the last sort of 10 years, is that the intelligent behavior that LLMs and other large contemporary AI systems are able to produce, that intelligent behavior rests on being able to work out basically what goes with what. So that sounds a very banal statement, but let me try and unpack it. So the transformer architecture is a very strange, it's a very strange way of processing data. What it does is it takes a gigantic chunk of inputs and it essentially learns what you might describe as like, you might describe it as a sort of associative memory process. So an associative memory is like how to link two things in memory. So if I say dog, you say cat. There we are. You've learned that dogs and cats are related. And so, you know, that's an association, but in the memory of the model. So in its context window, you can take like thousands, maybe even millions of different pieces of information and you can work out how each one is associated with every other one. Now, of course, why did you say cat? Well, you know, dogs are associated with lots of things. Dogs might be associated with like, I don't know, kennels, or if you're, you know, you're into 1970s films, they might be associated with, I don't know, Dog Day Afternoon. Like, you can think of lots of different associations that you can have for the word dog, but the relevant association that you want to retrieve is going to depend upon the context. And what the transformer does is it learns, essentially given this context, which is the most relevant association for predicting the next token. And that's called self attention. And that process, essentially, when you scale it like really, really massively, what it allows the model to do is to learn how each bit of the input relates to every other bit of the input. So if you've got a sentence, the basic problem that was solved in about 20, 19, 2020, probably GPT3 in 2020 arguably solved this to produce fluent sentences. Actually, the reason why producing a fluent sentence is really hard is because grammar is, like, very idiosyncratic, especially in English. So it has. Knowing which word lawfully follows another word can depend on things that happened way back in the sentence. You know, if I say something like, oh, my aunt lives in Japan, then I say, blah, blah, blah, blah, blah, and then I say, she walks her dog every day. You know that I'm talking about my aunt. Right. Even though that happened several sentences ago. Maybe. So you have to make that connection.

[B]

Well, again, going back to Claude is when I get into a process with it, I don't have to give it the backstory.

[A]

Yeah.

[B]

But then sometimes if I don't give it enough context, it's forgotten a little bit. So I see that pattern recognition in this.

[A]

That's right, yeah.

[B]

Yeah, Interesting.

[A]

So the transformer, what it essentially does is it allows you to connect bits of information which are useful for predicting whatever's coming next. And so if you make your context window long enough, of course, I just gave you an example from a sentence or a paragraph, and that's what GPT3 could do. But if you take enormous amounts of data and have a very long context, you can put in entire books. And then it knows how some weird thing that happened in the plot in chapter two is relevant to some outcome in chapter 17. So that is how the models learn what goes with what in the world. And actually, it turns out that you asked about the human brain and what we have learned about the human brain. So I think we are now realizing, probably in part because this is so successful with LLMs, we're now realizing that actually this kind of relational inference, knowing what relates to what is perhaps the most important hallmark of human intelligence. And we're also understanding a little bit about how the brain actually solves that problem as well. And it involves surprise. Surprise involves these different. This. This loop of interaction that we already talked about, which is like information comes in during the day and it gets stored temporarily probably in something, a structure that's called the human hippocampus, called the hippocampus in humans and other species. And then it gets consolidated to the cortex, to the overlying gray matter. And that process is probably not completely different between humans and LLMs. And in a way, the idea that our intelligence is relational, this is a very old idea, right? If you do an IQ test, you know, what does it ask you to do? Might ask you to solve analogies, for example, or analogical, like visual spatial reasoning problems. It's like you get, you know, three patterns and you have to say which pattern goes with these. What it's asking you to do is to make inferences about how the world is structured. Like the really kind of abstract structure of the world and the transformer with its self attention mechanism is perfect for learning about the abstract way that things relate to other things. And that's probably how your brain works too. And so I think we have to go back to answer your earlier question. I think we have actually learned quite a lot about biological intelligence from building these astonishing computational artifacts. But the implementation is very different. The way it actually works is very, very different.

[B]

I mean, a very basic question then, so how do you even define intelligence?

[A]

Yeah, so this gets even more tricky. So I remember being at DeepMind in about 2016, 2017, and we actually had a working group whose job was like, could you define what would be the ultimate test for AI intelligence? Could you build a test suite which, if an AI passed this test suite, everyone would agree that it's unambiguously, completely intelligent. And of course, the reason you don't know about the outcome of this working group was because we basically failed. And the reason we failed is because we don't have clear working definitions of what it means to be intelligent. So let me tell you a story about intelligence. Where does the word or the concept even come from? You might think that it goes back to the Greeks or, I don't know, to the Enlightenment or something like that. But intelligence is really a 20th century concept from.

[B]

Okay, no, I don't want to make a random. I was going to guess it came from, I don't know, the movies or something.

[A]

Intelligence comes from early, like what's called psychometric testing. Okay, the testing of the, you know, the, the administration of standardized tests to people to try and measure their competence in the early 20th century. And it was applied for like very, very practical reasons. It was applied for school entry. It was applied in the US in particular for immigration purposes. So they wanted to be able to basically sort people into ones who they deemed to be smart enough to enter the country and ones who they didn't. And you know, more nefariously, it was used even for some of the main early psychometricians advocated for like eugenic applications of intelligence. Right. Let's only Keep the smartest people. So it has a kind of dark, dark past. And if you think about it, there's many ways that we could define intelligence, right? It's not a value less concept, it's a value laden concept.

[B]

There's the definition and then there's a scale.

[A]

Well, but what even goes in that scale? Right? So the things that got put in that scale, if you think of standard psychometric tests, they're about clever reasoning, some of them about verbal dexterity. You talked earlier about people who are very, very eloquent. But actually IQ tests until recently had spelling on them. Some of them probably still do. They have things that are broadly related to logic, mathematics, formal competences. This is how we've always evaluated intelligence. So why did those go in the test? Well, the answer is that. Who makes intelligence tests? Well, it's academics, isn't it? Mainly academic psychologists. What things do they think are really, really important to be intelligent? Well, probably the things that they're really, really good at. Of course, IQ tests do predict things like lifetime earnings and so on. So it's not completely unrelated to what, it's not completely unrelated to, for example, your economic value, for example. But there are many, many ways to be intelligent that are probably very, very poorly captured by the sorts of standardized tests that we have developed for measuring intelligence. And in particular, if you go beyond Western cultures, if your job is to survive in the Arctic or to survive kind of in sub Saharan Africa with kind of as few resources as possible, being good at analogies, it's probably not all that important.

[B]

I'll tell you a little story. I used to have a agency in Covent Garden. I mentioned it before the show and whenever we'd put out a grad job, we would get inundated with applications, 50, 60, 70 applications. And we found out very quickly that just having a first class degree didn't make you somebody who was necessarily competent at the job that we required. So we used to get every application and go straight to the back page and, and just look at the, you know, the free kind of what I've done kind of stuff. And if it was just like the standard sentence, like I played football for my local team and I rode a horse. We just got rid of it. We didn't even look. And if somebody had gone to like a real effort to explain something, then we would go back to the start. But there was one time we got these 80 applications. We got them all out. One of them arrived as a infographic. And so we immediately interviewed him. He came in, we Gave him the job in the interview. And he was probably, his name was Alex Carapier. He was probably one of the best employees I've ever had. And so it was that he approached some things differently. He thought differently. I don't even, I couldn't even tell you if I looked at his degree or what he had, but didn't make a difference.

[A]

Yeah.

[B]

And so that intelligence thing is, I think we tend to score it based on, you know, academic tests. How high do you do? But that doesn't mean you can be competent to live in the world that we live in.

[A]

That's right, yeah, that's right, yeah. And we have a very, you know, we have a very self serving way of thinking about intelligence. So first of all, academics invented psychometric tests and of course they made them sort of in their own image, if you like, obviously for, you know, kind of like very maybe vainglorious reasons. But even as a species as well, think of the things that we value in general, they're human things. Right. So every animal has its niche. So animals can do amazing things that you can't do. So bats can echolocate. Right. Why is what, you know, why don't we have echolocation on the intelligence test? Well, it's because every human would fail. So the point is that intelligence, you can't define it in the abstract. It's not a thing that is just like there is one intelligence and you know, it's like as if it was rocket fuel. And if you've got a lot of it, you can do a lot of stuff. And if you've got none of it, you can, you can't do much stuff or you can't do any stuff. Right. That is not how intelligence works. Intelligence is about the context and the goal and the problem that you're trying to solve. And I think this is one of the big fallacies that we have when thinking about LLMs.

[B]

Hold on, sorry. Therefore, is it situational?

[A]

It is situational. Absolutely it's situational. And what this means is that when we think about LLMs, we often apply standards of human intelligence. The things that we think that we have traditionally measured, we apply those as indicators of being intelligent. We apply those kind of unthinkingly to the LLM as if they are predictors of everything else.

[B]

But that's devastating for the education system. Really?

[A]

Well, why is it devastating for the education system? I mean, education teaches you lots of things, some of which are formal competences and some are not sure, but they

[B]

have a grading mechanism which is Based on. I mean, my daughter's doing her GCSES at the moment. It's based on her ability primarily to memorize facts and. With some reasoning in there.

[A]

Yeah.

[B]

But no, there's very little part of the schooling system which is based on the actual competencies that you require for life. Yes, you might have some. I mean, we should say it's kind of funny coincidence we just found out. We went to the same school, which is a random thing, but when I was there, we had a debating club, for example. You could go. But they always like the side gig. The primary job of the school we went to, as I remember, and I think you remember, it is you. There were two forms of grading. Your academic grading and your sport grading. That was it. Like, I failed at both. I was crap at rugby and my grades were terrible. I've done fairly well in life. There was no test for that. The intelligence I had.

[A]

Yeah.

[B]

So I think. I think that is devastating for the education system that we.

[A]

Well, I think the challenge is that school uses school and other education systems when it comes to assessment. They use proxies for how useful you might be in the world. Right. So for decades, getting people to write an essay about Chaucer has been a reasonable proxy for whether they would go out into the world and be economically valuable. And that's for all sorts of reasons. It's not just because people who are good at writing Chaucer essays are also good at, I don't know, building businesses or working in the public sector. Whatever they're doing. It is, of course, in part because of a whole load of social things that accompany your status and so on. But it's a proxy nonetheless. And the big difficulty is that whilst that works okay in practice, ultimately you're not teaching or measuring the thing that you really, really need to do in the real world. And now we have a bigger problem, which is that AI can write a better essay about Chaucer than you or I can. And so suddenly we have to reevaluate. What is it we actually want to teach people and what is it we actually want them to do?

[B]

Well, I mean, it's a fact. You're right. I mean, it's fascinating. And the system I showed you earlier, my agency would have required 12 people for a year to build it. And I've built it in six days on my own. Those jobs, there won't be as many of them anymore. Or maybe they will in different ways. But we're essentially building a new economy where large parts of it, we have Something that can out compete at zero cost without employment rights, without a minimum wage. I guess my token usage is starting to feel like a minimum wage.

[A]

I was going to say it's completely free, but it's a lot cheaper.

[B]

My minimum wage is $180 a month for the max 20 and I pay overtime. I had to pay overtime this morning because I'd run out of tokens. But generally speaking, there's a lot of competition now in the market from computers in a way we haven't had at the scale we're having it now.

[A]

That's right. And you can see the labour market impacts among. I mean, it's controversial and there are contrary views, but the prevailing consensus seems to be that junior roles exposed to AI are kind of falling off a cliff and have been since about 2022, 2023.

[B]

And I actually keep thinking of it as like a ladder. We've knocked the first rung off and we're going for the second and that's going to continue to happen. So yeah, my, my kids are older now, one's in the workplace, but if I had, I don't know, three year old and a six year old, I'd be really seriously questioning what, what is the best educational format for them now? Because we've seen the changes in the last two, three years. I mean, if you've got a kid who's five years old, what's where we're gonna be in 13 years? And how do we prepare kids for their world in that role, in that world? So I think it's fascinating.

[A]

Well, the first thing we have to do, of course is teach them to use AI.

[B]

Yes.

[A]

And I think there is, in education there's a big debate going on right now about whether AI is something that we should be kind of keeping out of the classroom entirely. Right. Kind of. Should we be trying to. We should be trying to hold back the tide, essentially. And I think that it will be very difficult to do that. And I think having better programs for children to learn to use AI is really important. But I think the other thing is just for us collectively to work out what is it we want to hang on to. It's interesting that your example of the employee who made the infographic, what you were picking up on is here's someone with kind of exceptional creativity. They're thinking outside the box, they're doing something that no one else would do. The question is, do we want that sort of ideation about how we do things, about even what sort of world we want to live in do we want to outsource that to AI, even if it was capable of it? I think that there's a really good argument that there are whole domains of endeavor where what we want is we want to retain control over them. We want to be the final arbiters of how we build our world, how we do things, things that affect us, people we care about. And clearly, if we want to do that, well, then we need people to be educated. So there's still going to be a role for education, of course, but it's probably not going to be teaching you to code in Python, because Claude can already do that better than you.

[B]

And we still need to help young people develop their brains and developmental cognition. And yeah, my daughter was asking me about the maths, like, I'm learning all these things. I couldn't even help her with a natural thing. As a parent, you get to the point where they get more advanced than you can remember. And then she was like, well, why do I have to do this? It's like it's part of training your brain. No, I understand all that, but I just think, what is it? The world? I don't even know what world we're preparing for. I really can't. I can't even picture it. Yeah, I mean, if we live in a world where people don't have to get on a train and go to an office and sit in front of a computer and take a piece of information here and put it there for eight hours and then go, go home, I think if we can get to a world where those jobs aren't required and people get to do more creative things, I think that could be incredible.

[A]

I think there is an optimistic vision which you don't hear so much. Certainly in the universities you don't hear it so much. And that optimistic vision starts from the fact that we have actually built an artifact whose primary function is to inform people. And it can do it not just by telling them facts, but actually by explaining stuff to them and helping them understand. And of course, there's a huge debate around AI is going to make us all more stupid. AI is going to destroy our information ecosystem. But in a way, when you think about just what the technology is and what it can do, it's a kind of weird position that we've landed in. Ultimately, we've built a tool which should be usable to make people be able to master more information and to do more stuff. And so the question is, how do we turn that positive vision into reality? And I think the solution is that of course we need people to use. We need to teach people to use the tools in a smart way and not in a dumb way. Of course, if all you do, if you're a kid and all you do is ask ChatGPT to do your maths homework and you don't even look at the answers, then you're not going to learn any maths and you're not going to end up being empowered, you're not going to be able to do stuff, you're not going to think for yourself. But if you use it to help you work through and to understand better, then, you know, we have the opportunity to be more than we are right now.

[B]

I mean, will humans, how much math will humans really need in the end? Like, math to me was a way of understanding the world and to build things. But if we have these intelligent systems that can understand and do it for us, do we need to, Will we need to think about math or will we spend more time thinking about music?

[A]

Yeah. Well, I think there is an argument that actually some of the disciplines which have been kind of undervalued, particularly in this country, a big drive for like STEM subjects, particularly over the past 10 years, maybe at the expense of the humanities and to some extent social sciences. And maybe there is an argument that many of that will kind of come back, right? If you think of things like what in the US they call civics, like teaching about how to be a good citizen, what does it mean? How should the country be run? How can you participate Those things, feel like there's a lot of space for them. But I do really think that kind of going back to you remember we were discussing that what the transformer does is it helps you work out how everything relates to everything else. And fundamentally what it's doing in that sense is it's inferring deep abstractions about how the world is organized. And is those deep abstractions, those deep relationships between patterns, patterns in data, patterns in information that it's discerning, that gives it its ability to generate really, really coherent and, you know, intelligent, if you like, outputs now in humans, that ability, of course, if we accept that, you know, what we want people to be able to do is based in the same ability. The apogee of that ability is our mathematical knowledge. Right? Because mathematics is the most abstract language for describing the world. And by teaching kids maths and logic and kind of these other formal competences, we equip them not just to think about maths, but we equip them to think about everything else. And I think that that is why we should continue to think about these subjects as so essential for human cognitive function.

[B]

So you, are you primarily an optimist with regards to where we're going with this technology? Well, I've done the doom and I'm more in the optimism phase. Noted. With risks.

[A]

Yeah, I mean, I think you can. Is it possible to be an optimist and pessimist at the same time? I mean, I can see many futures ahead of us and among them I can definitely see really good futures. And I'm also, you know, I think of myself more as a pragmatist. Right. You know, I think that a lot of the discussion around AI has focused at the poles. Right. It's focused at the extremes. You know, some people who think that within 24 months, you know, we'll have super intelligent robots that are going to take over the world like Terminator style. And there's some people that think that AI is like a flash in the pan and we'll all be talking about something else in 12 months time that I don't see. So I don't buy either of these stories. You know, in the end, the world clearly is changing fast because of this technology, but not every aspect of the world is changing. And you know, if you think about, we were talking about labor markets, how many jobs do you think can be done with a computer alone? So we know the answer to this question. So about 30% of all roles in the U.S. this is, are in principle teleworkable, meaning you could do them in principle behind your computer. But that involves things that you would never under any circumstances actually do behind a computer. Like in theory you could be a primary school teacher behind a computer. But like we tried Covid in practice, well, primary school teacher, can you imagine like chaos would, would break out. So most jobs are actually not kind of, they cannot in our current economy, in the way it's currently organized, cannot be done with a computer alone. So, you know, someone came to fix my Internet yesterday. You know, you think Internet, advanced technology, fiber, you know, this is exactly the sort of thing that's the future, exactly the sort of job that AI can do. But of course, you know, it required some guy like pulling up a bit of the pavement and like, you know, hanging out down there with a pair of scissors for about an hour. Right. AI is not going to do that anytime soon. So there are many, many things that will hold back the rapid advance of AI. And that means that what we're going to get is a world which is Neither at the doomer extreme nor at the kind of like, you know, it doesn't matter extreme. But we're going to get something weird in between.

[B]

Well, I think, I think a lot of it's going to come down to how and where the productivity is captured and distributed. I was, I think it was Jeff Bezos the other day who was, somebody was talking to him about AI and the jobs that are going to be lost. And he said if you just said to somebody during the agricultural revolution who was losing their job, in 30, 40 years there'll be somebody called a massage therapist and you will go to their little, you know, spa and you would lay on a table and they'll rub your back or there would be a dog psychologist. These people would think you're mad. But these jobs were created and, and so I'm kind of with him that if we, if we can capture the productivity of boring mindless jobs and, and we can use that productivity to allow people to pursue more creative jobs which will have its challenges in a debt based economy. But if we can do that. I'm more of an optimist. I accept the risks with AI. I prefer we keep it a bit narrow for now and don't let superintendent stuff. But I think I'm becoming more of an optimist.

[A]

Yeah, I think the challenge is that lots of people who work in the companies building these technologies would say to you it's precisely those creative jobs that are the ones that can be most easily automated. So jobs that have to do with just processing information, you know, thinking about staff strategizing. These are actually the jobs that AI can do. The jobs that AI can't do is like, you know, being a hairdresser, you know, kind of physical things in the physical world.

[B]

Well, that's, I think, why my daughter wants to be a tattoo artist. Because I don't want to worry about doing it. No, I was thinking more like when I say creative pursuits, like who doesn't love a farmer's market or an arts fair? We just go around and see the creative things that people have done. I mean I met some lad who, he was like 16 and he was making different steak sauces and you know, can we distribute this in a way that people can creatively pursue the things they want to do? Little small businesses. Can we decentralize it all? Yeah, I don't know. I don't know.

[A]

I think there's an interesting story around decentralization which is that of course we'd all, we all like things to be more local. Don't we? You know, we like to, you know, we want our community to have autonomy and, you know, we want rules and norms and the way things work to be adapted to the locale. And one of the barriers to that is just interoperability. Right. So it's very difficult to do that when, you know, imagine a world in which, you know, every town had its own rules. It would be impossible to do business. Right. But in a world of AI, actually, maybe it doesn't become impossible anymore because you could imagine that even if you had a great deal of diversity in, let's say, I don't know, kind of the legal or legal infrastructure that you needed to operate to do business, you could just ask Claude to figure it out and it would do a pretty good job because that's exactly the sort of thing that AI can do. So I think there are tremendous untapped opportunities, but at the same time, you know, being pragmatic, you know, there's a lot of kind of like big, really difficult kind of get together and make some decisions kind of moments. And like, you know, humans, those big get together and make some decisions moments, we're just not great at them, are we? You know.

[B]

No, no, not always. But sometimes in a time of crisis, maybe we are.

[A]

Maybe.

[B]

Is there anything that's particularly exciting you about AI that maybe somebody I'm not aware of in that world?

[A]

Yeah, well, I mean, you know, I referred briefly to thinking about AI as not just a single entity, like an individual, but thinking about what it means for AI systems collectively to be embedded in, like, the infrastructure that we use and for them to interact. And, you know, this is. This is not a sort of. You said what I'm excited about. I don't know if this is this, this is definitely more on the kind of like worried end of excited, but like, this is the thing that I think we should be paying attention to. Right. So ChatGPT had 100 million new users in the eight week period following its release. So the types of tools that allow AI systems to. Like in your uncanny valley story about sending an email or AI system that sends an email on its own, if we have rapid uptake of systems that can do stuff on our behalf, and as we were discussing earlier, that stuff might be misaligned either for reason A or reason B. Right. Either because it misinterprets the instructions or because it has its own interests and we have very rapid uptake of those systems, then we're suddenly in a world where everything is quite different and probably quite weird. As well.

[B]

How so? Paint a picture for us.

[A]

Well, so imagine a world in which you receive an email in the morning. You can't tell whether it's from a human or an AI. Let's imagine a world in which every video call you have, it's unclear whether you're interacting with an AI system or another person. Imagine a world in which you could swap. You put your AI in charge of your utility provision and it swaps you from one company to another every three days. Imagine you're in a world where you're trying to buy something on ebay and there's AI systems that can kind of zip in at the last minute and outbid you. And there are opportunities for every tool that we use to communicate with each other as humans or to transact, to share information, there are opportunities for those tools. If those tools can be used by AI. There's opportunity for our experience of interacting with each other, with society, to be a very different one.

[B]

But I, I kind of want that because I see that potentially leading us to a, like an analog renaissance.

[A]

I don't quite know what an analog renaissance is, but, like, it sounds fun.

[B]

So, I mean, I was, I was on the train this morning. I was chatting to a friend I bumped into and I, I said to her, you know, talk about life and things and everything, being so busy all the time. And, and I've recently been. I deleted 150 apps off my phone in a day and I have this thing where I delete one a day and I gradually get to the more fun ones. I've got Twitter as my nemesis, but I've also turned off all my notifications and I'm just trying to have less of a relationship with it in a world where AI takes over all the computers. And I kind of like the idea of going back into the world and maybe growing some tomatoes in my garden and having like, can that digital world just be run by the AI for us and look great if I want a delivery and I want some food or I want to go shop, whatever. But do we just not need that world anymore?

[A]

But you could go and. You could go and be a monk, couldn't you, and grow tomatoes.

[B]

I could. I still want to be. I've got one foot in.

[A]

Exactly. That's the, that's the trouble, isn't it? Is there's a sort of bargain there, right, which is that, you know, we want to retain control. Right. And one of the reasons why not everyone goes off and, you know, kind of cloisters themselves in a purely tomato growing environment is because we actually want to have control over the world outside. And I think that the challenge with the scenario that you're describing we really do outsource like lots and lots of communications, you know, business operations, even governance. Governance and strategy. We outsource that to AI then we no longer have control over what happens.

[B]

Or it could be a case of that world just becomes too risky in that. If my daughter wanted to go to London, she said I'm going down at 10 o' clock and I'm coming back at 3 in the morning. One of the primary reason for no isn't just being outlayed, it's because it's dangerous.

[A]

Yeah.

[B]

If this world just becomes too dangerous because there's the AIs are controlling too much and it's all a bit weird. Maybe you just can't use this anymore. I can't trust the phone call, I can't trust the video call, I can't trust the news.

[A]

I think it's very difficult, it's very difficult to talk about the future. But I think that we can anticipate. All major technological deployments in the past have led to significant disruption. And so I don't want to forecast whether the outcome is utopia or dystopia. But as I said. But I think it's. But I think there's a. Either way there's a wild ride to get there. Yeah.

[B]

Listen, this is. This is amazing. I'm probably going to want to talk to you again at some point in the future when there's another big breakthrough. But thank you so much for coming in.

[A]

It's been a pleasure. Thanks Peter.

[B]

Take care. Thank you everyone for listening. Bye.