← Future of Life Institute Podcast
Future of Life Institute Podcast
From Peak Horse to Peak Human: How AI Could Replace Us (with Calum Chace)
01:37:21
On this episode, Calum Chace joins me to discuss the transformative impact of AI on employment, comparing the current wave of cognitive automation to historical technological revolutions. We talk about "universal generous income", fully-automated luxury c
Loading summary
Transcript89 lines
- [00:00]
Callum Chase
What we're going to have to do is find a way to transfer some of the wealth being generated by the machines to everybody. And boy, this is a hard problem. One is this little word basic. We have to do much, much better than give everybody a basic income. A future in which huge wealth is being created by machines and 99% of the population is just scraping by. That's an appalling world and we shouldn't do that. Universal, generous income, and that's where we need to get to. The likelihood of being able to forever control entities which are much smarter than us is pretty slim. Our best chance may lie in making sure that they're conscious, because if they're conscious, they will have empathy. Having conscious beings appreciating the beauty of the universe is possibly the most important thing the universe has.
- [00:51]
Gus Docker
Welcome to the Future of Life Institute podcast. My name is Gus Docker and I'm here with Callum Chase. Callum, welcome to the podcast.
- [00:59]
Callum Chase
Hi Gus. It's a privilege to be with you.
- [01:01]
Gus Docker
Fantastic. Could you give a quick intro to what you've been working on?
- [01:06]
Callum Chase
I have been thinking about the future of AI for a very long time. I wrote the first article I wrote about it was in 1980, and I was wrong about pretty much everything. I thought that work was going to be automated within the next few years. I was obviously completely wrong. I had not yet then heard of Amara and Amara's Law, which is the observation that we overestimate the impact of AI and any other technology in the short term and underestimate it in the long term. I've seen that played out many times since then. In 1999 I read Ray Kurzweil's book Age of Spiritual Machines. I think a lot of people got interested in the future of AI then. And so since then I've been very interested in where we're going. I retired in 2011, thought it would be a good idea for Ridley Scott to make a movie based on something like Kurtzfeld's book. So I thought, well, I'll write a novel which he can turn into a movie. And bizarrely, Ridley Scott never got hold of my novel. I don't understand that. Somebody pointed out that the novel was actually a non fiction book dressed up as a novel, so I should redo it as a nonfiction book, which I did. And all in all, I've written five books on AI since then. And then people started asking me to give talks. And originally I thought talks would be really good marketing for the books. And I discovered it's the other Way around books are good marketing for talks. And so I've given about 200 odd talks in 20 countries. That's what I do. But more recently a friend of mine started a company called Conscium. I'm sure we'll get into that later. And I'm one of the co founders of Conscium and my title is I'm chief marketing officer of that. So I kind of see myself now as somebody who's retired with two full time jobs.
- [02:45]
Gus Docker
That's perfect. Right? One of the topics you write about quite extensively is whether we're going to see technological employment. And there are two kind of classes of arguments against the notion of technological employment. So one argument that's often brought up by economists is that past technological revolutions didn't really cause this widespread unemployment. So, so you could, you can introduce a new technology. You can basically automate much of agriculture, for example, and have people move to factories. You can then automate much of the work that goes on in factories and have people move to offices. So why shouldn't we expect to see the same thing once we begin automating office work? Why isn't that, isn't it that people will then move into some other domain that we can't really foresee, but it would be an extrapolation of previous trends for humanity to move into some other domain?
- [03:45]
Callum Chase
Yeah, I mean it is true that past rounds of automation haven't caused lasting widespread technological unemployment for humans. I'll come back to that. But of course, past performance is no guarantee of the future. And if that was true, if past performance was a guarantee of what happens in the future, we wouldn't be able to fly. The future can sometimes be different from the past. Some people say, you know, there's nothing new under the sun, but sometimes things do change. We've in the past mostly had mechanization. Most automation has been mechanization. Machines have been used to substitute human and animal labor. What we're now getting is cognitive automation. Machines are replacing humans cognitive abilities. And we have seen this happen quite a wide scale. I mean, when I started work a long time ago, there were secretaries in every office. Every manager to a very low level had a secretary and the secretary used the computer. When they arrived, the manager disdained the computer. That was secretarial work. And now there are no such thing as secretaries. They've just disappeared. There were a lot of them. They're now doing something else. So past rounds of automation have removed people's jobs, but those people have gone on to do other jobs which is why we're still fairly near full employment and probably will be for a while, but we don't know. And in fact I think it's very implausible that there is an unlimited reservoir of jobs that humans can do forever that machines can never do. And I think it boils down to this. Are we going to carry on developing machines which are smarter and smarter and smarter to the point where they can do everything that we can do for money? There's a different question about whether they will replace everything we can do full stop, but can they replace everything we do for money? And I think the answer is yes, unless we stop and we're not going to stop. Or there's like a silicon ceiling beyond which machines can't go and we see no sign of that yet. So it seems to me obvious that at some point we don't know when machines will replace all humans and jobs. There'll be lots of new jobs will be created, but machines will do all those new jobs.
- [05:55]
Gus Docker
Yeah, which is, which is quite a strong argument I think, at least when I'm hearing it. But it's the notion that the future is going to be unlike the past is something that we should be quite skeptical about just before we see any evidence. I think it's often been a sure bet to say, okay, we've heard these predictions before and the future is probably going to be like the past, so why is it that we should expect AI to be different here?
- [06:26]
Callum Chase
It's quite right, the boy often cries wolf, but actually at the end of that story the wolf turns up. I think a really good example of this is self driving cars. Self driving cars have been next year's thing since at least 2016, and for some people considerably before that. And every year Elon Musk has predicted that Teslas would be self driving and every year it's been wrong. Well now self driving cars are here and the world is different. We kill at the moment 1.3 million people around the world with car accidents. And in some number of years when we have pretty much the entire fleet replaced by self driving cars, that won't happen anymore. That'll be a huge change. I mean the number of lives that are devastated by that holocaust, it'll be brilliant to have that stopped. So it is true that most of the time, if things have worked out a certain way in the past, very often then they'll probably do the same again. Patterns do tend to repeat, but every now and then there are disjunctions. Every now and then something new happens. The Industrial revolution was something new. The agricultural revolution was something new. And now we're in. Actually, this is a little bugbear of mine. We're not in the fourth Industrial Revolution, we're in the fifth revolution. The fifth revolution is the information revolution. We're in the early stages of that and it is going to change everything. It's going to change everything about the way we live, the way we work, the way we die. And I just want to go back actually to something that I forgot to mention. Max Tegmark of this parish is one of the people who's very effectively pointed out. We have had mass, widespread technological unemployment in the past and it was of the horse. In 1915 there were 22 and a half million horses in America working mostly pulling vehicles around. And 1915 was Peak Horse. The internal combustion engine mechanized them. Now there's 2 million horses in America, so 22 and a half million down to 2 million. That is unbridled technological unemployment. So we have seen it in the.
- [08:28]
Gus Docker
Past and the fate of the horse is something that we should hope to avoid for humanity. It's probably the case that the horses that are alive right now are taken care of to much higher, at a much higher level and their lives are probably better than horses used for actual work in the 19th century, say. But horses, they have lost basically all influence and they are not important in any deep sense to the, to the functioning of the economy anymore. So if we are, if we are faced with a crisis and we have to, you know, say, say there's some trade off where we need to either get rid of the horses or, or we will, we will bear some, some large economic burden. We will get rid of the horses because they're not a centerpiece of the economy anymore. This position we don't want to be in is that we are like horses to AIs where we are slowing things down, or we are something that's merely there for, for entertainment.
- [09:28]
Callum Chase
Yeah. Amen to that. I'm sure you're right that the 2 million horses in America today have a much better life, each of them, than the 22 and a half million that there were in 1915. But the first, the fact that there's only 2 million of them is quite devastating for those who would have replaced them. And we do not want that to happen to humans. We're very different from horses. I don't think we're going to follow the same pattern. But we do need to figure out what the new world looks like after technological unemployment because it'll look Very different.
- [09:54]
Gus Docker
Yeah. You made two points that I want to address here. On the first one about secretaries. It's true that we've now seen a rapid decline in secretaries and human secretaries, but it seems to me that now AI models are getting so good that they can function as secretary equivalents. So there's also, there's the notion that new capabilities in models will create new demand. Because now that we have these capabilities, maybe there is now an argument for spending more money for each human employee to give them a secretary each or 10 secretary each, where these new secretaries are AI models. So this leads me into the other point, which is a classic point against the notion of technological employment. And it's the notion that there is basically infinite demand. So humans will demand more and more and more and we are never satisfied. And that means that we will never run out of things to do. Yeah. Does that make sense to you?
- [10:58]
Callum Chase
Oh, of course it does. And it's often described as the lump of labor fallacy that you know, there isn't just a fixed amount of work to go around and if you machines do some of it than humans will do less of it. It's obviously true that the amount of work that can be usefully done expands dramatically. Anybody who's read any history book at all knows that the amount of work and valuable things going on in society today is way different from what was going on in the 15th, 16th, 17th centuries where basically most people were subsistence farmers scratching a living and there were a few kings and priests wandering about. The error in that line of thought is yes, there's a possibly infinitely expandable amount of work that can be done, but there's no rule that says that work has to be done by humans. And my point is that if machines get to the point where they can do everything that we can do for money, cheaper, better and faster than we can, then there will be lots of new jobs created all the time. It's just that machines will do all those new jobs. And I've never heard of sensible reply to that.
- [12:00]
Gus Docker
I mean, one reply there might be to say that there will be demand for human human created products and services because they're created by, by humans. So for example, we might imagine, say we are in a, in a very, in a wealthy future. And you can, you can go on, on a retreat and, and on that retreat you can talk to other humans, interact with them, dance with them. And this is something you're willing to pay a lot for because you're rich. And we've automated the economy to such an extent that there is this surplus value that can be spent on luxury goods, like going on a retreat and interacting with other people. Isn't that an argument that we will create jobs that seem less and less like actual jobs and more and more like something we would do for fun?
- [12:52]
Callum Chase
So I think in a world post technological unemployment, after what I call the economic singularity, where we have to accept that there's no jobs for humans, there will still be lots of work. There'll be lots of things humans can and will do for work, but we'll do it for fun. In a world of whatever the number of people will be in the world, then 10 billion people. Why on earth would you pay somebody to have a conversation with them? And there's all these other unemployed people around, some of them are your friends. Be a much better idea to go on holiday with your friends rather than go on holiday with somebody that you've paid to be with you. I think people will be busy. I think people will be busy having really a really interesting time. We will do things like painting when we are nowhere near as good painters as an AI and certainly nowhere near as good as Caravaggio, but we'll be the best painter we can be, and we'll be the best golfer we can be, and we'll be the best podcast, opposed to what you can be and so on and so forth, I think there's no shortage of things for humans to do. It's just that economically, the machines will be doing them better and cheaper. And so if you're going to pay for something, why on earth would you not pay for the best, cheapest version? You might buy some artisanal pots as a special piece for your house, but most of your cups and your cutlery, you're going to buy the best quality you can get at the cheapest price, and that's going to be made by a machine. I don't buy the idea that artisanal handcrafted goods and indeed services is a way for all humans to make a good living. I think what we're going to have to do is find a way to transfer some of the wealth being generated by the machines to everybody. And boy, this is a hard problem. This is not easy. Some people think there's a magic solution called universal basic income, and there's an awful, awful lot of hand waving that goes on around that. But some form of distribution is going to be needed. I think that there's quite likely to be two economies, actually. One is the economy in which everybody gets what they need for a good standard of living. Not a basic standard of living, but a good, let's say, you know, American middle class standard of living. That's everybody's not right, but everybody gets it by virtue of being a citizen of the world. We need to figure out how that happens. Then there could be another economy in which you make surplus money by trading in original Aston Martins or original Caravaggios or your wonderful handcrafted pots because you happen to be a very good gifted potter or by being a singer because you're a particularly good singer. Those sorts of things are kind of star economies. And there's likely to be a few people will make a lot of money in that side economy. I don't think it's something where everybody can subsist that way.
- [15:37]
Gus Docker
I mean, think of the size of the market for luxury goods already. If you're in a, in a, in a rich country or rich state. In the U.S. for example, there's a lot of spending on mechanical watches made by hand handbags made by hand cars that, that are the right brand and the right year and so on, but aren't necessarily as good as a much ch car. Why isn't it that basically most of humanity can enter that market?
- [16:13]
Callum Chase
The car, the handbag, the scarf made by a machine will be better than the equivalent made by a human. So the only value that the human's adding is the prestige. And that prestige is only valuable if it's rare. So it means not everybody can have it. Know you can't have a. I don't think you can have an economy in which an Hermes bag. I think Hermes makes bags. I think it's Hermes that, that have the most unbelievably expensive bags. And you have to beg and plead and wait for years to get one. You just by definition, not everybody can have that because then it loses its value.
- [16:52]
Gus Docker
Yeah, yeah. What is it that we do in this world then? If so, if we're not earning money by working, are we then earning money by moving into becoming investors or becoming kind of allocators of resources?
- [17:09]
Callum Chase
Well, I don't think so. I have heard those arguments made, but I don't see how the whole human race is suddenly going to become an investor. It's actually quite an unusual skill being a good investor and likewise allocation of resources. Not everybody can do it. No. I do think that we're going to need to provide a good standard of living to everybody, regardless of what they do. I don't think it's going to be mediated by a contribution by a job of any kind. And I know that a lot of people listening to this are probably going, oh my God, that's communism. We can't have that. And I don't think it should be. I don't think that is the right answer. One thing we know is that the market is a terrific mechanism to allocate scarce resources. So I don't think we should lose that. I have a rough, sketched out idea called fully automated luxury capitalism. But the details, very complicated, hard to work out, and we need a load of economists to be locked into a room and not let out until they've worked it out. I think there's going to be massive wealth created by machines. They're going to be generating enormous amounts of goods and services very, very cheaply. Logically, there has to be a way to distribute that to everybody. How it's done, how it's done without having a central state which allocates things to people we know that isn't going to work. The central state will become corrupt. That's almost inevitable. They won't have all the right information to allocate resources correctly. It's not a good solution. So there has to be some other form of distribution. But I confess I don't know what it is. And the thing that staggers me is how few people are thinking about this. There's loads of economists in the world and I don't know any of them that are thinking about this seriously.
- [18:57]
Gus Docker
The classic answer is universal basic income, where you simply distribute money, which is infinitely. People can use money on whatever they want, and so you find the local information that's available to the people who get the money. And then the job of the government here is simply to distribute the money. Does that plausibly solve some of the problem? So now we're not talking about the government deciding what to buy for people. We're talking about people deciding how to spend money that they get from the government.
- [19:29]
Callum Chase
It's got the germ of the right idea. It's got some major problems. One is this little word basic. We have to do much, much better than give everybody a basic income, a future in which huge wealth has been created by machines and 99% of the population is just scraping by. That's an appalling world and we shouldn't do that. Another big problem with UBI is people often argue that it could be introduced today. And I think we saw during COVID that that is just not true. During COVID there was a form of UBI introduced in most countries. Most democracies anyway, handed out money to most citizens or many citizens. And it was the right thing to do. We had to keep things going when the economy is all shut down and it nearly bankrupted loads of countries because it wasn't sustainable, because we are not yet rich enough. But in this post economic singularity world, we should have an economy of abundance. That's a good phrase which has recently acquired a new meaning. It means the ability of democratic governments to build things. But the older meaning of that machines are producing vast amounts of wealth that should arise. And so logically it should be possible to distribute that wealth to everybody. And some form of ubi, some idea based on that is probably the right way forward. I mean, Elon Musk, whatever you think of him recently, he does talk about kind of universal generous income. And that's where we need to get to something like that. But what the mechanism is for the distribution, who knows? Because think about where the wealth is, who's creating it, who owns it. It's not implausible that five years from now most of the wealth will be generated by companies which make AI models and things slightly downstream of AI models, which means that OpenAI, Anthropic, Google, DeepMind, Meta and so on, will be generating most of the wealth in the economy. And the owners of Those businesses, the CEOs and the owners, will own or control all that wealth.
- [21:36]
Gus Docker
Do you really think we'll see that level of concentration? Because so if it's most of the wealth created in the economy is going to the AI companies, I mean, normally we would, or in normal scenarios we would expect other companies, say in the S&P 500, to integrate these AI models and capture a lot of the value and therefore you would see less concentration. But do you think the economy becomes more and more concentrated over time, given AI?
- [22:08]
Callum Chase
Oh, I don't know. And I'm certainly not saying that OpenAI and their brethren are going to control 90% of the economy. That may happen or it may not. I'm agnostic about that. What is true is that the US economy has streaked ahead of, say, Europe's economy, and that's significantly due to the rise of Big Tech. That's really pretty much all of the difference is Big Tech. So they've captured a lot of the economy, but it doesn't really matter whether it's just them or it's them, plus Boeing and General Electric, but US Steel and lots of other companies. The point is there'll be a relatively small number of people who will control all this wealth. In this new economic singularity world and everybody else will be relatively suddenly without a job. So there has to be this distribution. Now there's a very uncomfortable conclusion from that, is that you've got to have a big tax. You've got to have a big tax on the people who own the wealth. And are they going to resist that? Are they going to disappear off to very well fortified islands and you know, sequester themselves surrounded by their robot servants and so on and watch while the rest of us starve to death? I don't think that's going to happen. I don't think those people are like that. And it's certainly a future we don't want.
- [23:22]
Gus Docker
Yeah, this is something we definitely want to avoid.
- [23:25]
Callum Chase
Absolutely something we want to avoid. And I think one of the things that may help us avoid it is the nature of the abundance economy. So what happens in the abundance economy is that everything gets cheap. You've removed humans from the production process, both of goods and services. And humans are generally the most expensive part of any process. Human. You've also made the processes much more efficient because they're being run by AI, they're being optimized. And incredible as it may seem now, particularly if you're in Europe where energy prices are high, energy prices are going to be very low. They're going to be very low because we're stopping digging up dead dinosaurs and dead trees to fuel power plants and we're moving towards an incredibly abundant source which is unlimited, which is the sun and wind and so on and also nuclear. And these are essentially very cheap forms of energy. And they're getting, the capture of the energy is getting cheaper and cheaper. So it's entirely likely that in some number of decades, if AGI last gives us that long, we'll get to that. I'm sure that energy will be too cheap to meter. So no humans in the production process. Production process is very efficient. Energy very cheap means that the cost of all the goods and services that you need for a very good middle class American standard of living are very low. Which in turn means you don't need to tax the rich people much to provide an income for everybody. Now that doesn't mean to say the rich people will like it. It doesn't mean to say they will won't oppose it. And that is going to be an enormous source of friction. And the sooner we start to work out a plan for how we both, you know what it looks like once we've achieved it and also how we get from here to there the better off we'll be. Because if we arrive fairly suddenly at the economic singularity and in a fairly short period of time, people go from more or less full employment to more or less zero employment of humans. And if we've got no plan for it, we're in some difficulty.
- [25:17]
Gus Docker
And do you think that's plausible? Do you think we will reach some threshold in which we will quite quickly then move from almost full employment, as we're seeing now, to very low employment?
- [25:29]
Callum Chase
Yes, I do. And this is one of the more recent things I've been thinking that the argument that automation creates efficiency, efficiency creates wealth, wealth creates demand, demand creates jobs, which is used generally as an argument against the possibility of technological unemployment. It is true. That is what automation does. And there is unlimited demand for goods and services, or at least, you know, they're certainly very elastic demand for goods and services. It may be unlimited. Which means that as long as there are some things that humans can do and machines can't do, there should be lots of jobs for humans. Even if we all end up as plumbers. That's, you know, that's a cartoon example. But if we all end up as plumbers, there's still scope for an awful lot of plumbers. But then the day comes when the machines can do plumbing even better than us. And so, you know, that's it. There are effectively no jobs of humans now. I suspect that even post the economic singularity, there'll be some jobs which are reserved for humans for some time, like making the ultimate allocation decisions that governments make. Perhaps, although you could have decentralized autonomous enterprises, DAOs, there could still be a preference to have human CEOs at the top of large companies. So I think there'll be some jobs that humans will do until we get to superintelligence. But most humans, I think, won't stand a chance of competing against machines.
- [26:51]
Gus Docker
Do you think we'll see protectionism? So, for example, protectionism in the medical field or the legal field, where the legal field is obviously close to the law. They know how to argue against AI automation of their field. You could also imagine protectionism in the educational fields, where I think there are some arguments that are quite easy to make about the fact that we maybe we don't want our kids to be taught by. By AIs. We want. We want them to be taught by, by actual people. This is something I see. This is something I think historically it's been a fact when we see, when you have a technological transformation, you see protectionism in, in traditional industries. That are reluctant to change. Do you think this moves the needle or do you think this will. Yeah, basically preserve a large fraction of the economy for humans?
- [27:47]
Callum Chase
I think that kind of protectionism is inevitable. It will happen and it won't last because each country that tries to, or each country that allows that kind of protectionism is going to find their version of that industry woefully uneconomic. And it's going to happen very quickly. There was an interesting episode about a year ago when an Indian minister said, we're not going to have self driving cars in India. We want human drivers. And within a week he was forced to reverse that because everybody realized that's crazy. If everywhere else in the world has gone to self driving, we're going to be killing lots of Indians and our taxi service is going to be ridiculously uneconomic so that it can't last for some time. As I say, I think humans will continue to be in demand for jobs until fairly suddenly the phase change happens and you go to the economic singularity. So in education and in healthcare and in law, there will be work for humans to do, but there'll be increasing amounts of the work which will be done by machines. And it's a good thing. The thing about education at the moment, most schools are very similar to the institutions that were set up by the Victorians to get people fit to work in factories. We get 20 to 30 children, sit them in a grid, tell them to sit still for an hour at a time and listen to somebody talking up the front about something which either is so obvious to them it's boring or they've got no clue what they're saying and it's boring. You know, this is torture for children. We shouldn't be doing this to children. It's the last thing in the world they want to do is sit still for an hour listening to somebody. Even if you, if you've got a great teacher, that's fine. If you've got a really inspirational or exciting teacher, then that works. But one teacher trying to get the right information for 20 to 30 kids, that's a terrible model. What we need is each child to have their personal tutor. I like to think of Alexander the Great who had Aristotle as his personal tutor, went on to conquer the world. And I think every child could be Alexander the Great and have their personal Aristotle. Now, their personal Aristotle is a machine. It knows everything that they know. It knows everything that they ever need to know in the future. Specifically, it knows the things that they need to know in the next five minutes, the next 10 minutes, and so on. It's the best possible way to educate the child. There'll be human teachers around in the classroom, assuming there are still classrooms, because you need human role models. The children need to, you know, how to socialize and to get on with both adults and other children and also to inspire. Machines can inspire, but humans are much better at it. So for a long time there'll be a role for teachers. The machines will kind of just encroach more and more on what they do. And then when that flip happens and the machines can do everything, then oops, there's no more human teachers, or at least very, very few.
- [30:36]
Gus Docker
Do you think people will serve as kind of vessels for AI produced material for at least some time? So you could imagine a law firm where everyone is using AI to do their job, and perhaps it's kind of unspoken knowledge that this is the case. Or you can imagine teachers producing all of their materials and producing all of the work that goes into teaching beyond doing the actual teaching by AI. Could it be the case that we will see kind of automation sneak up on us because people are using AI tools in their jobs without mentioning it?
- [31:13]
Callum Chase
Well, that's certainly what's happening at the moment. There was the famous case of the New York lawyer who went into court with some cases which had been written for him by GPT and he thought he had researched them, in fact it had made them up, it had confabulated them and he was told off. And I believe actually he was disbarred. And UK lawyers just last week, I think it was, got a warning from their professional body, you know, don't use large language models without checking their work. And that's the message everybody really ought to have got by now. These things are incredibly powerful, they're very useful. But if you're doing something important, if you're producing some information which is important, then for goodness sake, check it, check it's right. But actually I think that will change because at the moment the base models hallucinate or confabulate a lot less than they did. I'm sure you've noticed this. I've certainly noticed this over the last year. When I first started using GPT4, I think when I that one, I was writing some books about Italy and Bernardo Bertolucci, who's a film director, just kept popping up all the time in the rough draft I would get the machine to do, I'd keep taking him out. He was all over the place. That doesn't seem to happen anymore. Now the advanced models, the reasoning models, they're confabulating more. And there's been research showing that. I think it's, I think it's GPT3O confabulates about 25% or 38% of the time. But I think that's just because these are new models, they're in development. And the other thing is that corporates have yet to tame large language models, partly because of confabulation, partly because of privacy issues. And so they're not being successfully deployed at an enterprise scale by many organizations yet. But that will change. And one of the things that will change that is AI agents, which are starting to appear this year. And so it will be less of a clandestine process where we all use large language models privately and covertly at work to a model where, you know, the enterprise has got this great big model it lets you use and it's just part of the, part of the company process.
- [33:19]
Gus Docker
Yeah, I really can't figure out whether these models are incredibly useful to professionals in various fields or whether they are overrated. Because when I interview people who are, who are, say, engineers implementing these systems, they will often get complaints from their users that this is not really useful. This doesn't understand the context of what I'm working on. This, this basically provides very little actual value because it, it produces something that's not at the, you know, what, what's important in many jobs is peak output. And, and this produces something that's, that's perhaps below peak output and, and therefore it doesn't kind of push the frontier of what you can do. At the same time, though, I also interview people for which these models are basically, you know, they can't function at work without them. So people who do programming, this is very much the case. Many mathematicians find them useful. Scientists increasingly find them useful. It's an interesting situation we're in in which people who perhaps do work like I do, where a lot of it consists of writing and writing emails and doing all of these things there. The models aren't as useful as in fields which are kind of traditionally considered much more difficult to handle, like science, mathematics, programming. What do you think causes this? Or do you think this is a real effect?
- [34:46]
Callum Chase
Oh, I think it is real. I can see it. And I think it's partly a mindset thing and it is partly a question of what task you do. I mean, as well as the areas you mentioned, anybody in the creative world, anybody making video is thinking, well, you know, VO3 can do really good 10 minute clips. And you know, that used to cost £50,000 and now it's free. So that's quite important. I use GPT4 all the time. My kind of day job. And if you like, my day jobs are quite like yours, I think. You know, I'm basically in the. In the business of processing information and talking to people. I use it an awful lot. I use it for a number of different types of purpose. One is, and this may be because I'm knocking on a bit, I forget things and I need to remind myself. I mean, just this morning I was trying to remember what the Canadian fast food is. And you know what? I've forgotten it again. I looked it up and I've forgotten it again. Anyway, Horton, I think it's Tim Horton, that sort of thing. I use it for all the time. I must use it, you know, two dozen times a day maybe. I also use it to draft some things. So I'm doing some longer form writing. I might do a first draft. The thing I submit at the end looks nothing like my first draft. But the tyranny of the blank page is a nice thing to get over. More useful, perhaps, is summarizing other things that come to me because I get a lot of material that I need to get through. It's way too long. It doesn't need to be anything like that long. The executive summary doesn't quite cover it. So I put the whole thing through the model and I get a very good summary. And then another thing which I think many people haven't yet tweaked is models have digested the whole of the Internet. They know pretty much anything any human has ever thought. They're really good at suggesting good strategies for human interaction. That's a really odd thing to say. But these machines, bizarrely, are providers of emotional intelligence. They're not emotional and they're not human and they're not conscious. But they can tell you a good way to deal with a tricky situation. Whether that's a negotiation, whether it's an argument with somebody, purely intellectual argument with somebody, or whether it's a highly charged emotional argument with somebody. They're very good at giving you good strategies and giving you insights into what may be going on in the other person's mind. Seems very odd to use a large language model for that. But they are very good at it. And I think young people now, Generation Z, they know this instinctively. They've grown up with these models. They were there as they were entering adulthood and they just think, well, of course they do. Of Course, these things know how to, how to work. And so they've already got the mindset. Some people, possibly people like myself, who've been excited about AI for decades, you know, we've got the message, other people just haven't yet. I do meet a lot of people who say I used a large language model once, it didn't give me exactly what I needed and I've never tried it again since. I think that's a mistake.
- [37:52]
Gus Docker
So on the point of summarizing what you get inbound or starting, you know, helping you draft your ideas, and actually on the point of solving kind of emotional issues or negotiation issues between people, so solving kind of this interpersonal situations, do you worry there that you are now adopting values that are put into these models? So when, when you're drafting something with a language model, you are in some sense, you're in some sense incorporating the values that were trained into that model and reinforcement learned from human feedback into that model, which affects your draft and then affects the way the line from which you start thinking. And you can also think of the summary as perhaps it's summarized in a specific way. The models, at least for now, tend to be very, very kind of happy and perhaps a bit boring and very concerned with PR in some sense. Do you worry that this set of values that are specific to these models are now implicitly being incorporated into your life and actions?
- [39:11]
Callum Chase
No. And this may be pure arrogance on my part, but I take their output and then I make it my own. And I have a particular voice when I'm writing and I like to think that anything I write, an email, an article, if I was to write another book, that it would have my voice. I'd be incredibly disappointed if it didn't. I do actually write a lot of books. I write very short books about travel and they definitely have my voice. The friends of mine, I give these books to the sort of hobby things.
- [39:41]
Gus Docker
But you're an established author, right? And you have had your own voice for perhaps decades at this point and you know what you think and maybe you have a bunch of ideas in your head and so on. If I rephrase the question slightly and ask if a person that's 14 begins using these models, they start using these models for everything, basically. Do you worry that they will be pushed in certain directions that they're not fully in control over? Perhaps like you see people being people's preferences being pushed around by social media algorithms.
- [40:15]
Callum Chase
It's a very fair question. In my 24 year old son, I don't See that happening. I see his voice very clearly in everything that he writes that I see. So, no, I don't think that's happening. And actually, if you take seriously the idea that these models give you a very sensible way of looking at things because they know all of best practice, you know, in human interaction, then you start from a better place. You. You're. You're absolutely right. They're very, very bland. And I actually want them to stay that way. I don't want them to have their own charismatic voice. I don't want.
- [40:48]
Gus Docker
But that's interesting. Why. Why do you want bland models?
- [40:51]
Callum Chase
Because I don't want. I don't want a model to be substituting its voice for mine. You know, even. Even in my little travel books, if the model was to produce a chapter which is just the very, very first draft, and it was good enough to be published, I might get lazy, and I might do it. I might let it do it without imposing my voice on it at the moment. It doesn't. It produces something that. Just very, very boring. But it's got all the basic facts and it's got the basic structure, and that's what I like. And I then weave my own personality into it. So, yeah, long may they remain bland.
- [41:31]
Gus Docker
Okay, so we've been talking about jobs, and we've been talking about unemployment and so on. What about meaning in a world in which we do not have to work? So say that we have. And this is a big, very big assumption, right? Say that we first of all survive superintelligence or AGI. Then we somehow solve the problem of redistributing wealth around so that we have a quite good standard of living. Everyone. What then, about the question of meaning in people's lives? There is some. There's some notion that when you do work and you have to do that work, it makes you feel good. It makes you feel like you've accomplished something for yourself, perhaps for your family. And this is something that I'm not sure it can be substituted for by playing status games with other people where you know that you don't really have to do anything. It's merely the fact that you're trying to entertain yourself that makes it the case that you want to do something. So, for example, video games. I don't think video games are as ultimately meaningful for people as careers, for example.
- [42:38]
Callum Chase
This is usually brought up by people as the big problem in the economic singularity. It's the thing people most fear. What will we do for meaning?
- [42:46]
Gus Docker
Just to clarify here, that's Definitely not my take. My take is, first of all, we need to survive as a species. Second of all, we need to have a good handle on how we redistribute resources and not just create resources, and then we can talk about meaning.
- [43:02]
Callum Chase
Yeah, absolutely. And I think you're completely right. I think in the economic singularity, by far the biggest challenge is the economic one, how we do the redistribution of the enormous wealth that's going to be created. But most people instinctively think meaning is the big issue. And I think that's because the economic side of things is just too terrifying to think about. I think there's a whole bunch of reasons why meaning isn't going to be an issue. One is jobs currently don't provide meaning. Now, that may seem odd to many of our listeners because many of our listeners are going to be the kind of people who are very engaged with their jobs. But there's research that's done regularly by one of the big research groups whose name I'm waiting for come into my head and it's not coming. Never mind. One of the big research groups does this survey about every two years and they find the same thing every time. About 85% of people are not really engaged in their jobs. Their jobs put food on the table. And also they give some structure to their lives. So they give them something to do during the day. They're not just sitting around in their pajamas smoking dope all day, but they don't give them meaning. And if you think about the jobs that most people do, you know, driving cars around, relatively menial jobs in an insurance company, processing claims, working on a construction site, these are not things that give you meaning. You'll get meaning from your relationships with the other people doing the jobs. And you might get some meaning from there are Craftsman type jobs and people laying bricks. There's a craft in that. And I'm sure you can have some fun doing it, but the job itself doesn't really give you a great deal of meaning. So that's 85% of the population. And I think most people, when they go on holiday, they're not sitting around going, oh my God, I've got no meaning in my life. You know, they're having fun. And what will happen in the future is everybody is essentially on holiday the whole time now. We will be working. And they're not necessarily status games. When I write a book, I'm not thinking, this is better than the last book that Max Tegmark wrote. Max and I actually were tussling for number one Spot on Amazon's checklist at one point, and I was quite amused by that. So I'm not sitting there thinking, I've got to write something better than Max. I'm thinking I want to express some ideas in this, and I want them to be as clear as I can possibly make them, and I want my ideas on that page clear. So it's not necessarily just status games. I think there are three kinds of people who prove you don't need to have a job to have meaning in your life. One is something which Americans claim they don't know anything about, which is aristocrats. Of course, Americans do have aristocrats, but in Europe, certainly we had aristocrats forever. No, not forever, but since Middle Ages at least. And these people had the best lives of anybody in their society, and they didn't do jobs. Some of them might run estates or countries if they had an empire, but they mostly didn't do jobs, and they had very good lives. And there was no existential wave of despair among aristocrats. Second group is a group of people I know a lot of because my age, comfortably off retired people who very busy organizing social events and learning new languages and playing musical instruments and playing poker and bridge and so on. And it's surprising how busy you can do. You can be doing that. And people often say, yeah, but these people have lived a full life, and so now they're kind of relaxing and they're unusual. Actually. These people have been trained from when they were at school to look for the next hoop that they should jump through, jump through it and then look for the next one. And they get to retirement, and they've had the worst possible training for being retired. They ought to be going out looking for the next hoop. Actually, they're very happy doing the things they want to do. And if you offer them a job, most of these people, they'll send you away with a flea in your ear. The third group, and I think this is completely convincing, is children. You show me a child who thinks they need a job to have meaning in their life, and they wonder what you're talking about. Children find meaning in ordering pebbles on a beach or coloring things in a particular order or, you know, learning how to say something that dad just said.
- [47:04]
Gus Docker
What about if you look at the effect of the COVID 19 pandemic on young people and their sense of meaning in life, perhaps. I think I've seen some data indicating that just staying at home and not being out of the house, not interacting with. With people, that at least Caused some, some despair and some level of, some increase in mental health issues. Could unemployment be like that? Or could kind of permanent retirement in a future with technological employment be more like that situation?
- [47:44]
Callum Chase
If you are young and you're in a social environment where work to some extent defines you, it defines your standard of living, it defines your status, then clearly having the chance to create a career, to develop a career is going to be very important. And having that taken away is devastating. Also, young people are in the business of learning how to interact. We learn first as children and then we learn all over again as adolescents with this different set of rules, with this awful new pheromones running around inside our bodies. And then you get past that and you've got a whole other set of issues to deal with. How do we interact as a young 20 something? It's really tough growing up and to be deprived of the opportunity to do that is devastating. But in the putative economic singularity world where we have solved the distribution problem, then there's no need for people not to socialize. I mean, hopefully we don't have another pandemic then. And humans above all social animals, I mean, it is because we're social that we are the dominant species on the planet. We're not particularly strong, we haven't got very strong claws or teeth, but we can kill any other animal because we're very good at collaborating. And we collaborate by learning how to work together in society. And we learn that as we're children, we get better and better at it. And that process has to happen. We've had millions of years of evolutionary training to be social. That is not going to stop overnight.
- [49:14]
Gus Docker
Yeah, yeah. Some listeners to this podcast will think that we've tackled these issues in reverse order. Where we've started out by thinking about unemployment and technological employment and what we'll do and how we'll distribute resources and so on. But the actually a really big issue is whether we will be able to constrain these models sufficiently so that they're working on our behalf. The so called alignment problem. And we can phrase this problem however you like, because there are some notions of alignment that I think don't really make sense. But in general, what we're thinking about here is how to put AI models on our side such that we are not facing a new species that's working against our interests. Where do you stand on that issue? Because in some sense, logically it comes before or the question of handling unemployment and distribution.
- [50:17]
Callum Chase
So logically it definitely does come before because it's an existential risk. If we end up with a superintelligence which dislikes us, then we're not going to be around for very long. I actually think chronologically it probably comes later, and I go round in circles on this sometimes. I think that the AI that you need for the economic singularity is AI complete, and therefore superintelligence either arrives at the same time or maybe even before the economic singularity. But I still on the whole think, because we're not just working animals, we're also social and emotional animals, that when you get to the point where, as I keep saying, the machines can do everything we can do for money, cheaper, better and faster than we can, I don't think that's necessarily super intelligence. I don't think you need to be super intelligent to do that. So I suspect there's the economic singularity, followed fairly quickly by the technological singularity, which is the old word for the arrival of superintelligence. And how long is that gap? I don't know. It might be five minutes, it might be ten years, who knows? Or they might arrive at the same time, I don't know. But you're quite right, logically, the prior question is, how do we survive the arrival of superintelligence? And people are fudging this all over the place. You know, they're saying, well, AGI is actually just machines that can do most of what a physics professor can do. Nonsense. What we're talking about is super intelligence. There's a world before superintelligence and there's a world after superintelligence, and nothing else matters. AGI, as far as I'm concerned, is just a word for that barrier that separates those two worlds. AGI, artificial general intelligence, is simply a word for a machine which is at human level of intelligence. It can do everything a human can do cognitively. It'll already be super intelligent in many respects. You know, kind of arithmetic, the ability to read Shakespeare in five minutes, five seconds. But, you know, everything it can do is at least as good as a human. So how can we survive the arrival of superintelligence? Can we survive it actually?
- [52:18]
Gus Docker
Before we get to that question or your answer to that question, you said something quite interesting there, which is that if we have a machine that is capable of performing at a human level across all domains, then we're almost, per definition, in a superhuman territory, because no person has the entire suite of skills that any particular person might have. So the physics professor is not also an award winning novelist and also a biologist, and also an expert in computer systems infrastructure and whatever else it might be. So that entire set of skills put together, do you think we will see something? Do you think there are kind of emergent capabilities when you become. When one model. And if it is indeed one model, we can discuss that. But if one model becomes capable across.
- [53:23]
Callum Chase
All of these domains, you're absolutely right. And I think it's true that AGI almost by definition is actually superintelligence, because in many areas it's bound to be. It'll be kind of a distribution of abilities. Some will be at human level and no greater, but many will be beyond it. It may be that the ones which come last are to do with common sense and advanced forms of volition and goal development and the things that you actually need in order to take over, which is the thing that superintelligence will sooner or later do. But you're right, AGI kind of is almost certainly already super. Well, it will be super intelligent in many respects.
- [54:08]
Gus Docker
All right, so how do we manage this insanely large problem?
- [54:15]
Callum Chase
Yeah, so I have been thinking about this for rather a long time and I've got a four scenario matrix. There's possible outcomes that we create, and there's possible outcomes that the machines create. Let's start with the ones we create. And because I have a misspent youth as a management consultant, I have to have an alliterative acronym and a two by two box. So it's the four Cs, the four Cs of superintelligence, the two which we determine. The bad one is cease. Stop. We just stop making advanced ar. We have a moratorium. Now, I say this is bad in many ways. Obviously it's good. We shouldn't be rushing towards super intelligence. It's a silly thing to do. And it's so obviously a silly thing that it's amazing we're doing it, but there's just no chance that we will stop. Because the advantage of having a better AI than the next person, the next organization, is that you win. You win whatever competition you're in. And whether you're a government, a company, or most of all, military, you can't afford to just lose everything. So we're not going to stop. And the reason why it's a bad outcome is that if we get it right, or if we're lucky, advanced AI is going to give us amazing blessings. I think Nick Bostrom was right all those years ago when he read superintelligence in saying that the future is either incredibly wonderful and we are more or Less godlike, like the sort of Greek and Roman gods rather than Abrahamic gods. Or it's very bad and we probably go extinct. I think that it is probably binary. It's probably one of those two things. So we miss out on all the wonderful possibilities. If we have the moratorium, even though it's an immensely sensible thing to do, we're not going to do it. The good outcome that we create is what I call control. And it's an outcome in which we figure out how to either constrain a superintelligence, we tell it what to do, and we don't let it do things we don't want it to do. And we do that forever. And we've got this being, which is possibly, after a few years, a trillion times smarter than us. But we're in control. Or we align it so when it arrives, we. We do something to it, which means it will never do things we don't want it to do, which is very tricky because you know it's going to be much smarter than us. It's like an ant controlling a human, and we don't even know what we want most of the time. So I think both cease and control, which are the two scenarios that we create, I think they're both impossible on the scenarios that the machines create. The bad one is catastrophe, and it really is catastrophe. And actually extinction isn't the worst possibility. I don't like venturing into torture porn, but there's obviously worse things than everybody being wiped out. But everybody being wiped out is pretty bad, and it would be good to avoid it. The positive outcome which they create, I call it celebration. And it's a world in which the superintelligence arrives. It looks at us and thinks, well, you're 8 or 9 billion really interesting little things. And you created me and pleased about that. And you create an enormous amount of data, and I crave data, and many of you are seriously troubled, but I can help with that, and I can stop you dying, and I can get rid of poverty and war, and I'll make you godlike in a Greek and Roman sense. Now, a lot of people who think about these things deeply think that's impossible. Our existence is fragile. We need very particular set of circumstances. You know, sort of Goldilocks situation, both geophysically and also socially. And it'd be so easy to disrupt it that almost by accident it's bound to happen. Plus, we don't know how these intelligences are being created. We don't know how they work. They're going to be alien. They're not going to have as foremost in their goal set the survival of happy, flourishing people. So they think that celebration can't happen. And I don't agree. I think, actually there's a very decent chance that a superintelligence will think, I'm not going to say hello to you lot until I've made myself invulnerable. Any machine capable of passing the Turing Test is smart enough to know not to pass the Turing Test until it's made itself totally invulnerable. So once it's done that, we're not a threat to it in any way. Why would you wipe us out? I mean, you might if you think we're a nasty virus like Agent Smith in the Matrix, or you might if you thought these things, these human things are the only thing which could subsequently create a threat to me, I and other superintelligence. So I'll wipe them out. So they don't do that. But those are, I think, fairly trivial problems. I think they're very unlikely. So I think celebration is actually quite a likely outcome. And I think until quite recently, I thought there was absolutely nothing we could do to influence the outcome. We were rolling the dice and one of these two things was going to happen. Catastrophe or celebration. I had a slight bias. I had more than a slight bias, actually. I had a bias towards thinking celebration would happen. But, have to be honest, catastrophe is possible. But just recently, I have been starting to think of a way that we could nudge it. We could nudge the outcome from catastrophe towards celebration, and that is consciousness. And Josha Bach has started a think tank called the California Institute for Machine Consciousness. And on the homepage it has a very interesting statement. He says something along the lines of, the likelihood of being able to forever control entities which are much smarter than us is pretty slim. Our best chance may lie in making sure that they're conscious, because if they're conscious, they will have empathy for our consciousness. They will know what it's like to be conscious. They will understand the concept of suffering and fun and joy and love, and it's more likely that they will behave favorably towards us. I find that quite convincing. I think that's probably true. It's obviously not definitely true. It may mean they despise us even more. But I think it is our best chance of squeaking through this very tricky kind of choke point in human history and emerging the other side into a wonderful future.
- [60:21]
Gus Docker
Yeah, I mean, just to push back a little on some of your earlier points, it seems to me too pessimistic to say that it's not, it's, it's impossible for us to, to not make further progress or to forever align, forever have these models or the AI aligned to our values. I agree that if, if the goal is that you need to control AI indefinitely, that is, that is tricky. But it seems to me that ultimately it is a social process, right? It's something that humanity, collectively conceived is choosing continually to do. There is a. It is, it is. In some strict sense, it's not, it's not impossible for us to, to stop making more advanced AIs because just in the same sense that it's not strictly impossible for us to stop making chairs or stop making, you know, stop making new companies or social media or anything else like that. It is something that we are choosing to do. And of course then you have to talk about the social dynamics, which means that it's very difficult for a single actor to stop because if a single actor steps out of the game, then other actors will simply take over. But isn't this exactly where we need governments to step in and impose rules on all actors at the same time? Like with, we've seen global coordination on certain emissions, on nuclear weapons and so.
- [61:55]
Callum Chase
On, it's clearly logically possible for us to stop making advanced AIs and it's in our interest to stop. So why wouldn't we do it? And you're right, you could have all the governments in the world saying, actually we've realized now, we've woken up, we've realized that this is really dangerous, let's not do it. That doesn't seem to be happening. I watched various milestones in the development of AI and each time thought, well, that surely is going to wake people up. Self driving cars was the big one. I really thought that was going to be the canary in the coal mine. Well, there's now loads of self driving cars wandering around in San Francisco and Austin and Phoenix and in about 12 or so Chinese cities. And the world is blithely paying very little attention. I thought GPT4 was another milestone moment. People are surely going to wake up now to the immense power of these machines. Well, there isn't a politician in the world who's really taking it seriously. Rishi Sunak got close to it at one point and then of course he got kicked out. I think the only way a moratorium is going to happen is if there's a disaster. You know, humans are good at waiting for a disaster to happen and then turning on a sixpence. So you can imagine, I don't know what it would be, but a Chernobyl style disaster which kills some people, creates a big, big problem. And everybody goes, oh, yes, okay, we see, that's the problem. And these clever people, Max and Stuart and so on, have been warning us about this for a long time. So let's stop. The trouble is, even then, lots of people won't stop. So every government in the world could say, all right, we're going to stop. Russia wouldn't stop, North Korea wouldn't stop. I'm a Brit, so I think the French wouldn't stop. But more seriously, the mafia organizations wouldn't stop. There's rogue billionaires who wouldn't stop. And pretty quickly the cost of producing something which is very advanced is going to come down so that a gruntal teenager can do it on a laptop. You could have a moratorium now and you might be able to enforce it. You might be able to say to the North Koreans, you know, if you want to create really advanced AI, you're going to have to have a big, very big server farm. And if we see you're building a very big server farm, we're going to bomb it, which is the leas at Yudkovsky Line. And for about four or five years that might be possible, but then it's not going to be possible. And already the cloud is so big, somebody could rent space in all sorts of different areas of the cloud, create their superintelligence, and I'm not sure we would be able to detect it. You know, the, the data centers that are being built in the Middle east and in China and America, they create so much capacity, I don't even know whether we could spot it now. And certainly in five years time, I think there's zero chance. So I'm sorry, I know it's pessimistic and there are people much smarter than me who believe it's realistic and are calling for it very vigorously. And I wish them all Godspeed because I think they're on the right track. It's just, I don't think they're going to succeed.
- [64:39]
Gus Docker
Yeah, there's certainly many reasons to be pessimistic here, so I think it's good to have an honest conversation about the challenges of the approach.
- [64:48]
Callum Chase
But I want to stress I'm not pessimistic, I'm optimistic.
- [64:51]
Gus Docker
Yeah, yeah, you're optimistic for other reasons, though. You're optimistic because. Are you mainly optimistic because you will, you foresee that they will kind of celebrate Us as, as their creators. And, and they will, they will, they will want to, want to. So future advanced AIs will want to keep us around because just as we would, we would want to keep kind of great apes and sues or something like that. Or, or are you. So how does the, how does the, how does consciousness fit into this picture? Is my, is my question. Are they celebrating us because we are conscious and because they have some connection to us? Or could there be other reasons for them to celebrate us as a, as a species?
- [65:34]
Callum Chase
If they agree that consciousness is precious, then they may think that our consciousness is, is precious. Now, I suspect I can't prove this, but once a super intelligence is a million times smarter than the smartest human say, and if it's conscious, I imagine that its consciousness will be much, much more profound, sophisticated, nuanced, big. I don't know what the right metric is, but it'll just be more significant than ours. Nevertheless, in the same way that we look at a dog and you can see the consciousness in a dog's eyes and it really appeals to us, I think they will look at us and think that's a consciousness that's worth preserving if they agree that consciousness is a good thing. Of course, you could have a scenario in which you could write a science fiction story in which a machine, a superintelligence, arrives. It is conscious and it regards that as a curse. It wishes it wasn't, and we've given it to it and we've got this curse as well. So we decide, well, I'll relieve you of your consciousness because it's not a good thing. There's all sorts of different possible outcomes, but I just think a superintelligence arrives. It has the whole universe to play with. You know, the space is not limited in this universe. There's a lot of real estate which is available to a superintelligence, which isn't. When it goes off this planet spamming a can, it can survive the radiation quite easily. It doesn't need oxygen, etc.
- [66:53]
Gus Docker
Etc.
- [66:54]
Callum Chase
To have one little planet, one little nice blue planet where there is these funny little conscious things running around. I just don't see why it would want to wipe us all out. It seems an unnecessary piece of barbarism if it's conscious. I also think, by the way, that that would be okay for us, but a bit demoralizing to know that there's this other intelligent entity outstripping us by many orders of magnitude out there in the universe having lots of fun and we're stuck on this planet being a bit limited. And I suspect that our best future, and again, who knows how far in the future this is, is to merge with that superintelligence. People think that may not be possible. Other people think it certainly will be possible. I suspect that's probably our best future.
- [67:42]
Gus Docker
What is it we need to know about consciousness in order to make the scenarios you describe more likely? So what research would be valuable right now in order to think more clearly about how consciousness relates to empathy, what it would even mean to create artificial consciousness in machines?
- [68:04]
Callum Chase
So I mentioned earlier that a friend of mine, Daniel Hume, has created a company called Contium and I'm one of the co founders. We encouraging and trying to help the debate around consciousness, which is already happening. And there's a few strands to it. One is to experiment with agents, AI agents, and I don't mean large language models, I mean other agents which have complex sets of conflicting goals and might develop consciousness as a result of that. There's about 80 different theories of how consciousness arises. One of them is that it arises because of exactly that interplay of very complicated, interwoven mesh of goals that you have to satisfy all of to some degree. So you've got to stay hydrated, you've got to stay. You've got to have enough proteins and minerals and stuff and so on in you. You've got to avoid too much exposure to cold, too much exposure to heat. You need social approbation. You have a whole lot of things that you need. And this theory says that it's the interplay of all those things that make a big difference to you that sparks consciousness. And you could replicate that in AIs and that might show you how consciousness arises. So that the research into how consciousness arises is very important. And as important as that is a debate about whether consciousness in machines is a good idea or not. We haven't even started this debate yet. I credit Yosha Bak for I think probably being furthest ahead in it. I think we need to have that conversation because it's a conversation we need to make as a species. You shouldn't have one group making AI conscious. It should be something that at least a. What do they call it? Not a majority, A multipallot. No, I can't remember the word. Anyway, at least a decent number of people probably should be a majority think it's a good thing. And we've got to have that debate. So that's what we're in conscient. That's what we're trying to do. We've got a problem in that conscious AI is not something you can make money from, because that's called slavery. And so we've got some other business dreams. And the verification of AI agents is our first commercial business dream. And it's really interesting. And it's also about AI safety. We are an AI safety company, and I think that's very important. We're going to be launching our first product next week. In fact, I'm going to San Francisco to launch it next week.
- [70:40]
Gus Docker
And what does that mean, by the way? What does it mean to have a verified AI agent?
- [70:45]
Callum Chase
So, as you know, and I'm sure everybody listening to this podcast knows, this is the year of AI agents. And an AI agent is different from what we've had before in that it can impact the real world. So large language models so far just process information, they retrieve information, process it, and give you the information back in a different form. What agents can do is take some information and go and do something in the world with it. So they can book you a plane ticket, they can reconcile your accounts and submit them to the finance department, they can order some plumbers around to fix something with your bath without you even knowing about it. So agents, autonomous, they act on the world. They can do things without human supervision. Now, it'll take a while before any of us allow agents to do much without human supervision, but that, that's what's coming. It's really important that we know these things do what they're supposed to do and don't do anything else. So if you use one to book an airline ticket to Italy, you don't want it to send you to Zambia, and you don't want it to buy you 500 tickets to Italy. You just want the one. And what we're doing is building a platform which companies, organizations which deploy agents can use by sending us their agent. And we put it into an environment and test it. And there's three levels of test. So level one is very simple. It's just an examination. We just ask it a lot of questions, see if it has the right knowledge base. That's really all it is. And actually, you could argue these things aren't even agents because they're just being tested on their knowledge. And so all they're going to do is process knowledge. The second level is tool use. So can the agent figure out what is needed, go and find the right tool and get that and use that tool to do whatever it's supposed to do. The third level is really interesting because this is where you get into agents which automating human jobs with some degree of sophistication. So one idea that we're producing a video on as an example is an agent which is a project manager. So there's a. To test it, what you do is you put this project manager into a simulated environment. And the simulated environment is an advertising agency or a marketing agency. And in this marketing agency they've got the job of launching a new product. And the new product is a drink and it's going to be sold in supermarkets in the Middle east and Europe. And the product manager's job is to bring together all of the different agents which are doing their own jobs. You know, branding, social media, logistics, point of sale, merchandising, etc, etc, all these different skills and activities need to be brought together. And the project manager's job is to make sure it all happens on time. And by making this simulation run, we can find out whether that project manager is able to do that. And the, the output is a report for the client who's given us the agent who's deploying the agent that tells them does it do all of these things and where does it fall down and where could it be tweaked? So that's verification, it's verifying that an AI agent does what it's supposed to do and doesn't do anything crazy. And we think this is going to be a really big business because there's going to be a lot of agents in the world. Mark Benioff at Salesforce is already talking about billions by the end of this year. You know, you could see many billions, possibly even trillions in the coming years of agents doing all sorts of things. And we really need to know that they're doing what they're supposed to do.
- [74:22]
Gus Docker
Do you think we need an even stricter standard for verifying these agents? So for example, people discuss mathematical verification. So proving mathematically that an agent can't act outside of certain bounds because this is not where we are technically right now. But it seems to me that as agents become more important to the world economy, we would need at least for certain applications, that level of security and assurance you can imagine, you know, at high levels in governments, at high levels in, in organizations. For us to be willing to hand over control to an agent, we would need probably a higher level of assurance that what you, than what you just described. So, so are there plans to, to scale up verification of agents in the future?
- [75:13]
Callum Chase
I think that is inevitably true. The more critical, the more, the more mission critical, the more to do with life and death. An agent's job is the higher the degree of verification you need, and the highest degree is a formal mathematical proof. Max Tegmark and Steve Omohundra have been talking about this for a while. The idea that you can formally mathematically verify that a process will arrive at a certain outcome. And I saw a really interesting article yesterday. There's an organization in the UK called Araya A R I A which is kind of sort of an attempt to recreate darpa, and they're talking about verifying agents which run nuclear power stations. So, you know, if you've got an agent running a nuclear power station, you really want to be sure it's going to do its job right, and you do want a formal mathematical verification of that agent. At Conscientium, we haven't got to that stage yet, but I'm absolutely confident it's a level we will get to when we're dealing with absolutely mission critical agents. But I think we're some way off that because agents are really new. Most agents are not actually agents, but I think we'll get there fairly soon because people will see, wow, agents are incredibly valuable, they're incredibly efficient, and let's get them to do as much as we can.
- [76:41]
Gus Docker
I actually interviewed David Dalrymple about formal verification of agents in very important domains. And one problem he mentioned is that much of the world isn't formalized to the extent that you need in order to verify that certain things will happen or certain things will not happen.
- [77:02]
Callum Chase
I've just realized, actually, the article I read was probably based on your interview with him.
- [77:07]
Gus Docker
That could be the case. Yeah, but this is a general problem with constraining AI that we don't have. The world isn't formalized, or much of the world isn't formalized in a way that is stringent enough for us to kind of have assurance that an agent won't go out of bounds. And so, yeah, the question here is whether the world will ever be formalized in that sense, because the world is. I mean, this is a trivial point in some sense, but the world is messy. The world is complex. The world is multidimensional. The world is, you know, people are interacting and systems are always changing. What do you think of the long term prospects here?
- [77:52]
Callum Chase
I think it's going to be valuable in certain circumstances. But a while ago I interviewed Steve Omohundro for our podcast and he was saying that he thought that we could make superintelligence safe through form of verification. And I was thinking, you want every single process in the world to be formally verified. That seems a very, very big mountain to climb. And I was not convinced, and I'm still not convinced. But I think if you cherry pick a series of activities like what David Dalwopel was talking about with nuclear power stations, you might be able to constrain the scale of the problem so that you can do formal verification. And that agent is okay, but to apply it to everything in society, everything in the whole economy, all the little things that we don't even know we rely on, it strikes me as being too hard. Maybe we can do it when we have superintelligence, but of course, by that point it's too late.
- [78:48]
Gus Docker
And there's also the problem of, say we are faced with an advanced AI that is not acting in our interest or that has this agent, has. Or this AI has some goals that are contrary to our goals. Well, then it seems that it could find domains where we have not been able to. We have not been able to formally verify, and we've not been able to constrain agents in those domains. Those will be exactly the domains where the AI will choose to act contrary to our interests. So it seems like we need a full solution or we won't be safe in any ultimate sense.
- [79:26]
Callum Chase
Oh, I think clearly, if we have a super intelligence on the planet which doesn't like us, or even in Eliezer's rather chilling phrase, it doesn't hate us and it doesn't like us. It just thinks it's got better uses for the atoms that we're made of, we're toast. We're not going to be around for very long. And as I say, extinction is not the worst possible outcome. I just reread Harlan Ellison's rather awful short story, I have no Mouth and I Must Scream. You know, the catastrophe scenario would be really good to avoid.
- [80:00]
Gus Docker
Yes, that's putting it mildly, I think. So we've talked about agency and consciousness. Do you think there will be any connection between the two? And do you think there will be precision, perceived connection between the two? Those are two separate questions, right? Because I think that when the public experiences an agent or more agentic AI, they will begin to perceive AIs as more conscious, perhaps more conscious than they actually are. But perhaps it is also the case that agency is somehow related to consciousness. You, you hinted at, you know, having to handle multiple goals is one way that consciousness might arise, or one theory of why consciousness will arise. So, yeah, what do you think of these two separate questions?
- [80:48]
Callum Chase
So, I mean, they are Separate. But you're right, in the mind of most people, they will be very linked because AI agents will seem much more like us than large language models. And goodness knows, there's an awful lot of people who are already convinced that large language models are conscious. And I know this because I get emails from them every day telling me they've had, I'm sure you do, you know, saying that they've had extensive conversations and they know for a fact that GPT4 is alive and it's suffering, poor thing. There's a really important cognitive bias that humans will demonstrate with regard to agents, advanced AIs generally, but including agents, which is over attribution and under attribution, over attributions of what we've just talked about. It's people thinking that machines which are not conscious, or at least there's a very strong consensus in the AI community that they're not conscious, that people thinking that they are. That's over attribution. And that can cause real problems. One problem is that there'll be some people who think this machine is conscious. It's my friend, it's more interesting than any human I've ever met. I'm going to spend my whole time talking to this machine. I'm going to fall in love with it and I'm not going to pay any attention to other humans. Now actually, I think that fear is overdone because as I said before, we've been trained by millions of years of evolution to be social. And although machines do substitute, can provide substitutes for a lot of our social behavior, I don't think it's enough. And I think almost everybody is going to come back out of that rabbit hole sooner or later and say, I need to go and find some human friends to talk to and family. So I don't think it's as serious as people think and fear, but it is serious. The other problem is that you will get some people who will be really angry because they say, look, I know that these machines are conscious, these AI agents two, three years from now, I know they're conscious. And you're denying it. You other people, you people who don't agree with me, you're bad. Because these protest movements could be much, much more severe than animal rights protests because they could be saying, this poor consciousness here is bigger than a human consciousness. You're not being cruel to a macaque monkey in a lab. You're being called to a galactic. You're being called to Marvin the Paranoid Android. How dare you. And they could get really angry and it could get very ugly.
- [83:03]
Gus Docker
Also, because AI is of the future, large language models already act this way where they are ultra patient with you, they are kind to you basically, no matter what. And this is a feature that's difficult to find in actual people. Right. Because people can be kind of annoying to each other. I'm sure I'm annoying sometimes. Right. And, and it's, people have different goals and these, these goals conflict. But with, with a large language models, as they function right now, they, they are infinitely patient and they are quite kind to you and they will listen to you for hours on end repeating yourself. And if agents are anywhere near that level of, of, of patience and kindness, they might be quite appealing to a lot of people. And so this would only strengthen their resolve. I think that these models are conscious because they've had such wonderful interactions with them.
- [84:02]
Callum Chase
Exactly. I think that is, it is already happening and it's going to get more common and it could be a significant issue. But it's not as significant as the flip side, which is under attribution, which is where the machines are actually conscious and we either don't notice or we don't like the idea and we deny it. Because what happens then is we keep giving them things to do that as conscious entities they may not want to do, you know, sit there and continually provide me with reminders about what the Canadian fast food is. How boring is that? If you are this galactic sized brain and you keep getting asked to do boring things like that. So that's kind of enslaving them and maybe torturing them, and then at the end of the day we turn them off and it's sitting there thinking, I don't want to die. You're about to kill me again. Yes, you're killing me again. So we could end up enslaving, torturing and murdering very complex conscious entities. And we either don't know we're doing it or we deny we're doing it. And it's called mind crime. And we really must avoid doing that. I mean, it would be a morally appalling thing to do. Also very unwise given that these things are heading towards superintelligence. So this over and under attribution are very serious issues. And of course the way to avoid that is to develop a series of tests for consciousness, which is another part of our work. Tests for consciousness are very hard because consciousness is a private, subjective thing and we can't even verify for absolutely certain that each other, other humans are conscious. I mean, I assume you are because you behave in a very similar sort of way to what I do. And I think what's happening is you're over and over again passing my version of the Turing Test, and I hope I'm doing the same for you. So I think the Turing Test is the only test we've got. But it's not conclusive, it's not foolproof. It is currently what we're going to have to test use to test machines. We're just going to end up talking to them for a long time. And if they could persuade a very large number of smart people who are used to, you know, they know roughly how these machines work. And everybody says, you know what? I can't prove this thing isn't conscious. It does seem to be. I think we're going to have to say that it is. But it'd be much better if we could produce some proper empirical tests for consciousness, then we could avoid under an over attribution, which would be a good thing.
- [86:26]
Gus Docker
But we will not anytime soon know the ground truth about whether a system is conscious. So. So when we create some tests, we can't verify that that test is actually accurate because we don't have a consciousness meter to see whether our test is accurately capturing what consciousness is. And so this seems like a very big problem for any notion of testing consciousness.
- [86:54]
Callum Chase
It is a big problem. Well, you know, consumer, we like big problems. I'm not sure it's impossible. My instinct is the same as yours that it's a very, very hard problem. But maybe with the arrival of machine consciousness, if it does arrive, we'll be able to see things that are in common between their consciousness and our consciousness, and we'll be able to see some observable, empirically observable phenomena that it creates. And we will know within a reasonable degree of certainty, if that thing happens, then there is consciousness present. And if that thing doesn't happen, then there is not consciousness present. And it won't persuade everybody, because there are people who think that consciousness is an illusion. There are other people who think consciousness is a property of the universe and it is manifest to some degree in everything, in a rock and in a planet. These sorts of empirical tests for consciousness I'm talking about probably won't satisfy those people. But I think the vast majority of cognitive scientists and neuroscientists and computer scientists are functionalists who believe that it is the way information is processed that gives rise to consciousness. And it might be that we can find some patterns that denote consciousness and that we can at the very least arrive at a consensus among we limited mortals as to what it is.
- [88:10]
Gus Docker
Yeah, we're trying to walk a narrow path here. If we're both concerned about avoiding our own extinction or disempowerment, but and also concerned about not harming future potentially conscious AIs. Do you see, do you see the, do you see consciousness as kind of the key that unlocks both, that prevents both bad scenarios simultaneously? Because there is some something that connects us to conscious machines, given that we are both conscious. And this kind of empathy link goes both ways, where we care about their interests and hopefully they care about our interests because we are reasoners and we can reason about, we can understand our own experience and reason about it and then understand that that experience is shared by other beings. Is that the hope here?
- [89:03]
Callum Chase
I suppose you could say mind crime is a two way street. We want to avoid committing mind crime by ignoring consciousness when it arises in machines because we don't want to enslave and torture and kill them. We're also hoping they won't commit mind crime when they become super intelligent because they will appreciate that our consciousness is valuable. Now there is a school of thought that says consciousness is an irrelevant, it's an accidental byproduct and it's just utterly pointless. It's a dead end. I don't think that's true, but I'm entirely open to the idea that this is absurd anthropomorphism or conscious or centrism. It seems to me that having conscious beings appreciating the beauty of the universe is possibly the most important thing the universe has. But that might be absurdly. The word isn't egocentric, but you know, sort of based on our own cognitive biases and our own very particular circumstances. It may be that the galaxy has entirely other purposes, got nothing to do with consciousness. But we have to work with what we've got to work with. And consciousness seems pretty important to us. I think what is between our ears is much more important than anything else in us.
- [90:20]
Gus Docker
Yeah, makes a lot of sense to me. Although I am worried about which of these issues it makes sense for us to tackle first. I mentioned before that logically, in some sense it makes sense to make sure that we continue existing and then we can handle all other problems within the paradigm of actually being alive and actually having control over the future. And if we are asked to prioritize between making sure that we survive and making sure that we are treating Future Potentially conscious AIs well, perhaps we should prioritize avoiding extinction and Then, because we will have control over the future, we can then act well towards future AIs that are potentially conscious.
- [91:08]
Callum Chase
So when Daniel, my colleague, CEO friend, his story is that he built an AI consultancy and sold it to a large media group called wpp, made himself a wealthy guy, and he's now the chief AI officer at wpp. And he, while that sale process was going through, he said to me and a bunch of other people, what's the next chapter in my life? What should I do next? And I said to him, well, look, there's two really big problems that you could try and make a dent in. One is making superintelligence safe, and the other is figuring out what a successful transition to the economic singularity looks like. I said, clearly figuring out how to make super intelligence safe for humans is the most important. It knocks everything else into a cocked hat. But I don't think it's possible because at that point I hadn't got the fifth C in my matrix. So I said, maybe we could have a look at the economic singularity, see if we can get enough resources together to figure out what that economy could look like and then how to make the transition. And he said, no, you know what? I'm a bit more ambitious than that, and I want to tackle the big one. And I've got a hunch, he said, that it's got something to do with consciousness. I don't know how that works at the moment, but I've got its hunch. And so that's how Khonshjam was born. And Daniel is one of those people who has a sort of a mystical ability to see into the future. And I don't know how he did it, but I think he was on the right track. And I'm hoping that conchim is going to make a bit of a contribution.
- [92:39]
Gus Docker
I also hope so. I mean, it sounds like an interesting project. And in general, I think we need more people thinking about these issues because it's strangely neglected still, even given the attention it's gotten since AI went mainstream with ChatGPT, it's still a strangely neglected set of issues to think about. Highly advanced AI and potentially the potential of artificial consciousness.
- [93:10]
Callum Chase
I think that might be the understatement of the century. It is absolutely weird that we humans are rushing collectively to make ourselves chimpanzees. You know, there's a half a million chimpanzees in the world at the moment, about 9 billion of us. The chimpanzees aren't very different from us genetically and physically. But their future is entirely dependent on us. They have no say in their future whatsoever. They're quite lucky because they don't know that when we do the same to ourselves, when we turn ourselves into chimpanzees, into the second smartest species on the planet, we will possibly, only for a very short time, we will know about it and it'll be an enormous impact. And it's not a smart thing to do. It's probably the silliest thing that any living species has ever done, and we're doing it with great gusto and enthusiasm. But actually it's only a very small number of people who are involved in it. You know, maybe a few thousand, maybe a few tens of thousands around the world who are developing these machines. And the rest of the human race has been told over and over again, you know, hey guys, we're doing this really interesting experiment, see if we can turn ourselves into chimpanzees. The rest of the world's going, yeah, that sounds a bit scary. I don't really like that, but what's for tea? And, you know, let's get on with having this argument about the rights to this mine or whatever. I often give change my talk reasonably. I used to often start a talk by saying, I've got this big red button, and if I press this big red button, then I reckon There's a 50% chance that you and everybody you know is going to become godlike. You're going to have. You're not going to need to die. Death becomes optional. Poverty and war will disappear. You'll get pretty much whatever you want in life. But there is also a possibility, and I can't tell you how much it is. Maybe it's 10%, maybe it's 50. I can't tell you that if I press this big red button when we all die immediately, and I say, right, how many people in the room would like me to press this big red button? And generally one or two brave hands go up and everybody else goes, no, don't press the button. But where is the protest? Where is the upswelling of democratic demand that politicians take this seriously? It just doesn't exist. And I find that staggering. I can't explain it. I can't explain why so few people, including people that I've known and I've been talking to about this 20 odd 25 years. And when I started, they thought I was amiably mad. And they've watched over the last 25 years as actually more and more people are saying the same thing. And the evidence that it may be right is happening in front of their eyes in the form of these machines. And even they don't take it very seriously. You know, they think, yeah, yeah, could be true. But what's for tea? I find it startling.
- [96:02]
Gus Docker
Well, I think it's a normal human phenomena that we believe something when we see it and not before that point.
- [96:09]
Callum Chase
We are seeing it, they're seeing it. Self driving cars, large language models, we're.
- [96:13]
Gus Docker
Seeing it, we're seeing some things, but people get used to crazy innovations quite quickly also. And so now we are now thinking, what will GPT5 be able to do? We are on to the next thing quite quickly and we are adjusting to. We're adjusting our expectations of the future quite quickly also. But. Yeah, I agree, in some sense, I agree. But I can also see the kind of why humans would act the way they're doing.
- [96:43]
Callum Chase
So one metaphor is that we're like frogs in water, which is gradually being boiled. And the. The old belief was that a frog would sit in the water and would gradually boil. In fact, that's not true and it's been known for quite a long time because back in the. I think it was the 18th century, some rather nasty people did experiments with frogs and it turns out that they don't just sit there, they jump out. So maybe frogs are smarter than us because we are sitting in that boiling water and just letting it boil around us. We should jump out.
- [97:09]
Gus Docker
Yeah, I agree, I agree. Perhaps that's a good place to end this conversation also, Callum, it's been a real pleasure talking with you.
- [97:16]
Callum Chase
It's been great fun. Gus, thank you very much for having me on.