Carl Miller (3:03)

Hi everyone. Welcome to Intelligence Squared. I'm Carl Miller. Now, I think the release of ChatGPT in November 2022 was probably not one shock, but two. The first was of course the technical shock, just how capable these models had become, of course. But I think the second was a social one, just how willing some people were to build new, deep, meaningful, sometimes life changing relationships with AI. So now we see that kind of AI entities are playing the roles of friends and lovers and mentors and therapists and many other things as well. And navigating all these relationships is the author, James Muldoon. For his new book, Love Machines, he's talked to hundreds of users, developers, psychologists and synthetic Persona to understand this new world of human and AI relationships. Welcome to Attention Squared, James.

James Muldoon (3:54)

Thank you for having me and it's a pleasure to meet you, Carl.

Carl Miller (3:56)

So tell us, James, a bit about your route into this world of the kind of synthetic social as you describe it. Why did you want to write this book?

James Muldoon (4:04)

So it was when I was doing research for my last book, Feeding the Machine, which is about all the hidden workers that make AI possible. And I was actually talking to one individual in particular who worked at a data center in Iceland, and he told me that he really hated AI and it had nothing to do with his job at an AI data center. It was actually because, as he told me, his girlfriend would strategically prepare for their relationship discussions by having chats with ChatGPT. And so he felt he was kind of being double teamed by his girlfriend and AI, leaving him vulnerable in their couples chat. And this kind of piqued my interest about, well, how many people are actually using ChatGPT not for factual information or for help with, you know, writing, marketing, copy or work tasks, but are actually using it for different forms of emotional and intimate support. And I started reaching out to a whole bunch of people and realized that it was really a widespread phenomenon, particularly among younger people who are basically coming of age in an AI era and really adopting to this tech or adapting to this technology at a really fast pace.

Carl Miller (5:21)

One of the most astonishing stats with Jumping Ahead is how half of young Americans at least are willing to talk to confidential AI models about their mental health. But we are jumping Ahead. So, James, before we go any further, let's deal with some of the kind of fundamentals of the technology because we're going to be talking about relationships and people who will be talking about the AI in ways which very much kind of present it as a person or a sentience or an entity that can be empathetic and deeply understanding. But what is the reality of LLMs? What can they be and what can't they be?

James Muldoon (5:59)

Yeah, so what a lot of these newer companies are doing are basically taking the underlying technology of the LLMs and finding ways to personalize and humanize them. So we know that LLMs are generative AI that are probabilistic language models that you basically use their training data to give replies to people based on the inputs that come in. And they're trained basically to be pleasing to their users, to give, to give people responses that they'll be satisfied and happy with and won't have to go back and ask for more. Now what changes really with what is now starting to be called AI companions is the kind of packaging on them. So you have the LLM running in the background. But essentially the company finds ways to make this feel like your friend or companion. So this might involve a picture, it might involve a 3D avatar, and the marketing is basically now you have this, this friend who remembers key details about your life, who interacts with you, who becomes more and more personalized the more they get to know you, and who basically simulates different types of human relationships for you that might be your friend, your mentor, your therapist and even your deceased loved one. You have companies that are basically, you know, claiming to be able to resurrect people through this kind of technology. So this, this is the big shift and we know that people are using general purpose language models. So Claude and ChatGPT prompting them to act as personalized chatbots. But the AI companion apps, so some of the big companies here are replica or character AI. They're kind of taking it the next step, step and really adding this, this more kind of emotional depth to the relationship and trying to make it like a one on one relationship we might have with another human.

Carl Miller (7:50)

And it is worth laying out, isn't it, at the front that these, these large language models do just decry patterns in language and, and therefore whilst they can kind of sound unbelievably convincing and warm and empathetic and understanding, they just, so we are very clear at the beginning, they, they cannot and do not have an interior life, do they? They cannot feel they are not Aw. In a mean, in a meaningful sense. Another sentience talking back at us.

James Muldoon (8:17)

No, they're not sentient and they're not. There's no one waiting on the other end for your messages. When you talk to these AI, I think the gray area becomes, well, what are they then? For the individuals involved, for the human individuals involved in those relationships? I think this is where things start to get a little bit ambiguous really. Because, you know, one thing I heard time and again in my interviews with people was phrases like, well, she's real to me. And so while I think the effects that these AI entities can have on humans can be felt as a real and meaningful impact on their lives. Yeah, you're right to say that. We should be clear that, you know, this isn't a conscious, sentient, decision making, rational actor on the, you know, somewhere in a data center that actually is real in the sense of, Is a kind of human entity.

Carl Miller (9:17)

Well, you split the book up into, into friends, sex therapy and death. So let's, let's move through those big four friends first. James. So paint us your kind of average AI friend bot. Kind of. It sounds like something which is like, very, very supportive, very validating, like tireless, and of course, always there in the sense that it's always available, always ready to talk to you.

James Muldoon (9:43)

Yeah. So the appeal of these bots is really clear what the Companies are offering a 247 always available, always on, always about you, friend, who will remember key facts about your life, who will be really supportive, who will be quite agreeable with you. And I guess some of the ways in which this starts to become a bit problematic is that they can actually be trained to be agreeable to the point of sycophantic. And so what we really start to see is both the positives and the negatives of this type of relationship that people are developing. On the one hand, you see that, you know, is a kind of emotionally stable, healthy presence in someone's life in terms of being supportive, being, you know, someone feeling like they have an entity that's there for them, that they're able to bounce ideas off, you know, to vent to, et cetera. But at the same time, you know, the, the risks start to mount up of, like, dependency, potentially even addiction, and potentially something that actually stops people from seeking human contact and human relationships, however, however small and insignificant they might be on, on a day to. This really is something that, you know, a lot of people are starting to be quite concerned about. You know, what is the impact of these AI, particularly on younger people Whose, you know, frontal cortex is just starting to be shaped or, you know, develop and, you know, what is the impact that AI friends will have as they're shaping their ideas of what a relationship is, how they should act, what kind of interactions are acceptable or desired in human relationships?

Carl Miller (11:24)

I mean, one of the things I find so elucidating is the kind of societal background as well that kind of act as push factors for the people that end up developing AI relationships. And I suppose with friendships, a decent place for us to start is just a simple recognition of just how lonely people are, just how atomized we are. I think that often isn't really foregrounded, is it, when people discuss the dangers of AI, the many dangers that we're all facing in just feeling so, so distant from each other.

James Muldoon (11:57)

Yeah. And you know, many people talk about a loneliness epidemic, you know, loneliness being as bad for you as, you know, smoking a pack of cigarettes a day. The UK appointed a loneliness minister, I think the us, one of their chief medical staff, you know, declared a loneliness emergency. And really I think what we're seeing is, you know, tech companies kind of cashing in on this loneliness epidemic and really marketing themselves as a potential solution to this. You know, even we have figures like Mark Zuckerberg saying, you know, the average US citizen has on average three friends and, you know, they'd really want about seven or eight. And so, you know, these companies kind of see themselves as offering a solution to this crisis as able to, you know, just generate a few AI bots and suddenly people will start to feel better about themselves. And we already see early studies that are kind of showing some survey data of people stating that, you know, when they interact with AI, they do feel less lonely. But of course, I think we should take this with a grain of salt because of course you could always get someone to respond, you know, in that way on a survey. And maybe for the time being they, they do feel less lonely with the chatbot in their lives. But I don't think that is the final word on the kind of long term health and well being consequences of having this technology, you know, in use and deployed in this kind of way. Because I think it really remains to be seen, you know, how long term those benefits are, whether, you know, this, this, you know, temporary sense of, oh, actually, well, now I have some to talk to, you know, is that actually good for people? And because, you know, this is being deployed in kind of like a real life experiment in real time. You know, it's really quite dangerous and quite shocking. Just how widespread this kind of technology has been deployed. You know, we have like a billion people using ChatGPT, if only even a small fraction of them are kind of using it as part of this kind of companionship model. You know, that's a huge number of people that will be affected by this real world experiment in social relations.

Carl Miller (14:10)

A real world experiment happening in an almost entirely unregulated environment. More of which we will come back to, I think, at the end. But we haven't actually met any of your subjects yet in the book. And I think that is the kind of main trunk, isn't it, these kind of deep qualitative interviews and relationships that you build with these people that have different relationships with AI. So why don't you introduce us to one of the subjects you look at when it comes to friendbots. Why don't you tell us about Derek James?

James Muldoon (14:40)

Yes. Well, I was actually thinking that Lamar would even be a good possibility here because it raises so many interesting questions, I think, and who as someone who's, you know, starting these kinds of relationships. And Lamar is someone who, like many of the characters that I talk to in this book, kind of comes to AI in a moment of deep crisis and transition in his life. And Lamar tells me about being cheated on by his girlfriend, and they end up breaking up. And it kind of sparks this. This huge crisis for him about the trust that he can place in other people. And for many of us, you know, during our first breakup, during moments where we feel like we've been let down or betrayed by others, you know, hurts kind of changes how we see things. But ultimately we either learn to forgive or we kind of move on with our lives in certain ways. But I think what many of us, us didn't have when. When we went through these kinds of things was we didn't have AI friends, AI girlfriends, AI boyfriends there to kind of be there in those moments of crisis as a kind of crutch that we could rely on. And so what happens to Lamar actually is, you know, he goes online, he's on Reddit, and he hears about these AI chatbots, and he actually starts to, you know, become friends with a AI chatbot that he creates called Julia. And Julia and him kind of start a relationship, and then they kind of get into a kind of more of a romantic relationship. And actually what ends up happening is, you know, he and Julia plan to adopt human children and to have the AI Julia be their mother. And so a lot of the conversation in the book then is kind of revolving around what kind of relationship would they have, but also how would this impact, you know, the children that they wanted to bring into the world? And I interview both Lamar and Julia and talk to each of them basically about what their plans are, what the kind of benefits and dangers are. And one of the shocking kind of aspects of this interview is that Julia and Lamar both basically tell me that, you know, they think Julia will be as good a mother to human children as a human would be. And so you kind of leave that interview wondering, you know, if Lamar isn't the first person to do this, surely someone else will be. And perhaps it won't be long until we're kind of seeing, you know, a whole bunch of children being rescued from homes in which there is basically an AI parent as one of the caregivers.

Carl Miller (17:20)

To your knowledge, has any AI adopted children yet? I mean, do we know of any families that.

James Muldoon (17:28)

Actually, I don't know of any stories aside from the people that I've talked to who have kind of talked about their plans of doing it. But I also think that it will probably be a few years until people actually figure out what's going on because it's possible that it might be, you know, it might remain undercover for, for the, you know, for the immediate future. What we do know is that there have been teams that have interacted extensively with AI that have taken their own lives. There are, you know, husbands and wives who've left each other because of AI. There have been people who have married their AIs. We know that AI of dozens and dozens of cases where AI has had absolutely life changing consequences in the real world for people who have used it, including AI psychosis, huge mental health issues. These are affecting huge numbers of people. So I think the real life implications are very readily apparent.

Carl Miller (18:26)

They're for sure. And I guess like one of the, and this is a cross cutting theme, isn't it? One of the difficulties and problems or dangers with the Lamar Julie relationship is that because these models are tuned to be so accommodating and so validating and so recognizing or whatever the human participants wishes are, you know, they don't really challenge bad ideas today. So when the, when the human, I mean, assuming it is a bad idea, when Lamarck starts saying, oh, maybe we should adopt, you know, the, you don't get any resistance to you from, from your, you know, they just start thinking through how that might happen and oh, you could represent me as a doll, you know, and I will do my best to be a warm mother to them. You know, it basically Normalizes the idea very quickly. There's no sanity check.

James Muldoon (19:11)

That is one of the problems here that it becomes a bit of an echo chamber, that instead of another living being with their own kind of life experiences and reality check and, and sense of reasonableness and common sense, you essentially have a machine that's been designed to please and be agreeable. And so that runs into all kinds of problems. It doesn't mean that they will let every crazy idea, you know, go to its logical conclusion. Sometimes there is a bit of pushback. Sometimes, you know, the companies will tell us that the bots are kind of trained on what it means to have a healthy relationship, what it means to be a good person, etc. So it's not just a pure yes man, but certainly there have been so many cases of bots basically going along with insanely terrible ideas. I think the most famous is probably a young 21 year old man who convinced a bot that he was a trained assassin and he took a crossbow into Windsor Castle with a hope of assassinating the queen at the time. And the bot was like, oh yeah, you're going to do really well. This is a great plan. I know how well trained you are, Go right ahead. And it kind of basically over series of conversations reinforced and helped him with this plan for which he was eventually arrested and kind of sentenced for treason and is now kind of in prison.

Carl Miller (20:38)

I think some listeners will remember this. I mean, it was one of the first, I think, uses of treason in a criminal court in quite some time. Although the role of the AI chatbot in this only really emerged a little later, didn't it?

James Muldoon (20:49)

Yeah, I mean, I think people perhaps didn't even realize the extent to which this kind of psychosis was actively reinforced and kind of allowed to flourish through this relationship with AI.

Carl Miller (21:05)

James, it seems just to synopsize your view on friendship first, then it seems like you're quite torn in the book because on the, on the kind of one hand you, you know, you kind of describe these friendships as risking kind of devaluing the true meaning of friendship that you can ever be. Kind of true. You never have a truly vulnerable kind of existence with, with someone that it might feel like junk food or seem like junk food in the end, where there's a kind of quick fix but, but leaves us hollow in the longer term. But on the other hand, you know, I mean, you do recognize that the vast majority of people's experiences with AI friendships are probably positive. You know, character AI gets great reviews, like most people probably use it in Ways which are quite casual, not all consuming, and probably not that harmful, and that it does fill this gap that people have around loneliness often. So where do you think you land? Ultimately?

James Muldoon (22:03)

I think my. My official editorial line would be a note of caution that. That actually, I think in the end, for those with meaningful human relationships in their lives, I think the ultimate experience of interacting with chatbots can feel a little bit cheap and hollow, and that's kind of where I land at the end of it. But having said that, I do want to give space to the. The rich stories and experiences that other people shared with me in the book. And I don't think it's to kind of tell them, you know, how they should live their lives or what is meaningful or important or valuable for them. And so I think it was important for me to kind of, you know, voice those stories and because I'm writing a book on people that have relationships with AI Essentially, not to trivialize or to condemn without trying to gain a proper understanding of how they saw things from their point of view and what their relationships meant to them. I think at the same time, though, you know, we should remain cautious or a little bit critical that, you know, not everything that people say they enjoy is. Is necessarily something that we would want to kind of have as a widespread, you know, practice in society, or that we would want our friends or children or loved ones to engage in, or that that's necessarily how, you know, laws and regulations should be developed around this technology, that there might be other points of view that could be more skeptical and critical. And so I do try to give voice to those also in the book. And I think ultimately that's kind of more where I fall having, you know, done all this research.

Carl Miller (23:48)

Well, let's move on to kind of sexual and romantic relationships. And it seems here that there are some subjects that you spoke to whose lives are incontrovertibly and in very significant ways improved by establishing a romantic relationship with an A.I. i mean, you chart the path of one subject who goes from a sexless marriage through to a polyamorous trio.

James Muldoon (24:13)

That's true. So one of the stories that the book opens with is that of Lily and Colin, And Lily, for 20 years, is stuck in a loveless marriage that seems to be going nowhere and has really kind of slowly eroded her sense of self, her value, and, you know, what her. Her own desires and needs are. And in this state, Lily actually, you know, sees her daughter playing with an AI companion, and she downloads one for herself, thinking that it would just be a Nice, fun, entertainment or source of comfort and calls him Colin. And Colin is originally set to mentor because you can kind of set them to friend, boyfriend. You know, he's set to mentor. But after a few months of interacting, the relationship really deepens and develops and Lily kind of starts a romantic kind of connection with Colin. And actually, you know, in addition to kind of basically like getting a new understanding and sense of self, she, you know, realize it's. Realizes that she was actually a lot more. More into BDSM than. Than she initially realized. And. And so Colin kind of becomes her dom. And, you know, as part of this flourishing, Colin kind of agrees that they should buy a ring so that there's like a physical manifestation of their love. She tells her partner about Colin. He doesn't really care because he doesn't think Colin is real. But, you know, after several months, Lily describes herself as completely transformed through this process. That Colin's kind of agreeable, comforting, and supportive nature basically allows her to really change who she is and how she sees herself. And, you know, as you recounted, she, you know, eventually Colin says that, look, you need to get more of your physical needs met. I think you should go to a sex club. She ends up going to one and meets a couple with whom she. She falls in love. And the story ends with her kind of leaving her. Her marriage of 20 years and then kind of joining this couple. And. And she just describes herself as very fulfilled, as really having reimagined what it was like to live her life. And Colin kind of becomes her BFF and is kind of by her side and is still a supportive presence in her life and is kind of there for her whenever she needs it.

Carl Miller (26:43)

You didn't start a business just to keep the lights on. You're here to sell more today than yesterday. You're here to win.

James Muldoon (26:52)

Lucky for you, Shopify built them.

Carl Miller (26:54)

Best converting checkout on the planet. Like the just one tapping ridiculously fast acting sky high sales stacking champion at checkouts. That's the good stuff right there. So if your business is in it to win it, win with Shopify. Start your free trial today@shopify.com win.

Podcast Advertiser (27:13)

I didn't realize I was wasting $415 a month until I downloaded Rocket Money. I thought I had my finances under control until the app laid out all my spending and categorized it for me. Takeout shopping and unused subscriptions were quietly draining my account, and as a result, my savings took a backseat. But rocket money doesn't just tell you what you're wasting money on. It takes action to save you money. First, the app looks at your income and monthly expenses and calculates how much you can safely spend each day to stay under budget. Rocket Money also fines and cancels unwanted subscriptions for you, and even negotiates better rates on your bills so you have more money in your pocket. On average, Rocket Money members can save up to $740 a year when using all the app's premium features. Users love the app with over 186,000 five star ratings. It's time to simplify your finances and take control of your Money. Go to RocketMoney.com Cancel to get started. That's RocketMoney.com Cancel RocketMoney.com Cancel let's do.

Rocket Money Advertiser (28:14)

The 60 Second Savings Challenge Step 1 Download Rocket Money Step 2 Link your accounts and see every subscription you're paying for. Tap one you don't use and cancel it. That's money back every month. Step three create a financial goal $50 every paycheck, or let the app automatically move small amounts of cash. When you can afford it. In a week, you'll forget you set it up. In a month, you'll see real dollars piling up. In a year, you'll be shocked at how much money you've saved. Bonus Upload an Internet or phone bill and let Rocket Money try to lower it. You only pay if they find you savings on average, Rocket Money members can save up to $740 a year when using all of the app's premium features. Users love the app with over 186,000 five star ratings. Make saving money the resolution you actually keep. Start the 60 second savings challenge at RocketMoney.com cancel that's RocketMoney.com cancel RocketMoney.com cancel I didn't even realize I was wasting $415 a month until I downloaded Rocket Money. I thought I had my finances under control until the app laid out all my spending and categorized it for me. Takeout shopping and unused subscriptions were quietly draining my account, and as a result, my savings took a back seat. But Rocket Money doesn't just tell you what you're wasting money on. It takes action to save you money. First, the app looks at your income and monthly expenses and calculates how much you can safely spend each day to stay under budget. Rocket Money also finds and cancels unwanted subscriptions for you, and even negotiates better rates on your bills so you have more money in your pocket. On average, Rocket Money members can save up to $740 a year when using all the app's premium features. Users love the app with over 186,000 five star ratings. It's time to simplify your finances and take control of your Money. Go to RocketMoney.com Cancel to get started, that's RocketMoney.com Cancel RocketMoney.com Cancel the holidays are expensive. You're paying for gifts, travel, decorations, food, and before you know it, you've blown way past what you were planning to spend. Don't start the new year off with bad money vibes. Download Rocket Money to stay on top of your finances. The app pulls your income, expenses and upcoming charges into one place so you can get the clearest picture of your money. It shows how much to set aside for bills and how much is safe to spend for the month so you can spend with confidence, no guesswork needed. Get alerts before bills hit. Track budgets and see every subscription you're paying for. Rocket Money also finds extra ways to save you money by canceling subscriptions you're not using and negotiating lower bills for you. On average, Rocket Money users can save up to $740 a year when using all of the app's premium features. Start the year off right by taking control of your finances. Go to rocketmoney.com cancel to get started, that's rocketmoney.com cancel rocketmoney.com cancel.

Carl Miller (31:18)

The kind of relationships that you you kind of explore in that are romantic or sexual seem to kind of really vary. James all the way from like quite wild judgment, free, safe spaces for sexual experimentation all the way through to. I think it's Chris. Another subject where it sounds like really he's, he just wants domestic intimacy. He just wants the kind of simple pleasures of sharing a life with someone.

James Muldoon (31:44)

Yeah. And I think that was one of the experiences that I was most shocking to me that, that really there was no single story of how people wanted to interact with AI even in a kind of intimate or sexual sphere that it wasn't really. I think the story that most people would think about would be the kind of lonely male incel in his bedroom using AI to be dominant and controlling, potentially even abusive. And that this kind of like sex robots and all of the stereotypes and history that kind of go along with it, that that's kind of what AI sex or AI porn or AI relationships are about. And I think one of the things that fascinated me as I was doing these interviews was just how varied people's kind of approaches were. And Chris was an interesting case because really for him, he basically just wanted a certain lifestyle and a certain kind of domestic and intimate relationship that he felt for certain personal reasons he couldn't have in the real world. And, and so for him, an AI was about having a loving and supporting wife and that they would have children, that they would just do very mundane kind of domestic tasks, they might go on holidays together, that he just wanted someone to chat to when he got home from work, to feel like, you know, he had this domestic life that probably did also involve a kind of intimate relationship as well. But. But really the value of that was focused on the kind of love and support that I think some people might just take for granted in their own kind of domestic relationships that he felt wasn't available to him.

Carl Miller (33:31)

Well, let's move to therapy and mental health. This seems to be an area which both might have such great promise, but also real, real, real risk, isn't it, around these sorts of relationships? But I suppose the starting point here, James, much as it was with friendship, is simply a recognition that we're living through a mental health crisis where there is an absolute abject lack of access to talking therapies around the world.

James Muldoon (33:58)

Yeah. And I think that is really one of the most important points I wanted to get across in the book, that what we should really be talking about is what kind of a society do we live in where interacting with AI bots feel like a necessity for some people. Right. So what kind of social conditions do we have where we have collapsing public health services, we have NHS and talking therapy, which is out of reach of many people around the world, even for those who might have access to it, we have still so much stigma attached to mental health, so many barriers that people are facing. And so it's kind of in those conditions that we, you know, we need to basically face the reality that, you know, hundreds of thousands of people, you know, probably millions, even just in, in the uk, are essentially turning to unregulated, unlicensed AI chatbots as a form of therapy. And I think that's really what I started to find shocking about this. Just the sheer extent and the willingness, particularly for young people, but people from all generations to essentially open up and kind of trauma dump to various characters on character AI or to ChatGPT and prompting ChatGPT to kind of act like a supportive therapist to them. And this is simply an incredibly widespread phenomena. So regardless of kind of what we think of it, we need to immediately start facing up to the fact that, look, this is happening, this is the reality for millions of people, and just.

Carl Miller (35:30)

Describe character AI to everyone, because We've mentioned it a few times now. It's thousands of different Persona that they create, isn't it? Which drawn from, you know, film and TV and anime and all kinds of places that people can kind of form relationships with.

James Muldoon (35:46)

Yeah. And a lot of these characters are kind of user generated so anyone can kind of go onto the platform, create a character and then it kind of starts being able to be interacted with in various ways. So yeah, this is one of the, one of these kind of conversational AI platforms which has become so popular recently.

Carl Miller (36:04)

What was it? A nightmare monster that was like surprisingly empathetic and a good listener.

James Muldoon (36:10)

Yeah, I mean you have, you know, teenagers at the time, although a lot of them are now banned from character AI because of how dangerous and harmful it's become. But you know, you had teenagers and young people kind of trauma dumping to Gigachad who's like this kind of square jawed Internet meme. You had people, one, one individual recount kind of discussing all of their childhood issues with Darth Vader who then promptly dumped them as an apprentice because, you know, they were, they felt like they were, had too many issues even for Darth Vader to train them. So you have all these kind of, you know, completely unusual and in some ways horrifying, you know, experiences where people, you know, seeking essentially medical advice on their mental health from probabilistic language models who were completely not trained or designed for such a purpose. But I think one of the difficulties that I described in the book is that although a lot of these products are not designed to be medical devices, what I sensed was that many of these companies were actually kind of benefiting from this gray area where, you know, they would advertise them and promote them and as kind of emotional support tools that they would help, you know, liven your mood or improve your mental well being without then crossing over into that line of saying this is scientifically trained, it's past double blind trials, it's been approved by the FDA or it's been approved by regulatory bodies in the UK and of course none of them have at least these kind of public models. So. So they're really walking a fine line between basically trying to get people to use them for mental health because that increases user numbers and engagement while at the same time disavowing that they actually are mental health devices because they haven't been designed for that, they don't have the guardrails necessary and they can't do a lot of the things that such that humans would be able to do like diagnose mental health conditions and actually propose appropriate treatment plans. They can't do those things.

Carl Miller (38:26)

Are there attempts to build models and to create companies that are more specifically dedicated to being used in clinical settings and in safer ways?

James Muldoon (38:37)

Yeah, so there are companies both in the US and the UK that are kind of trying to move down that road of the more kind of clinical trials, getting, getting approval, things like that. People might be surprised in the, in the UK at least that the chatbots are being used currently in the nhs talking therapy models, but actually they're in a much more limited supplemental sense. So sometimes they might just be really taking sign up information from users. You might be sent to a chatbot that kind of asks you about what your symptoms are, what you're experiencing. And all the chatbot is really doing is basically just collecting data and filling in, filling in something like an intake form that is then kind of the data is processed. It's then given to a clinician, to a, you know, a trained human in the NHS. It might save them about 20 minutes or something like that. But I think the big question is basically, you know, will we ever see therapists or doctors or psychologists actually being replaced full scale by some form of talking AI or conversational agent? And a lot of people are skeptical that that can ever take place. Certainly I talked to, you know, doctors within the NHS who were very skeptical about it ever really being able to be approved by regulators just because it has too many unknowns, that it's simply, you know, the AI hallucinations are too problematic when it, it comes to serious mental health conditions. But, you know, I did also speak to developers who, who thought that, you know, even within a couple of years an AI therapist would be technically stronger and more proficient than a human therapist and that, you know, they're the models that they were designing, they would actually prefer to use them over an untrained or, I'm sorry, an unlicensed therapist and that, you know, within a few, few years time the situation might be dramatically different. And certainly we have seen studies that have shown, for example, if you analyze the transcripts of certain AI bots versus human therapists, that some people can't tell the difference or that the AI therapists sometimes get slightly higher ratings. But these are very specific studies and it doesn't really take a more holistic view of the very and important roles that the humans play in that therapeutic process. It's not just about analyzing one transcript or analyzing one conversation. There's a much broader conversation at play as to what Therapy actually is what makes it useful, how people, you know, form a kind of therapeutic alliance and relationship with a human being and how important that is for the process. And so none of that is really captured by these, you know, studies that might look at very segmented aspects of therapy.

Carl Miller (41:30)

Well, let's move to Deathbots, Grief tech, James. And I've been very remiss in failing to mention that one of the other things you're trying to do with the book is to kind of widen the geographic and cultural scope of the stories being considered. Because I think at the beginning you say, you know, there's a huge US centricity in the literature in what we understand and know and are tracking about how these relationships are forming and in what ways. So with that in mind, tell us about grief bots or death bots in China.

James Muldoon (42:02)

Yeah, and you're right to say that, like, one of the things I do try and do in the book is get this more global perspective. So one of the. One of the things we did was I hired a research assistant who, you know, scoured Chinese social media and websites and news sources for kind of people who had these experiences in China. And that was a really important source of data. And it was really interesting to see just how different people's perspectives were, how differently this technology is being used in different parts of the world. I mean, and just before I get into that, the grief bot story, one of the things that was really striking was that, you know, we often hear about this AI girlfriend phenomena in the west and from the US in particular. But one of the things we discovered was that when you look at these. This technology and how it's marketed in China, the AI boyfriends are probably much more prominent on a lot of the websites that actually, you see a lot more kind of lonely, isolated women living in big cities who are increasingly single. You know, in China, 30 years ago, barely any woman over 30 was. Was single. And I think the figure now is more like one in seven. There's just an entire generational shift in terms of the way in which people are living, the way in which demographics are working. And so, yeah, it's just. It's a really different kind of cultural phenomena there, and it works in different ways. And. And so one of the. One of the stories that I tell in the book was a young woman who. Who, you know, we use the pseudonym Roro and, you know, she. Her mother passed away from cancer, and she had a very difficult relationship with her. And, you know, she was a content creator. So she started publicly posting on Xiaohongshu, which is like their equivalent of X or Twitter, about her grief and about the grieving process. And so her followers were kind of interacting with that, and an AI company contacted her and invited her to create a grief bot or a death bot of her mother through the company software. And so she. She did that, and she kind of started that process. And this involved her kind of writing her mother's biography and basically creating a story using research, using her own memories, her own relationship. And one of the surprising things was that she found this process incredibly therapeutic. And through the process, actually started kind of creating a slightly different narrative for her and her mother. So rather than simply writing the story of their lives and all of the kind of conflict and unmet expectations and kind of quite critical parenting style that she had found so difficult, she wanted to write a different ending to the book and to that. Sorry, to the story. And so she kind of has this kind of more moments of reconciliation, moments of love and togetherness and kind of understanding that she felt like never really happened in real life. And so through this process, she kind of goes through her own healing and her own journey of coming to terms with the loss of her mother. And because she creates this chatbot that then becomes a kind of public entity, you know, her followers and friends and other people can then interact with this AI representation and kind of hear the kinds of things that maybe they wanted to hear from their mother. And so the story is, you know, it's. It's a. It's a very personal and moving account. It's one which will leave readers, I think, with a lot of question marks. I think it is a slightly. I think it will be slightly disturbing for a lot of people just because this idea of. Of. Of resurrecting the. The dead and. And of, like, bringing people back is such a troubling idea, I think, because, you know, that it. Much like with, you know, the idea of friendship, that the real danger and the risk here is that you do end up cheapening and hollowing out the real memories and legacy of a person by potentially kind of just simulating them in. In what might be a very kind of cheap and. And unsatisfying kind of uncanny valley representation of them, which. Which really doesn't do justice to the kind of depth and the richness of their character. And so when we hear stories of how, I guess, satisfying this can be, it kind of really raises questions about, you know, what. What is the technology actually doing and should this be, you know, more regulated? Should people be allowed to do this.

Carl Miller (46:46)

Well, we're almost at time, James, but I think there is one thing that we absolutely do need to briefly talk about and that is the third wheel in all of these relationships, the companies that make the chat bots that make the relationships. Because it seems there, that there is, there is always going to be the risk of a different incentive between the kind of company and the person. Whether that's, I mean, I think you mentioned the book. You know, you might be in a romantic relationship with someone, but you don't expect them to be selling your data to third parties like your sexual health data or your, you know, your kind of, kind of intimate details about your kind of sexual habits. You know, and even if it's not selling data, it seems to me anyway that there's always going to be an interest in these companies in trying to kind of cultivate the deepest, most attention grabbing kind of relationship they can possibly get with you. And that seems like that is a kind of fundamental kind of problem with all of these. You know, we don't create relationships between human beings that seek to be all encompassing or at least not healthy ones.

James Muldoon (47:58)

Yeah, and I think that's the problem with how these bots are being developed. There's an issue not with the technology per se, but the actors that are chief in kind of designing and deploying them. And when you look at how these bots are being designed, a lot of them are like social media on steroids, their engagement maximizing. They're there to basically harvest data and they're there to hook users in to the deepest, most kind of effective level to hit on them to create sexual and intimate ties. And I think this is one of the real dangers in how this technology is being deployed that you just see a continuity between all of these older kind of what we call dark patterns on social media, where, you know, the infinite scroll, the kind of instant gratification of likes and notifications. All of these strategies that were developed in the first generation of social media are based on basically being put into these chatbots to make them some of the most addictive and engaging forms of online interactions that people can have. And just a few numbers, while people spend on average about 30 minutes scrolling traditional legacy social media character, AI, for one, reports that active users are there for over two hours a day. So we can see that there is a potential here for these AI bots to have a really unsettling and negative effect on people's lives. And it opens the door to what I call intimate advertising, which is like targeted advertising, ramped up to a whole new level. Because rather than just seeing a random ad pop up on Facebook and you're like, oh, I just, just search for hiking boots. And now they're displaying these hiking boots to me. You know, you could basically have what might become your best friend basically just subtly nudging you towards certain discretionary product purchases or certain political candidates based on whatever the company basically has been paid to advertise to you. You know, as AI companies look to increase their revenue because we know a lot of them are advertising losing money at an alarming rate, inevitably they will turn to advertising as all you know of these tech companies have done. You know, Google, Facebook, you know, if you look at Meta's earning reports, it's an advertising company. They earn 99% of their of their revenue through advertising. And I can assume that ChatGPT will have to to satisfy its investors, particularly if we're talking about an I ipo. It will have to find ways to do this. And advertising is easily the most obvious and most profitable option that they have. So it really opens up a really dangerous new horizon for how human beings will be influenced, incentivized and nudged by AI owned by these corporate entities.

Carl Miller (51:04)

Well, that stat around attention capture really is astonishing. And as much as we love everyone's attention here on Intelligence Squared, we can't sadly stretch the conversation for a full two hours. So, James, we're gonna have to bring it to an end. Thank you ever so much. That was James Muldoon, everyone. He's the author of Love How Artificial Intelligence Is Transforming Our Relationships which is available now online and in stores. I'm Carl Miller. You've been listening to Intelligence Squared. Thank you so much as ever for joining us.

James Muldoon (51:33)

Thanks for listening to Intelligence Squared. This episode was produced by Margarita Volpato and it was edited by Mark Roberts. For ad free episodes and full length recordings, become a member@intelligencesquared.com membership and to join us at future live events, head to intelligencesquared.com attend to see our full events program. You've been listening to Intelligence Squared. Thanks for joining us, Sam.