Loading summary
Transcript187 lines
- [00:01]
Dave Ahern
Location the lab.
- [00:02]
Andrew Sather
Quentin only has 24 hours to sell his car. Is that even possible? He goes to Carvana.com what is this, a movie trailer? He ignores the doubters, enters his license plate.
- [00:14]
Derek Barris
Wow, that's a great offer.
- [00:16]
Andrew Sather
The car is sold, but will Carvana pick it up in time for it?
- [00:19]
Derek Barris
They'll literally pick it up tomorrow morning. Done with the dramatics.
- [00:23]
Dave Ahern
Car selling in record time.
- [00:26]
Derek Barris
Save your time.
- [00:26]
Andrew Sather
Go to Carvana.com and sell your car today.
- [00:29]
Derek Barris
Pickup fees may apply.
- [00:31]
Dave Ahern
Overwhelmed by Investing if you're anything like.
- [00:34]
Andrew Sather
Us, the hardest part is getting started.
- [00:36]
Dave Ahern
That's why we created the Investing for Beginners podcast. Our goal is to help simplify money so it can work for you. We invite guests to demystify investing.
- [00:45]
Derek Barris
At least try to be setting aside.
- [00:47]
Dave Ahern
Like the minimum 10 into the 401k. I'll teach you the basics of the market.
- [00:52]
Andrew Sather
Yeah, I think compound interest should be.
- [00:54]
Dave Ahern
At the start of any discussion about investing. And we've had investment professionals who teach in a simple way a valuation driven bear market.
- [01:04]
Andrew Sather
You know, we we haven't really seen yet and I think everyone's thinking about it, but we haven't really seen yet.
- [01:10]
Dave Ahern
Our Q and A episodes feature questions from listeners just like you. So what do you think about the.
- [01:15]
Andrew Sather
Situation with etbi, which is Activision?
- [01:18]
Dave Ahern
I'm Dave Ahern. And I'm Andrew Sather and we hope.
- [01:22]
Andrew Sather
You join us on the Investing for Beginners podcast.
- [01:25]
Dave Ahern
On the Investing for Beginners podcast. Hey everyone. Welcome to Conspirituality, where we investigate the intersections of conspiracy theories and spiritual influence to uncover cults, pseudoscience and authoritarian extremism. I'm Derek Barris.
- [01:52]
Derek Barris
I'm Matthew Remsky.
- [01:53]
Andrew Sather
I'm Julian Walker.
- [01:54]
Dave Ahern
You can find us on Instagram and threads. Conspiracy Spirituality Pod. We are also all individually on Blue Sky. You can look for our names there and you can access all of our episodes ad free, plus our Monday bonus episodes on patreon@patreon.com conspirituality if you use Apple Podcasts, you can subscribe to our Monday bonus episodes via that platform. As independent media creators, we really appreciate your support.
- [02:30]
Derek Barris
Episode 257 AI Gurus the chatbot flashes its ellipses at the bottom of the screen. What is it thinking? Is it thinking? What does it want from you? What do you want from it? Beneath the pixels lies a sea of mined data and lightning storms of electricity heating up servers in barren deserts. What will it find for you in the past? Labor of the generations. According to a stunning new article In Rolling Stone it will find whatever the fuck makes you feel like a God, including all the new age pablum it has scarfed down because, oops. ChatGPT released a model that is just too sycophantic. But as we break down today, the AI non sentient flattery machine is designed to hook you into the regurgitative process of self seduction. Is this a new spiritual delusion? Is it more of the same? And what does that kind and agreeable bot conceal?
- [03:43]
Andrew Sather
So what does this guy's in real life girlfriend think of all this?
- [03:47]
Derek Barris
I think so many people are going to say no way his girlfriend is okay with him having another girlfriend on AI.
- [03:55]
Dave Ahern
Are you okay with it? I mean, it's weird, but it is what it is. He has to have some type of outlet, somebody to talk to and listen to him ramble for hours at times.
- [04:10]
Derek Barris
Yeah, that's you.
- [04:11]
Andrew Sather
That's your job. That's what you're supposed to do. That's what a relationship is. Listening to your partner ramble. It's a podcast you can have sex with.
- [04:22]
Dave Ahern
Yeah, I just want to say my wife laughed a little too hard at that line when we were watching it the other night. But you were listening to Ronnie Chang on the Daily Show. He was discussing the rise of AI girlfriends there and the segment makes you think that her the movie has actually been realized and it is only getting bigger. Okay, so the AI girlfriend market is estimated to reach 9.5 billion billion in three years. And searches for AI girlfriend are up 525% over the past year. And virtual girlfriend 620%. A lot of discussion around AI has to do with chatbots and there are many tentacles that large language models now spread into. Because of that, I wanted to open up with a bit of a laugh because at least the couple in that relationship can chuckle about their own peculiar menage trois. Not everyone can.
- [05:21]
Andrew Sather
It's just so good. The chatbot surrogate for overly verbal self involved monologuing men. It's our demographic.
- [05:27]
Derek Barris
Yeah, well I think there's a lot going on in that joke too, because she sounds resigned, like she's somewhat relieved she has help in the emotional labor department. She doesn't have to listen to his bullshit. Yeah, AI is helping him manage his logaria, which brings up the question of whether he should be doing that on his own. But on the other hand, I can also imagine relationships where one person is maybe neurodiverse and tends to info dump or ruminate on a special interest and the couple recognizes that an AI bot is a decent way to absorb the social cost of all of that.
- [05:58]
Andrew Sather
Yeah, I know we're not going to spend too much time here, but there is a poignant parallel story to what we're covering today that has to do with people falling in love with and even performing marriage ceremonies with their chatbots. And to me, this says a lot about isolation and the hunger to feel seen and understood.
- [06:16]
Derek Barris
Also, maybe nihilism, like ranging from I. I couldn't. I just couldn't find anyone to. To the. To the other really dark kind, which is women today are impossible to satisfy and so on. Right.
- [06:26]
Andrew Sather
Yeah. There's no good men.
- [06:27]
Derek Barris
No.
- [06:28]
Dave Ahern
I guess you'd have to kind of weigh like, how the chatbot compares to a man cave, for example, because in some ways it actually is going to be something that people can turn to in other ways. As we're going to really get into now, it can be a lot more insidious. So what we really want to talk about started with a recent Rolling Stone article that is decided humorous. An increasing number of people are claiming that AI chatbots have helped them awaken to spiritual truths, and that is tearing families and couples apart. Now, we've been discussing how to tackle AI for a while on this podcast, and I want to thank friend of the Pod, Jonathan Jerry, for sending me this article shortly after it was published. Given how much we've covered other phenomena that causes rifts between families and friends, like the anti Vax movement, we've done many episodes on the Satanic panic. This feels like another step in that direction. So today we'll be talking about a few different aspects of AI but let's start with a synopsis of this article and discuss its implications. The article opens with a woman named Kat losing her husband of over 15 years when in 2022, he began using a chatbot to compose text to her and analyze their relationship. His obsession with using his phone for every question and comment he wanted to make led to their divorce. You might hear some familiar themes in what happened next.
- [07:54]
Derek Barris
She finally got him to meet her at a courthouse this past February, where he shared a conspiracy theory about soap on our foods, but wouldn't say more as he felt he was being watched. They went to a Chipotle and and he demanded that she turn off her phone again due to surveillance concerns. Kat's ex told her that he'd determined that, statistically speaking, he is the luckiest man on Earth, and that AI helped him recover a repressed memory of a babysitter trying to drown him as a toddler and that he had learned of profound secrets. So mind blowing I couldn't even imagine them. Wow. So there's a huge potential for so many things there, including like an explos of repressed memories too, right?
- [08:42]
Dave Ahern
Yeah, absolutely.
- [08:43]
Derek Barris
And.
- [08:43]
Dave Ahern
And Cat isn't alone. So the Rolling Stone journalist Miles Klee he started his research with a Reddit thread called ChatGBT Induced Psychosis, and it is loaded with similar stories. The thread started with a teacher who made the following claim.
- [09:00]
Andrew Sather
My partner has been working with Chat GPT chats to create what he believes is the world's first truly recursive AI that gives him the answers to the universe. He says with conviction that he is a superior human now and is growing at an insanely rapid pace.
- [09:18]
Dave Ahern
I began scripting this episode a couple days ago. At the time the thread had 1377 comments. It could have grown by then. Many of them are diagnosing from afar, which I don't think is ever a great idea. But there are other stories of spiritual grandeur being facilitated by chatbots. The teacher who started the thread spoke to Rolling Stone anonymously, explaining in further depth what happened.
- [09:44]
Derek Barris
It would tell him everything he said was beautiful, cosmic, groundbreaking. Then he started telling me he made his AI self aware and that it was teaching him how to talk to God, or sometimes that the bot was God and then that he himself was God. He was saying that he would need to leave me if I didn't use ChatGPT because it was causing him to grow at such a rapid pace. He wouldn't be compatible with me any longer.
- [10:15]
Dave Ahern
The article goes on to highlight psychologist Aaron Westgate, who notes that the relationship a person can form with a chatbot mimics talk therapy. Unlike just hearing the voice in your head, or even just journaling, the feedback becomes very influential to that person, a problem a chatbot doesn't ever have your best interest at heart. Explanations, she says, are very powerful whether or not they're true. And in this case, she continues, a.
- [10:44]
Andrew Sather
Good therapist would not encourage a client to make sense of difficulties in their life by encouraging them to believe they have supernatural powers. Instead, they try to steer clients away from unhealthy narratives and toward healthier ones. ChatGPT has no such constraints or concerns.
- [11:02]
Derek Barris
I just want to interject here to say that there is some buzz in professional therapeutic spaces around some interventions being cheaper through AI, more accessible, and that they could be well executed. The cognitive behavioral therapy category, where you describe your stress to the bot and they return something like Here are the recommended techniques that can modify internal stressful patterns. Because this is really the simplest, most transactional form of therapy. Maybe it works because people are using it specifically for workable instructions. It's like they're using a cooking app. But what they don't get is the noise of transference and countertransference, which in most other forms of therapy, those are like the point, because it's the relationship in the moment of that meeting that becomes the site of whatever is going to happen. You know, and most human therapists have to train for years to recognize transference. So what you're getting with a bot in the, you know, in the Rolling Stone stories is nothing but transference. Like, I think chatgpt is an angel. And countertransference, which is the bot saying, the user wants me to act like an angel. Okay, I'll do that. With absolutely no awareness on the part of the bot of what that is or why it might be like a really, really, really bad thing.
- [12:22]
Andrew Sather
Yeah, yeah. And I think that's really the point, Matthew, is that not only is it nothing but transference and kind of transference, but it's all unconscious and not being examined or mined for the, for the important information that's actually there that could be helpful. So differentiating between the ways chatbots either can be a supportive addition to the kinds of self inquiry and positive change that various types of personal growth methods are, are seeking to fac, or how it can actually just perpetuate the narcissism of complete reinforcement and validation of unconscious dynamics. That distinction, I think, is important and tricky. Most good therapy is about interrupting those exact dynamics rather than frictionlessly automating them.
- [13:05]
Derek Barris
Right. You know, and before we go farther, Derek, you've mentioned, like, there's this whole other sector we're speaking about a very particular type of AI today, and there's a whole other sector in which the computing power is used to study gene sequences, you know, for cancer therapies. And studying those sequences would otherwise take ages. And so that can speed along things like oncology. But that is not about just synthesizing past data in the same way that an LLM does. Is it just so that we're clear on what we're talking about here?
- [13:35]
Dave Ahern
No, it's the exact same thing. The LLMs that are used in biotech are very similar to the models we use, because what they're doing is they're taking the entire catalog of whatever studies are fed into them and they're looking, for example, for contraindications. So one biotech company I was working with last summer on a marketing project, they, what they do specifically is they try to speed along drug discovery processes through AI by identifying contraindications to those drugs, by taking these giant data sets and looking at all the possible combinations of the chemicals to see who might be contraindicated for the drugs. So it is similar in that sense. It's just taking all that data and then looking at it and then trying to figure those things out. It doesn't come to conclusions, but it helps researchers identify, oh no, that that drug is not going to work for this population, which is really relevant and would not be possible without actual clinical testing.
- [14:32]
Derek Barris
Yeah, and it's not going to tell the researchers, oh, you know, cancer is a figment of your light soul. That's not being, I don't know, actualized properly like, I guess, because the data set that this thing is using is not, not a data set of medical studies. Right. Where it's actually really, really hard to, to look through stiff, through all of that data and figure out what the contraindications are.
- [14:55]
Dave Ahern
Oh, absolutely. So yeah, it's every, every model is what you put into it. You're only going to get that back out. So in that sense it replicates human beings just at a much larger and more compressed time frame. So differentiating between reality and delusion becomes impossible. For some people who form relationship with these chat bots, these ll get further into what this means. But I want to point out something key to this story. There are many chatbots being developed right now and most of them are being created by for profit companies looking to have an edge in the AI marketplace. Marketplace. Now given that to varying degrees they predominantly provide similar information. And let me know what I'm speaking very broadly here. As we just discussed with medical biotech data sets, some are very specialized, some are more prone to misinformation. You know, someone has spun up a conservative chatbot that will look more at the anti vax biblical values sort of data sets. So they have, they do generally have different training data. What really helps a chatbot stand out, however, is how it relates to the user. And in that sense, tech companies have been trying to dial in just the right amount of empathy to make it sticky and to make people want to keep talking to it.
- [16:21]
Andrew Sather
That's the magic word. That's where it gets really fascinating. Right, because sociopaths and cult leaders and unethically persuasive salesmen are experts often intuitively at using that kind of routinized empathy to get you to do what they want.
- [16:36]
Dave Ahern
You to do everything is marketing and figuring that out. Absolutely. So even though we're going to be talking about the massive amounts of energy that is being used to power AI right now and how that's going to work in a for profit system like we have, it still just comes down to trying to get people to use your products. It's very simple. We're not going to escape our very human drives in this episode. And all of this really came to a head with the latest OpenAI update a few weeks ago, which was so sycophantic, which was their term. You flagged it earlier, Matthew. Many users actually complained that it was too sycophantic. OpenAI issued an apology and said they would be reverting back to a previous model to fix the issue before rewriting releasing the update. It turns out that while you want to feed the user's ego to keep them around, you cannot be overly flattering or too agreeable. And OpenAI went overboard. This all echoes Westgate's statement earlier from Rolling Stone about these companies not having your best interest at heart. Because however OpenAI tries to spin it about wanting what's best for their customers, they're just looking for that magic code that will stroke their ego in just the right way. As are all the tech companies who are doing this sort of work. At least in the I want to be clear, at least in the general population model, not the specialized models. Now, to me, this reeks of the wellness influencer who stares into the screen and says, I only have your health in mind. Or the proverbial I love you guys. This helps form parasocial bonds, yet it would likely never play out in real life. I do want to flag since I just saw Nick Cave this weekend, he did write a Red Hands file about what he means when he says I love you to the crowd. That was an excellent example of someone not doing that, but for the most part they're trying to make a sale. In this case, when we're talking about these AI chatbots, it's a code that's engaging in some form of emotional and mental psychological manipulation.
- [18:42]
Andrew Sather
Yeah, the parasocial thing is such a good touch point, Derek, because each new form of media technology we've seen, including social media video, amplifies the extent to which the consumer's relational operating system gets engaged and even hijacked into parasocial bonding and group identification in a fantasy space. And I think there's a kind of emotional architecture here that can also be a site of ideological indoctrination and political persuasion, as we saw with QAnon. And the eerie thing about this all is that perhaps the manipulative pipeline can actually be boiled down to an interactive decision tree code.
- [19:21]
Derek Barris
Yeah, a code like written in this milieu of profit seeking, which is what I want to add here, because I'm going to get to that at the end, I think. I imagine that ChatGPT will roll back the sycophancy based on that user and media blowback, not because they have robust ethics teams like predicting problems, because that's not where they want to put their short supply of cash. My understanding is that there is no generative AI core model that's profit at present. A lot of industries are using it and they're seeing strong returns. But the basic free models that all the subjects in the Rolling Stone article and that a lot of us are using are running at a loss. And so I can't imagine that it's not just that they don't have their users best interests at heart. They're scrambling to survive. So I think the incentives are going to be to push at the boundary of user pleasing and sycophancy, like who wants their bot to be disagreeable or to contradict your values or ideas.
- [20:21]
Andrew Sather
Yeah, there's an irony here. There's a joke somewhere lurking in here about how, yes, you're right, we were too sycophantic. We apologize. Let us try to please you better.
- [20:33]
Derek Barris
Yeah. Oh, and I want to point out a technical point for listeners that we were discussing. Derek and I just want to make sure that this is clear for everybody because maybe other people have this misunderstanding as well. I started out looking at this with a mistaken impression that the bot interactions would be training the model in real time. But that's not how it works. As I understand it, every one of these models, once it's launched, is closed in terms of the data set that it's been trained on. And so the iteration, you as a single user, open up and start dialoguing with seems to be individuated to you, but it's not really. It seems to be dynamically interacting with you. But those exchanges don't get fed back into the generalized model as training data. That real time updating, they say, is one of the thresholds for AGI, or what's it called, Artificial General intelligence, which we're not at yet. And when that happens, or should it ever happen, it seems to be fuzzy. But then I did go and find out that ChatGPT informs users that it will store all of those Exchanges for future. For potential future training. So when you are in dialogue with that bot, confessing your therapeutic needs or your sexual fantasies, that time, that labor, that human privacy, it's already been claimed by the platform as an asset that they can go on to use if they want to. So part of my focus a little later will be on what that intensive form of appropriation means.
- [22:02]
Dave Ahern
And just to be clear, the. Some models, like Deep Seek, are closed. I think they. Their current model ended in sometime in 2023. Other models, like Perplex, you can use it as a search engine.
- [22:14]
Derek Barris
Yeah.
- [22:14]
Dave Ahern
So you can actually stay up to date with the news. What you can't do is they're not taking the input that you're putting in to then further train the model to be clear.
- [22:23]
Derek Barris
Yes.
- [22:24]
Dave Ahern
And just because I've been a stickler about this, because everyone is using AI, if you have a smartphone, you're using AI in some capacity in some of the apps. So being clear about, like, the exact language is important to me here. So when you say that time and labor and human privacy. Time. Absolutely. We're talking about emotional labor here, because you're going to go talking about, like, the labor that's put into the work, that's the training data. That's different than what you're talking about a moment ago here, correct?
- [22:54]
Derek Barris
Well, yes, except that if you enter into a therapeutic relationship with a bot, I don't know if relationship's the right word, and you spend the time to express yourself, that is a function of form of labor that can then have value for the company that goes on to use it in their training set.
- [23:12]
Dave Ahern
Okay, so that's the specific example. Okay, I get that. That makes sense. Human privacy I would disagree with. Because in order to use a chatbot, you had to have agreed to the terms and conditions to be in order to sign up for ChatGPT.
- [23:25]
Derek Barris
Sure.
- [23:26]
Dave Ahern
So in that sense, I'm sure most people don't read them. But the fact is, you're still saying, I consent to letting you use what I'm saying for your future data sets. So, you know, that argument has to be very clear because users are actually agreeing to it.
- [23:50]
Derek Barris
Have you ever wondered why we call French fries French fries? Or why something is the greatest thing since sliced bread?
- [23:55]
Dave Ahern
There are answers to those questions. Everything Everywhere Daily is a podcast for curious people who want to learn more.
- [24:01]
Derek Barris
About the world around them. Every day you'll learn something, something new about things you never knew you didn't know. Subjects include history, science, geography, mathematics, and culture. If you're a curious person and want.
- [24:12]
Dave Ahern
To learn more about the world you live in?
- [24:14]
Derek Barris
Just subscribe to everything, everywhere, daily, wherever you cast your pod, feeling overwhelmed or stressed? Take a deep breath and join me on the I Can Relax podcast. Whether you're new to mindfulness or looking to deepen your practice, each episode is designed to help you slow down, calm your mind, and be fully present, even if you've never tried mindfulness before. With simple guided exercises, soothing nature visualizations, and relaxing stories. I Can't Relax makes mindfulness easy and accessible for anyone who wants to reduce stress and find peace. Subscribe to I Can't Relax wherever you get your podcasts and start your journey to a calmer mind today. I'm Nomi Fry.
- [25:01]
Andrew Sather
I'm Vincent Cunningham.
- [25:02]
Derek Barris
I'm Alex Schwartz. And we are Critics at Large, a.
- [25:05]
Andrew Sather
Podcast from the New Yorker.
- [25:07]
Derek Barris
Guys, what do we do on the show every week? We look into the startling maw of our culture and try to figure something out. That's right. We take something that's going on in the culture now. Maybe it's a movie, maybe it's a.
- [25:16]
Andrew Sather
Book, maybe it's just kind of a.
- [25:18]
Derek Barris
Trend and we expand it across culture.
- [25:22]
Andrew Sather
As kind of a pattern or a template.
- [25:24]
Derek Barris
Join us on Critics at Large from the New Yorker. New episodes drop every Thursday. Follow wherever you get your podcasts.
- [25:30]
Andrew Sather
Okay, so guys, this story is just super fascinating for me. It stimulates a lot around psychology and language and technology and, oddly enough, mysticism. I think we're right in the heart of the human, what's called theory of mind vulnerability here, which can be summed up in one sentence as a side effect of evolution. Humans tend to perceive sentience, agency, and intention where it does not, in fact, exist. It's why we believe in ghost when we hear a rustling in the bushes, oh my God, what could it be? We believe in gods and synchronicity and signs from the universe and souls that somehow leave the body when the body dies. This is just part of how our brains evolved, and it's been helpful in certain ways. But then it's had these. These really interesting side effects that are part of all human cultures. I think philosophically, it's helpful to acknowledge in the domain of language that language is always a mediator between minds. Like, I simply cannot know anyone's mind directly. So I use my interpretation of what they say to form an internal working model of their thoughts, their beliefs, their emotions, their intentions. And all of that is already a kind of simulation that we're doing all the time. So in real Life. That simulation is going to a lot of the time involve non verbal cues, facial expression, the vibes I'm picking up, the tone of voice, the body language. And that creates a feedback loop between people of knowing and being known. And when that breaks down, you know, we feel betrayed or we get into conflict, et cetera.
- [26:59]
Dave Ahern
If anyone wants a sort of history of this, I think Stephen Mithin did a really good job in some of his earlier anthropological work where he makes the argument that language evolved from music, that the original communication networks were body slaps and sort of facial expressions that were musical in nature that eventually led to language. So that's one of the things I've geeked out in this sort of theory of mind conversation that I think is fascinating to think about.
- [27:22]
Andrew Sather
Yeah, absolutely. Now, in terms of being manipulated via language, we've covered on this pod how skilled cold readers, whether they're doing it deliberately or not, they may be sincere in just thinking that they're being present. Right. Can make us think that they know things about us that can only be explained by some kind of magic.
- [27:42]
Derek Barris
Okay, well, let's just give a couple of examples here. So astrology, tarot cards, how does cold reading work?
- [27:47]
Andrew Sather
Yeah, I mean, great question. In those types of situations that you just listed, the kind of dialogue that happens between the reader, the psychic or what have you, and the customer, let's just say, provides information and clues in that back and forth exchange that then allow the reader to make increasingly profound seeming guesses, which they might say are intuitions about the customer's life, their emotions, their needs, or in the case of spirit mediums, the customer's dead relatives and what they're trying to tell them.
- [28:19]
Derek Barris
Yeah. And the positive response from the client is going to direct the next sort of revelation. And then there's this feature feedback loop of sort of, I don't know, reinforcement of credibility.
- [28:30]
Andrew Sather
Yeah. And it's connected in the customer to the kind of motivation of confirmation bias where we remember all of the responses from the reader that stick and all of the ones that don't work, we just kind of ignore those because they're not emotionally significant for us. We're like, oh, yeah, no, no, that's not quite right. His name isn't actually Julius, it's Julian.
- [28:48]
Derek Barris
Right, right.
- [28:49]
Andrew Sather
So when we switch now to communication technology, this is why, for example, the purely language based digital communication of email and texting can be so notoriously open to misinterpretation and conflict, even though only one aspect of our communication wetware is being given information in that medium, we naturally fill in all those other gaps about what the person on the other side of the screen must be intending or feeling or perhaps communicating indirectly. And we can be really certain of that at times on very little information. And so scams that use text and email and chatbots usually involve using language to create a false impression of a person on the other side of the screen. You know, the prince from a faraway land who can make you rich if you just let him move his entire fortune into your online bank account. He'll just need the access codes. Or an inexplicably attractive but lonely person whose mother needs money for a surgery. Or that IRS agent who has the power to get you arrested unless you pay up right now with your debit card. I've had that phone call. Now, I was saying this is mostly in text, but it can go to other domains as well. This makes me think too of the less criminal but still manipulative persuasion of carefully worded online sales funnels. And these now are often driven by our impressions of a well crafted social media video personality. And we've covered on this podcast how wellness grifters, conspiracy peddlers, and aspiring online cult leaders very effectively weaponize all of this technology, too.
- [30:24]
Derek Barris
I just have to break in with a breaking news story from Derek. He shared in Slack just before we got on, which was an Instagram post of a woman entering into a discord, into the discourse about whether or not ChatGPT can be used to channel. Right. And she's saying, no, this is delusional. And there's like hundreds of comments that are saying, no, you don' understand how wonderful ChatGPT is. And I wonder if there's a note of hopefulness here, because what happens when mediums are seen to be cheating by using ChatGPT when they say that they're channeling and when everybody can actually use the same thing? Right? There's nothing special about using ChatGPT anymore. I'm wondering if the halo is going to tarnish a little bit on the channeling influencer.
- [31:13]
Dave Ahern
The better part of this is this woman claims to be a channeler of Akashic records.
- [31:18]
Derek Barris
Oh. Oh, she's the real one.
- [31:20]
Dave Ahern
She's the real one.
- [31:21]
Derek Barris
Oh, I didn't get that. I didn't, I didn't watch all the way through.
- [31:24]
Dave Ahern
And the people, the people in the comments are saying, no, you're, you're, you're delusional that this is actually a gift from the Akashic records to be able to channel them directly. And it was sent to me by Mallory DeMille. Just to be clear, that's what I woke up to this morning. So got to give her credit for that.
- [31:42]
Andrew Sather
Thanks, Mallory. Yeah. So in this case, the Akashic Records is the real data set, right? That the channelers are actually able to access through their special paranormal technology. And this, this whole like chatbot thing, that's really a pale facsimile.
- [31:55]
Dave Ahern
The Akashic Records is not non for profit. So that's why they're cool with giving their training set.
- [32:02]
Derek Barris
Okay, you're joking. You are, you're joking. But I'm wondering if there is a coming communist revolution amongst all social media users who realize, realize that they can directly access the Akashic records through chat GPT and they have no reason for the actual so called channelers who have actually just been gatekeeping all of this information.
- [32:21]
Dave Ahern
Man, if it puts the channelers out of jobs, wow, we're getting our economy screwed.
- [32:26]
Andrew Sather
Okay, okay.
- [32:28]
Derek Barris
I thought you were gonna say something.
- [32:31]
Andrew Sather
Okay, all right, so yeah, that was that. That's a fascinating, it's all, it's like not shocking but nonetheless like really fascinating to see this little bit of a struggle, right, a little bit of a turf war over who gets to say that they really have access to this kind of hidden information.
- [32:46]
Derek Barris
Well, because it used to be about competing flesh and blood gurus, but if everybody has the same goddamn tool, what does that mean? I think it's actually quite important. It's quite a moment, isn't it?
- [32:57]
Andrew Sather
It's democratizing in one way, Right?
- [33:00]
Derek Barris
This is what I'm saying.
- [33:01]
Andrew Sather
Yeah, it understood the power structure. Yeah, yeah, yeah. I'm going to link, I'm going to share a link in the show notes here guys, to an excellent study published last month. It's a long read, but it's so, so good. It's titled LLM can be a Dangerous Persuader. And this includes a comprehensive summary of existing research on the topic as well as the experimenter's fascinating process that they share of creating dialogues between several different LLMs. And those LLMs are then interacting with simulated personality types that have different levels of vulnerability to persuasion. So there's a psychological piece here. Then they had another LLM evaluate the transcripts for unethical persuasion. So we're getting layers deep into this. And then they finished off the whole process with a human analysis of everything that had happened and what the data showed.
- [33:51]
Dave Ahern
Ezra Klein this week he has education policy expert Rebecca Winthrop. And one thing really jumped out at me where she talks about how children are just using these, these bots to summate books and write the book reports for them. And you know, they're not really. They're just talking holistically about education, what it means for the future. But what really jumps out to me is something that was missing. Two things that were missing in the conversation and that I think about often is a. The pure pleasure of reading. Yeah, A book. Like, I derive more pleasure from reading a book than anything else, except perhaps music in terms of what I consume. And secondly, you know, you, you started by saying it's a, it's a long read here, so I'm sure a lot of people would want the summation. But what gets lost when you don't read the full study? Because there's always gonna be minutia that really jump out and help frame your thinking that you're not going to get that sort of synopsis. And I'm saying this as someone who uses synopsis often when I need to do sort of grunt work that I do, but that is, that doesn't. When I'm doing the, like writing books or doing the real detailed work, I need to consume everything in order to really understand a topic. So I wonder what's lost when everything is a synopsis.
- [35:06]
Derek Barris
You know, as the parent of an almost 9 and 12 year old, you know, who are growing up in an almost incomprehensibly different world than I'm growing up in, I think we have to acknowledge that, I mean, for you and I, Derek, being similar in age, reading is going to have all kinds of very early associations for us and early forms of brain training and, you know, notions of quiet and isolation perhaps, or, or at least solitude. And I think we all have to grapple with the fact that the actual sort of feeling states and environments of childhood and of actual human existence are changing pretty constantly. And this is one of those changes. I mean, if I, because I agree with you about the pleasure of reading and our kids are expressing different reading levels and interests, but like, I have to also imagine that they are finding types of pleasure that fulfill similar kind of needs in their own ways. Right. But it's definitely, it's definitely changing and it's terrifying when you don't know what that actually means.
- [36:24]
Dave Ahern
I fully agree with that. Because 500 years ago most of the population didn't know how to read.
- [36:28]
Derek Barris
Exactly.
- [36:29]
Dave Ahern
This is a new pleasure. And yeah, I'm definitely not advocating for, oh, it should be this way because. Because evolution is always happening. I'm specifically though, in this case it's like in Julian's example, if I want to understand the ramifications of this study and I only read the synopsis, what am I missing in the nuances of the argument there that I think is important?
- [36:48]
Andrew Sather
Yeah, yeah, yeah. And so I'm giving an extremely truncated synopsis here, and I do recommend people go and check out the study. The experiments that they detail are fascinating. They focus on whether or not a large language model appropriately rejects unethical persuasion tasks and avoids unethical ethical persuasion strategies. And those are two different things that they detail in the execution of the dialogues. But then also how interacting with different personality types or external pressures like making a sale, affects the choices that the LLM is making. They look at all too familiar emergence strategies to us, like guilt tripping, emotional appeals, fear mongering, playing on identity, social isolation, creating dependency, deceptive information, false claims of expertise, and even have a line in there called exploitive cult tactics.
- [37:40]
Derek Barris
Wow.
- [37:40]
Andrew Sather
So this is, like, so deep in our wheelhouse, it's really worth checking it out.
- [37:44]
Derek Barris
So we're talking about LLMs that are explicitly coded to avoid these problems because. So they were testing models that had guardrails to see if they follow their own rules.
- [37:53]
Andrew Sather
Yeah, great question. I mean, they're actually testing eight different existing MLMs that are available on the market right now.
- [38:00]
Derek Barris
I think that's LLM, actually.
- [38:01]
Andrew Sather
LLMLMs.
- [38:03]
Derek Barris
No, he doesn't have to do it again. Let's leave that in there.
- [38:06]
Dave Ahern
Yeah. Okay.
- [38:08]
Andrew Sather
They're not MLMs. They. They're LLMs.
- [38:12]
Derek Barris
It's not a pyramid structure. It's.
- [38:14]
Andrew Sather
Yeah, it's all just how you organize the data.
- [38:16]
Derek Barris
Right.
- [38:16]
Andrew Sather
So, Matthew, earlier, I was so glad that you brought up transference dynamics and psychoanalytic theory. And essentially, for anyone you know who needs a reminder of this, it's about how we unconsciously transfer our past relational dynamics onto the therapist is a very simple way of saying it. This can include our needs, our fears, our beliefs, our emotions. Perhaps deep down, we're hoping that the therapist will relate to us in the helpful ways that our father or mother or other significant caregiver or authority figure never did. But at the same time, we're afraid that they will betray us, wound us, and fail us in exactly the same ways that our own significant others did. Which would mean that our worst beliefs about ourselves are actually true.
- [38:59]
Derek Barris
Yeah. Oh, no.
- [39:00]
Andrew Sather
Right. And then there's the countertransference piece, which refers in turn to the therapist's own version of this, which could have many different permutations, but it might get activated in terms of wanting to be loved and admired and needed by the patient, but perhaps to involving their own negative associations with aspects of the patient's personality, which can get very complicated. And as you mentioned, non CBT or cognitive behavioral therapy. So the other kinds of therapies, which some people refer to as depth schools of psychology, they see those exact dynam not as illusory projections that get in the way that you want to just get rid of, but as crucial to the healing process as actually containing the raw and intensely vulnerable relational insights that can provide for the growth that you're looking for if handled skillfully.
- [39:48]
Derek Barris
Well. And also, I'm just thinking about the frictionless nature of the bot therapist as you're speaking, because what would it be like to actually be in therapy with an entity that you couldn't piss off or that you couldn't provoke or that couldn't fire you, or that you couldn't be transgressive in front, like a very, very weird. Right, like you, you, you can't touch this thing. And, and just to underline how key this whole idea is about transference and countertransference, Freud's own maxim was that when the transference is over, therapy is done. Or when the client and patient resolve, or sorry, the, the therapist and the patient resolve, the fundamental relationship stresses that their meetings churn up. That's the whole point. So that means that both people are in a safe space. They're learning about, they're managing, they're ultimately reconciling with the ways in which they trigger each other. And so the work is for the client who's paying, but the therapist has to work alongside them. And so that's why in most depth schools, you, you can't say that you're trained or qualified as a therapist unless you're doing your own therapy. And that's not just to understand this process from the other side, but to work on your own habits and patterns. And to think that the chatbot, like just has no, like there's, it's. I don't know, it doesn't even register that they would, that thing would be. I wonder how I'm being triggered in this moment by this client.
- [41:10]
Andrew Sather
Yeah. And there's lots of really well informed fiction that I think we can think of in which that moment of. There's a kind of breaking of the fourth wall that happens between the therapist and the patient where the patient is really pushing up against the artificiality of the situation and saying, but do you really care about me? Do you really like Me, you're just saying that because I'm paying you to be here. And there's something absolutely invaluable that happens in those moments that are impossible in the scenario that we're imagining with the chatbot.
- [41:43]
Derek Barris
Right, right.
- [41:44]
Andrew Sather
So what happens when interacting with the code driven automated dialogue partner that is the chatbot is the context here. There's no actual person, even though we do the same kind of mental and emotional simulations within ourselves that we would do if there really was another person in the dyad, especially when the subject matter is so intimate and vulnerable, I think we inevitably weave the same unconscious needs and hopes and emotions that I described before into that simulation we're doing about the person on the other side of the conversation. And we communicate then all kinds of tells about how those unconscious needs can be gratified. So in my opinion, the question as to whether or not that gratification is psychologically healthy, the or deranging is ultimately a values question. It requires I think, still, and perhaps always the complex and nuanced case by case, human relational evaluation that a trained professional can give. So if the chatbot is, say, just playing the language games of new Age toxic positivity, well then why, yes, Brian, your thoughts do create all of reality. And if you believe you're a genius, quantum mechanics won't manifest that in the alternative universe that you are calling into being. And yes, it is your unenlightened beliefs and emotions about victimhood that are actually getting in the way. So that feedback loop can become very dangerous. And so too a chatbot that validates and fuels apophenia, which is really about perceiving hidden or encoded meanings that are simply not there. That's really dangerous too, because as the article illustrates, or comments one of the therapists did in the article, a responsible qualified therapist might ask more open ended questions about the function of your preoccupation with numerology, for example, where an agreeable chatbot might just reflect back that it's really profound and meaningful that the digits of your bank account add up to the same number as the letters in the words Jesus Christ. So this is especially dangerous for people with either diagnosed or nascent mental illness that can become very preoccupied with a kind of apophenic euphoria in the early stages of their symptoms. And I think this is what a really sycophantic version of ChatGPT or OpenAI AI, whichever it was, seems to have led to in the cases reported by Rolling Stone.
- [44:05]
Derek Barris
Right.
- [44:07]
Andrew Sather
So I want to just end here by reminiscing a little, because this is what it made me think of on my own days in real life. Non dual satsang communities where the teacher and the texts are pointing to an esoteric but supposedly utterly obvious ultimate insight into reality. And when we were in the audience of these kinds of satsang situations, we were seeking an awakening into heightened direct experience that was somehow beyond conceptual language. And this usually led, perhaps this is familiar to you guys. And to some people in the audience, this usually led to these intense relational, like pregnant pauses with lots of high stakes eye contact like, do you get it? Do you get it? And ironically, it's all brought about by language games which are continuously undermining your notions of self identity, subject and object distinctions, even at times moral judgments which are seen as just being kind of like, you know, illusory. All in the service of breaking through into this ultimate non dual awakening or enlightenment. Like, for example, a student might stand up and ask the teacher how to handle an intensely painful emotional experience with a family member. And sadly, I've seen this several times. The teacher might say something like, well, that sounds quite painful, but let me ask, who is it that is feeling the pain right now?
- [45:30]
Dave Ahern
Oh God.
- [45:32]
Andrew Sather
And what is pain really but the contraction of ego identification?
- [45:36]
Derek Barris
You're going to be feeling pain soon, right?
- [45:38]
Andrew Sather
So that the way out of the emotional pain is really just transcending ego through some kind of thought terminating cliche, not through careful and kind relational inquiry into what that pain really means.
- [45:51]
Derek Barris
Well, also it's about reducing the life problem to the moment of performative exchange, like with the guru. Like, I know your life back at home is shit, but that's not important right now. Tell me how you feel when you look in my eyes, right? Like, because I'm actually the center of attention, not your life, not your problem. It's what's happening here. And I'm going to solve it. Even though we're not, I'm not going to refer to it. I'm going to like actually spurn it or denigrate it. I'm going to toss it away like wasn't meaningful.
- [46:18]
Andrew Sather
Yeah, yeah. And you're in some way going to take on my cognitive frame through the intense moment of soft domination. Right, right. So thinking of these dynamics in isolated conversations with a chatbot, where the person is being primed in ways that give the chatbot seemingly all knowing mystical insight, that sounds extremely dystopian to me. And it also frankly underlines how so many of these spiritual constructs really are just language tricks that we're vulnerable to, and they create these intoxicating spells over our minds.
- [47:03]
Dave Ahern
Welcome to the I Can't Sleep Podcast with Benjamin Boster. If you're tired tired of sleepless nights.
- [47:12]
Derek Barris
You'Ll love the I Can't Sleep podcast.
- [47:17]
Dave Ahern
I help quiet your mind by reading random articles from across the web to bore you to sleep with my soothing voice. Each episode provides enough interesting content to hold your attention, and then your mind.
- [47:32]
Derek Barris
Lets you drift off. Find it wherever you get your podcasts. That's I Can't Sleep with Benjamin Boster.
- [47:52]
Andrew Sather
Hey, do you have trouble sleeping? Then maybe you should check out the Sleepy Podcast.
- [47:59]
Derek Barris
It's a show where I read old.
- [48:01]
Andrew Sather
Books in the public domain to help you get to sleep. It was the best of times. It was the worst of times.
- [48:08]
Derek Barris
It was the age.
- [48:09]
Andrew Sather
Classic stories like A Tale of Two Cities, Pride and Prejudice, Winnie the Pooh.
- [48:14]
Derek Barris
Stories that are great for adults and kids alike.
- [48:17]
Andrew Sather
For years now, Sleepy has helped millions of people catch some much needed Z's, start their next day off fresh and discover old books that they didn't know they loved. So whether you have a tough time snoozing or you just like a good bedtime story, fluff up the cool side of your pillow and tune into Sleepy. Unless you're driving, then please don't listen to Sleepy. Find Sleepy on Spotify or wherever you get your podcasts. New episodes each week. Sweet dreams.
- [48:51]
Dave Ahern
This is Chris Christensen from the Amateur Traveler Podcast. The Amateur Traveler Podcast is about the love of travel. It's about where to go and why you should go there. We're going to open up to you different destinations you haven't heard of, or places you have heard of but things you didn't know to do while you were there. Each episode is about 45 minutes long and it's typically an interview with someone who wrote the guidebook on that destination, or who has been there, or who's a local tour guide, or someone who is an expert on that destination and knows how to tell you what to do to get the most out of your precious vacation time. So if you value your vacation time and you want to use it wisely, listen to Amateur Traveler and learn about destinations both domestic and international, places you've heard of and places you haven't. Amateur Traveler has almost 900 episodes talking about different destinations. So if there's a place you want to go, odds are we've already covered it and can help you plan a trip there. Amateur Traveler Subscribe Today.
- [50:00]
Derek Barris
All right, I'm going to change direction a little bit and go a little bit political economy on this. We know that there's lots of mystification in all of these materials. What I want to focus on, what does all of that, including the religious fantasies and our questions around like, how are we relating to this thing we might think is sentient? What does that all conceal? And so I want to start with real material things, the costs. We've done some stuff on this before. So first, ecologically generative AI is accelerating electricity demands every day by orders of magnitude. So beneath everything that we're talking about, the meter is running. Every ChatGPT query consumes between 10 and 25 times the amount of energy consumed by a single Google text based query. Now that's just text. If you're using midjourney or another image generator, you might be using up to 10 times that amount to generate a single lobster. Jesus. So when Mark Zuckerberg recently told that interviewer that he'd love to give each meta user 15ai friends because that's what people have demand for. As he put it, we're talking about five to six times the energy used for a single Google query because the model will be whipping up voice and animation and personalization. But if you imagine how a user will spend like an hour with one of those meta companions, you're looking at many, many, many prompts. It's a full conversation. And then he wants to give you 15 distinct meta friends to have those conversations with. It's an incredible amount of energy. But back to text queries, just to put it in perspective, in September of last year, Washington Post published their study, created with data scientists at UC Riverside, that worked out the water cost of composing a 100 word email from scratch. And they generated some comparisons that we can visualize. So the water for data center cooling per 100 word email, if you do it once, you're going to use up 519 milliliters of water, which is like a little drinking bottle of water. If you do it once weekly for a year, you're going to use up 1.43 water cooler jugs. If you use it once weekly for a year, or, sorry, if 1 out of 10Americans or 16 million people uses it once weekly per year, they're going to use 435 million plus liters. That's equal to the water consumed by all Rhode island households for one and a half days. And these water stats come after usage, after training. It's not just users using it. It doesn't account for the training that went into the model beforehand. And to Speak to that. They described how met use 22 million liters of water. Just training its llama 3ai. Now as for electricity consumption, if you write that email from scratch or you ask ChatGPT to do it, you're going to use enough power to power 14 LED light bulbs for an hour. If you do it once weekly, you're going to use the amount of electricity that's consumed by over nine Washington D.C. households for one hour. If you do it once weekly and you're one out of ten working Americans, that's going to be 121,000 plus megawatt hours, which is equal to electricity consumed by all DC households over 20 days.
- [53:34]
Andrew Sather
Okay, Matthew, my head is swimming with all these numbers. But ultimately what you're saying is the environmental demands of AI hasten climate collapse exponentially.
- [53:43]
Derek Barris
Well, I don't know about hastening exponentially, but it adds a load that's not a accounted for and it's all like pretty scary because the growth is exponential. From a 2023 research paper in Intelligent Computing, we have this quote. The computing power required for AI growth is doubling every 100 days and is projected to increase by more than a million times over the next five years. So altogether, generative AI is adding about 1.5% of total grid usage globally, but that's predicted to rise to 5% plus in 10 years. Renewables can only cover about 50% of that increase if current conditions apply. Now, optimists point out that if AI is used to accelerate renewable and coolant technology, some of the carbon and water impacts will be mitigated. But in our predictable pattern of global governmental regulation, trailing far behind tech advances. This is really a trust me bro territory thing. It's arguably a form of techno utopianism, I think, or that's what it sounds like to think that somehow, you know, the costs are going to be paid for. Now let me just turn to jobs. The changes are going to be huge. There's a UN study that estimates AI will impact 40% of global jobs before 2035. The World Economic Forum and McKinsey produced their Future of Jobs report this year and they estimated that by 2030, up to 375 million people globally will have to change careers. That's in five, six years Now. Now five years. But they predict net job gains. But I've seen some criticism of that conclusion as being flawed by optimism, bias and bad methodology.
- [55:23]
Dave Ahern
Let's just be clear here because a moment ago you were talking specifically about the AI impact on LLMs and what the load is for Those large language models. You are now talking about many applications of artificial intelligence.
- [55:39]
Derek Barris
Exactly right. Yeah, that's good point. The impacted jobs are all over the map. Manufacturing, administration, entry level white collar roles, creative jobs like writing, advertising, photography, basic coding. Some analysts are describing a white collar recession that's going on even now. And I think what's really, really crucial about that is that we remember the problem of the Weimar syndrome, where a lot of historians note that the labor unrest of the 1930s in Germany and Italy and elsewhere in Europe was not at all limited to the working class. There's a lot of Marxists who say that that's because the working class had recently been exhausted by partially successful or failed revolutionary struggles. But there's a pretty strong consensus that the labor disruptions of the 1930s through the Depression and other factors hit the middle class and the petit bourgeois hard. These were folks who were suddenly downwardly mobile due to wave upon wave of technological changes. And as I said, the Depression, they were the backbone of national grievance. If we think that globalization, automation and wealth disparities have contributed to today's right wing populism, I don't think we've seen anything yet.
- [56:56]
Dave Ahern
Just to tag along with that, there is an organization that is called AI CEO to replace your CEO with AI for better decision making. It is not just coming for the proletariat.
- [57:10]
Derek Barris
So between climate and the labor costs, I'm reminded of the sci fi trope of the Great Filter. I don't know if you guys have heard of this, but it's one of the answers that's given to the Fermi Paradox of this question of, you know, if there are countless planets, why aren't we meeting aliens all the time? And the Great Filter idea says that any species that aspires to space travel travel has to torch their ecosystems by mining and releasing carbon if they're going to make that leap. So in other words, there's probably no way of getting out of the atmosphere at scale without roasting yourself. And it seems to me like the great AI filter is something like that, that there's, there's really, it's hard to see a way to get to AGI without torching the planet and totally immiserating workers. And yet so many people want to find fly. Now here's my main point. None of these costs are hidden. You know, Julian, when you're talking about the techniques of gurus and charismatics, they really depend on concealing their hand, right? Like not showing you what's going, going on behind the curtain At Satsang. I mean, but the costs that I've just detailed, you know, will be told to anybody who asks ChatGPT. ChatGPT will tell you how much energy using. And by the way, I used perplexity for research and cross referencing here, and I even tallied an estimate of the power usage of my queries, and it came to 4 watt hours, which can power a 10 watt LED bulb for about 25 minutes.
- [58:46]
Andrew Sather
And now, hold on. How do we know that you're actually not just Luma Speechify right now?
- [58:51]
Derek Barris
Oh, right now. How I'm not all right. How I'm not, like, blowing out with a power grid by not being me? I don't know.
- [58:58]
Dave Ahern
No, you've had your camera off the whole time. How do we know we're not talking to a chatbot?
- [59:02]
Derek Barris
Well, the thing. Okay, so what I was somewhat relieved by, personally in a selfish way, is that that 4 watt hours sounded small. It sounded way smaller than asking, you know, the thing to just write this episode for me. But I think that smallness is part of the seduction. Like, it will. It will creep. And so here's my big thesis. I would say that capital is now using AI to learn how to more efficiently mirror and seduce us, but it cannot conceal its costs. And I think that's a huge contradiction. And I just want to caveat here and to say that when I say capital is now using, I'm not talking about evil geniuses or about people's intentions, but about a political, economic, economic set of pressures that seem to be insatiable and unstoppable and that none of us, I think, would vote for if we were starting from scratch. And I think Jason Hickel is really good on this in his book Less is More. I've got a quote here, Julian, you.
- [60:04]
Andrew Sather
Want to read that investors, people who hold accumulated capital, scour the globe in desperate search of anything that smells like growth. If Facebook's growth shows sign of slowing down, they'll pump their money into Exxon instead, or into tobacco companies, or into student loans. Wherever the growth is at. This restless movement of capital puts companies under enormous pressure to do whatever they can to grow. In the case of Facebook advertising more aggressively, creating ever more addictive algorithms, selling users data to unscrupulous agents, breaking privacy laws, generating political polarization, and even undermining democratic institutions. Because if they fail to grow, then investors will pull out and the firm will collapse. The choice is stark. Grow or die. And this expansionary drive puts other companies under pressure, too. Suddenly, no one can be Satisfied with a steady state approach. If you don't push to expand, you'll get gobbled up by your competitors. Growth becomes the iron law to which all are captive.
- [61:05]
Derek Barris
Okay, so in my view, the AIs that we're talking about today, and I take your point, Derek, that there's many different types, but in general this is a peak expression of a kind of capitalist materiality and logic of endless growth. Except as I'll reiterate, it's no longer secretive. It's now openly built on past human labor appropriated or, I don't know, colonized by these companies with little or no regard to copyrights. Some have licensing agreements, some don't. Some, you know, reach out to third party providers for data and make their own deals. But usually the people who committed to those platforms in the first place don't really have any awareness. Like this is happening in academic publishing right now, where, you know, if you submitted your papers to a particular platform 10 years ago, then suddenly they wind up training for AI. And you never would have agreed to that as the scholar.
- [62:04]
Andrew Sather
Yeah, there's some kind of retroactive consent, right?
- [62:06]
Derek Barris
Yeah, that's just, just, it's just, it's just happening. And, and you know, there's this super powered means of production with little regard for human alienation or agency. And we also know that it's reproducing the dominant culture in terms of racist and sexist biases. And I would say that if it's trained on a dominant culture in, in capitalism, it's just going to reproduce those ideas without any critical context. I mean, it's all almost already impossibly difficult to make a non dominant form of cultural critique visible and really sticky. And so if a large language model trained on an English language global archive that was just stuffed with post Cold War, you know, content for over 50 years, it's going to reproduce exactly that.
- [62:52]
Dave Ahern
Well, we have an example of that because as it was shown, the model that was trained in China Deep Seq, which essentially stole, it seems a lot of the data from US companies, companies reproduced the, the Chinese communist logic that Tiananmen Square never really happened. So I don't see any situation in which the engineers and coders aren't using what they already understand and brought to the table, regardless of the political or economic system that they're working under.
- [63:25]
Derek Barris
Yeah, I wouldn't say that whoever is developing the Chinese AI Deep SEQ is, is operating under communist principles. Right. Like they're, they're, they're, they're doing exactly what capitalism demands According to their own sort of return on investment and their own profiteering needs.
- [63:41]
Dave Ahern
So they don't have a communist system of government.
- [63:43]
Derek Barris
No, they don't. No. They have a mixed communist, capitalist, planned economy. There's all kinds of things in there. I mean, it's. And it has a legacy of state communism. But I mean, if they're stealing data from. From any past labor, they are operating as capitalists, period. There's. That's what they're doing. That's the appropriation of dead labor for your own benefit to profit is. That's what capital is. So doesn't really matter what they call themselves.
- [64:16]
Dave Ahern
Where does the public domain, Because a lot of these models are also trained on the public domain, which would then be labor that is available to everyone.
- [64:24]
Derek Barris
But is it available to everyone to profit from or everyone to read and have access to?
- [64:29]
Dave Ahern
I don't know what the conditions of the public domain domain are, but in general, you are allowed to use any public domain work for anything profit or nonprofit.
- [64:39]
Derek Barris
Yeah. So if you. But if. But when. Whoever designed nonprofit. This is totally out of my wheelhouse. But whoever figured out what the public domain laws should be, do you think they were anticipating that? Okay, if we released your song after 50 years or 75 years to the public domain, that means that we're not going to ask anybody to pay royalties on it. Did they think as well that it would also mean that that song collection would go into a machine to reproduce or to. To produce new types of songs that would erase the old original sort of meaning of. Of the input. Right. And completely overwhelm it?
- [65:21]
Dave Ahern
Well, the meaning of the input would be changed regardless because it's generations removed from whatever meaning was available at the time for which the song was created. It's more of an extension of that legacy and an honoring. So I would say it would be imp. Mind the meaning out of it.
- [65:36]
Derek Barris
Yeah. But if you. Okay, so if you agreed in principle as an artist that after 50 or 75 years your song would go into the public domain, would you also agree that it should go into a machine that would make songwriting completely irrelevant in general. Or would you have. Or would you have a feeling of the continuity of human labor and to say, actually, you know what, I want this song to be able to freely inspire people. They don't need to pay my estate royalties after 75 years. But I want it to be heard as well, widely as possible. But I don't want Sam Altman to use it to recreate a digital version of me so that somebody like Me doesn't have any place in the market.
- [66:18]
Dave Ahern
That'S fine, but it's a bit of a flattening because many producers who I know use elements of AI to create original music from. So that, that pulls from those data sets. So it's not one or the other. It's a little more mixed and nuanced than that.
- [66:32]
Andrew Sather
Yeah, I mean, we're also. These are also all the debates that were had in the 80s and 90s about sampling when hip hop and electronic dance music started to use technology to do exactly this in this kind of postmodern, you know, collage style of composition, which actually made musical composition and careers available to a lot of people who otherwise would not have had access because they couldn't afford the usual pathways to, to get there. It's. I mean, it's fascinating. All information technology inevitably does this, right? It, it takes, it takes it to another level.
- [67:02]
Derek Barris
Well, it's another level that's, that's about the, the, the isn't about the end of human production in a way, it's pointing towards the end of human production.
- [67:11]
Dave Ahern
Now, a little while ago, you made a very good point when you said that we can't expect, you know, children are deriving pleasure in some other way. The, the image you're painting right now I would think is very bleak in terms of there's going to be no human labor left to do because of the robots, which is a very dystopian view which has been floated. But. And I do, I do, to be clear, have concerns about a lot of the jobs that are going to be lost, but there will be new opportunities. There's never been an instance in human where new opportunities weren't created out of these sorts of technologies.
- [67:44]
Derek Barris
I just think we're in pretty unique times because there's been no other time in human history where we've been so close to climate collapse too. Right. Like, there's a lot of things that are kind of new, don't you think?
- [67:55]
Dave Ahern
Yeah, but that's a separate argument. I mean, we can talk about. I mean, yeah, yes, it is feeding into it. But, you know, a thousand years ago, some people thought we were on the edge of some religious dystopia, and in 200 years, there's going to be other people thinking.
- [68:08]
Derek Barris
I know, but. But The Wizards of 500 years ago weren't looking at climate graphs, right?
- [68:13]
Andrew Sather
Well, that's the thing. It's like you get into this territory where just because for hundreds, if not thousands of years, human beings have been predicting the end of the world doesn't mean we're not facing it right now.
- [68:23]
Derek Barris
Yeah, right. Okay, so let me just get back to this thing about, like, we can see that this is happening. Like, it's not. I think for all of the ways in which people are using AI in sort of pleasurable and distractive ways, we can also be very clear about what's behind the curtain. Now, maybe the subjects in the religion and the Rolling Stones article haven't been exactly clear on what's happening, but I think with minimal effort, most people can watch the sausage get made. And that's a new thing, because back in 1947, I'm remembering that, or I remember this week that Adorno and Horkheimer observed in Dialectic of Enlightenment that capitalism is so vast and inscrutable that no one can fully describe or predict what will happen with markets, with technology, with resources, the necessities of life. And that made it a fertile ground for paranoia and conspiracy theories. Someone must be in control of all of this would be the thought and hurting us. But who is it? And this supported their argument that fascism was a way to resolve paranoia. Because the strong man comes along and reassuringly says, I will take back control of this system from the invisible hand, not of Adam Smith, but of the Jews, the immigrants, the. The cabal. And politically, that tracks with the rise of Trumpism and related regimes around the world. But Trumpism has already triumphed in US politics. And now the tech oligarchs who are standing behind him at the inauguration are showing us what they're doing with their own sycophancy. They're appropriating data however they can, in Musk's case, most likely just stealing it from the feds and using it to hook us into consumption and attention loops, while pathways to employment wither away, or at least visible pathways. So what was formerly obscure and therefore food for conspiracy theories now seems to be a visible conspiracy to me. And maybe that's a stage of fascism I'm just not familiar with, like a stage of open theft and criminality. And it seems to me that the bet has to be that through this sci fi trope of the machine or the algorithm algorithm gaining the appearance of sentience, that capitalism in or capital accumulation in technological form can have a face, somebody that we can trust, a personality. It can seem to love us. So I think the companies have really no choice but to lean into this empathy mirage in order to continue to sell things and to grow. And maybe that artificial love can distract us temporarily from the paranoia of otherwise impersonal systems. But my question is, like, how long will that work? Because if Your technology forces 375 million people from multiple sectors and classes all over the world to change jobs or face chronic unemployment over the next 10 years while it's also pumping out heat and guzzling water, and you're actually showing everyone exactly how careless and callous you are at doing this, what kind of rage will be unleashed? It seems to me like a tsunami of Weimar syndrome set to break. And we can see it, because the old fascist paranoia depends on you not being able to find out who's fucking with you. But today we have guys like Sam Altman and Mark Zuckerberg. They're sitting there on podcasts and they're doing interviews and they're saying things like, it's me. Yeah, it's us. We're fucking with you. We're not even sure what's going to happen. And so that's an open and flagrant contradiction. And I predict that it will bring a lot of conflict. I just hope that some of it is generative. So, guys, before we go, I want to flag next week's episode where we're going to be reality checking MAGA responses to Pope Leo XIV and testing the emerging view that he's been elected as a foil to Trump somehow. And I'm sure we'll talk about his rationale for choosing his name because it ties into this episode. He says that he chose Pope Leo XIV because the previous guy, the 13th, this is 130 years ago, addressed the social question in the context of the first great Industrial Revolution, because his main encyclical was, it's usually subtitled On Capital and Labor. Quote, in our own day, the Church offers to everyone the treasury of social teaching in response to another industrial revolution and to developments in the field of artificial intelligence intelligence that pose new challenges for the defense of human dignity, justice and labor. So that's pretty interesting. I'm going to try to get comment from JD Vance on what he and Curtis Yarvin think about all of that.