Loading summary
Transcript93 lines
- [00:00]
A
We're talking the Pope and AI here on ChinaTalk with AKB Zakaria, ChinaTalk's resident Catholic, John Clark Levin of Kurzweil Technologies
- [00:13]
B
and Tim Huang today representing. I'm the deputy director of the Institute for Christian Machine Intelligence.
- [00:20]
A
Beautiful. The Pope, he's got takes. Past popes have had many, many, many takes as well. I have seen since learned there were parts of it that felt a little CCP doctriny, but there were also many parts that did not feel like that at all. Perhaps we'll explore this theme over the course of the episode, but I think we should start with the concept of unencyclical. Why is the Pope writing a five hour audiobook's worth of content about AI and humanity?
- [00:55]
C
Pope Leo said twice in the first three days of his pontificate that AI is the greatest new challenge facing humanity, not one of a list of like six or seven things. The thing and that really jumped out to me because one can easily imagine an alternate scenario in which a new Pope had said humanity's greatest challenges like gender ideology or on the other side, climate change or even something anodyne and obvious like war or poverty.
- [01:24]
A
Like, like that's really striking. There's a lot of like hungry and cold and war stricken poor people out there. And as he's talking all about this, I'm just like, what about all of the human suffering which is happening today?
- [01:41]
B
In some sense, the Catholic Church is like every other very, very large global organization, right. That it has to be able to set intentionality and that a big part of its kind of ability to leverage events like shape what's happening in the world is sort of its ability to kind of wrap its interests and its priorities in sort of the work of the moment. Right. And actually I think about this in some ways because, Jordan, to your point, I think a lot of what you see in the encyclical is about kind of exactly the topics that you're mentioning being wrapped into the AI discussion. Right? And so I think almost the one way of thinking about it is like the, the Pope is using this kind of like being able to.
- [02:17]
A
Is this just like a news hook for like all the like.
- [02:21]
B
Well, I think it's more than a news hook. Right. Like, I mean, because I think actually, I mean, you know, you only get to do your first encyclical once, Right. And so I actually think this actually like augurs a much bigger bet on the power of the Catholic Church that what we're seeing actually is going to be significant. It's going to be with us for some time, and that a lot of things in the world are going to be worked out through the lens of this particular technology. And so I think that's one of the reasons why you see this being, you know, the first one that's being dropped, as it were.
- [02:52]
C
I think Pope Leo clearly sees the AI transition as analogous to the Industrial Revolution and as epochal in its impact. And so that frames his approach to engaging with the world, because just as the Industrial Revolution impacted not just the economy, but social relations and politics and warfare deeply and profoundly across a full century, in a similar way, Pope Leo expects AI to do the same into the substance.
- [03:27]
D
Because I feel like when I see on either, like the Internet or X or whatever, when I'm on social media, too often, half of the people are saying, oh, my gosh, the Pope has declared a fatwa on AI. It's over. It's like, you know, a crusade against AI. And the other half are saying, the Pope is super AGI pilled. So where is, where is the balance? What is the actual substance?
- [03:52]
C
Pope Leo, through this encyclical, evinces a generally balanced view about AI. He clearly recognizes its great positive potential, but also its capacity to cause grave harms. And the encyclical focuses more on addressing the harms than the upsides. But that's really to be expected as a genre of theological writing, Encyclicals are more about acting as a corrective to toxic views in the world and calling people to stop harmful behavior than they are about cheerleading things that are already going well. So the fact that Pope Leo focuses more on the harms does not remotely mean he's declaring a sort of Dune style Butlerian jihad against AI.
- [04:41]
B
Yeah, but I think it actually goes a little bit deeper than that. I think in 2026, almost the idea of you being a skeptic or you being AGI pilled is actually, in fact, something of like, an obsolete distinction. Like, I was talking to a friend recently who's working on kind of an rfp, a grant rfp, and thinking a little bit about, like, well, we want to really define the world between nothing happening and everything happening. And I'm kind of like, well, yeah, that's the world that we're in right now. I mean, part of the challenge is like, it's a big deal, but like, how do we actually characterize that? It's like a very, very broad kind of spectrum. And so I guess, I don't know, like, you know, the encyclical is Very much of a moment. Right. That I think it kind of represents, like, this transcending of these kind of distinctions that have for a very long time anchored discussion. Like, everybody agrees that we're in the middle of some kind of takeoff now. The question is, like, what kinds of directions are you moving in? And like, what should be prioritized?
- [05:33]
A
I'm sorry, guys, I still, we need a. We still got to pull back for a little more pressy in context. Like, what are insignical? Like, what does it mean when a Pope says something like this? Like, how do Catholics around the world process their religious leader having these types of takes?
- [05:52]
D
Is it infallible?
- [05:55]
C
No. An encyclical is basically an open letter addressed not just to the entire Roman Catholic Church, but to all people of goodwill all around the world. This is something that the Pope undertakes on his own initiative. So it doesn't require the entire Curia, the cardinals, the bishops. It doesn't require that apparatus outside his own office as Pope. And so the encyclical reflects a Pope's own personal teaching on, on a subject of concern to the Church and to humanity more broadly. It carries with it the weight of Catholic teaching, but it is not considered to be infallible.
- [06:45]
B
Yeah. And I think, Jordan, I think at the core of your question is like, does this matter if we can just kind of cut to it? And I think we're like, we, we are about to find out how much this kind of matters in terms of practical action that the Church is going to take. You know what, what I would recommend, I think, is like, that there's been a lot said about the encyclo, but it sets forth an intention. Right. It sets forth an agenda. And so I think what we're really waiting to see now is, okay, well, what kind of bureaucratic muscle gets put behind this agenda? You may have seen the news prior to the encyclical launching that there was this commission that was being set up between kind of multiple ticastries within the Catholic Church. I think that's where the action's going to be. Right. Because it's like, okay, now we've set what we think are the big problems here. What are we going to do around it? And so, you know, I think like, almost the jury is still out in terms of the long term significance of this. And part of it, I think, is just seeing how Catholics around the world are going to respond to this message and do things that are aligned to it.
- [07:41]
A
Let's go around the horn with everyone's, you Know what everyone thought was most interesting?
- [07:46]
B
I think the two sort of most interesting things in my mind, right. I think one of them was that the encyclical kind of opens up. You know, I think in some ways the most important thing that it does is just say, look, Catholic Church has agency over where this technology is going to go, right? It says, we can either build Babel or we can rebuild the walls of Jerusalem. I think that's actually quite important because traditionally, most of the religious discourse I've seen around AI tends to be, this thing is happening, how do we hold back the tide? Or this thing is happening, how do we fight it from happening? I think the change institutionally to say, look, there's a lever, we can decide where it should go is one of the things that's actually most significant, right? And I think we should get into that. I think that's like one of the things that I thought was most exciting about it and opens the question of, okay, so, like, where do you want to push it and how do you go about pushing it? I think the second thing that I'll highlight that I thought was really good is this kind of Section 99, Section 100, which has become so controversial, particularly on social media, you know, I think is a really interesting one, right? Which is, okay, to what degree does the Catholic Church believe that what is going on in these systems is the same as what humans are engaging in or distinctly different, and if so, how different? Right? So there's also, like, big, open theological questions that are opened up. And again, I think in my mind, you know, this is related to the work that I do. I think it's basically setting like a research agenda, like a theological research agenda, about how do you make the fine distinctions between this, like, humans, non humans, and then AI, which may be some secret third thing. And so, at least for me, you know, out of a very long document, those are the two that immediately kind of caught my mind. I think the third one that I'll just mention, you know, was kind of just on reflection, I was briefing a bunch of Catholic bishops earlier in the week, and I think one thing that's really percolating down to folks who, you know, like, are really in doing the work of, you know, being in the clergy is all the education stuff. And I suspect there's going to be a bunch of action on the ground that takes place specifically around that. So maybe three things to throw on the table, all of which I think are, like, super deep and very interesting.
- [09:50]
A
Well, let's let some intersperse some passages from this. So from the Babel thing. The task that stands before us is that of being builders of communion rather than architects of Babel. We are to be servants of the coming kingdom instead of lords of towers destined for ruin with the heart of a shepherd and a father. I ask everyone to abandon the construction of yet another tower of Babel and to join forces in building up the common good so that humanity will never lose its beauty and once again will come to recognize the human heart as the place where God desires to dwell.
- [10:26]
D
Can I. Can I quote a tweet that the pope pulled from 99 to 100 that I feel like a bazillion responses to?
- [10:33]
B
That seems fine. Yes.
- [10:34]
D
Yeah. It reads, artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships, and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience. They may imitate or even simulate, but they do not understand what they produce, for they lack the effective relational and spiritual perspective through which human beings grow in wisdom.
- [11:00]
B
I think it's, it's. I think it's an important statement, right? I mean, I think it's a strong sort of virtue statement that there will be sort of a line, right? And I think this is part of the problem with, like, a lot of AI work right now, which is, okay, so Anthropic is going to come out with a paper that says, look, there's these emotional concepts inside the model, right? It is important to keep 2 thoughts in your head which are very similar, but actually importantly distinct. The first one is actually experiencing those emotions in the way that we are cognizable on those emotions as humans. Right. And then also the idea that, like, those concepts can be represented in a way that causes the machine to behave in a very human like way. Right. Those are actually pretty. Two pretty distinct things. I mean, I think, like, ultimately what the Pope is pointing out is sort of like the theological difference between the two that, like, you can totally say, even though the machine is engaging in, say, thought, it's not really thought in the sense that thought is almost inherently a human act. Right. And I think that's kind of like the thing that is causing a lot of confusion online because it's, it's too not. It's basically too nuanced for Twitter. It's my opinion.
- [12:04]
C
So, yeah, the, the problem is words like imitate and simulate and understand are used there almost as theological terms of art. There's a lot of lading of theological presuppositions about ensoulment, for example.
- [12:23]
B
Yes, that's right.
- [12:24]
C
That a common colloquial understanding of those terms does not carry. So even though I think Pope Leo is correct in the narrow theological sense he likely intends about today's model, I'm worried that a lot of people might get a misapprehension from that. If you tell the average person that AI is incapable of understanding, for example, they're likely to infer that AI won't be able to do many kinds of dangerous things that inhumans require deep understanding to do.
- [13:01]
B
Yeah, and I think this is actually, like, it's a great signal of just how I think impoverished our language is around some of this stuff, right? Where we're like, you say understanding, you know, I think let's just distinguish the two. Like, maybe there's like a lowercase understanding and uppercase understanding, right?
- [13:14]
C
Yes.
- [13:15]
B
And I think the lowercase is like, lots of objects can engage in things that are. Are the act of understanding. You know, I think basically the, like, Christian position here, though, is like the kind of understanding that humans engage in is definitionally a kind of act that only humans do. Right. But I think, John Clarke, to your point, right, like, when you talk about understanding, people snap to like, oh, you're talking about a capability, right? And I think that's, like, that's not what's at stake here. And I think part of the problem that's being played out is, like, people see the word and I think they're snapping to like that, Right?
- [13:45]
C
Yes.
- [13:47]
B
This is, I think, one of the reasons why I'm advocating that everybody on every podcast I'm going on to get really into Thomas Aquinas. And I think you should get into Aquinas because he's got this notion of the species, right? So that there are these kind of like, essentially forms in the mind that can be shared by all sorts of souls, right? Vegetable souls. Right. Like vegetation, animal souls, human souls. They all contain species. Right. And what's so good about that is basically you, you. You decouple these two ideas. So the. The idea of installment from the idea of capability. And that actually, I think, is a very rich thing which actually should inform our technical research on the ICMI stuff. I really want to do like a technical research, like a seminar at some point. I met a guy who's trying to rally a bunch of kind of sort of like, Muslims that have a machine learning background. And I was like, we should do an interfaith technical research seminar that would be Fire. Yeah, yeah. So I think it's one of the things I'm working on right now.
- [14:41]
A
So let's continue around the corn. John Clark, what angles of this struck out to you?
- [14:47]
C
I was particularly interested to see whether and to what extent this encyclical would engage with AGI or superintelligence. And it turns out that it did not at all. AGI and transformative economic impact, existential risk are not mentioned. And that was actually a surprise because the church's most recent previous flagship document on AI, a note called Antiquet Nova from 16 months earlier, did at least mention those issues.
- [15:21]
A
Maybe it's time for another absence. I mean, there was no. There was like a little bit of like when I see democracy used in documents like this, where, where are we on that? And maybe relate that to the sort of China discussion, particularly around like AI and autocracies.
- [15:39]
B
Jordan, of course, I think coming from someone who's hosted China Talk for so long, your immediate thing is like autocracy versus American democracy. I almost read this kind of in the context of democracy of method. So I think a lot about the fact that, you know, the companies have for a very long time wanted AI safety and alignment to be a field which was very kind of like neutral and objective and kind of not needing to touch all sorts of the difficult things that come. When you start thinking about like embedding values and systems and what has that unintentionally has created, basically because of the virtue of these industries having these like very strong natural monopoly type effects, is that you end up with literally an aristocracy of AI alignment and safety. I think a little bit about kind of. And I think this is like one of the things I'm most interested in the space is the idea that we live in this really cool world though, where AI agents themselves are really good at doing AI research because this is literally what the labs are using them for. And what that allows is anyone with a different set of normative priors to be like, I'm going to do alignment too. Um, and so I actually think like when he says democracy, obviously I think what he has in mind is democratic backsliding and, and sort of systems of governance. But I actually also see in this kind of like the idea of a democratization of who gets to say what these models are aligned to and even how we define safety at all.
- [17:02]
D
Right.
- [17:02]
B
And sort of the idea that you can rebuild from first principles, a lot of these techn, like these sort of technical methods and kind of democratize that I think is actually a really key part of the story here, and you can almost see the entire encyclical in this light, is to basically say, like, there's this AI thing going on. You know, if you're talking about moral philosophy, you're sort of on our turf. And by our turf, I mean the Catholic Church.
- [17:24]
D
I'm a little bit kind of along Tim's lines. You know, the encyclical uses the word democracy a bunch, but I don't think it's necessarily the Pope saying, down with the ccp. It's more this democracy that's like, you know, the Church sees itself almost as a representative of the downtrodden. Right. Especially with the overwhelming majority of Catholics being from the global south and the sort of emphasis on laborers or things like that. I think it's talking about almost this local democracy that whatever policy you're enacting, you're going to have to get input from the people that you're enacting that policy on, and it's going to impact them the same way. I don't know how much that says about giant superstructures of you have a giant republic versus you have a party apparatus that does it this way. I think that's the more of the lowercase d democracy vs an uppercase d democracy.
- [18:16]
C
A very striking line from the encyclical is the need to disarm AI. And in that passage, Pope Leo speaks explicitly about the arms race dynamics that are currently playing out in AI and basically tells the world to knock that off. And that seems very relevant to current efforts to find some path toward bilateral engagement between the US and China around this race toward AGI.
- [18:49]
D
Yeah, can we talk about that paragraph a bit? Because, I mean, maybe it's because I work at chinatalk, but I was surprised it only got one paragraph of disarmament. Yeah, so the disarmament paragraph, which doesn't specifically name the US or China, because the point is to be much more subtle than this is. Disarming AI means freeing it from the mentality of armed competition, which today is not limited simply to the military context, but is also an economic and cognitive phenomenon. This entails a race for ever more powerful algorithms and larger data sets driven by the desire to secure geopolitical or commercial dominance. To disarm means discrediting the assumption that technical power automatically confers the right to govern. So I think, you know, this is pretty clearly, especially when it's time, geopolitical dominance is kind of screaming about US China, AI competition, saying, hey, knock that off. But I feel like so many of the worries that the Church is finding in this sort of AI race or AI acceleration is at least partially justified by labs or by governments on this sort of geopolitical race dynamic. I'm surprised it doesn't interact with it more in terms of saying, hey, there's a reason we're not doing this. This is maybe a more practical path of going about it. And I can almost worry that some people might interpret it as a church saying, guys, why don't we just hold hands and sing Kumbaya? I don't know, what do you guys make of this passage? Why don't you think the Pope said more about it? Is there something between the lines that we're missing?
- [20:15]
C
I think the Vatican's strongest engagement on this can come not through an encyclical, but using the diplomatic power and perceived neutrality of the Holy See to both mediate potential bilateral engagement between the US and China, but also to convene global south nations around these concerns. The billions of people, Catholic and non Catholic, in Latin America, Africa, South Asia and Southeast Asia, who currently have no voice, no vote, and no say in the risks and harms that may be imposed upon them by this US China race.
- [20:57]
B
Look, I think the Holy Father's telling you, right? Like the quote is, to disarm means discrediting the assumption that technical power automatically confers the right to govern. Right? So this is, this is certainly an attribute of national competition, right? We need to win the AI risk race because by doing so we will ensure America's continued dominance over the international system. It is also relevant to the domestic discourse as well. Right. Which is, well, once we have AI fully adopted, you know, within this industry, my company will be the one that dominates this sector. Right? Even the idea that like, oh, well, we're going to call, you know, the heads of all these AI labs to D.C. and they're going to opine on the moral impact and social impact of the technology. Is this as well, Right. I think this is all in the context of disarmament. And so I guess I think to the question earlier about, like, why don't we hear more about it? I think you hear more about it earlier where there's this broad discussion about the centralization of power. Right? And that part of the concern here is that you're giving these systems like kind of control over many aspects of society. And that control for many reasons is centralized. One of the jokes that I've been kind of like turning over in my head is like, well, maybe we need a homeschooling movement. For AI, we need a parochial school movement for AI. Right? Like, maybe this is the idea is like part of the idea is like we need to think a little bit about not just the democratization of these systems, but sort of their democratization of their application. Like what are they even used for? And what do you think people. People think they can get out of it? Right. I think is kind of what's at stake here.
- [22:30]
D
So is the Pope anti export controls?
- [22:32]
B
I guess I did see someone do like a. This totally justifies our. At open source AI, which I think is a little. It's a little cute, you know, I mean, again, part of the problem is such a long document is you can, you can of course attach lots of agendas to it. I don't know if you're going to have to ask him yourself. You should send a message to feedback at Vatican Ca or whatever.
- [22:56]
A
So I got to pull the back Kobla camera for a second. Like, where is he getting this? Does he have staff who are like on Twitter all the time? Like, how much of this is him writing it? Like, like what, like to what extent is this like all Pope versus like Pope and friends when you read something like this?
- [23:17]
C
Encyclicals are very much Pope and friends sorts of documents where typically, the Holy Father will retreat to the summer residence at Castel Gandolfo with several trusted theological advisors, and they will hash out the key issues collaboratively, work on drafts, and then over a period of months, loop in more and more advisors that shape the document with broader perspectives. So even though Pope Leo should be considered the single author of the document, it certainly embraces a range of perspectives.
- [23:57]
A
Do we think he has like a Claude code account? Like, what has he built?
- [24:07]
D
He just called Chris Ola when he needs that perspective.
- [24:11]
B
Chris, I need this programmed. Get on it. Yeah.
- [24:14]
E
How many.
- [24:15]
B
I mean, Jordan, when I say, what's
- [24:16]
A
the Vatican's token budget? Okay.
- [24:20]
B
I mean, I actually think the Vatican needs to put money down and have their own alignment lab. Like, I think they actually really need to get in the game in a serious way. But I mean, Jordan, to your earlier point, I can get into that idea. But like, I think that one thing I'll say to you is I think everybody would be very surprised just how kind of forward thinking, I guess the Catholic Church is in some ways. I went to a conference many years ago that was hosted by the Vatican. It must have been like five to 10 years ago. And it was about AI. But I met the guy who was basically like, well, you know, we Have a working group that is trying to figure out, like, what the Catholic Church would do in a First Contact scenario. Do aliens have original sin? Should they be? Is this a group that we'd want to try to convert? Right. Like, how would we engage with them? And I was like, totally makes sense, right? You are a really big institution that needs to think on millennia timescales. Of course, of course you invest. You've got at least one guy who works on these types of problems. And so I actually think like, you know, there's a vision of like, ah, isn't it funny that the Pope's doing this thing on AI? But I think you'd be surprised. Like, I think they've got the resources to invest in foresight and they do. Right? Because I think you've got to plan for every constituency. You're running this enormous institution. So in the same way the government has this, I think we shouldn't be surpr. The Catholic Church has it as well.
- [25:31]
A
What's the first context answer? Oh my God. We need a whole series on.
- [25:35]
B
I never found out more. I mean, I think it was the Vatican's version of classified. I was like, can I hear more about that? And she was like, I'll talk to you later. So, no, they're not going to tell me about that. Okay.
- [25:47]
A
Tim, why does the Vatican need its own AI alignment lab?
- [25:52]
B
Part of my worry about the encyclical is that I think it's very easy to have very strong kind of rhetoric around the technology. And that is valuable on its own account. Right. It will shape the preferences of how Catholics think about adopting this technology and using this technology. But I think we also live in an era where tech is upstream of lots and lots of different things and that like, by the time you are debating whether or not the chatbot should do this or that, you're already, you've already lost the game. Right. The decisions that are being made here are at the researcher level, at the fine tuning level, at the data collection and curation level. And so while it would seem very strange for the Catholic Church to do this, I think it's not implausible to say that they should be investing in GPUs, they should be doing their own research. They should be rallying global Catholics that work in these labs and have machine learning expertise to work on their technical research agenda. And, you know, I think the thing that I'm most interested in is sort of the idea that, look, it turns out that Christianity is like very well represented in the training data. It's very deeply represented in the training data. And so the concept that you would come up with alignment techniques that are the same or even better than secular approaches to the same problems seems like very empirically plausible to me. And I think if the Catholic Church starts doing that, right, starts releasing results, being like, oh, yeah, anthropic can do this, but check out what we can do. And if you think you can do better, you need to be our benchmarks, or we can beat your benchmarks. Like, that starts to really influence the technology at a very, very deep level. That is the kind of level that you would need to do if you're serious about shaping the direction of this technology. Like, how do you get away from Babel? How do you build the wall? Well, like, building the walls of Jerusalem requires technicians, engineers. Right. And I think, like, we should, we should take that metaphor very seriously or that illusion very seriously. So, yeah, that's my riff.
- [27:43]
A
There was a lot of discussion in the encyclical about, like, the role of states, and he kind of defers to states on a lot of things. Tim, how do you see the, like, you know, UK isi, like, US Government role in aligning models, or just like, states more broadly, I guess, and like, the, like, how that, how you see that interacting with the religious institutions, having their own.
- [28:11]
B
I mean, I think it, like, raises really. Yeah, I think it raises really interesting and difficult questions more than it might initially appear on its surface. Right. So if you take seriously the idea that religious priors will lead you to very different technical research agenda, the question is, okay, is UK AC representative of the kinds of risks that global Catholics think are important in this domain? If not, why not? Shouldn't the state necessarily be potentially prioritizing some of those concerns, not even of Catholicism, but, like, religion in general? That's, I think, a really difficult, interesting question. The other one I'll maybe throw out, which I think is maybe intentionally a little bit provocative, is we talk a lot about the American AI stack, the Western AI stack. I don't know. Is the American AI stack a Christian AI stack? We probably should have that conversation. I think it's a really interesting set of questions and one that I think should be fiercely debated. And so I think the encyclical kind of makes life difficult for states in some ways because it now says, look, part of your agenda now also has to confront, like, a religious question.
- [29:16]
A
Does this, like, should we be bearish on all the decentralized religions that, like, can't get their shit together and, like, get a data center and an alignment team?
- [29:27]
B
Well, I don't know. I mean, obviously it would be cool if the Vatican was like, we're going to just like get an allocation of h, you know, GB2 hundreds and like build the data center. I mean, I do think that would be cool as hell. But I do think that more importantly, we also live in a world where again, what I talked about a little bit earlier, you can do really interesting mechanistic interpretability work on small models now, right? Like Quen 3.5B. I've been doing work on it. It's really fascinating. It's really rich. The internal representations are really, really interesting. And so I think generally I don't know if there is a structural Catholic advantage, which sounds like Jordan, the question you're asking is I do think that actually now lots and lots of people can play in the alignment game and you know, if you can scrape together the money to buy a DGX Spark, you're like, you're in the game.
- [30:14]
C
But I found it really striking how many non Christian and non theistic researchers at the Frontier Labs have actually been looking with great anticipation toward Pope Leo's words on this. Because at the Frontier Labs, individual researchers might often feel a little bit queasy about what's going on, but they look around them and everyone else seems to be going along with what's being built. So they think, well, maybe that's just a me problem. So when there's someone with the neutrality and moral stature that Pope Leo has speaking out on these issues, that acts as a moral coordinating signal far beyond the confines of just the Roman Catholic Church.
- [31:02]
D
I'm curious as to the relationship between the Vatican and the labs, because clearly there's some relationship, right? If anthropic's present at the event, I guess, you know, like, first, what's it? Why anthropic? I don't know if there's an answer to that. Why only anthropic? At the exclusion of others, maybe. And also again, from the China angle, if you're trying to disarm AI, if you're trying to cut the entire world to be on board of this, why wasn't Deep Seek also there or something like this, you know, I hope they
- [31:29]
C
will be in the future. As to how all of this unfolded. Bishop Paul Tighe, who was one of the lead authors of Antiqua Nova in January 2025, went in person to the Frontier Labs in San Francisco earlier this year and kind of did a tour there, getting a sense of where this technology actually was at the Frontier and got to meet with Chris Ola, then I suspect that was the stage at which Hola got looped in. But I see Ola's participation less as an anointing of anthropic than appreciation for him personally as a researcher. He is the father of the mechanistic interpretability field and is very, very highly regarded at all the Frontier labs. He's worked at all three of them, actually. So I see his presence as being more as a scientist first than a representative of the company that he co founded.
- [32:31]
D
Are you bullish or bearish on the idea of, like, you know, we already say the United States government sucks at understanding AI, or the Chinese government sucks at understanding AI. The Vatican is also, you know, it's a little bit cloistered and it's older men. How well do they, like, how. How much understanding does Bishop Ty have when he goes around and visiting these labs?
- [32:51]
C
I've been doing this outreach since June of last year in Rome at the Vatican, and I've been really, really pleased and impressed by how much receptivity there was to evidence and argumentation around these issues. So even when an older cleric theologian is quite skeptical of transformative impact from AI, they are still listening thoughtfully and authentically to the case that I'm making and many other scientists are making, trying to learn more about the technology and discern the path forward. So, overall, I am bullish on the Vatican's epistemics, trying to figure this out going forward.
- [33:36]
B
Yeah, I mean, I just flip it on its head and like, it's about time for us to just recognize that AI is a real boomer technology. You know, it's literally like you just talk to it like a human. And in fact, if you. If you don't even know how to use a computer, you can call wallet and have a conversation with it. Right. What AI has done is allow computing to become a lot more democratized. And actually, some of the users that understand it most intimately are the ones that are not kind of traditionally computer native, in some ways would be my point of view. And so I don't know if age really kind of counts in it in some ways. Right. I do think that, weirdly, the tech kind of flips a lot of our usual instincts about this stuff around.
- [34:14]
A
And I think that's the answer to John Clark's question of why the people who've built this are really curious and excited about this. It's because they haven't read Aquinas. Right. And, like, if you were so, you know, if you've Been riding this wave for the past four, six, ten years. Like you're just staring at code and trying to like, improve your model all the time and like, are sort of aware at some level that these, these, you know, these models are going to have like big societal moral implications. But like it's, it must be, I don't know, it's almost a relief that like there's someone out there who's like, send it setting you some direction. I mean, I feel like, you know, even back in the, in the heady days of 2023 when Sam Aldwin was like, you know, genuinely, or not asking for regulation, like there's a part of it where I think these labs, they just like, want someone to like, tell them how to do this in a way which is not going to like, you know, rip society apart and like take away any meaning people. People have in their lives.
- [35:33]
B
And also might there. And I think there might be a real demand in a very serious way among the top leadership of these companies. Even not from like a practical business standpoint.
- [35:43]
D
Right.
- [35:43]
B
Like, you know, in many of these labs, it is literally a group of people who have been constituted to build a disembodied perfect intelligence whose main purpose is to like cultivate virtue in, in the users. Like, I feel like a lot of them are kind of confronting like how sort of like deeply religious, you know, the, the kind of project is that they're working on. We're, we're going to see how that tribe evolves over time. But I think I certainly sense among many of them a real sense of like, okay, well, yeah, you start to do feel a little bit religious about this technology and you know, maybe organized religion is one way that they help to kind of make sense of what it is that they're involved in.
- [36:18]
C
It's also really striking to me that although Anthropic was co founded by a bunch of effective altruists, they have empirically, in the process of training, successive iterations of Claude converged on something much more like virtue ethics as the right framework for developing Claude's character. So in a way, that's an interesting validation for the Christian approach to ethics that comes from their empirical contact with what makes Claude behave better or worse.
- [36:54]
B
Yeah, and I have a theory that I'm chasing after right now. I really want to see if I can get kind of like an empirical way to show it. But it is true that maybe like, if you, if you tend to be like, well, we're going to be very deontological for these models we're going to specify a bunch of rules. This maybe makes sense in an earlier era where the models are primarily rule following, but I am sort of interested in, like, scheming risk. Right. Like, it turns out, like, as the intelligence gets better and better, its ability to reason past a set of rules you create becomes better and better. And so there's kind of this maybe, like, this natural gradient in the actual, like, scaling of the model that makes virtue ethics the better alignment strategy later on, because any set of rules you propose, the model can get around or reason around in a halfway predictable or understandable way. And so the only way you can get alignment at that point is the model itself needs to have a sense of, like, what is the good that it is attempting to achieve, which I think is, like, really odd and hard to kind of, like, put a number to, but I think is, like, one way and one thing that you really should be exploring if you're interested in sort of computational theology as a field.
- [37:55]
D
I didn't know computational theology was a field.
- [37:58]
B
Well, it is now.
- [37:59]
C
Yeah.
- [38:01]
D
Okay. I just want to keep harping on the China aspect of this, because I
- [38:05]
B
feel that it is called China talk after all.
- [38:07]
D
Well, I just mean that it's like the Pope. I mean, he's, you know, he can't. He was born in the US and obviously there's such an engagement with the US Ecosystem and, like, a deep understanding of it. Right. I mean, there was, like, I was amazed by the paragraphs on transhumanism. That's just like, you know, directly talking to Peter Thiel or something like this.
- [38:31]
C
An encyclical subtweet?
- [38:33]
D
Yeah, it was an encyclical subtweet at Peter Thiel. But there's, you know, clearly, I think this lack of understanding of the Chinese system. Right. And, you know, a lot of that's due to, like, the sort of complicated relationship the Vatican has with China. So I'm, you know, I'm. I'm kind of pessimistic or I'm kind of, you know, not so optimistic about what this encyclical can mean for impact on, like, China. Or I've, you know, even people talking about, oh, the Vatican could be a mediator on the. Between the US And China in this arms race. But, yeah, like, where is that stature coming from? I don't know. Clearly, the church has to have some sort of impact on China if they want this to go well, and I don't see the line of impact there.
- [39:14]
C
Well, China wants to have economic access and diplomatic Power in the Global South. And to the extent that Pope Leo can convene the Global south and act as a voice for the Global south, that can, at least on the margins, shift Beijing's incentives around how they approach AI and AGI diplomacy.
- [39:38]
A
Tim, do you have a view on AI's impact on, you know, helping authoritarian governments, like, stay in power?
- [39:53]
B
Just that I think, like, technology is in some ways a little bit of a sideshow. And maybe that's like a little controversial to say. I think it's wrong to believe the vision that authoritarian regimes can use technology to retain control is assuredly true, but it is also just assuredly true of technology in general. So I don't know. I guess I've never found the argument particularly compelling that there's some necessary gradient that makes the technology more or less authoritarian, if that makes sense. It kind of is nonsensical to talk about whether or not social media has a strong democratic or authoritarian direction in the same way that it's kind of nonsensical to talk about whether AI has a strong democratizing or authoritarian direction because it turns so much on these aspects of design. Right. Like when we say social media, is social media by necessity something that contains a large scale content recommendation algorithm? Like, I guess as it's been instantiated, but maybe not really within its category. Like, I just think it's like you end up in detail and, you know, I used to work at Substack. I think one of the funniest things about Substack, I like it, it's a great platform. But it's like it started out as like, everybody's going to have their own email platform. And then after a while they were like, it's really hard to find for people to find content on Substack. What if we created an algorithm that recommended content to you and so you can like, start on completely the other place and you kind of felt you like, get dragged towards these attractor states for the technology. So I don't know, I think it's just like, it's. It's hard for me to reconcile.
- [41:21]
C
Yeah, I thought it was very powerful in the encyclical where although Pope Leo does heap a fair amount of criticism on tech CEOs who are creating these manipulative algorithms, he does not let all of us, the users, off the hook. He's very clear that we have a shared responsibility to promote a healthy climate in the digital world. He describes cyberspace as a battlefield. And he's absolutely right about that. And in a Catholic sense, the Recommendation algorithms that serve these often toxic purposes are actually neutral in intent. They're just trying to maximize ad dollars. But in so doing, they reflect our own vices back to us. When we are wrathful, the algorithm learns to show us more rage bait. When we guzzle flattering lies, the algorithm muscles truth off our screens in favor of propaganda. So I think it's really important that Pope Leo took the stand that he did in this encyclical, calling all of us to that shared obligation and not letting us act as though this is purely manipulation by the billionaire class.
- [42:44]
B
Yeah, I agree. I like that as well.
- [42:46]
A
Let's go around with some closing thoughts. Tim.
- [42:50]
B
Yeah, I guess the final maybe thought, while it's easy to get distracted by all the social media discussion, I think if you're a Christian or a Catholic, I think the best way to try to figure out whether or not the encyclical is having effect is just like, you know, in your community. And I think that's kind of like what I would kind of focus our attention on because, you know, Twitter will be onto its next thing, you know, next week, so Springboard island.
- [43:14]
D
Akab I think the most interesting aspect of this is the Vatican sort of acting as a proxy for nations that maybe don't have enough like social capital to be at the bargaining table, particularly just like these Global south countries. So I'm very interested to see if they will have more of a voice going forward or if it will sort of just go back to the same.
- [43:36]
C
I think this encyclical is a very strong start, but it's best suited as an opening to the conversation, not as its final word. If AGI really is coming soon, then Magnifica Humanitas would not be a suitable final word on this issue. It clearly envisions a world that is not impacted in the way I expect AGI will impact the world. So I'm hopeful that as Pope Leo and the Church engage more deeply with these issues, we may see an AGI focused encyclical in the coming years. I also found it very striking that at the presentation ceremony, Cardinal Czerny, who is one of the lead theological advisors on the encyclical, said during its public presentation, quote, A related question much debated today is whether and in what sense we can speak of consciousness or conscience in relation to the most advanced artificial intelligence systems. It is a serious question, one that deserves attention and further study. And for the Church, that's huge. Explicitly just feet away from the Pope opening the door to AI consciousness and moral patienthood. Nobody fainted. The Swiss Guards didn't skewer him with their halberds and drag him out of the room. Those remarks would have been circulated internally in advance, and if Pope Leo didn't want that signal to be sent, it very probably wouldn't have been so. Although I was hoping that Pope Leo would include the further study line in the encyclical itself, that's kind of the next best thing that we got there during the presentation.
- [45:27]
A
All right, stay tuned for more AI religion content here on man, we really got to rename this soon. China Talk. Thank you, Tim. Thank you, Akiv. Thank you, John Clark.
- [45:41]
D
It's been a pleasure.
- [45:43]
E
The civilization of love will not arise from a single spectacular gesture, but from the sum total of small and steadfast acts of fidelity that serve as a bulwark against dehumanization. The primary choice is not between a yes or no to technology, but between constructing Babel or rebuilding Jerusalem, between a power that claims to dominate the heavens and a people who work together to rebuild the walls of fraternal coexistence. Technology has the power to heal, connect, educate, and protect the our common home. But it can also divide, exclude, and generate new forms of injustice. In practice, technology is never neutral because it takes on the characteristics of those who devise, finance, regulate, and use it. Today, the main drivers of development are private, often transnational parties endowed with resources and the capacity to intervene that surpass those of many governments. Tech Tech technological power thus takes on an unprecedented, predominantly private aspect. Current AI systems are more cultivated than built. Developers do not directly design every detail, but to create a framework within which the intelligence grows. As a result, fundamental scientific aspects, internal representations, and computational processes remain unknown. So called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature the relationships and and do not know from within what love, work, friendship, or responsibility mean. Nor do they have a moral conscience. They may imitate language, behavior, and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce. Artificial limitation of care or support can become particularly risky when it enters context where real relationships and emotional bonds are lacking. The danger is not so much that a person may believe they are communicating with another person. They may gradually lose the very desire to form genuine human connections. The the algorithm is not neutral. The use of AI is never purely technical. When it enters processes that affect people's lives, it touches on rights, opportunity, status, and freedom. Important decisions concerning employment, credit, public services, or reputation risk being fully delegated to automated systems that do not know common compassion, mercy, forgiveness. So the hope that people are able to change. Entrusting an algorithm with the power to select who is worthy, and not without anyone bearing responsibility for that judgment, is to hand over the task of redefining the boundaries of human possibilities. In this process, political responsibilities lost. We cannot be satisfied with merely calling for the alignment of AI with human values without openly discussing the ethical frameworks involved. A more modern moral AI is not enough if that morality is determined by a few. To speak of the common good means exposing this new form of epistemic economic and political asymmetry. In naming the new monopolies of AI to speak of, solidarity obliges us to recognize the hidden, often exploited workers who sustain algorithmic systems. To speak of justice requires questioning the global distribution of power that decides who can train these models and who is merely subjected to death. Disarming AI means freeing it from the mentality of armed competition, not only military, but economic and cognitive. A race for ever more powerful algorithms and larger data sets, driven by the desire for geopolitical or commercial dominance. To disarm does not mean rejecting technology, but preventing it from dominating humanity. It means freeing technology from monopolistic control and restoring it to the parent of human cultures and ways of life. AI is driving profound changes in public and political communication. Disinformation did not begin with AI, yet today it finds a powerful amplifier in AI. The ability to manipulate content, images and videos exposes people to biased or misleading perspectives. Democracy does not consist of rules and procedures alone, but of a solid concordance with the facts and a genuine commit to the good of individual in society. Indifference to truth leads slowly but surely to a descent into totalitarianism. It is not permissible to the wind trust lethal or irreversible decisions to artificial systems. No algorithm can make war morally acceptable. AI does not remove the intrinsic inhumanity of conflict. It can only bring conflict about more quickly and render it more impersonal. The decision to use lethal force cannot be delegated to OPAC or automated processes, but must remain under effective, self aware and responsible human control.