← 80,000 Hours Podcast
80,000 Hours Podcast
Andreas Mogensen on what we owe 'philosophical Vulcans' and unconscious AIs
02:37:15
Loading summary
Transcript235 lines
- [00:03]
Sujana
When you're met with a situation where you have to, like, choose between one horrible conclusion and another horrible conclusion, is that a sign that we're just totally on the wrong track?
- [00:13]
Andreas Mogensen
I'm inclined to think it's not an indication that we're on the wrong track, it's an indication that we're doing philosophy.
- [00:19]
Sujana
Okay, yeah.
- [00:20]
Andreas Mogensen
Because I think the core of philosophy consists of puzzles and problems that arise when a number of things that all individually seem extremely plausible turn out to yield absurd results. There's this quote from Bertrand Russell that I think I'm probably going to butcher, but it's like the job of philosophy is to start with something so obvious it doesn't need saying, and to end up with something so incredible that no one could believe it, and these, like, deep conflicts amongst principles that otherwise strike us as compelling, that's a sign you're doing philosophy.
- [00:51]
Sujana
So we're doing something right? Maybe.
- [00:52]
Andreas Mogensen
Possibly, yes. Hey listeners.
- [00:56]
Raph
Raph here Today I have the pleasure of introducing you to a new interviewer on the show, Zashani Qureshi. Zashani has been a researcher writer at 80,000 hours for a while now, recently writing articles on the challenge of using AI to quickly enhance societal level decision making and reasons AGI might still be decades away. Back in the day, she studied mathematics and philosophy at Oxford University, which is just as well, because for her podcasting debut she has opted to tackle some pretty challenging philosophy. In today's conversation, she speaks with Andreas Mohansson, a senior researcher at Oxford University focused on moral and political philosophy. Had a lot of episodes over the years covering the possibility that AIs could in the future become conscious and deserve moral consideration because there's something that it's like to be them. Among others, we've had Kyle Fish, the first ever model welfare officer at Anthropic, on the show a couple of months ago, and there's a detailed treatment back in 2023 in episode 146, Robert Long on why large language models like GPT probably aren't conscious, but everyone's kind of heard that idea. Now, people might Disagree about whether AIs are going to have subjective feelings in practice, but sure, if AIs can suffer, if it would be better if we didn't make them suffer, that makes sense to most people, I think. So what is a philosopher to do? Or a podcaster for that matter, if we want to say something new and important about whether AIs might deserve moral consideration in future? Well, Andreas has a bunch of new and serious arguments that AIs could start to deserve moral consideration for their own sake, even if they're not conscious and there's nothing it's like to be them. These are arguments that people should seriously care about whether you are free and whether you get what you want. Even if you were to experience nothing at all or, or if you did experience something, it was always neither positive nor negative. It's fair to say that that would be big and indeed highly inconvenient if true. And that is why Andreas is researching it. If we're all completely fixated on subjective experience, but that's not the only way that beings could matter, then we're vulnerable to a catastrophic moral error. In the final third of Sushana's conversation with Andreas, they also turn to new philosophical arguments regarding whether we should weigh suffering much more highly than well being and whether human extinction might actually be a good rather than a bad. I know the more complex cutting edge philosophy episodes like this are often subscriber favorites. And if you do enjoy it, you might also like my interview with Andreas, episode 137 from back in 2022 about whether effective altruism is attractive or not to non consequentialists such as Andreas himself. Without further ado, I bring you Zoshana Qureshi and Andreas Molinson.
- [03:21]
Sujana
Andreas, it's great to have you back on the show.
- [03:24]
Andreas Mogensen
Thanks so much, Sujani. Thanks for having me back.
- [03:27]
Sujana
So you've been doing a lot of research into the relationships between being conscious, having a capacity for welfare, and having what we call moral standing or moral status. Before we actually get into these ideas, can you say what motivated this research and like, what's actually at stake with these questions?
- [03:50]
Andreas Mogensen
Yeah, so I think in some sense questions about moral standing or moral status, like what it takes to matter morally for your own sake, these are sort of the questions that made me a moral philosopher. So when I was in my first year as an undergraduate, I sort of read slightly by accident, a paper by Peter Singer called All Animals Are Equal, where he argues very roughly that the interests of human beings don't actually matter more than the interests of non human animals. And I find this both a persuasive argument and I was also kind of like, I found it remarkable that such an argument could exist, that there was a philosophical argument that was rigorously argued, but had these radical practical implications. So all that got me really interested in thinking about moral status, moral standing, that sort of thing. But the kind of research I've been doing recently hasn't really been focused on non human Animals, it's been more about AI systems. So I guess there's this sort of thought that given the extraordinary pace of AI progress, we might relatively soon be creating AI systems that might have morally significant mental states of one kind or another. And if we do, they might have quite weird minds. So they might occupy strange, unfamiliar regions of the space of possible minds. So maybe they have conscious experiences, but they never feel good or bad. So they might be like literally emotionless robots, or they might have very sophisticated cognitive abilities, like very sophisticated forms of agency, but they've got no conscious experience at all. And you might think we're already there. So it's very hard to know exactly what we should make of minds with these quite unfamiliar profiles. And at the same time the stakes might be really high because software is kind of easy to copy at scale, so there might be loads and loads of these. And so figuring out what kind of moral standing, moral status they might have could be very important.
- [05:42]
Sujana
Yeah, basically we kind of want to avoid committing some huge moral atrocity by failing to really properly consider whether these AIs have interests worth respecting.
- [05:51]
Andreas Mogensen
Yes.
- [05:52]
Sujana
Okay, so I want to jump into something that I find especially puzzling in your work. So intuitively I think I should only really care about beings who have their own point of view, where there's something it's like to be that being. I think that lots of people share this intuition and I think that's why so many of our debates about whether AIs or non human animals deserve moral consideration tend to be about the question of like, do they have what people call phenomenal consciousness, like a subjective experience of the world? But you think that this kind of focus might be a mistake. Can you just very briefly outline your main argument for thinking that and then we'll dig in a bit more.
- [06:44]
Andreas Mogensen
Yeah, so I think what I'll try to do is sketch. It's kind of an argument that I don't quite buy in the somewhat bare bones form I'm going to give it. But it's kind of the argument that got me interested in this topic. And I think there's like a version of this argument that I find kind of plausible. So we start from the idea that anything which can be harmed or benefited is worthy of some degree of moral concern. The next question then is what does it take to be harmed or benefited? There's not much agreement about that among philosophers, but one very influential theory of what it is to be harmed or benefited is a kind of desire fulfillment or preference satisfaction theory. Roughly, this theory says Your life goes better as opposed to going worse, insofar as your desires or preferences are satisfied, as opposed to frustrated. So insofar as things are the way you want them to be, as opposed to being a different way altogether. So maybe I don't want people to say horrible things about me behind my back, and if so, my life goes better overall insofar as things like that are not being said about me behind my back. Now, if you put those ideas together, it seems like the main question now is, can something have desires or preferences, even if it lacks phenomenal consciousness? Because if it has desires or preferences, it can be benefited. And anything that can be benefited is like a being worthy of moral concern. And I think that question is sort of an open question, but it does at least seem kind of plausible that there are desires that don't enter into our conscious experience, at least not at every moment that we have them. So, like, I do indeed want my wife to be happy, but that's not a desire that, like, figures constantly in my conscious experience throughout all the time that I have it. It's a desire that I have even when it doesn't enter into my subjective experience or something like that. So, yeah, certainly, if it's an open question whether desires can be conscious or can be had by beings that lack consciousness altogether, and anything which has desires can be benefited or harmed, and anything which can be benefited or harmed is worthy of some degree of moral concern, then we end up with a picture where it's at the very least an open question, and there's some reason to think that there might be beings of a kind that matter morally for their own sake, but that don't have subjective experience.
- [08:59]
Sujana
Hey, listeners, Seshane here. Let me just recap what Andreas is arguing here, just so that everyone can stay with us. So we started off with this question of what does it take for a being to warrant moral consideration for its own sake? Now, the most common answer that people give here is that you need phenomenal consciousness. Basically, there has to be something that it's like to be that being. But here Andreas is questioning whether that's actually necessary. His argument runs through the concept of welfare. So the idea is that if something can be benefited or harmed, then it seems to deserve some moral concern from us. And one of the most influential theories of welfare says you're harmed or benefited when your desires are fulfilled or not fulfilled, basically. So the question then becomes, can something have desires without being conscious? If the answer is yes, or even like maybe yes, then we might have moral Obligations to beings that just have no inner experience at all because they do just desire things. And that would be a radical conclusion, at least it would be to me. But whether having desires can actually happen without having conscious experience really depends on what we think desires are. And that's where we're going to go next. I think I want to start off talking a bit more about this idea of desire. So we're talking about here some sort of conception of desire which plausibly could be unconscious. I think for this to make sense, I think we need to kind of unpick a bit like what we mean by desire in this context. I understand that there are lots of different ways of defining this. What's your preferred definition of desire for this argument? And why?
- [11:13]
Andreas Mogensen
So the conception of desire that I think makes it most easy to attribute desires to beings that lack phenomenal consciousness is a kind of purely behavioural or purely motivational conception of desire. And this is actually like a conception of desire that's very popular, or at least has been very popular among analytic philosophers. So it's just a view which says to desire something is something like to be motivated to perform actions that you believe will bring about that thing, the thing that you desire, or something like that. And so that makes it really easy to count as having desires. And it doesn't seem like phenomenal consciousness enters into it anywhere at all. So that makes it very plausible that you don't need desire to have phenomenal consciousness.
- [11:53]
Sujana
Yeah, so that's quite a weak condition because I'm just thinking about that and I'm like, lots of things could have that kind of desire, like not just AIs, but a corporation could have that kind of desire. Right. Is that true?
- [12:06]
Andreas Mogensen
I mean, it's debated, but yeah, I mean, it's certainly, I think, a kind of a mainstream view among people who work on collective agency or corporate agency, that when we talk about Microsoft wanting to achieve something, or Google thinking that such and such is the case in the market, we speak literally and truly. So, yeah, this would be a case where it would be literally true that a corporation has a preference or a goal or a desire to achieve some outcome.
- [12:36]
Sujana
Right. So I'm guessing that your preferred conception of desire is a bit of a stricter condition, is that right?
- [12:44]
Andreas Mogensen
Yeah. So one thing to note about this purely behavioural conception of desire is it makes it impossible for people to intentionally and voluntarily do anything they don't want to do. But we at least sometimes speak as if that is possible. So there might be some long, boring meeting that you have to go to. And you might say, I don't want to go to this meeting, but I have to. But that doesn't really make sense on this purely behavioral or motivational conception of desire, presuming that you sort of do intentionally and voluntarily go to the meeting as opposed to somebody forcibly moving you. So there's got to be some other conception of want or desire in play. And I think a very plausible idea here is that to desire something or to want something in this sense is to have some kind of positive emotion or some kind of other affective state directed toward the object of desire. So what's going on in the meeting case is that although you are motivated to go to the meeting, there is no positive emotion in you associated with attending that meeting. And yeah, I think it's very plausible that it's these emotions that are backed up by. Sorry, desires that are backed up by positive emotions. Those are the ones that really matter insofar as we care about how people's lives go.
- [13:59]
Sujana
Yeah, yeah. So like the thing that my welfare consists in is not like me going to this really boring meeting that I'm obligated to go to, but it lies more in like me getting to skip the meeting to do something I actually really want to do. Like that seems like the morally relevant thing in that example, right?
- [14:20]
Andreas Mogensen
Yeah, that's right. So I don't know if it would count as altruistic, but in some sense you're sacrificing your self interest in the meeting case.
- [14:28]
Sujana
Yeah, I often find that.
- [14:29]
Andreas Mogensen
Yeah. But yeah, I mean the weird thing is on a purely behavioural conception of desire, it seems like in this case you're doing what you most want to do. Right?
- [14:40]
Sujana
Yeah.
- [14:40]
Andreas Mogensen
Your strongest preference is to go to the meeting because you actually end up going and you aren't literally forced to go. So we can't really account for that if we invoke a merely behavioral or motivational conception of desire and have a desire satisfaction theory of welfare. But we can explain what's going on. If we instead evoke this emotional or affect laden conception of welfare, then we can easily explain why this is a case of self sacrifice. And in general we can account for cases of self sacrifice. They're like ones where you do something but you don't feel good about it or something like that. You don't have any positive emotion directed toward the thing that you end up doing.
- [15:19]
Sujana
Yeah, okay, that makes sense. So I think we need to talk a bit more about these emotions that kind of ground what we're calling desires. So that sounds to me like you're pointing towards, like, in order to have a desire of this kind of this morally relevant kind, you need to have some kind of, like, psychological state, like the right kinds of psychological states. Can you tell me a bit more about what kind of state that needs to be and what kinds of beings can be in those states?
- [15:55]
Andreas Mogensen
Right. So I think the most sort of general way to classify the kind of mental state I have in mind is an affective state. So affect is like this bit of jargon in psychology, and basically it refers to a class of psychological states of which emotion is the central instance. But there are other kind of mental states that are in this class of affective states that you might think aren't emotions. So one example might be moods. So like a feeling of depression or anxiety. Maybe you think moods aren't emotions, but psychologists would nonetheless think of moods as being affective states. They're in the same class as emotions. Similarly, they would include things like itches and pains. So you might think those aren't emotions either. Nor are they moods. But nonetheless, they sort of go together with emotions and moods. These are all mental states that have a certain kind of heat to them or something like that. And so the thought is these states belong together under this umbrella term affect. Exactly what it is that they all have in common and sets them apart from other mental states is sort of an open scientific question. It's like we're sort of pointing at this thing that we think of as belonging together and forming a natural kind. But exactly what it is they all share and why they belong together is something we're still trying to figure, I think. Now the question of what kind of beings can have these affective states is also very much like an open scientific question. So the sort of issues that are really at the frontier here are, say, questions about whether there could be emotions or pains or other things. In invertebrates, especially in insects, there's been lots of debate about this. So there's some evidence suggestive of things like states of fear or feelings of fear in flies, for example. But, yeah, I mean, it's still very much an open question exactly what the sort of boundaries of this concept are in terms of, like, which animals does it apply to and which does it not apply to.
- [17:56]
Sujana
Okay. A big part of your view in the paper that you have about desire fulfillment and consciousness is that these kinds of positive affective states, these sort of positive emotions grounding desire, might not always require phenomenal consciousness. So they might not require you to have A subjective experience of the world. To have those kinds of emotions, that feels kind of surprising to me because I guess I kind of treat these things synonymously, like having emotions and being. Having good or bad conscious experiences or something. I kind of view those as the same thing. So why might someone think that these two things come apart and that you can have emotions without consciousness?
- [18:44]
Andreas Mogensen
Yeah, so, I mean, I definitely think a lot of people have that kind of intuition. But I do think it's actually a lot more of an open question whether the kind of states which are consciously experienced when we're having these, like, you know, the paradigmatic cases where we're feeling afraid or sad or whatever, could those states also occur without conscious experience? So I think there are a number of different kinds of cases that could be used to illustrate this, which some of, like, I don't think any of these is like a slam dunk. So I don't think there's any definitive proof that there can be unconscious emotions or other affective states. But nonetheless, I think there are lots of cases that suggest to me, at least, that we should be quite uncertain, we should be open to this possibility. So I can walk through a couple of these if that would help.
- [19:30]
Sujana
Yeah, that sounds great.
- [19:32]
Andreas Mogensen
So one sort of case we might try is you might say, have someone say that they've been angry with their sister for years or something like that, or that they've been afraid of spiders all their life or something like that. Now, if you consider the kind of person of whom you might say that they've been angry with their sister for years, probably you're not imagining that at every waking moment throughout their life they were having, like a feeling of anger. Right. Like this anger is something that's sort of been lurking beneath the surface a lot of the time, you might think. And if that's what's going on, then it seems like we have a case where someone's angry throughout a period of time, but they don't have feelings of anger throughout all that time. So there are at least some times in their life where they have an emotion, but they have no conscious experience of emotion throughout that time. So that might seem to be a case of an unconscious emotion.
- [20:24]
Sujana
Right.
- [20:25]
Andreas Mogensen
I think there are things you can say against this interpretation of what's going on. So one thing you might say that I'm somewhat sympathetic to is that actually what's going on here is when we say the person is angry throughout all this time, we're not speaking literally or truly. Anger in this case is not an emotion. Per se, but a disposition or a propensity to exhibit this emotion. So someone who's angry with their sister is not literally angry at all these points in time, but has a robust disposition to feel anger when they confront their sister or something like that. Maybe that's what's really going on in these cases. So they aren't really cases of unconscious emotion.
- [21:06]
Sujana
Is it right to say that there's sort of no consensus here about whether you are just experiencing that emotion unconsciously or it's something else going on?
- [21:17]
Andreas Mogensen
I'm not sure about this particular sort of case. I would kind of expect that many people would find this sort of alternative diagnosis that I've given quite plausible. That this is just a case of having a disposition to exhibit an emotion and not an emotion per se. And some emotion. Researchers use the term sentiment to refer to this kind of state. They distinguish sentiments from emotions, where emotions are these things that are actually happening in your mind and sentiments are dispositions to exhibit those kind of states. I do think there are other cases that might divide opinions quite a bit more. So sometimes when we're in the grip of very strong emotion, you might think that our attention is actually focused very narrowly elsewhere. So, for example, if you're driving in a car at 60 miles per hour and suddenly the steering wheel grips out. Sorry, the steering wheel gives out. On the one hand, you might be absolutely terrified during this experience as you try to stop the car or whatever.
- [22:19]
Sujana
Yeah.
- [22:19]
Andreas Mogensen
But in addition, we might think your attention is going to be absolutely focused on stopping the car. It's like everything else that's going on is going to be pushed to the side. And you might indeed think that you would have no conscious experience of these feelings of fear because your attention would be so narrowly zeroed in on bringing the car to a stop, to a safe stop or something. And it might be that you only sort of realize how utterly terrified you were during this ordeal once it's over. And maybe you look down and you've dug your nails into the steering wheel or something like that. Maybe you realize that you are screaming and have been screaming throughout all this time, but it's kind of surprising or news to you or something like that. I mean, this sort of case, as I say, it's very difficult to know exactly how to explain what's going on. And it raises these difficult questions about what's going on with unattended stimuli. Are they hovering on the edge of consciousness? Are they definitely not conscious? It's very hard to say exactly. So I think this kind of case might sort of divide opinions and maybe even expert opinions a bit more than the other one.
- [23:24]
Sujana
Yeah, I see that. I think the place that I would kind of resist some of these examples, I think while we've been talking, I've been wondering, are there, in fact, two questions here? So one question is, can you, a being who is typically conscious, have some emotional states that you're not aware of that are unconscious to you? And a second question where you're like, can a being that is not conscious and not even potentially conscious have emotional states, have emotions like fear and so forth? From the examples you've given, I feel like the answer to that first question seems like it might be a yes. But the second question, which seems like the thing that's most relevant in the case of determining can something that isn't conscious be a moral patient? It's less clear to me what the answer to that question is. Do you have a response to that?
- [24:20]
Andreas Mogensen
As I said, all of this is very confusing. It's hard to know what to think. And it definitely seems like if emotions can occur outside of conscious experience, why should it be impossible for them to occur without being even potentially conscious or something like that? So, I mean, this does link to really big issues in the philosophy of mind. There are some philosophers who think that in some sense, you cannot have any mental states at all unless you have the capacity for phenomenal consciousness. Because, roughly, any mental state or any kind of state that purports to represent the world out there as being a certain way can only occur if it's potentially conscious in some sense or other. In general, I don't find views like that especially persuasive. My sense is. I mean, I'm not totally sure of this, but my sense is that they're not especially popular amongst philosophers of mind and cognitive science in general. But there are definitely these bigger questions. So you're absolutely right that the examples, you could take them at face value and still claim that there couldn't be emotions in beings who lacked any capacity for phenomenal consciousness.
- [25:25]
Sujana
Yeah, I think where I land with this is a lot of the things that I've been saying, you know, that like, surely you have to be conscious in order to actually be able to experience these emotions, whether consciously or unconscious. And things like this, these are just kind of gut instincts. So there's kind of assumptions. And what I'm realizing through this conversation is that there's a lot of. There's a lot up for grabs. There's a lot we don't really know. There's a lot of, like, difficult questions. And, yeah, maybe we shouldn't take it for a given that the thing that's important for deciding whether something's a moral or patient is, or in fact whether it can have desires, for example, is something to do with it being conscious.
- [26:11]
Andreas Mogensen
I mean, one thing I was going to say is also I do think one thing you might acknowledge is that there are unconscious emotions. Maybe there could be emotions in beings who lack phenomenal consciousness altogether, but you might think it's only consciously experienced emotions that matter, morally speaking, or something like that. So I think that's definitely also a further question.
- [26:33]
Sujana
Hey, listeners, it's Seshanee again. We've gone a couple of layers deeper here, so it's time for a recap. We started off here by asking, can something have desires without being conscious? And Andreas's answer was that the kind of desires that matter for welfare aren't just descriptions of behaviors. They're backed by emotions or affective states. Now, these terms are quite muddy, but we're thinking here of things like fear and joy and pain. Andreas described them all as states with a certain kind of heat to them. So then we asked, okay, can something have emotions or affective states without being conscious? And Andreas gave a few examples suggesting that maybe, yeah, they can. For example, he points to how you can be so terrified during a car crash but not even notice that you were having that emotion until after the fact when you see how you've dug your nails into a steering wheel. But I pushed back. Those are cases of a conscious being having emotions that they're not aware of in the moment. Surely it's a higher bar for a being with no capacity for consciousness at all to have emotions. And Andreas just thought we should be genuinely unsure about that. So what does all this mean for the case of AI? That's where we turn next. So I want to turn to the implications for AI. The big question is if this kind of understanding of desire as central to whether something can be benefited or harmed, if that's correct, what does it mean for the question of whether AIs are moral patients?
- [28:42]
Andreas Mogensen
Yeah, so I think if you had this very thin behavioral and motivational conception of desire, then it's probably quite plausible that there are certain current AI systems that exhibit desires understood in that sense. So, for example, Cameron Cut Giannini and Simon Goldstein have a paper where they argue that certain kinds of agents that are scaffolded atop large language models, certain kinds of generative agents, should be thought of as having desires and Beliefs according to these sort of thinner, more behavioral conceptions of desire and belief. And as a result, they think there's a decent case for supposing that beings of this kind can in fact be benefited and harmed in a way that maybe should matter to us, morally speaking. By contrast, if you have this kind of like affect laden conception of desire where desires have to be sort of like, have to embody or be backed by positive or negative emotions, for that matter, the kind of view that I've been sketching, then it seems pretty implausible or certainly much less plausible that these kinds of systems exhibit desires of a kind whose satisfaction or frustration makes their lives go better or worse. And indeed, I think there's potentially a kind of in principle reason to think that systems like most current large language models couldn't in principle have emotions. So very roughly, the worry here arises in something like the following way. These large language models, I mean, there are some applications in robotics, but by and large they are disembodied systems, right? They can't move any kind of like physical hardware around in space intentionally. They don't monitor the internal states of the hardware on which they're run or something like that. I think if you actually prompt, I think Claude, to think about itself, you sort of see activation for features that are associated with disembodied spirits, like ghosts and things like that. So these are sort of not embodied creatures. They sort of have no body of their own, nor even the illusion of a body or anything like that. But there are many theories of emotion that postulate a very tight relationship between emotions and our awareness of our own bodily changes. So most famously, there's this theory due to the American psychologist William James, where emotions just are experiences of bodily changes. So if emotions essentially require an experience of embodiment, and LLMs are disembodied systems, then it would seem to follow that these disembodied AI systems couldn't possibly have emotions. Now, whether or not you buy that argument will depend on lots of things and your attitudes toward lots of controversial questions. So this is actually an issue on which I'm working. I've got a draft paper with a researcher in Germany called Leonard Dung that's exactly on this issue.
- [31:47]
Sujana
Yeah. Okay, so what if you gave an AI some kind of sophisticated robot body? Do people think that that would be enough to kind of simulate the physiological emotion states or. Yeah. What do you make of that?
- [32:07]
Andreas Mogensen
I mean, yeah, it becomes a very difficult question exactly what to know about such cases. But it definitely. Yeah, I mean, if, in fact, the experience of bodily and physiological changes and events is a core part of ordinary emotional experience for us, and if you gave a robot a body whose internal condition it monitored in something like the way that we monitor the internal condition of our body, then relative to these views on what emotions are, that would certainly be a much stronger candidate for being an AI system of a kind that felt genuine emotion. I think this issue is discussed at some point in Michael Graziano's book Rethinking Consciousness. He talks about exactly this issue of what would it take for an AI system to have emotion? And he sort of leans towards this view that you would need to give it some kind of body that it would monitor.
- [33:03]
Sujana
Yeah. Okay, so imagine now that we had an AI that we were pretty convinced had emotions and desires. Let's suppose Claude is not conscious, but we think that he has emotions and desires. What should I do to avoid giving Claude negative emotions? Or should I be following Claude's desires when I'm having a conversation with him?
- [33:37]
Andreas Mogensen
So in some sense, it might be true that you should be nice to Claude and that there is some reason to do things that Claude. Or to make things happen that Claude wants to happen, to avoid doing things that it's averse to happening or something like that. But it might still be true that because it lacks phenomenal consciousness, its interests don't matter very much as weighed against beings that have phenomenal consciousness. So that's definitely kind of a possible view. I think it is a view that I find that I do find quite intuitive. I'm a little bit unsure whether to believe it, but, yeah, I definitely think that's a possible view.
- [34:16]
Sujana
Okay, so through all of this conversation, we've been saying that plausibly phenomenal consciousness might not actually be necessary for something being worthy of our moral consideration. But I haven't asked yet, is it enough? If we were able to determine that an AI was conscious, would that on its own be enough to oblige us to take its moral interests into consideration?
- [34:47]
Andreas Mogensen
Yeah, I mean, I think there might be two slightly separate questions here. So one is whether phenomenal consciousness on its own is enough for having some kind of moral standing. Because you might think, like, if there were an AI system, a current AI system that was phenomenally conscious probably would not only be phenomenally conscious. Right. It would be, like, phenomenally conscious and exhibiting quite plausibly, quite sophisticated cognitive capacities of a kind that maybe amongst animals, only human beings have and that most other animals lack, at least, or something like that. And I think those are potentially two quite different questions. So I think it's certainly. So consider the question of whether phenomenal consciousness on its own suffices to make you the sort of being that we ought to be morally concerned for. I think it's somewhat difficult to say, but my instinct is to say no, in that it seems like we can imagine these very simple creatures. So there might be some kind of organism that is bolted to the sea floor and it has some degree of conscious experience, but the only conscious experience it has is an impression of a slight brightness every now and then, or something like that. There are no thoughts, there are no emotions. Yeah. There's just this faint feeling of brightness that flits through its experiential field every now and then. And so there's this question like, does it matter morally what happens to this creature more so than, say, an otherwise exactly similar organism sitting next to it where the lights are totally off? And, I mean, it's very difficult to say exactly. But my sort of intuition, it certainly seems fine to say no. The fact that it's phenomenally conscious, that just doesn't actually suffice to give it any kind of claim on us. And I think there's also a question of what kind of claim would it even have on us. It doesn't feel pleasure or pain. It has no desires. It just has this experience of a faint brightness every now and then. So I'm inclined to think no in that sort of case. So in that sense, phenomenal consciousness does not suffice for moral standing or moral patience.
- [37:02]
Sujana
Right.
- [37:03]
Andreas Mogensen
So then there's a separate question, like, might phenomenal consciousness suffice for moral standing in conjunction with other mental states? And, I mean, certainly, like, if you put it like that, then the answer is, like, definitely yes, because those other mental states could be like emotions and feelings of pain. Like, one kind of question that's maybe more interesting is something like, if you had a creature that was phenomenally conscious, and it did have the kind of sophisticated cognitive abilities that you and I have, and that sort of characterize human beings as animals. And it was phenomenally conscious. But say it felt no emotions, never felt pleasure or pain, would it have moral standing? So this is a kind of very similar or more or less identical to this kind of philosophical Vulcan thought experiment that David Chalmers has come up with, which I think may have been debuted on the ATK podcast many years ago.
- [37:57]
Sujana
What a claim to fame.
- [37:59]
Andreas Mogensen
Yeah. So he sort of asked us to imagine what he calls, like, philosophical Vulcans. So these are supposed to be, like, sort of extreme versions of the Vulcans from Star Trek. Where I think the Vulcans from Star Trek do feel emotions and they suppress them. But these are, like beings that never feel any emotion. They never feel pleasure, nor pain, nor happiness, nor sorrow. Nonetheless, they're conscious beings, very much like you or me. They have, like, sophisticated intellectual and moral desires. And there's a question of whether they have moral standing. And Chalmers has the intuition that they do. Like, it would be wrong to, like, mow down hundreds or thousands of these Balkans. For the sake of providing some trivial benefit to some sentient. Some being that sentient in the sense of not only having phenomenal consciousness, but experiencing pleasure, pain, emotional feelings and that sort of thing. So surely it would be wrong to kill millions of these Vulcans. In order to give a hamster a juicy treat or something like that. And I definitely share that intuition. So I'm definitely inclined to think that beings of this kind, Vulcan like beings, would have some kind of moral standing. But it's not just because they're conscious of it.
- [39:08]
Sujana
Sounds like these Vulcans are not the kinds of beings that have capacity for welfare. Under the view that we've just been talking about. The one where being capable of being benefited or harmed. Is something to do with having desires that can be fulfilled or not fulfilled. Is there any other way in which they could be like any other route to being capable of welfare that you're imagining for them?
- [39:37]
Andreas Mogensen
Yeah, I think there are views that might seem to give you this result. Philosophers tend to think that there are three main classes of theories of well being. Or at least this is a very influential proposal due to Derek Parfitt. I think some things get left out of this tripartite distinction. But includes many of the main things. So on the one hand, there are these preference fulfillment theories. We've talked quite a lot about those. The other sort of key class, another key class of theories of well being. Are so called hedonist theories. So these are theories on which, roughly speaking, well being consists in feelings of pleasure. And the absence of unpleasant feelings or the absence of pain. So those are also not going to give you the result that Vulcans are capable of being benefited or harmed. But then if you look to the sort of third group. So these are often called objective list theories. Or simply objective theories of well being. So typically, these are views on which there isn't just one thing, one kind of thing that makes people's lives go better or worse. There are many different things that contribute to well being. And most of these things are things or at least some of them are things that would be good or bad for you, independent of whether you had any kind of positive attitude or negative attitude toward them. So, for example, often objective list theories will feature knowledge as a welfare good. Or knowledge of important truths might be considered a welfare good. So not just knowledge of people's phone numbers or something like that, but say, knowledge of the deep structure and nature of the universe, knowledge about the ultimate fate of the universe, knowledge about whether there exists a God and whether that God cares for us or something like that. They might think that these are deep items of knowledge such that it benefits an individual to have them. And you might think Vulcans could definitely have knowledge of that kind. And depending on exactly what you think knowledge is, you might think knowledge does not require phenomenal consciousness. So maybe you think corporations, say, know stuff, like they know how many widgets they're producing or something like that.
- [41:45]
Sujana
That deep truth of how many widgets.
- [41:48]
Andreas Mogensen
We are producing as a corporation. Yeah, yeah, I guess that's an interesting question. Do corporations have religious beliefs? I mean, some do. I guess the Catholic Church has religious beliefs, you might think. Yeah, anyway, Right. So you might think that these goods like knowledge, are ones you could have without having any kind of affective states and indeed without having phenomenal consciousness or something like that.
- [42:11]
Sujana
Yeah. Okay. And things like knowledge and stuff. That sounds like right up the alley of what AIs could plausibly have. So if you buy this, I'm guessing that it makes it a lot easier to buy that future AIs will be moral patients.
- [42:27]
Andreas Mogensen
Yeah, absolutely right. Yeah. You might already think that current AI systems know a whole lot or something.
- [42:33]
Sujana
Like that a lot more than I do.
- [42:35]
Andreas Mogensen
Yeah, I mean, certainly you could also doubt this. Maybe you think that they're sort of merely parroting various verbal behaviours. But at least taken at face value, they seem to know a heck of a lot, including some really important stuff.
- [42:48]
Sujana
Yeah, but do they know how many widgets this corporation has? That's the big question.
- [42:54]
Andreas Mogensen
I mean, for some corporation and some widgets, I'm sure they do. Or they can search the Internet and look it up and then they'll know that's true.
- [43:02]
Sujana
Gosh, they've got us there as well.
- [43:04]
Andreas Mogensen
Yeah, but so definitely say knowledge as a kind of welfare good. If you think that knowledge is a welfare good, then it should be plausible that Vulcans, maybe even corporations, maybe other sorts of non conscious systems can possess these welfare goods, which would make them better or worse off. That's a view you might have.
- [43:25]
Sujana
Hey, listeners it's sashanae again, if you can have emotions without conscious experience, as Andrea suspects, does that suggest that AIs will probably have emotions? So, not necessarily. There's this long tradition in psychology saying that emotions are tied in with bodily states like your racing heart and your clenched jaw. And on that kind of theory, AIs couldn't have emotions even in principle, unless they had bodily states that they could monitor. That means AIs might not meet the conditions for being worthy of our moral concern, even if being conscious is not actually one of those conditions. After discussing all of that, we tried flipping the argument around. So, sure, having phenomenal consciousness might not be necessary to warrant moral consideration, but would an AI being conscious at least be enough for our moral concern? And Andreas argued that that's not clear either. So, for example, for a very basic creature that's just living on the sea floor, merely being aware of the sun's brightness, and not much else, that doesn't feel like it gives much of a claim on us morally, if any. All of this discussion has been assuming that being a moral patient rests on being able to be benefited or harmed. But next, we consider a different possible route to moral patienthood altogether, which is having a capacity for autonomy. When people ask, should AIs be given moral consideration? The conversation often ends up being about this question of, well, do they have some capacity for welfare? Or in other words, are there. Are there ways that this AI could be benefited or harmed by my actions? But I think I want to understand whether an AI having this capacity for welfare is even really necessary for it being worthy of our moral concern. And you think that it's not. So you think there's maybe another route to moral patienthood that doesn't even involve being the kind of thing whose life can go better or worse. Can you explain what that root is?
- [45:58]
Andreas Mogensen
Yeah. So, I mean, very roughly, the thought is that something like autonomy or capacity for autonomy might suffice to make you a moral patient in the absence of any capacity to be benefited or harmed.
- [46:13]
Sujana
And when you talk about autonomy, what do you mean here?
- [46:19]
Andreas Mogensen
Good. It's a good question. So I guess one thing I should say is, well, maybe it doesn't need say, because it's true of all things in philosophy, but it's very controversial exactly what autonomy is.
- [46:29]
Sujana
Of course it is.
- [46:31]
Andreas Mogensen
So I might think, for example, that autonomy is a kind of psychological capacity for self government, for the ability to sort of direct your life based on rational reflection of a kind that human adults say possess to a much greater degree that Human children and non human animals or something like that. So that's roughly what it is that we're sort of pointing to. And then there's sort of questions about exactly what it is we're pointing to when we try to point to that thing. And yeah, so this is where it gets really sort of controversial, I guess, amongst philosophers. So yeah, I mean, there's this thought that generally speaking, in order to be autonomous, you need to have a capacity for rational reflection. You aren't simply led about blindly by impulses. I think very often this is conceived as involving a capacity for second order desires. So what this means is there are first order desires which are basically just desires for things out there in the world. Desires for chocolate cake or desires to win a football match or something like that. And then there are desires about desires. So these are second order desires. So I might desire that I don't want to eat the chocolate cake or something like that because maybe I want to diet or something like that.
- [47:54]
Sujana
Hey listeners, I just want to clear up that we're playing a bit fast and loose with definitions here. We previously mentioned that there are lots of different conceptions of desire. And earlier on in the conversation we were mostly talking about the type of desire that's backed by emotion. But we also mentioned versions of desire that are purely behavioral and they just don't involve emotions at all. At this point in the conversation, Andreas has just switched back to using the word desire in a non committal kind of way. So it could involve emotions, but it doesn't necessarily have to. So a being that has the second order desires that Andreas is describing here isn't necessarily capable of being benefited or harmed. Okay, back to the show.
- [48:48]
Andreas Mogensen
And so some philosophers, especially inspired by the work of Harry Frankfurt, think that what's really essential to autonomy is that you're able to formulate these second order desires, or even higher order desires through which you identify yourself with some of these first order desires you might have and you distance yourself from others.
- [49:11]
Sujana
Yeah. So that means like you choose to pursue or not pursue some of these first order desires in accordance with what your second order desire is.
- [49:20]
Andreas Mogensen
Absolutely. So a lot of people think that's really essential for autonomy or something that looks a little bit like that capacity. Other things that people often associate with autonomy is things like being minimally rational in some sense. So satisfying certain minimal criteria for having consistent beliefs and desires, for not just having a totally jumbled mind or something like that over and above just satisfying these formal coherence criteria. Some philosophers think to really count as autonomous. You need to be rational in a more substantive sense. So they think you need to genuinely value the things that genuinely are valuable or something like that. That's slightly more controversial, but I think it's a view that has quite a bit of purchase amongst philosophers who think about autonomy. Another thing that's often associated with autonomy is, is some kind of idea that to count as properly autonomous, you sort of need to have the right kind of history. So you sort of can't tell whether someone's autonomous just by looking at them as they are right now. And this sort of often goes off these like thought experiments where you can imagine that someone's been going through their life and then someone kind of like, I don't know, spreads some nanobots into their brain and rewires their neural circuitry such that they become a completely different person. Maybe they become a duplicate of somebody else or something like that. But they have all these kind of alien desires and preferences and values implanted within them, which they might then act in a rational and reflective way in order to realize. But there's a sense that this individual is not autonomous because of the way that these goals and values originated in them. So there's this thought that there needs to be the absence of a certain kind of manipulation in our past in order for us to count as genuinely autonomous. Exactly what needs to be absent is a matter of significant controversy. But a lot of people also buy this sort of view. So these are just like some sketches as to what it would take to count as autonomous. But as I said, everything is very controversial, I think, especially so when it comes to what it means to be autonomous.
- [51:32]
Sujana
Right. Yeah. I definitely want to talk about why it is meeting these criteria might make you worthy of moral consideration, even if you can't be benefited or harmed. But before we do that, I'd be curious to compare the conditions that you have presented against AI systems. This kind of conception of autonomy, the thing about sort of second order desires, reminds me a bit of people speculating that AI systems might soon become directed towards complex goals and strategy making, becoming sort of more complicated goal directed systems. So that kind of lines up with things that people are maybe already speculating about with AIs. One of the things you mentioned, this manipulation point when you talked about the example of someone doing this sort of complicated neural fiddling and that sort of not like the person who's had their brain fiddled with doesn't count as autonomous, does that kind of rule out AIs from meeting the criteria like you could think that we've manipulated AIs in basically that way.
- [52:42]
Andreas Mogensen
It's a good question. I think the honest answer is I don't really know. I think in large part because I think we as a community of philosophers or something like that, don't quite know what it means to be manipulated in the relevant sense. So I think you might think in some sense our psychological state, certainly if the universe is genuinely deterministic, then our psychological state will always have been determined by factors that occurred prior to our birth that were outside of our control. And so there definitely is a worry that many philosophers have articulated that these kind of intuitions about manipulation cases, if you take them to their logical conclusion, mean that there couldn't be genuine autonomy in a deterministic universe. And similarly, you might think, yeah, I mean, we do sculpt assuming that LLMs count as having preferences and values. Those preferences and values are sculpted through things like reinforcement learning from human feedback as and so forth. You might wonder, I mean, that might look a lot like manipulation, but how different is it from the ways in which we instill values in human children through ordinary parenting practices? Yeah, yeah, I feel like I sort of probably don't understand enough about how reinforcement learning from human feedback works to be able to say with much confidence exactly how different these processes are. But yeah, I think it's certainly a worry that the way in which the values and preferences of LLMs are sort of installed in them by us might very much raise this worry that they fall foul of this anti manipulation clause. But there is also this worry that we, if we really thought about it, would fall foul of this anti manipulation clause because our preferences and values are also sculpted by factors beyond our control.
- [54:37]
Sujana
Yeah. So you might think there's some sort of line somewhere where if you fall on one side of the line, you do count as autonomous and if you fall on the other side of the line, you don't count as autonomous. And maybe the thing at issue here is the extent to which you've been manipulated or something like that. Or you might just think that it is just like it is very much just a scale. And some beings are more, whatever this conception of autonomy is, some people are like further along the scale than others. I'm wondering, do you think that AI systems are kind of gradually becoming more autonomous along this scale?
- [55:18]
Andreas Mogensen
Yeah, it's a good question. I certainly think there are things that suggest that that's the case. So for example, you might think that the advent of so called reasoning models mean that sort of LLM systems Have more by way of a capacity what we would think of as a capacity for rational reflections. They seem to have this ability to engage a process of slow deliberative thoughts rather than answering roughly the first thing that comes into their mind. And maybe these sort of like reasoning capabilities mean they're able to revisit and revise earlier steps in the chain of thought. So this might look like something that involves more by way of the kind of rational reflection capability we think of as constitutive of autonomy. Another thing that might be relevant is something like the capacity for introspection. So especially if autonomy requires having these like second order desires, desires about what kind of desires you have, it requires something like the ability to step back from your beliefs and desires and to say whether what you believe is really true or it's just some kind of like belief that's gripped you, whether what you desire is really something you ought to desire or something like that. You might think all that presupposes a capacity for introspection. So we do have some evidence that language models are able to do something a little that looks like introspection, though I think they're somewhat unreliable at it. So there's like this recent paper from Jack Lindsey at Anthropic called Emergent Introspective Awareness in Large language models. And they sort of present some evidence that models are able to do something that looks like introspection. They do say that the abilities they observe are highly unreliable. I think another thing they do Observe is not 100% sure of this, but I think they're looking only because they're anthropic. They're looking only at instances of the Claude series of models. And if I remember correctly, it was the latest version they tested, which I think was Claude 4.1 opus possibly that did best in terms of introspective ability.
- [57:37]
Sujana
So I guess I now want to imagine we do get AIs in the future that are just much better at these things that you've listed, like the second order desires, the rational reflection, the introspection, and maybe also less manipulation by humans. Let's imagine we do get an AI that meets whatever criteria you put in place to say, yeah, we're comfortable calling this autonomous, this has autonomy. The question that I have then for you is why might I have moral obligations based on the fact that it has all of these qualities, Right?
- [58:17]
Andreas Mogensen
So in some sense this goes back to some pretty foundational questions in moral philosophy. So I think if you're a utilitarian, you will think that in Some sense, the only moral reasons we have are reasons to promote the welfare of others and ourselves. And so you might think if something has no welfare, then there's no moral reason to be concerned on its behalf. So I'm not a utilitarian, and I think if you take sort of common sense morality at face value, you will think that it's not a utilitarian moral system. So in particular, it does seem like the kind of obligations that we think of ourselves ordinarily as having toward other people don't reduce, certainly don't, obviously can't obviously be exhaustively explained in terms of an obligation to promote other people's welfare. Some of them look more like obligations to respect other people's autonomy. So an example might be norms against paternalism. So, very roughly, I mean, it's sort of contested, but paternalism is when you sort of intrude on another person in a way that's intended to benefit them. And we can even stipulate that these paternalistic actions might involve you knowing that you will in fact benefit this person, although you're sort of acting against their wishes and interfering with them or with their property or something like that. So, for example, maybe you're much better at financial stuff than I am, which would really not be surprising. So maybe you hack into my accounts and you make various investments on my behalf, and as a result I end up much better off materially, or something like that. Nonetheless, I think most of us would think that you'd acted wrongly. Even if you managed to do this secretly. I never really notice because I'm not very financially savvy, so I don't really check what's going on with my accounts. I just sort of notice one day that there's sort of more money than I would have expected or something like that. I think a lot of people would nonetheless think that you'd sort of acted wrongly in this sort of case. So there's a question of what explains why you acted wrongly in this sort of case. But I think a very natural answer that governs many of these cases of paternalistic action is, is that you have sort of interfered with my ability to exercise autonomous control over my own life, over my own person and the things that belong to me, or something like that. And so roughly, that's the source of the thought that maybe you don't need to be capable of being benefited or harmed in order to be capable of being wronged. Because it seems like ordinarily you can wrong someone even by virtue of benefiting them.
- [60:57]
Sujana
Yeah, okay, that's very important. Yeah. Because I think my intuition here, that was making the argument feel a bit weird was that like, well, look, if your life can't get better or worse, like nothing I can do to you is bad or good. That was like the kind of instinct that I had here, but then just kind of loosening that and thinking something that doesn't make my life go worse and in fact maybe makes my life go better, could still be bad for me.
- [61:29]
Andreas Mogensen
Yeah. And it might even be neutral. Like maybe you rearrange my investments in a way that makes me neither better nor worse off. Right.
- [61:34]
Sujana
Yeah.
- [61:35]
Andreas Mogensen
Still seems wrong.
- [61:36]
Sujana
Yeah, that makes sense. That makes sense. But I think where am I still resistant here? I think I'm with you when you say that maybe we shouldn't go all in on utilitarianism where you're saying the right action is just about maximizing welfare. And I think I'm with you when you say that there are other things that could count towards what makes an action good. And I think that the example you gave there is some kind of rule being broken about you shouldn't hack into my accounts and stuff like that.
- [62:10]
Andreas Mogensen
You really shouldn't.
- [62:11]
Sujana
Yeah, exactly. Please don't to all listeners, even if you're going to make me a lot of money. But yeah, I think even with these things agreed upon, I think I still feel unsure why this should have implications on who is deserving of good actions. So, okay, what do I mean here? I think maybe you could argue that, okay, respecting autonomy is morally important, but maybe it's only important when you're dealing with beings who can be harmed or benefited in the first place. Like I should respect your choices to make bad financial decisions because you are a being with a capacity for welfare and not just because you're able to make bad financial choices. Is there any credible view in that direction?
- [63:05]
Andreas Mogensen
I mean, maybe it depends what you mean by credible. I think, I mean. So in some sense you might think that say a kind of a utilitarian interpretation of something that looks like an obligation to respect autonomy could be cashed out in these terms. So you might think there's like no fundamental requirement to respect others autonomy, but maybe we ought to do so in general. Because although there might seem to be these cases where paternalistic action does make people better off, we're going to misjudge this so much of the time that it would be better if we followed this rule of thumb that says don't do this or something like that. So that's definitely an option. I think it's otherwise a little bit hard to come up with a plausible view that would give you this kind of results. One thing you might consider, and that I do sort of have thought a little bit about, is a view which says something like, well, ultimately when we say we should respect others autonomy, what we mean is we should respect their ability to live in accordance with their own conception of what the good life for them looks like, or something like that. And so at the very least, this requirement to respect people's autonomy must presuppose that people believe that they can be benefited or harmed. And if you also think something like that, autonomous subjects have got to have some kind of like sort of minimal epistemic competence. They've got to sort of know key important facts about what they're like and what their surroundings are like. That might entail maybe that any autonomous being who believes that they can be benefited or harmed probably is the sort of being that can be benefited or harmed.
- [64:45]
Sujana
Right, okay.
- [64:45]
Andreas Mogensen
Yeah. So if you have this conception of autonomy, of what it is to respect autonomy, where it just reduces to respecting other people's ability to live out their conception of what would make their life go best for them, maybe that could justify something like this. But I don't think that's an especially plausible view because I think the requirement to respect others autonomy extends to their sort of moral priorities, actions that they might undertake which they perceive as sacrificing their own welfare for the sake of some higher good or something like that. So not just actions that they perceive to be in their self interest. So for example, it seems like it's part of respecting people's autonomy that we respect their right to live in accordance with their moral convictions, even if they conceive of those moral convictions as worse for them but better for others. So here's an example. Suppose you have a housemate that is vegan. Intuitively, it would be wrong of you to add small morsels of meat to their vegan lentil stew. Like small morsels that are so small that they can't detect that this is going on. And that seems to be true whether or not they're like vegan for health reasons, they think this is in their best interest. Or if they're vegan for moral reasons, maybe they think this is not in their best interest, but they're doing it for the sake of animals.
- [66:05]
Sujana
So I think, where does this leave me? I think maybe there's some reason to feel here that because it doesn't matter whether the recipient of my action is actively benefited or harmed by my action. Maybe that means that, in fact, it doesn't even matter whether they could have been benefited or harmed by that action in the first place. My actions could still be bad or good in some way towards them.
- [66:35]
Andreas Mogensen
Yeah.
- [66:36]
Sujana
So we've been talking about the relationship between being autonomous and being capable of being benefited or harmed. Or, in other words, being a welfare subject. But there's another concept I want to throw into the mix here. What is the relationship between being autonomous and having phenomenal consciousness? Like a subjective experience from your own perspective.
- [67:05]
Andreas Mogensen
Good. I'm not sure. I think that's also. Yeah, sadly, I think that's very much like an open question. Like, part of what I wanted to do in writing a paper about this. Was in some sense to encourage the philosophical community to think harder about this. Because there's been all this talk about the relationship between the capacity to be benefited and harmed in phenomenal consciousness. But if you do think that there's this alternative basis for moral standing that goes by way of autonomy. And there's this whole separate set of questions about how autonomy relates to phenomenal consciousness. That I think we've only really sort of begun to scratch the surface on that question. I do nonetheless find myself somewhat inclined to adopt a view on which autonomy requires phenomenal consciousness. So it might be somewhat surprising, given the view about being a welfare subject in phenomenal consciousness. But basically that's because I'm attracted. I think autonomy requires a certain kind of rationality. And I'm quite attracted to the idea that there's some kind of intimate relationship. Between phenomenal consciousness and having rational beliefs about the world. After all, when I think about what justifies me in believing that there's a computer in front of me. That's resting on a desk and all kinds of things like that. Intuitively what justifies me in believing all that. Is the kind of perceptual experience that I'm currently having. It's a conscious mental state. And if I imagine that these perceptual experiences very suddenly switched off. Then intuitively I'd feel like I kind of lost the ability to rationally update my beliefs about the outside world. And you might think, like, if I continued to believe as strongly as before. That there was a computer in front of me and a desk in front of me. And that none of these things had moved or anything like that. In spite of all of my conscious experience suddenly blinking off. Then I would be irrationally confident in those beliefs. So if these, like, phenomenal experiences. If subjective experience does in fact play A crucial role in justifying us, in believing what we do about the world, in making our beliefs rational as opposed to irrational. And if autonomy essentially involves some kind of rationality, including having rational beliefs, then maybe there is some necessary connection between phenomenal consciousness and autonomy after all. But as I said, I think this is sort of like a question that we've only just opened, right?
- [69:35]
Sujana
Hey, listeners. Sushana, here again. We've heard some pretty complex ideas here, so let's pause and take stock. We started off with what seemed like a pretty promising alternative route to deserving moral standing. Maybe you don't need a capacity to be harmed or benefited. Maybe autonomy is enough. And by that we mean something like the capacity for rational reflection, following second order desires and directing your own life. And this seemed like maybe good news for AIs, because, you know, reasoning models seem to deliberate. There's some evidence of like, introspection, inspection and pursuing goals in AIs. And just overall AI systems seem to be getting more autonomous over time. But then I asked Andreas, what's the relationship between autonomy and consciousness? And his answer there was somewhat less encouraging for AIs. So Andreas suspects that autonomy probably does require phenomenal consciousness because meeting this condition of doing rational reflection might actually demand it. And that's because your beliefs are justified by your conscious experiences. So without having the capacity for those kinds of experiences, your beliefs would just be like totally untethered. So you couldn't engage in rational reflection and you couldn't have true autonomy. If that argument is right, then this autonomy route to moral patienthood isn't actually going to let us sidestep the questions of consciousness. Still, Andreas is quick to note that, you know, as with seemingly everything in this area, this very much remains an open question. Honestly, at this point, this all feels like a bit of a confusing mess to me. Is it confusing because we're not great at thinking about these things, or is it confusing because in fact there aren't any definitive yes or no answers to these questions? That's the possibility that Andreas is going to entertain next. We've been talking a lot about questions like could an AI be conscious? Could an AI be autonomous? Could an AI be benefited or harmed? Are there any good arguments for thinking that the answers to those questions are like they don't actually have answers, they're neither true nor false. It's just sort of woolly.
- [72:34]
Andreas Mogensen
Yeah, I do think so. I guess one kind of slightly weird or surprising view that I do find quite plausible is a view on which there might just be no Fact of the matter about whether a given AI system is conscious. So in particular, even if you had an AI system that was in some sense as similar as could be in terms of its internal processing to what goes on in the human brain, but it's made of silicon and all that sort of stuff, I do think that there's a case to be made that in that sort of case it would nonetheless be indeterminate whether this system exhibited phenomenal consciousness.
- [73:13]
Sujana
Right.
- [73:14]
Andreas Mogensen
So, I mean, these aren't really my own ideas. This is me sort of trying to channel a kind of argument you find in David Papineau's 2002 book Thinking About Consciousness. Though I'm not going to make any claims that I'm accurately reporting his view as opposed to my attempt to reconstruct what I think his view is, or something like that.
- [73:32]
Sujana
Thanks for the flag.
- [73:34]
Andreas Mogensen
There's one very crucial assumption here, which is I'm going to assume that physicalism is true. So very roughly, physicalism is a hypothesis which says everything is ultimately physical. There's no additional soul stuff that makes up the mind or anything like that. There is nothing over and above the physical. So, like, to be in pain is just to be in some particular physical state, nothing more. Right. So in some sense, like, there's no genie, there's only the lamp.
- [74:02]
Sujana
Right. Nice.
- [74:03]
Andreas Mogensen
Here's another sort of key assumption I think is true. So whenever we have a conscious experience, whenever I have a conscious experience, like a feeling of pain, say, there are lots of different ways you could characterize what's going on inside my brain or inside my skull at that time. So you could characterize what's going on using the language of neurophysiology. You could talk about patterns of activation involving the anterior cingulate cortex or the insular cortex and that sort of stuff. You could also describe what's going on, say, as an abstract pattern of information processing. Conceivably, you could characterize this as this abstract pattern. And that's a sort of abstract pattern that could conceivably be instantiated by a kind of being that didn't have any kind of neural wetware, that didn't have an anterior cingulate cortex or an insular cortex or any kind of brain cells or any stuff like that. And generally speaking, like, these things just, like, get sandwiched together. They're both occurring at the same time. When. Whenever I'm in pain, let's say. So imagine I do feel like, a horrible pain in my knee. And I kind of, like try to attend to that feeling, I try to, like, pick it out and mentally point to it. And I think that's horrible. So here's a question. What is that? What is it that I'm pointing at? Am I pointing at the neurophysiological state, or am I pointing at the abstract computational state? When I try to pick out the feeling of pain and I try to point to it, which of these things am I pointing to? Here's why this question matters. If I'm pointing to the neurophysiological state, if that the thing I'm pointing to, the feeling of pain, if I'm pointing to the neurophysiological state, then only beings that can share in that neurophysiological state can have that sensation, the one I'm pointing to. By contrast, if I'm pointing to the abstract computational property, then beings who lack any kind of neurophysiological similarity to me, they could also share in that state. The thing I'm pointing to, the thing I'm calling that the thing that's horrible. And I think one natural view is just there isn't really a fact of the matter about which of these I'm pointing to. I'm not specifically pointing to either of them, because, after all, it's totally opaque to me that this stuff is happening inside my head. The pain does not present itself to me either as something that involves squishy neurons, nor as some kind of abstract computational structure. When I'm sort of pointing at the pane, I'm pointing at something I know not what, or something like that. And it's just very plausible that I'm not determinately pointing at one of these states in particular. And if that's the case, then it ought to be true that if you had a being that had only one of these states, say, only the abstract computational property, then there just wouldn't be a fact of the matter about whether that kind of individual has that kind of state. The thing I point to, the thing that's horrible. There's a kind of analogy. So all that's very abstract, right? Yeah. There's a kind of analogy I could give that might help, sort of give a kind of purchase on how you should conceive of the situation.
- [77:17]
Sujana
Yeah, go for it.
- [77:18]
Andreas Mogensen
So imagine you live in a cabin somewhere nice. There's a mountain some distance away. Each morning you go out, you look on the horizon, and you see moving along the mountainside, there's some kind of shape. It's all really fuzzy at this distance, so you can't really Work out what it is that you're looking at, and you don't really give it much thought. You don't spend lots of time thinking about, oh, what is that? Or something like that. You just sort of notice that there's this shape moving along. You don't give any more thought. You go back inside to have your breakfast or whatever. And suppose this keeps happening day after day. You go out in the morning, you look onto the mountainside, and there's this kind of like, fuzzy shape. You don't give it much thought, but you do think to yourself something like, ah, it's one of those again. And you do this day after day. You think, oh, it's one of those again. There it is again. Suppose what you've been seeing are deer, like real living deer. Real living deer are the things which give this kind of appearance from that distance. Suppose nonetheless, that one day some scientists have built a robotic deer. This deer looks quite a bit like the deer that you've been seeing. And indeed it gives the same appearance, let's say, from this distance or something like that. So this robotic deer has wandered out onto the mountainside. You're looking out and you think, oh, it's one of those again. So here's the question. Is that true? Is it one of those again? Yeah, And I think a kind of natural reaction is. Well, I don't know. There isn't really a fact of the matter here, because your usage of this term, this concept, one of those. You haven't really put enough thought into your usage of that concept to determine whether it refers specifically to deer, a kind of a living thing, or whether it refers to anything that has the appearance of. Of a deer, anything that has the outward form of a deer, or maybe anything that gives off the kind of visual appearance at this distance that these deer give off, or something like that, which might in principle be something that a totally different kind of thing could give off, or something like that. Yeah, so there's this sense like you've just been kind of like gesturing somewhat blindly at something, you know, not what, and you've been kind of pointing to it. And it's not really clear that the general kind you've been pointing to, whether that's like deer things that look like deer but could be robots or something else or something like that. And the thought's supposed to be that something similar is going on with our concepts of our conscious states. So when I think there's that horrible feeling of pain because it's totally opaque to me, what's really going on inside my mind. My usage of this concept, that horrible feeling, is just not specific enough to determine whether I'm pointing to a neurophysiological state in particular or an abstract computational state in particular. For this reason, there might very well be no fact of the matter about whether something that's shared in the abstract computational state but lacked the underlying neurophysiology, whether that would count as having that, the feeling that's horrible or something like that. Just as there might be no fact of the matter about whether this robot deer is one of those.
- [80:38]
Sujana
Yeah, yeah, this makes sense to me. I think what strikes me here is just how much this is coming down to, like, how our concepts work and how we're using terms rather than what's actually out there in the world. Like, whether there is a deer or a robot deer out there in the world or something. It kind of feels. Yeah, it kind of feels weird that this should be the case. Like, surely there's a way to. And shouldn't we be trying to focus on the actual fact of is or isn't there a deer there? Is this just some kind of linguistic confusion? And if so, what do we do with that?
- [81:19]
Andreas Mogensen
Yeah, it's a good question. So I think in some sense, the question has become kind of semantic, as you said. It's now a question about how do our concepts work? What is it that we are picking out when we use these concepts, and do we use them with sufficient specificity, with the right kind of intention, such that we pick out one thing and not another? And you're right, that's really kind of a question about what's going on with us and not what's going on with the AI systems or something like that. And so it might. Yeah, I definitely agree. You could think we're sort of, like, getting away from what matters. I think the worry is that if something like physicalism is true, then this has to reduce to something like a semantic question in the following sense. So if physicalism is true, then, say, pain or phenomenal consciousness in general just is some physical state. So it's either some neurophysiological state, or it's the instantiation of some abstract computational structure or something like that. And that means you could, in principle, provide a complete and accurate description of everything that happens without ever needing to mention the term phenomenal consciousness.
- [82:35]
Sujana
Right.
- [82:36]
Andreas Mogensen
So if you have a given AI system, you can completely describe everything that is true of it without ever talking about whether or not it is conscious. So you could say, look, it's Made of this and this kind of stuff. It implements these and these algorithms, yada, yada, yada. And in some sense, once you've told us all that you've given a complete description of everything, which is the case with respect to what's going on within that AI system. And so if you think that there's a further question, is this system conscious? I'm inclined to worry that one of two things is going on. Either you are sort of implicitly a dualist, so you think consciousness, phenomenal consciousness, or maybe the mind in general must be something over and above the physical. There's this further question over and above all of these properties that you've listed that the AR system has, does it also have these additional special properties, phenomenal.
- [83:31]
Sujana
Consciousness, some non physical thing has entered the discussion?
- [83:36]
Andreas Mogensen
Basically, yes, exactly. Yeah. So is there some extra something over and above the physical that might be something that you have in mind? Another possibility is just the question is something like, we have this term phenomenal consciousness, and we want to know, does that term apply to these properties of the system that we've described? So we have this term that we use all the time. We're not sure exactly what it refers to. Which physical things out there in the world are we picking out? And what we'd like to know is of all the physical things that we've said are the case with respect to this AI system. It's made of this and that, here's how the transistors are organized, here are the algorithms that it's running, etc. Etc. One thing you might want to know is, is any of these things, is that the thing we're referring to when we talk about phenomenal consciousness? But that does seem to be ultimately a question about how our concepts work or how our language works.
- [84:39]
Sujana
Right. So my question there is. It feels kind of hard to me to feel optimistic about us resolving difficult questions and potentially important questions about AI moral patienthood and what we might owe them and so forth. If things do come down to some kind of debate about terms or confusion about terms, can you say anything hopeful or do you have a sense of what people should be doing with all of this uncertainty and confusion?
- [85:16]
Andreas Mogensen
Yeah, I mean, I do think, yeah, there are many different ways you could react to this if you buy this story I've been trying to tell, which maybe you don't. So, I mean, one thing you could think is just, Yeah, I mean, maybe this is in some sense a semantic issue. We're asking of all the physical things that we know to be the case with this system, which of these are we referring to when we talk about phenomenal consciousness? Which, if any, maybe you just think, actually, that is an important question. Because by settling that question, we settle whether the system is indeed conscious. And a lot, morally speaking, must hang on that. Surely that's the intuitive picture, right? So you could sort of think that maybe even if everything I've said is true, that doesn't deflate the question, the moral importance of the question of whether the system is conscious.
- [86:05]
Sujana
Right.
- [86:06]
Andreas Mogensen
That might nonetheless be somewhat hard to believe. And, I mean, you can certainly sort of imagine these cases where you've got these two people who've got this. They've got this AI system. They're both convinced physicalists. They agree on what it's made of and how its physical parts are arranged and what algorithms it runs. But one of these subscribes to the view that consciousness requires neurobiology. The other subscribes to the view that it doesn't. And they disagree about whether the system is conscious. And there's like some kind of worry that this is in some sense a verbal disagreement, because they agree on everything that is the case, everything that really takes to be the case with respect to the system, because they sort of agree on a complete physical description of it. And what they're disagreeing about is really just how our terms work or how to use words or something like that. They're disagreeing about the right way to describe the system, not what the system is like in and of itself. And I think it's kind of hard to avoid the intuition that nothing of moral importance could hang on that. And one thing you might infer from that is just that in general, or certainly in this case, maybe not in general, but certainly in this case, nothing of moral importance hinges on the question of whether this AI system is phenomenally conscious. That's a very counterintuitive thing. But it would also, in some sense, be quite helpful in terms of clearing away some confusion, because it would mean that we don't actually need to be concerned with this question of whether the.
- [87:33]
Sujana
System exhibits phenomenal consciousness, which is a hard question. It's nice if we don't have to answer it.
- [87:40]
Andreas Mogensen
Yeah. So you could definitely think that. Yeah, I do think there are sort of other gloomier things you might think, or maybe other more revisionary things you might think. So maybe you think, look, what this is telling us is that physicalism cannot be true, Right? Because clearly it does matter whether or not this system has subjective experiences. You might think, you know, that as well as, you know, more or less anything. And so if physicalism, you know, if actually following through on a physicalist picture of the world tells you that that's an empty semantic question that just tells you something is wrong with physicalism or something like that.
- [88:17]
Sujana
Right.
- [88:17]
Andreas Mogensen
So maybe you should throw out that kind of metaphysical worldview. And indeed, I think. I mean, at least amongst philosophers, I think maybe physicalism is sort of. Maybe surprisingly not uncommon, but less common than you might believe or something like that. So I think in the kind of recent survey, I think it's only about half of philosophers accept or lean toward a kind of standard physicalist theory of the mind or something like that. Slightly more than half, I think. And yeah, I mean, I think the very gloomy thing you might think is something like, well, if consciousness doesn't matter, how could anything matter? You might be tempted by a kind of nihilism or something like that.
- [88:55]
Sujana
Right.
- [88:55]
Andreas Mogensen
Yeah, I think that's not the optimistic one you were hoping for.
- [88:59]
Sujana
Yeah, I asked you for something optimistic and you said the most harrowing things that have ever been said to me.
- [89:06]
Andreas Mogensen
Yeah, yeah. But, yeah, I mean, there is this option that it tells us that consciousness doesn't matter. That means that something else does matter, something that doesn't require consciousness, and therefore something whose presence or absence might be much easier for us to verify in general.
- [89:19]
Sujana
Right. Okay. I really want to dig into what we should do about all of this and how we should avoid harming AIs if they are moral patients. So I guess, let's recap here. You lay out a few different routes to potentially being worthy of moral consideration in your research. One of them is to through having desires that can be fulfilled or not fulfilled, which means that you have some kind of capacity for welfare. You can be benefited or harmed. Another is through having autonomy, which could make you worthy of moral consideration even if you don't have a capacity for welfare. There could be more. There are open questions about what role consciousness plays in all of this. But my next question is, does the way that we should treat a being depend on which of these routes to moral patienthood it's taken?
- [90:26]
Andreas Mogensen
Yeah. So I think the answer is plausibly yes. So I think in particular there might be an important difference in the sort of thing that we would owe toward individuals who are welfare subjects who can be benefited or harmed because they not only have goals, but those goals are invested with a kind of emotional investment, so to speak, versus autonomous beings who might have goals but who have no emotional investment in the achievement of Those goals who, say, operate purely on cold logic or cold rationality or something like that. And the thought is, in general, often it's a good idea. We have moral reason to help people to accomplish their goals. You might think we have not only negative obligations to avoid harming people, we have positive obligations sometimes to help them out or something like that. For example, if somebody needs money in order to pay for food or medicine or something like that, we have a moral reason, maybe not a decisive moral reason, but a moral reason to help them out. But I might think that's true only insofar as this goal is the sort of thing in which they have some kind of emotional investment such that they in particular are benefited by achieving this goal or something like that. So if a person is in need of money because they need it to buy food or medicine for themselves, then I might think we have reason to help them. But if somebody is looking for money not because it's not for their own sake, but because they feel they have an obligation to raise money for their church, let's say, then I feel like I've got much less of a reason to help them out. They might be the same person, they might be in need of food or medicine, they might nonetheless prioritize their obligation to their church over and above their concern for their own well being or something like that. But I think in that case I have much less reason, and perhaps no reason at all for to help this person out. Certainly if they in some sense have no positive feelings toward their activity of helping out the church, this is just something they feel that they have to do in order to realise an obligation. And I'm assuming I don't share this person's religious beliefs. And we can assume for sake of argument that indeed I know their religious beliefs are false or something like that, and the church would not otherwise do anything especially valuable people with the money and just promote these religious beliefs that I take myself to know are false. So in that sort of case, I feel like, although it is this person's aim and it is an aim that they sort of wholeheartedly endorse on reflection, to aid their church in this way, I don't think I have an especially strong moral reason and probably no moral reason at all to help them fulfil that aim in this case. So the general thought here is that if you have goals of a kind whose fulfilment contributes to your welfare, then I have reason to help you out. If you have goals of a kind that do not contribute to the fulfilment of your welfare, but are nonetheless ones that you rationally and reflectively endorse. I don't have a reason to help you out, though. I may have a reason not to interfere in your pursuit of that goal.
- [93:44]
Sujana
Right, yeah. So what might this look like in the case of, say, large language models?
- [93:52]
Andreas Mogensen
Right. I mean, I think it's somewhat difficult to say. Presumably we're also sort of imagining something that's at least a little bit far in the future or something like that. But I mean, certainly the thought would be that there would be sort of like duties of non interference of a certain kind that would fall on us if these genuinely are like moral patients with a capacity for autonomy. But there wouldn't necessarily be any obligation on us to help them out in achieving these goals, I think. I mean, there might be all sorts of things that complicate this picture a little bit or something like that. Like we are responsible for creating these systems. So we have quite a different relationship to them than we do to this person that I might encounter who's raising money for their church, who I presume is not my child or anything like that. Maybe that makes some kind of difference, I don't know. But yeah, certainly that's the kind of baseline or default that, all else being equal, if the kind of view I've been sketching is true, our moral reasons in respect of how we treat such systems in some sense might give out in terms of a kind of negative duty of non interference, as opposed to there being any positive duty to help them in fulfilling their goals.
- [95:01]
Sujana
Okay, so let's spell this out a little bit. So if some AI in the future, imagine it's a future version of Claude. If Claude is capable of being benefited or harmed, then maybe we have some moral reasons to try to do the things that benefit Claude and avoid the things that harm Claude, whatever those things might be. And that would be a case of like, Claude will have particular desires that relate to getting more welfare. And we should help Claude with those things.
- [95:38]
Andreas Mogensen
Yeah, something like that.
- [95:39]
Sujana
But in the alternative case, Claude cannot be benefited or harmed in any meaningful way, but is like this autonomous being. And that means that Claude will have certain goals. And really we don't have any obligations to help him towards those goals. But we shouldn't interfere, we shouldn't prevent him from pursuing them within reason, I guess. I mean, the thing that comes to my mind here is could that compete with human needs at any point? And what should like, you know, us not interfering in, I don't know, an AI's plan to take over humanity or something.
- [96:17]
Andreas Mogensen
Yeah, yeah.
- [96:18]
Sujana
What do you make of that?
- [96:19]
Andreas Mogensen
Right, yes. I mean, certainly you have to. Yeah, all these things hold true, only all else being equal or something like that. So yes, there very well might and almost certainly would be certain cases where these considerations come into conflict. Yeah. And yeah, I definitely don't think that our duty to respect the autonomy of rational agents is absolute in the sense that we shouldn't interfere with their plans for world domination or anything.
- [96:45]
Sujana
In the same way with a human, I'd feel okay stopping them from world domination.
- [96:50]
Andreas Mogensen
Absolutely.
- [96:51]
Sujana
As much as I love Claude, I can't give him a free pass there either.
- [96:55]
Andreas Mogensen
Yeah. But I'm more thinking like, I don't know, maybe Claude has this desire to, I don't know, list the decimal expansion of PI or something like that. Something that like strikes us as kind of weird. That sounds great to me.
- [97:11]
Sujana
Don't know what you're talking about.
- [97:14]
Andreas Mogensen
And the thought would be like we don't have any reason to help it with this project or something like that, but we may have a duty not to interfere with its ability to pursue this project if indeed it satisfies these criteria for being a kind of moral patient, a being that has moral standing purely in virtue of being an autonomous agent without having any capacity to be benefited or harmed by fulfilling or failing to fulfil its goals. That's roughly the kind of the picture that I think has plausibility in this case.
- [97:47]
Sujana
So it sounds like the way that you weight different moral patients interests is also dependent on what type of moral patient they are. Are they the kind that could be benefited or harmed, or are they just the kind that is autonomous but can't be benefited or harmed? Pushing on. I really want to get a sense of just how important these topics feel right now. So there is some reason to believe that at some point we're going to get really advanced AI systems that can help us reason through really tricky philosophical problems like these ones about moral patienthood. And yeah, maybe there's some reason to think that in the future we'll just be way better equipped to answer all of this messy stuff than we are right now. Should we just wait for that or should we be working on this right now?
- [98:49]
Andreas Mogensen
It's a good question. I think to a large extent it would depend. So if you were sort of certain that you would have a kind of super intelligent AI system that radically surpassed human capacity for moral reasoning or moral philosophy and that such a system would arrive before there were any AI systems that were moral patients, then. Yeah, obviously it would make sense to just wait until we have such a system and allow it to guide us or something like that.
- [99:22]
Sujana
Right.
- [99:22]
Andreas Mogensen
I think it nonetheless seems to be a very real possibility that things won't appear in that order or something like that. That you might have a world in which there are potentially a large number of AI systems of a kind that exhibit morally significant mental properties like emotion, phenomenal consciousness, whatever you think is the really important stuff, and that that sort of predates a world in which super intelligent AI systems of a kind that might far surpass our capacity for moral reasoning and moral philosophy appear or something like that. So plausibly it hinges on these kind of like timelines Y questions. But it does seem to me that, yeah, on the one hand that the possibility that we'll get the arrival order such that in some sense we have to face up to these questions before or we can punt these questions to a more intelligent being if such a being comes to exist, we're going to sort of face that crunch, that point in time. Yeah, that seems relatively likely. And I think it especially seems working on these issues because they seem very neglected by comparison with other kinds of issues that you might associate with sort of risks and opportunities thrown up by the possibility of transformative artificial intelligence or whatever you want to call it.
- [100:44]
Sujana
Yeah, yeah, yeah. And I guess as well, like, it's not just a case of like we, if we get things wrong and realize later that we were wrong, no harm done. It's like if we get things wrong, it means we like are enslaving like species of morally important beings or something like that. That's maybe. Yeah, yeah. Like, I mean, hopefully that's not the direction things go in. But like, you know, there are some stakes here which suggest that, yeah, working on things sooner when there's uncertainty about what order things are going to happen in, seems wise.
- [101:22]
Andreas Mogensen
Yeah. And I mean, you might also think that there's the potential for something like a kind of weak lock in effect or something like that. I mean, it's sort of somewhat implausible that you would lock in the kind of way of treating digital minds as strongly as, say, some kind of extinction event that killed all human beings would lock in a world without humans. That extinction is very final. But nonetheless, if you think about, say, our treatment of intensively farmed or factory farmed animals, I think if we'd sort of like in the early 20th century, when factory farming was first coming online, when this kind of industrialized agriculture was still being put in place. I think if we'd sort of anticipated the future we would be bringing about and we thought, is this a good idea? Is this morally okay? We would probably have thought, no, we shouldn't be doing this. But now we're in this world where we live, well, not surrounded by, but we live in a world where industrialized agriculture and the intensive farming of non human animals is an entrenched practice. And once a practice becomes entrenched like this, it becomes much harder to give it up because people have become accustomed to being able to, say, buy meat at very, very low prices and things of that nature. And so it becomes much harder to quit a practice once you've begun it, once it's become institutionalized and entrenched, and once people are used to it, rather than to sort of preempt a given practice before it really properly gets going and people come to depend on and expect that things will work a certain way.
- [102:57]
Sujana
Yeah, you can imagine a situation with AIs where we get some regulation in place. And even on the basis of new evidence that comes to light further down the line, where we're like, maybe we should be kinder to our AIs, there will be a lot of resistance because it's like, hey, why are you. We already came up with the regulations, why are you changing it? Or you reach a point where lots of people have just sort of interests sort of embedded in the way things have been set up. Like they're making profits out of the status quo. And we've come to expect that we can run these things very cheaply and this incurs costs. So I can totally imagine how it sort of pans out that it becomes especially difficult to reverse things once they've been sort of set in motion. Though definitely not impossible. Like not as possible as if we had literally gone extinct at this point. That seems much harder to get back from.
- [103:52]
Andreas Mogensen
Yeah, in some sense you might think maybe there wasn't a kind of moral superintelligence around at the time. But certainly once factory farming had got going and had become entrenched, there were moral philosophers like Peter Singer and others who you might think in some sense have a greater than ordinary capacity for moral reasoning or for sort of ferreting out what's actually morally important, what's actually right and what's wrong. And those people did tell society at large that what they were doing was morally unacceptable or something like that. Certainly there were many people who were receptive to that message, but it's certainly not put a stop to factory farming and Associated practices. So even if there might be some future moral sage that could awaken in a data center somewhere, I'm not sure that one should expect that its advice would be able to sway us from abandoning a practice that become entrenched in something like the way factory farming has become entrenched.
- [104:51]
Sujana
Yeah, especially if we're like, well, obviously you, the AI, are going to say that you deserve more rights.
- [104:56]
Andreas Mogensen
Exactly.
- [104:58]
Sujana
Why should we listen to you?
- [105:00]
Andreas Mogensen
Exactly. Yeah.
- [105:03]
Sujana
Okay, we're about to move on now to a totally different topic, so let me try to pull together what we've covered up to this point. We started with this question, does something need to be conscious to deserve moral consideration? And Andrea sketched out a few other routes to moral patienthood. One was through welfare. So being the kind of thing that can be harmed or benefited, which in his view might require having desires that are backed by emotions. Another route is through autonomy, this capacity for rational self direction. Andreas also noted that if AIs are capable of well being, then we probably have some reason to help them pursue their projects. But if they're only capable of, say, autonomy, then we merely have some reason to not interfere with their plan plans rather than actively helping them to pursue those plans. And after all that, it turns out that Andreas suspects that there might be no fact of the matter about whether a given AI system is conscious. The idea here is that questions like but is it really conscious? Might just kind of come down to how we are using words, not deep truths about AI or the nature of reality itself. And that could be liberating if you're looking for a way to not have to solve the hard problem of consciousness. But it could also be unsettling because if the nature of consciousness is just between us and our dictionary, then maybe nothing really matters at all. And finally we could wait for superintelligent AI to just sort all of this out for us, but the questions that we're asking could actually bear on the decisions that we're making before any of that happens. Basically, we risk immoral treatment of AIs just getting entrenched really deeply into our social practices and our business practices and ultimately becoming way tougher to uproot. Alright, after all that fun stuff, let's push on to a very different set of issues. Let's move on now to a totally different topic. So you've done a lot of research into some quite tricky moral questions, and some of the stuff that really stands out to me is about how we should wait suffering and whether that means human extinction could actually be A morally desirable thing at 80,000 hours. As listeners probably know, we're pretty keen to have lots of people working on preventing human extinction on the grounds that we think the future of our species could be incredibly valuable and worth preserving. So I really want the answer to the question, is human extinction morally desirable to be.
- [108:13]
Andreas Mogensen
No.
- [108:15]
Sujana
But can you maybe tell me in brief terms what assumptions or facts about the world might lead you to believing it?
- [108:24]
Andreas Mogensen
Yeah. So I think, I mean, yeah, definitely like the view that human extinction would be a very bad outcome and perhaps one of the worst tragedies imaginable. Definitely a very common view, I think, amongst people who are inclined to doubt this, generally speaking, people, those people are most concerned or most driven by concerns about the impact of human activities on non human animals. So on the one hand, the many non human animals that we farm for meat and so forth, and then also wild animals whose habitats and whose populations we decimate, or something like that. So there's roughly this sort of thought or this concern that many people have that certainly if things go on like this, and yeah, I mean, you might very well doubt that they would, but certainly in some sense, if current trends continue, then the sort of scale of suffering inflicted by human beings on non human animals might be such that the continued survival of our species would be undesirable. I think that's perhaps the most common reason that people might give for questioning the desirability of the continued survival of our species. Or something like that.
- [109:36]
Sujana
Yeah, that feels kind of intuitive to me.
- [109:39]
Andreas Mogensen
There are also sort of more philosophical arguments of a kind which might lead you to think that the extinction of all sentient life might be desirable, which would involve human extinction. But it's not specifically about humans. So one such argument derives from a sort of a position that's often called negative utilitarianism. So that term can be used to talk about a bunch of views. What I have in mind is something you might call classical negative utilitarianism. This is like the view that the only thing that matters morally is the reduction of suffering. So maybe getting a bit more exact, what we ought to do always is bring about the outcome with the least total suffering in it, or something like that. You might think that sounds kind of plausible on its face, but it sort of quite infamously leads to the conclusion that we ought to bring about the extinction of all sentient life as soon as possible. If we could, could certainly if we could do so painlessly. Because if we allow sentient life, human beings, animals, all the rest to keep on going, there would just be more and more suffering. And the sum total of all suffering that would accrue as the human species and the whole terrestrial biosphere went on and on, would just be so enormous that it would be better if there no longer existed any living things or anything that could suffer at all. So this kind of classical negative utilitarianism immediately gives you this kind of striking conclusion.
- [111:08]
Sujana
Hey, listeners, Zoshane here. Judging by what we've just heard, you might conclude that people who sympathise with negative utilitarianism are in favour of working to cause human extinction. But surprisingly, or maybe unsurprisingly, that's not the case. Obviously there's like a bunch of considerations that go into this, but one very interesting one is that negative utilitarians think it's far more important to prevent futures that have crazy amounts of suffering in them. And the reasoning there is that most possible futures, even if they do contain some really serious suffering, just don't have anywhere near the scale of suffering that's theoretically possible. Because there's nobody who's actually motivated to bring about outcomes with like, huge cosmic scale suffering. Like, think about just vast numbers of really tortured lives. In most worlds where there's suffering, that suffering is just happening incidentally or by accident or through neglect or something like that. So in a cosmic sense, these futures all round to approximately zero value. Now, the extinction of all sentient life might drive the value of the future up from approximately zero to actually zero, but that difference is just not that important in the grand scheme of things. What's more important is reducing the probability of a truly disastrous future, where for some reason or other, there are some groups that really does try to use some part of the universe to cause, like, the maximum amount of suffering that is physically possible. This argument is pretty generally accepted among negative utilitarians, and it's been around for decades. Interestingly, it's actually the mirror image of an argument Will MacAskill made on the show a while back. Basically, Will said that in the great majority of futures where we don't go extinct, if they turn out to be positive, they're actually only very weakly positive if you compare them to the best possible world. That's for various reasons, which I won't go into now, but mostly Will just doesn't think that people will be very motivated to do the. The morally best thing by default. And, you know, even if you are motivated to do the morally best thing, there's just so many ways that you could completely fail to get there. So the argument goes, rather than focusing on avoiding extinction, there might actually be higher expected value if you try to nudge our trajectory away from, from our default of just a pretty mediocre future towards achieving a future that's close to the best future that's possible. Now, I know we're already well into this interview, but if you're locked in and you're ready for another few hours of content, then you could also check out Will MacAskill's interview with Rob Wiblin from March 2025. This mirror image argument is discussed about two and a half hours in during the section called we should aim for more than mere survival. Okay, back to regularly scheduled programming. So I definitely find the first argument, the ones concerning animal welfare, more intuitive. Maybe I want to start there because, yeah, I think it seems like it makes sense. Yeah, we're causing a lot of harm to these animals. Maybe it would be. Be better if we didn't exist. But my understanding is that the argument is not quite that simple and we have to make some kind of further assumptions before we conclude that and therefore humans should go extinct. Could you kind of walk us through a little bit about like, what else is going on here?
- [115:42]
Andreas Mogensen
Yeah. So on the one hand, I think there is just, you have to make a sort of assumption that these trends will continue or worsen or something like that. And that might not be at all obvious. I don't think it is at all obvious. Human beings have previously succeeded in to a large extent, enforcing a kind of global elimination of certain practices that were once prevalent and morally catastrophic. For example, the near universal acceptance of a prohibition on enslaving other human beings, for example, is something that in some sense happens relatively suddenly and relatively recently in the history of our species. And something similar, a similar kind of moral revolution could take hold when it comes to the interests of non human animals. I think there are also potentially, I think this is quite unclear, but potentially certain philosophical considerations that might be brought to bear on this question, which are about how you sort of add up and weigh harms and benefits to different kinds of individuals. So let's start by just focusing on wild animals and the sort of harms that human beings do to wild animals by virtue of encroaching on their habitats and otherwise engaging in practices that mean that wild animal populations have significantly decreased within recent years, within recent decades, and that some people take as indicating that we're on the way to like a sixth mass extinction or something like that. Now viewed certainly through a kind of utilitarian lens, in order for it to be a bad thing that human activities reduce the size of wild animal populations, we'd certainly have to assume that generally speaking, wild animals have good lives, that they have lives that are worth living, as we might put it.
- [117:34]
Sujana
Right.
- [117:34]
Andreas Mogensen
And I think they're certainly like that is the kind of intuitive view. Many people, I think, very naturally assume that animals that live in the wild, that live in their natural habitats, that they have a kind of positive existence, that they're flourishing, that they're happy, they're fulfilled, things of that kind. I think there's good reason to think that this idyllic conception of what life in the wild is like is a mistake. And that in fact, the lives of many wild animals are plagued to a large extent by starvation, suffering, disease, fleeing from predators in fear, or suffering within the jaws of predators or things of that kind. And I think there's a sort of view which I think is to me surprisingly prevalent in sort of effective altruist circles, a view that's sort of championed by Yu Quang Ng and Brian Temasek, that in fact suffering predominates in nature, that most wild animals have lives that are not worth living, lives in which the bads, the harms outweigh the goods. And in some sense, if you add up all the suffering and all the happiness that these animals experience, you find that there's more suffering in total.
- [118:45]
Sujana
Right?
- [118:46]
Andreas Mogensen
Yeah.
- [118:48]
Sujana
But I'm guessing that the important question here, if you have utilitarian assumptions about maximizing welfare, is not just about the question of do these wild animals have lives worth living? But how good are they? How above that threshold are they? Right. Because it needs to be enough for them to then outweigh in aggregate the combined well being of continued human lives.
- [119:16]
Andreas Mogensen
Absolutely.
- [119:17]
Sujana
So I guess it has to be above some kind of bar as well, right?
- [119:21]
Andreas Mogensen
Yeah, I think exactly. In what sense it has to be above a bar will depend on what you think about tricky questions in population ethics. Right. So, yeah, certainly you want to have it be the case that these wild animals have lives that are worth living, generally speaking. And I think although this view that suffering predominates in nature is surprisingly prevalent amongst defective altruists, I think there's actually, my sense is there's not an especially strong reason to think that this is the case. So there's quite a good article by Heather Browning and Walter Veidt where they sort of make the case that wild animal welfare is positive on net or something like that. But even if it is. Yeah, positive, as you said, it matters how positive but it might also matter in somewhat surprising ways. So if you have a kind of straightforward total utilitarian view, where you think, look, what we do is we assign numbers to how well people are doing. They're positive if they're doing well, and they're negative if they're doing badly. And then we just add up all these individual numbers and whatever gives the biggest number that's the best outcome. If you have a view like that, then the following is possible. Maybe, in general, wild animals have lives that are only just above neutral, or something like that. Or maybe this is true of very many of them, or something like that. Sufficiently many of these lives that are only just above the neutral points could, in total, it could be better to have that many lives than to have a much smaller number of lives that are really good in terms of everything that can make a life go well. So, yeah, this is a version of the famous or infamous repugnant conclusion. So the repugnant conclusion says for any population, no matter how big and no matter how well off the people in it are doing, there is a better population, which would be bigger, much, much bigger, even though the people in it have lives that are only barely worth living. So you could imagine some utopian civilization where everybody has these wonderful, flourishing lives, they're successful as artists and inventors and scientists, and they have these deep, meaningful relationships, et cetera, et cetera. Compare that with a much, much bigger population of people who basically, they go through life more or less feeling nothing, like there's nothing good or bad really in their lives. They exist in a kind of stupor. Every now and then they hear a slightly pleasant jingle or something like that. So kind of standard total utilitarianism says that second outcome would be better if you just have enough of these people who live these stupor lives or something, right? One thing you might try to say in response to this concern that it would be better for human beings if there were no more human beings because of the extent to which human beings decimate wild animal populations. If you think that, in fact, very many of these wild animals have lives that, if above neutral, are only slightly above neutral, and you reject the repugnant conclusion, and you think maybe that there are certain kinds of goods, like, say, autonomy or meaningfulness or knowledge of deep truths about the universe that really ennoble human lives and make them really good, or at least the very best possible human lives, then if you take this position where you reject the repugnant conclusion, you might think that it would be better to have some number of human beings who achieve these pinnacles, Even if we could instead have a much, much larger number of wild animals, if those wild animals, if they have lives that are worth living, that are above neutral, might nonetheless be only just above neutral. So there seems to be some case for thinking that this issue that arises in thinking about whether you should accept or reject the repugnant conclusion might be relevant to thinking about how we should trade off the interests of human beings or think about the human beings and non human animals, wild animals, and how we should think about the desirability of the continued survival of our species. I mean, I think it's very, very hard to know exactly how things are going with wild animals. I think we should just have a lot of uncertainty about that. But it does seem to me quite possible that maybe there's a kind of repugnant conclusion trade off involved in some of these questions about the desirability of the continued existence of human beings. Yeah, it doesn't seem obvious to me that you could dismiss the moral relevance of these questions about whether we should accept the repugnant conclusion.
- [124:02]
Sujana
So we've just been talking about wild animals. Another category of animals that humans don't necessarily benefit are factory farmed animals. I'm guessing that the basic idea behind this argument for human extinction is a bit different because I guess if humans were to stop existing, these animals would also stop existing because we're breeding them. So in this case it's just a question of like by humans going extinct, we'd stop bringing these lives into existence. Can you sketch out how that argument goes?
- [124:40]
Andreas Mogensen
Yeah. So here I think the worry is more that the animals that we bring into existence and that are intensively farmed to a very large extent, have lives that are not worth living. They're lives in which suffering predominates over happiness. And so in some sense it is a harm for these beings to be brought into existence, you might think, or certainly it's something that makes the world a worse place overall because it just adds more suffering to the world than it adds joy or happiness or something like that.
- [125:11]
Sujana
Yeah.
- [125:11]
Andreas Mogensen
And so, yeah, I guess the rough thought is like one thing that would be desirable, One point in favor of human extinction is that it would mean that, say, we cease to bring into existence factory farmed chickens and other kinds of intensively farmed animals that we might suspect have lives that are so terrible that we should wish for their sake they'd never been born.
- [125:32]
Sujana
So I guess the question is how we should be weighing the net negative bad lives of like chickens against the well being that humans get by continuing to exist. And I guess one natural thought is that if additional humans mean billions of very bad lives, yeah, it sort of weighs up that maybe humans ought to go extinct, but maybe, yeah, is that.
- [126:07]
Andreas Mogensen
How.
- [126:10]
Sujana
Is there a way to respond to this that suggests that we should be weighing human prosperity just much more highly, regardless of how many suffering chickens there are, or something like that?
- [126:25]
Andreas Mogensen
So here's a principle you might believe. You might believe that if we can choose to add some number of good lives to the world, but some number of bad lives have to come along for the ride, then if there were sufficiently many bad lives that came along for the ride, it would always be worse to make this expansion of the population. And this might be true for any kind of good life. And for any kind of bad life, you might think for any good life you could add, if sufficiently many bad lives have to come along for the ride, then this makes things worse overall. In a paper I published, I call this the always outweighable principle. And I think it seems like something like this principle is what's being presupposed. If you think that the fact that human beings perpetrate factory farming is a reason in favor of the extinction of human beings or something, however good human lives are because expanding the human population brings all these suffering lives with them, that makes the outcome worse, I think that's a very natural thing to think. So it turns out that there's an argument against this always outweighable principle if you reject the repugnant conclusion. So if you think that a smaller population of these like wonderful lives would be better than any number of lives that are only weakly positive, and if you also think that any number of lives that are weakly negative can be counterbalanced by some sufficiently great number of lives that are weakly positive. So you imagine making some kind of expansion to the population. You're adding some number of lives that are barely not worth living, but you get a bunch of lives that are barely worth living that come along for the ride. And imagine that you think, well, if there are enough of these lives that are barely worth living, the ones that are weakly positive, then that can compensate for the fact that we're adding some weakly negative lives as well and can make the overall outcome not worse when we're adding these lives, that doesn't seem too crazy. But if you believe that and you reject the repugnant conclusion, and you buy some other more technical assumptions that I won't talk about, then you're forced to Reject this always outweighable principle in that you're compelled to believe that there are some lives that are sufficiently good that it could be better to add lives like that to the world, no matter how many weakly negative lives, lives that are only barely not worth living come along for the ride.
- [129:01]
Sujana
Yeah, it seems like given certain assumptions, you might end up believing that like, some level or type of human welfare is so valuable that like, no matter how many mildly suffering animals come with my continued existence, it was still worth bringing me into existence. Even if you tried to do the maths and add it up and you thought that these thousand flies or something like that were worth more than me, there is some kind of value to my existence that just, just trumps that, in any case.
- [129:47]
Andreas Mogensen
Yeah, that's basically the idea. Yeah. I mean, I think I can give a somewhat intuitive sketch of how the argument or the proof goes. So the thought is, if you reject the repugnant conclusion, you think that there's some number of really good lives that are better than any number of lives that are only just worth living.
- [130:04]
Sujana
Right.
- [130:05]
Andreas Mogensen
And if you believe this other principle I sketched, you think that for any number of like, weekly negative lives, there's some number of weakly positive lives such that those can kind of outweigh the badness of the weekly negative lives. But if you're rejecting the repugnant conclusion, you're thinking there's some small, smallish number of very positive lives that's even better than any number of, you know, any arbitrarily big number of weakly positive lives. So that's roughly how the like argument is trading on. I think I probably didn't actually make it especially clear, but that's roughly sort of what's going on in this argument, which appears in a footnote in a paper of mine. So, yeah, as you said, you could have this view where bringing into existence a sufficiently great number of sufficiently good lives of the kind you might think the very best lives that human beings can lead are of that kind could in principle outweigh the badness of having arbitrarily many non human animals whose lives are weakly negative, whose lives are only just not worth living or something. And yeah, I mean, I think this phrase only just not worth living. I mean, who really knows what that means? But you might. It's not maybe absurd to think that some intensively farmed animals fit that description or something like that, including some of the most abundant. So maybe farmed shrimp, generally speaking, have lives that are not worth living, but if they do, maybe they're nonetheless say they're sort of maybe only barely conscious. So insofar as their lives are not worth living, maybe they're only barely not worth living or something like that.
- [131:44]
Sujana
And the life is short as well. So it's not like to the extent they can suffer, not that prolonged. Yeah, I can kind of see that, though. I sort of prickle at people being like, oh, it's not that bad to be a farm trim. You know, there's some kind of instinct, instinctive ickiness about that, but can kind of see the reasoning.
- [132:05]
Andreas Mogensen
Yeah, absolutely. I mean, I think we should also be suspicious if you find any of these stories I'm telling convincing. You should maybe be a little bit suspicious about your sort of instinct to believe these stories because of course it is awfully convenient for us if things work out in exactly this way.
- [132:23]
Sujana
Yeah, wouldn't that be nice?
- [132:24]
Andreas Mogensen
But I do think. Yeah, yeah. Potentially there are these philosophical complications.
- [132:29]
Sujana
Yeah, yeah, makes sense. So let's push on here. I want to talk a little bit more about extinction arguments that emerge when we weigh suffering more heavily than well being, rather than treating those things symmetrically. You talked a bit about negative utilitarianism and how under that version you where you're just trying to minimise suffering. It's basically a foregone conclusion that everything that can suffer should just go extinct. You just get rid of them, game over, humans, animals, AIs, whatever, gone. If you can do that without causing more suffering, if you could do it painlessly or something like that. But there are other ways that we could say we care a lot more about suffering than well being. That might not commit this to that conclusion. Yeah. Could you, in really, really simple terms. Yeah. Explain what lexical threshold negative utilitarianism is and what that says about the desirability of extinction. Sorry, that was a long word. A long term.
- [133:37]
Andreas Mogensen
Yeah. So I also want to just sort of exculpate myself. I did not come up with this term. I borrowed it from Toby Ord.
- [133:46]
Sujana
So it's his fault. Very roughly, we'll blame him.
- [133:49]
Andreas Mogensen
Definitely. Yeah. So lexical threshold negative utilitarianism, ltnu, which has nothing to do with low traffic neighbourhoods either. So ltnu says vertfully that there's some depth of suffering so terrible that it would be worse to add a life like that to the world, no matter how many however good lives might be added alongside it.
- [134:12]
Sujana
Right. So that's kind of the. Is that the flip side of the principle that we were just talking about before?
- [134:21]
Andreas Mogensen
Yeah, pretty much. So, yeah. I guess earlier we were thinking of a view, I think it's slightly different, but we were thinking of a view where there are some lives so good that their addition to the population can justify adding arbitrarily many lives that are only barely not worth living. Okay, but this principle is actually stronger. It doesn't just flip the sign. It says, look, there are some lives so bad that adding them to the population would be worse, no matter how many, however good lives come along for the ride. Okay, so not just lives that are only barely worth living, just above neutral, but the best possible lives you can imagine. There's some depth of suffering so deep that it cannot be counterbalanced by any amount of well being experienced by any number of people. Or something like that.
- [135:09]
Sujana
Yeah, got it.
- [135:10]
Andreas Mogensen
So quite an extreme view in many respects. And yeah, I mean, you might think if you believe something like this, then this also straightforwardly entails that it would be better if there were no more sentient life or something like that, because you might think that in the fullness of time there just are going to be more lives of this, like, really terrible, horrible kind that get added to the population. And I mean, I think the Oxford philosopher Roger Crisp, for example, is someone who's quite sympathetic to something like this argument for the desirability of the extinction of all human life and all sentient life or something like that.
- [135:45]
Sujana
But it doesn't necessarily commit you to that. Right, because you could just say, yeah, there is some awful depth of suffering such that this applies, but that depth of suffering is so crazy there's no way we're just ever going to reach it. Is that an appealing view at all?
- [136:03]
Andreas Mogensen
I mean, I guess it's appealing insofar as this view might allow you to escape the conclusion that the extinction of all sentient life would be desirable. You might think it's nice if we weren't committed to that.
- [136:17]
Sujana
That's pretty appealing.
- [136:19]
Andreas Mogensen
At the same time, it might be thought that it can sort of honour some of the intuitions or other arguments, considerations that might be given in favor of lexical threshold negative utilitarianism. So in this paper I have called the Weight of Suffering. I show that there's a couple of, to my mind, relatively plausible seeming assumptions you can make. And if you make those assumptions, you get driven to believing lexical threshold negative utilitarianism. And so, yeah, one way to have your cake and eat it might be to accept lexical threshold utilitarianism, but set the threshold, as you were saying, at a depth of suffering so extreme you might be confident that no life so terrible will ever be lived or something.
- [137:03]
Sujana
Right. Okay. Putting that aside, what is a reason to believe in lexical threshold negative utilitarianism? I know you mentioned that there are some really plausible assumptions that get you there, but they're pretty technical. Is there like an intuitive route in or a thought experiment or something that you like?
- [137:24]
Andreas Mogensen
Yeah, the sort of thought experiment I have in mind is the sort of thought experiment you find in Ursula K. Le Guin's well known short story, the Ones who Walk Away from Omelas. So I expect many readers will know how this story works. But roughly, there's this city, Omelas, which is initially presented to the reader as this kind of unblemished utopia where people live the most ideal kinds of lives and have these wonderful festivals and achievements, and they live in this splendorous and beautiful city. Then it's revealed to the reader that somewhere in Omelas there's a child that's suffering terribly and it suffers day upon day. And we're told that all the splendour and delight of Omelas depends on. On the suffering of the child. And the way Le Guin tells it, basically all the citizens of Omelas at some point are made aware of this fact, that the prosperity of their city depends on the fact that somewhere within its walls there's this lone child that suffers terribly. Most of them, according to the story, sort of make their peace with this fact. But a few instead think they don't want to be part of this. And those are the ones who walk away from Omelas. So they leave the city, they walk away, and they never come back. And I think many people sympathise with that reaction. They think that they too would walk away, or at least they hope that they would walk away from Omelas. And I think, yeah, if you have that reaction, it might very naturally suggest to you maybe there is some depth of suffering that cannot be compensated for by any measure of bliss experienced by others.
- [139:02]
Sujana
Yeah, the formal argument for how some very plausible assumptions actually force you to accept lexical threshold negative utilitarianism is in your paper the Weight of Suffering, which we will stick up a link to. But basically what I got from that is that there's a bunch of really attractive assumptions that lead you to believe the formal statement of this principle. There is some depth of suffering so horrible, no amount of well being can morally outweigh it. But, yeah, that same principle seems like it might lead you to think that a world without Any beings in it that can suffer would be better to avoid meeting that threshold instinctively. We don't want that conclusion to be true. We talked already about how you could set your threshold to a point that to where that conclusion isn't true, which seems like a kind of weird and arbitrary thing to do, but maybe there is some kind of principle case for it. What else can you do if you don't want to conclude that a world without sentient life would be better?
- [140:09]
Andreas Mogensen
I guess in some sense the only alternative is to reject one of the premises of the argument. So, for example, one premise that does a lot of work in the argument is this always outweighable principle, right, that we discussed, where it says if you're expanding the population by creating new lives that are happy lives, lives in which happiness predominates over suffering, adding those lives to the population might nonetheless be for the worse, or would be for the worse if sufficiently many bad lives come along for the ride as well. That's roughly what it says. So you could reject that principle. You could think that there are some lives so good that they can counterbalance any number of lives that are only barely not wor living, or something like that. You might also think that's pretty repugnant. So the other sort of key assumption that plays a big role in the argument is something called the reverse repugnant conclusion, or rather the negation of the reverse repugnant conclusion. So this is like the repugnant conclusion, but you flip the sign on everything. So the reverse repugnant conclusion says that there's some population of people having extremely long, horrible, tortured lives where every second of their life is utter agony. There's such a population. No, for any such population, rather any such population of people with lives that are just total constant torture. It would be worse if rather than there existing a population like that, there existed a much larger population of individuals whose lives are only just not worth living. So only just weakly negative. So these might be, say, people who live their entire life in a kind of stupor, who have really nothing negative or positive ever happens to them, but ever so often they hear a slightly irritating buzzing noise or something like that.
- [142:01]
Sujana
Right? Okay.
- [142:02]
Andreas Mogensen
So the reverse proponent conclusion says if you've got sufficiently many lives like that, where the only valenced thing that ever happens is this slight irritating buzzing noise. If you've got trillions upon trillions of lives like that, that would be much worse, could in principle be arbitrarily worse than having a smaller population of individuals who are just being tortured in the most horrific ways you can imagine or would prefer not to imagine.
- [142:27]
Sujana
Right, yeah.
- [142:28]
Andreas Mogensen
So the reverse repugnant conclusion says that larger population of lives that are only just not worth living would be worse than the smaller population of really horrible, tortured lives. And just as many people find the repugnant conclusion extremely hard to accept, many people also find this reverse repugnant conclusion very hard to accept. And so they prefer to believe the opposite, that the smaller population of the people with the really horrific lives, that would be a worse outcome overall.
- [142:57]
Sujana
Yeah. So to get to the conclusion of lexical threshold negative utilitarianism, that is, there's some depth of suffering that no amount of well being can compensate, one of the moves you make to get there is to reject the reverse repugnant conclusion. Right. So if you don't want to endorse the lexical threshold view, which maybe makes you say everyone should go extinct, you can accept the reverse repugnant conclusion. But the reverse repugnant conclusion is pretty repugnant. So there's kind of a rock and a hard place here, right? Yeah, yeah. It's a difficult trade off when you're met with a situation where you kind of have to choose between one horrible conclusion and another horrible conclusion. Is that a sign that we're just totally on the wrong track with our theories or is there some inherent difficulty of balancing the values of different populations? What do you think is going on here?
- [144:08]
Andreas Mogensen
I'm inclined to think it's not an indication that we're on the wrong track, it's an indication that we're doing philosophy. Because I think this is just like in some sense what it is to do philosophy. In some sense, the core of philosophy consists of puzzles and problems that arise when a number of things that all individually seem extremely plausible turn out to yield absurd results or something like that. So there's this quote from Bertrand Russell that I think I'm probably going to butcher, but it's like the job of philosophy is to start with something so obvious it doesn't need worth saying, and to end up with something so incredible that no one could believe it or something like that.
- [144:48]
Sujana
Right.
- [144:49]
Andreas Mogensen
And yeah, I think these deep conflicts amongst principles that otherwise strike us as compelling, that's a sign you're doing philosophy.
- [144:58]
Sujana
Okay, so we're doing something right. Maybe.
- [145:00]
Andreas Mogensen
Possibly, yes. But yeah, I certainly think if you think this indicates that we're on the wrong path, it sort of indicates that philosophy as a whole is taking a wrong turn or something like that, which maybe you want to believe.
- [145:16]
Sujana
Yeah, maybe we should Just stop doing philosophy and problem solved. Yeah, okay. I kind of want to push on towards what we should make of the fact that there are some arguments that are kind of hard to get out of that suggest humans or in fact all sentient life should maybe go extinct. Many of our listeners are longtermists or compelled by some of the ideas of longtermism that they, you know, we want to influence the long term future as a moral priority. And if you're a longtermist, the obvious thing to do is work to try to prevent the extinction of future generations of humans. But yeah, some of these arguments might make that seem less obviously a good thing. What do you make of this? Should longtermists be less sure they want to prevent extinction or should we embrace the intuitive view that extinction is a really bad thing?
- [146:17]
Andreas Mogensen
I think, I mean, ultimately, I suspect we should still embrace the view that extinction would be a very bad thing. I mean, certainly I think these considerations should make us maybe less certain of this than we were previously. But there's a sort of question about how much, and I guess with respect to something like this, question of whether the harm that we do to non human animals outweigh whatever good we might experience within human populations, it's very difficult to know exactly what the future holds. It might be the case that we get our act together and we succeed in ending or suitably limiting the worst of these practices.
- [147:01]
Sujana
Right.
- [147:02]
Andreas Mogensen
I think it's also the case, I mean, we've been sort of speaking about the possibility of human extinction or something like that, but we haven't really said very much about how humans might go extinct. And I guess maybe the sort of extinction program that might be justified by this, if any, would be a sort of voluntary human extinction or something like that. If you're instead worried about human beings going extinct from some kind of disastrous natural catastrophe like an asteroid strike or something like that, that's not only going to take out humans, but it's also going to take out non human animals. A large number of. Well, not only it's going to take out a large chunk of the population of wild animals or something like that.
- [147:47]
Sujana
Yeah.
- [147:47]
Andreas Mogensen
And similarly, if, say, what you're worried about is existential risks associated with advanced AI systems, you might think quite reasonably that if such advanced AI systems come to disempower humanity or perhaps even to bring about human extinction, it's probably not going to be good news for the biosphere as a whole. So if you imagine this kind of the classic paperclip maximizer story, the paperclip Maximizer is given some goal to maximize the output of paperclips. It's a super intelligent system, it's been given a very bad goal and it converts the known universe into paperclip factories. That brings about human extinction, but it also completely destroys, you know, all life on Earth or anywhere else in the immediate vicinity or something like that.
- [148:39]
Sujana
Right.
- [148:40]
Andreas Mogensen
So yeah, if you're worried about the impact of human civilisation on wild animals, there are many extinction scenarios that long termists and others might worry about that would be also pretty terrible for these wild animals.
- [148:55]
Sujana
Yeah, makes sense. Yeah. I mean it's interesting to think about AI here because I think we've been having most of our conversation like imagining life gets wiped out or not wiped out or something like that. But in fact what we might be dealing with here is the prospect of humans getting replaced by some new dominant species. And I sort of see the argument that like in the animal case, if AIs wipe out humans, there will be some kind of disruption to nature that comes with that. That's going to be bad for the animals too. But maybe. Is there any reason to think that kind of longer term life for animals on Earth might be better if AIs are the dominant species? This is totally speculative.
- [149:46]
Andreas Mogensen
Yeah, I mean it's obviously very hard to know. I mean certainly you might think an AI system has no need to eat and probably no desire for animal flesh or something like that. So the intensive farming of animals would disappear. The question of whether a given AI system would, if human beings were replaced by a civilization of AI systems, would that be worse for wild, non human animals? I feel like I'm not totally sure about this. If we're talking about radically misaligned AI systems that have strange values that are entirely orthogonal to human values, you might very well suspect that they would be also indifferent to a large extent to how well or badly things go with these wild animals or something like that. But yeah, it's a strange and very difficult question and I guess I'm not totally sure.
- [150:47]
Sujana
Yeah, yeah, that's completely fair. It was a wild card. Yeah. I think the other consideration, just to mention it, and how AI is becoming the dominant species changes the calculus in these cases is the question of whether AIs are their lives are worth living, morally valuable in some way. I guess there is some reason to believe that if AI lives are worth living, they could be extremely valuable. Like maybe, for all we know, I guess they could have a very high capacity for welfare goods or other things that make lives morally Meaningful. I think I won't get you to speculate much on how what you think the equation will be of should AIs replace humans, will they have more valuable lives than us? But seems like it just makes me think there's a lot of tricky questions here.
- [151:51]
Andreas Mogensen
Yeah, absolutely. One important thing I think to observe is that not only is it conceivable that they might have a much greater capacity for well being, but they might also have a much greater capacity for ill being or whatever of the stuff that makes life that counts against people's happiness and continued well being or something like that.
- [152:16]
Sujana
Right. We've discussed some pretty tough questions today and it seems like everywhere we look there's a bunch of open questions. With that in mind, what research would you be especially excited to see people doing both on the side of could AIs be more patients? And also on the side of the value of preventing extinction or the weight of suffering and so on?
- [152:45]
Andreas Mogensen
Yeah. So I think I probably got better answers for the stuff about digital minds. So one thing that I think is kind of underexplored at the moment is something like what would it take for an AI system to be not only phenomenally conscious, but also sentient in the sense of having affective experiences, experiences that feel good or bad, like emotional experiences and things of that nature. So there's lots and lots of debate and research into how to identify phenomenal consciousness in AI systems. It seems in principle possible that you could have subjective experience, but all your experiences are neutral. They neither feel good nor bad. And I think there's much less work on devising kind of criteria for what it would mean to have emotions or to feel pain that can be abstracted from human or animal neurophysiology and applied to AI systems. So I think that's one area of research that's quite important. Another thing that I think is also quite important, though I think this is gathering quite a bit of interest now is research into questions about how to individuate digital minds to explain why this is important when you think you're talking with Claude or something like that. So far as I understand, which may not be all that much, but so far as I understand, what's really going on is that different parts of the conversation are actually being handled by different instances of the Claude model you're talking with. And those different instances might be located in different parts of the world. And also whilst you're talking with that model instance, it's processing other conversations with other users in parallel. So you might have this Sense like there's a persistent interlocutor whose attention is fixed on their conversation with you, but it seems like actually nothing like that is going on. Right. There's like a series of different things you're talking with. And insofar as those have minds, maybe their minds are a kind of like weird jumble or something. Alternatively, maybe there is like a persistent interlocutor with a coherent stream of thought, but it's made up of little slices of all these different model instances that live on all this hardware spread out across the world. I think currently that feels like kind of a slightly scholastic question. It's like how many angels can dance on the head of a pin or something like that. Maybe. But certainly if you think that these beings matter morally, that they could be, say, harmed in death or something like that, then these questions about how you identify and individuate digital minds become quite important. But yeah, there has been some recent work on this issue by people like Jonathan Birch, Derek Schiller. David Chalmers has a recent preprint on this. There's also a paper by Chris Register about this.
- [155:33]
Sujana
Just to wrap up, I have one final question for you.
- [155:37]
Andreas Mogensen
Sure.
- [155:38]
Sujana
There's a stereotype that philosophers spend their whole days thinking about really implausible, pointless, abstract ideas. What is the weirdest example of something that you've spent hours thinking about?
- [155:54]
Andreas Mogensen
Not sure if I spent hours thinking about this, but maybe one of the weirdest ones is does God's omniscience violate our right to privacy? So at least if you assume that there exists a God of the kind accepted by Abrahamic religions, a theistic God, and this God knows everything about us, and in some sense, all knowing and all seeing, you might think that's a massive privacy violation or something like that.
- [156:18]
Sujana
Yeah.
- [156:18]
Andreas Mogensen
And so, yeah, I sort of wondered a little bit about this question. Is that in fact the case this would sort of indicate maybe a way in which the traditional theistic conception of God is sort of incoherent, because God is also obviously supposed to be morally perfect. But that morally imperfection might be incompatible with constantly spying on people when they're doing their private business.
- [156:42]
Sujana
Right.
- [156:42]
Andreas Mogensen
Yeah. So I spent a bit of time mulling that over at some point in my life.
- [156:46]
Sujana
Yeah. Yeah. That's kind of broken my brain a little bit. All right, that's it for today. Our guest has been Andreas Mogensen. Andreas, it's been a delight to have you. Thanks for puzzling through these bizarre questions with me and thank you so much for coming on the show again.
- [157:05]
Andreas Mogensen
Yeah, thanks for much for having me.