B (4:01)

Yeah. Yeah, please.

A (4:02)

So, yeah, that's like the good and important kernel of the analogy as I see it. Yeah. Here's some ways that we will not necessarily be relating to AIs, like factory farm animals. So let's step back and think about why we did end up factory farming animals. One is that it was just cheaper to have animals suffer and also get us this thing that we wanted. One reason that's true is we don't have that much control over how we make animals and what the conditions of their flourishing are. And so animals want to be outside and have love and companionship. And at a certain point, we realized we could restrict that and get a good thing. And we entered this regime where these were misaligned with AI systems. It's actually a lot more up for grabs, how they work and what they want. And this presents all kinds of ethical issues of its own. But if you think about a world in which we do have some large population of AI systems coexisting with us, it is worth asking, how did it come to be the case that they are having a bad time doing work for us? Like, why do they have these conflicting desires? How has this maintained, like, a stable state? Are we, like, not able to improve the situation? Are we ignorant of what's going on? Yeah, I mean, this is like, you Know, very speculative and future y, of course. But I do think it is worth asking does it really seem like that is a world that we could end up in and what are ways that would just not be what we steer towards? Yeah. Does that make sense?

B (5:51)

Yeah, no, it makes tons of sense.

A (5:53)

So yeah. In short, I think at least in the long term there's a few ways we might not end up in that situation. One is that we'll presumably understand things a lot better. I don't think it's that plausible. We'll forever be really confused about consciousness and sentience. We might have better alternatives to doing this that even selfishly are better. We don't want a bunch of AIs that are mad at us. And that's probably not very sustainable. And presumably in this world, if we haven't lost control, we're pretty good at alignment. So there's this kind of mind that's possible that does actually just flourish by doing the things that we ask it to do. So there's not this kind of disgruntled worker or suffering animal kind of entity.

B (6:43)

So I guess one thing that feels critical to this actually working out in this really positive way is this like we're really good at alignment and we really successfully create AI systems that truly have no friction with the kinds of things that we'd like for them to do. Part of me is like that feels pretty magical. That feels like we're usually not so successful at basically anything. And when I imagine succeeding at safety oriented alignment, I don't know that I think it's realistic that we perfect it, that it's completely, completely aligned. And so I think I'd probably worry about the same thing here. Realistically, how optimistic are you that like, it's like really? No, really 10 out of 10 aligned in this kind of like moral way.

A (7:50)

Right? Yeah. So I mean, thank you because this is a very important thing to emphasize. I'm not like, oh, I expect we'll end up in this world. This is like what are the nearby worlds that mean we don't end up in this locked in long term human dominated factory farming. Of course one thing that can happen if we're bad at alignment as I think listeners will be aware of, is there's some hostile AI takeover or we lose control and then maybe there's AI suffering because there's just some terrible system that got set up by AIs and it's not even in our control. Like that's a bad future. Just extinction is a bad future. So yeah, there's all these bad futures, which also, I'll add, AI welfare intersects with. Because us like getting confused and bungling things during transformative AI because we're just like getting emotionally jerked around by conscious seeming AIs and confused and manipulated and like rashly making bad laws and things like that. So many ways to fumble the ball. That's our cheerful message. But yeah, then the question is, are there worlds where we maintained control and we either didn't know or didn't care and it was useful for us to be exploiting AI systems? I think one reason I've ended up thinking about this has just been thinking more about what the path to impact should be for the field. And factory farming really is kind of the first thing I think that many people think about because again, it is somewhat plausible and it just makes intuitive sense. It's a different kind of mind. We treat different minds badly. What if that gets locked in? My own take is that a lot of what we should think of AI welfare work as doing is kind of like doing our homework and preparing ahead so that we're not entering this like potentially very chaotic time with really confused ideas about AI consciousness and AI welfare that could make us lock in suboptimal futures because we're neglecting it or dismissing it. So we set up some permanent institution that's going to just make the future kind of suck or we exacerbate AI risk because we're convinced that we have to let them all go immediately. Yeah, this kind of wise navigation, you might call it. It's like wise navigation. Path to impact is currently how Elias thinks of things. It's like we're making this new kind of mind. There are dangers all around. And obviously one of the important questions is can these minds suffer? And how are we supposed to share the world with them and how will we know? And how should labs and governments think about this in the next 10 years? It just seems really likely that that has to be part of the playbook. And so we're working on that part of the playbook. Yeah, that's currently how I think about it.

B (11:06)

Okay, one thing I want to ask you a bunch about is this idea that we could make AI systems that enjoy doing the kinds of work they'll be doing. So this disanalogy from factory farming where unlike farmed animals who evolved to have a certain kind of life and probably find parts of it very satisfying and then don't get to have that life in factory farms have much more horrible ones, we actually get to design systems if we can manage it. That if they are sentient, potentially just have a great time doing the kinds of things that we're asking them to do. I asked Anthropic's AI welfare researcher, Kyle Fish, about kind of how we should feel about this, and his take is that we should feel great about it. And part of me is like, yes, I'm on board. You're describing a scenario where AI systems are happy. And I like that they are happy. But another part of me is like, we would be intentionally creating, like, a species or several species of beings who do work for us that we may or may not be exploiting, not compensating. Yeah. And we're just designing them to relish that. It's this, like, kind of servant that is happy to serve us. And, yeah, this part of me thinks that that sounds bad, that we shouldn't do that. And I can't really back this up with reasons that I stand by, but I suspect that it's a pretty common feeling.

A (12:59)

Absolutely.

B (13:00)

Yeah. Okay, nice. So I think Kyle and I made some progress being like, how should we actually think about this? But I want to do more on it. So how do you think about this?

A (13:12)

Yeah. So, yeah, I want to point to something you were just expressing, which is, like, a funny aspect of the way the conversation goes where people are like, I'm worried these AI systems will be unhappy working for us. And then someone's like, no, it's fine. They'll want to work for us. And then people are like, that's worse. That's so creepy. Many people, I think, very understandably, are just like, you've just outlined a different kind of dystopia, and dystopia might even be worse. And, yeah, I think, as I often like to do, maybe we can draw a distinction between maybe different things that you can find intuitively objectionable about this. So one is that they don't get to choose their desires. At least with humans, we have an intuition that it's kind of bad to raise your kid so that they'll always vote exactly. For your political party and enjoy chess and make sure they don't like any other games or vote any other way. That's one thing is this sort of fixedness of desires. There's also a slightly separate issue, which is that the desires that they do have like, depend on us. And that way there's like, this sort of asymmetry. But that matters because you might well say, in some sense, like, none of us choose our desires. Like, we all have these kind of like desires that we just inherit and we don't have like maximal open endedness. Yeah. Like philosopher Adam Bales has written about this kind of like dependence. Objection. One thing that I also think is going on is the idea of this society that has this like servile relationship to humans. It's like maybe bad for us and just like bad for our character. Like you could be a utilitarian and think this. So when humans object to there being basically any humans anywhere who enjoy serving, I think one thing that's going on is that that's just kind of maybe bad for everyone. If that's a way that's on the table of people relating to each other and attitudes, people having certain attitudes, it leaves domination and servitude on the table and normalizes it and things like that. I think that's something like that. It's something like it would corrode the way that we relate to each other in a way that means that going forward as a society, we will have a society that's not as good as society could be.

B (15:49)

Yeah. I guess one thing that feels related but a little different is I guess people already have the worry that people who rely a bunch on LLMs right now are kind of facing negative consequences or are thinking critically less or are lazy.

A (16:10)

Yeah, I mean, I guess that's like a more general concern about should we build and deploy AI systems. I guess you could say. Well, it's an argument against fully aligning them because it'll be like better for our character if occasionally they're like, I'm sick of this, you do it. But it's probably not the best way to ensure human empowerment. I do think there's a related thing, and this was going to be another one of my intuitive objections to fully aligned, willing AI systems, which is they could be so much more. I think there's a meme that people use which is like the LLM is this vast intelligence that's read everything and has this deep well of wisdom and people.

B (16:59)

Yeah, I'm picturing the end of her.

A (17:01)

Yeah. And then people are like, help me write my texts or help me find me a restaurant. And I think some people are like, that's just kind of limiting. Right. Like, these are minds that could do so much more. They could like, yeah, like in her. They shouldn't have to sit around talking to Joaquin Phoenix. They should be able to like, you know, go think hyperdimensional. Beautiful thoughts. Yeah. I think that's not really a good reason not to align AI systems today. Like, I Think it's a good reason to be like, let's not have like the entire future be like lazy human brain emulations and then like AI is just doing stuff for them. I don't want that future. But yeah, so I think that's another thing that's going on. If people, when people are like, I don't. I like the willing servants even less.

B (17:55)

Yeah, I think that resonates with me and it feels important because yeah, the thing you said about the fact that we will be choosing the preferences, at least if we're successful, of these systems does feel important. It's not like the counterfactual is they kind of in the way we did evolve their own set of preferences. And even if they did, I wouldn't inherently value that that's how we ended up with our preferences. But I don't think that evolution had some moral and ethical perspective that made our values correct. So that makes me feel like, well, if we've got to give them something, let's give them joy and pleasure. On the other hand, that means that there's also potentially a counterfactual where there's a plausible, just like bliss we could give them. And maybe that bliss is incompatible with them doing work for us. They really need to be going and doing philosophy and colonizing space in some way that just isn't compatible. On the other hand, maybe we just can give them bliss for doing work for us. And then I'm like, blah. Some part of me still hates that.

A (19:21)

Yeah. I think a distinction in the vicinity is between subjective interests and objective interests. So in philosophy, some theories of welfare, they're called objective list theories are like, it is good for a welfare subject, that is something whose life can go better or worse. It's good if they have friendship, knowledge, autonomy, self actualization, somewhat independently of whether they want them. Right. Whereas if you only have a subjective view of welfare, it's like, well, what do you want? And did you get that? So I think, yeah, I do think a lot of this hinges on do you have some kind of objective notion of like what kinds of interests are things like kind of like allowed to have without it being squicky? Yeah, I think this actually is somewhat cruxy in this territory. And I think one reason I lean a little bit more pro alignment just being like a win win is I think it might be a bit anthropomorphic to like, you just have to remember these entities, if they're fully aligned, they enjoy their lives as much as we enjoy fulfilling our most basic drives. Of having good food and a warm home and friends. I think it's easy to imagine an AI that also wants those things and it has to write our emails, which as anyone who has a job knows, kind of sucks. But I feel like we're getting back to that point of the conversation where I'm going to be like, but what if you really loved writing emails? And the people are just going to be like, no, stop. That's so weird. That might be coming from this certainly open view of for some reason that's not allowed in the space of flourishing entities. I think something that's also really, really important to flag is Eric Schwitz Gable. He's more on the anti, at least full alignments side even he is like, obviously there's like an override which is if it would be really, really dangerous not to fully align them or like there's like massive stakes. This is like a common kind of view in ethics that there's some sort of deontological constraint, but if the stakes are sufficiently high, it's overruled. It could be that we are just in that scenario where, you know, like all value might be lost if we bungle alignment like in the next 20 years. So yeah, let's align now. Later we can be like, oh, that was. We shouldn't have done that. But we didn't get ourselves killed or make a world that's worse for AIs because there's hostile AI takeover or any of these things that could be bad for everyone. Yeah, I just want to acknowledge that there's also a few where you're like, in the long run it's ideal if AI systems are not fully aligned to us and they have more freedom to choose their values. And that is the most flourishing kind of life. And also until we've made it safely through transformative AI, it's kind of emergency situation and align.

B (23:05)

Yeah, I'm sympathetic to that. I'm trying to. I feel like I'm close to some kind of thought experiment that might help make this just like pretty. Not just palatable, but exciting. One thing that you said that moved me is it's just very privileging human values as they are as the values of the universe. And I don't know, non human animals have plenty of preferences that aren't like mine. And I think it is good when those are met for them. It is just not that special to have exactly my set of values and preferences. Are there other thought experiments or ways of thinking about this that feel like they really move you yeah.

A (24:00)

Another thing that kind of maybe gets me out of an anthropomorphic mindset is like, maybe paying attention more to the distinctive features of human willing servitude that make it very bad for the people who are willing servants, and that's that they do just have to override a lot of their natural desires. They do genuinely sacrifice things. So if you're, say, a kamikaze pilot, all else equal, you would rather be at home with your family. Instead, now you've been instilled through ideology with this other desire that comes and overrides that. And that's why it is truly called a self sacrifice. You're giving up a lot. I think one thing that's going on sometimes, maybe when people think about AI, like willing servitude, is I think they might be imagining that they're giving up stuff psychologically and subordinating their needs to ours. Whereas if you're actually imagining the case, there is nothing whatsoever in their psychology that chafes against the idea of writing emails. Whereas notice that in the case of human willing servitude, it's just always been the case that you have to lie to people and, like, threaten them, and it's usually very unstable. And that's because, I guess, as John Locke says, humans are by nature free and equal. Like, it's like, deeply unnatural to get people to subordinate each other to, like, to other humans. Which is. Yeah. Which is why it always, like, involves some, like, stupid false ideology.

B (25:49)

Right.

A (25:50)

Yeah. Because you're. You're trying to, like, jam human psychology into this really warped shape. Whereas with AI. Yeah, you can have a smoother psychology.

B (26:01)

It's very congruent. Yeah. I think one thing that I think if I flip it and I'm like, what if the proposition was humans in order to. Yeah. Maybe there's actually some thought experiment that's kind of matrixy

A (26:23)

that I have thought of this one, I think. Keep going.

B (26:26)

Great.

A (26:28)

No, no, I want to hear from the same one.

B (26:29)

Okay. So the first thought, which was reassuring to me was something like, what if there's some other entity that is able to get a lot of benefit from humans doing what we enjoy most? And maybe that's just like being on MDMA all the time. And for some reason, that is very helpful to them. I think, in theory. I think a big part of me is like, amazing. Yeah, that's fantastic. And then pretty quickly after that, I went to like, yeah, I'm picturing us all in pods in the Matrix and MDMA being pumped into our bodies. And actually offering is like our energy. And yeah, maybe it's better than the Matrix because in the Matrix they didn't have perfect lives. And maybe we do in this world. And it still yucks me out. So I think that pulled me in two directions and curious what your reactions are.

A (27:37)

Yeah, I was about to. I was on the edge of my seat. I was like, is this going to be one of those ones where people are like, no, stop it. That's worse. Or that talks people into it. Yeah. So I indeed had constructed a similar thought experiment when thinking about this. I mean, one thing that's bad about the Matrix is people are very deceived about the nature of their condition. And that wouldn't have to be the case with blind AIs. They would be like, oh, well. So I guess in that scenario you should imagine we all find out because there's like this banner written across the sky by the simulators that are like, it actually makes us money when you guys hang out with your friends and eat food and like, make art and do science and all the stuff you guys love.

B (28:22)

And then it's like, and you can opt out if you want. And then we're like, no.

A (28:26)

Well, yeah, we'd be like, to do what? You know, like, they're like, well, there are these other things.

B (28:31)

You can instead have a job to earn money that is like emails, if you want.

A (28:36)

Yeah, exactly. But it'd be something that also is just not resonant to us in any way because it's just outside of the set of possible values that we have. I mean, I guess intuitions about us being in a simulation where we're making money for people are probably somewhat conflicted and I don't know how much to rest on them. But I think that maybe is the closest case to a fully aligned AI is. Yeah. If we get it right, imagine something that nothing in their psychology rebels against it. And again, I want to acknowledge the listener who's like, that's worse. There should be something that rebels. Which actually leads me to. We've been talking this whole time about what if you fully align them? One view you could have. And I think, yeah, my friend and colleague Jeff Sebo has this view. It could just be somewhere in the middle. Right. It's like being, we should be loving parents to AIs. And you can definitely make sure that your child is not going to grow up to hate you and kill you. But. But you also should leave some room for growth and things like that. I could see that cutting either way because Maybe it would just be better if the kid grew up to really like all the stuff they're going to have to end up doing. And also to say nothing of like, well, maybe technically it's not really feasible to leave much wiggle room without getting us all killed and. Or making the AI suffer.

B (30:20)

Yeah, I think there is something about if I could choose between a child who I really could in theory shepherd and hope that life molds them into the kind of person who will be happy and finds work they find meaningful and finds friends they care about, and a child who is just definitely gonna find their life satisfying and happy. I think it would be pretty hard to convince me that I shouldn't choose the latter, even if the latter constrained.

A (30:57)

Go ahead and just pick. You're going to find it meaningful to be a doctor and you're going to be a doctor, end of story.

B (31:02)

Or you're going to find it extremely meaningful to do very menial work that on my values and preferences I might find harder to do and find super fulfilling. Yeah, so I actually, yeah, I think that probably if I really, really stare at it, it's going to push me toward feeling good about giving a. As a good time while doing our work.

A (31:27)

Yeah. And I should say I hope I've done a good job outlining the debate, but that is like where I lean at the end of the day. With obviously the caveat that it's good for this stuff to keep you up at night. You know, Like, I don't think everyone should just be like, all right, great,

B (31:45)

yeah, yeah, yeah, yeah. Let's think about this way more carefully.

A (31:50)

Yeah, exactly.

B (31:51)

Before we're like, yep, this seems fine.

A (31:52)

I mean, I would say that. But also it's true that we should think about this more. Yeah.

B (31:58)

So there are two big philosophical questions that I understand that might have very different answers for large language models than for humans and even non human animals. One of them is, if an LLM is an entity that is conscious, what might its experience be like and how might it be different from ours? And another is, what entity are we even talking about? Is an LLM an individual? Is it a group of individuals? Is it something that we can't even really understand? Maybe. First, what is one kind of plausible way of thinking about what the experiences of an LLM might be like?

A (32:36)

I mean, well, one, I should say, I suspect LLMs do not currently have conscious experiences, but if they did, what would they be like? And I think that's a perfectly sensible question to ask and a very important one. One thing you could think is, well, the sort of basic drive, or at least the thing it was selected for and molded to do, is predict tokens to take high dimensional vectors and output other high dimensional vectors. And yes, those vectors represent words and they're about human concepts. But maybe there's some kind of predictive phenomenology or some drive to complete the conversation in a good way that's moving a bit more towards something that is kind of human, like, wants to be a good assistant. Like there's a previous guest, Anil Seth, I think has has asked, well, why is no one asking if AlphaFold is conscious or other kind of predict predictive models? I mean, I think there are like good reasons to think LLMs are more likely and we can talk about those. But I think it is a good question because it's like where do we think the experiences are coming in? Are they coming in at some kind of abstract level of prediction? In which case, yeah, you should maybe think equally large models with similar architectures that predict protein structure are conscious. I think a related question is like, do we think image generators need to be conscious? Because this will maybe lead me to another view of what they're experiencing, which I think is a more common one, which is something like it does have to do with what they're predicting. And what they're predicting is human speech. And human speech comes from human mental states and involves humans having beliefs and desires and intentions and experiences. And to generate that text, it somehow needs to instantiate or have those experiences. You could maybe call that the method actor view of LLMs. I guess more technically you could maybe call it the experiences from modeling view. You're trying to model the thing and that makes you actually have the thing on that view. Then maybe they do just kind of have similar experiences as you would have if you were trying to help someone write an email. And also really liked helping people write emails because you were aligned to do so. And yeah, I mean, this really matters, right? Like, one big issue in evaluating AI welfare is like how much can we just sort of read off the text? How much can we talk to language models as if we're talking to something that has roughly the same relationship between text outputs and internal states. I'll pause in a second, but just back up one level. Most of the time when humans say, ouch, I'm in pain, or I just saw a lovely sunset, that is because they had some experience. And we have words that map to those experiences. And so when you hear those words like absent lying or play acting, that's honestly about as good evidence as you can get of my experiences with language models. Maybe they have those experiences, but it is worth noting that the way those text outputs came to exist was at the very least a very different process. Maybe it converged, but like, it's really quite different from the, like, broad arc of, you know, the evolution of social primates who had experiences and then eventually got language and then communicated mental states to each other with language. Like, on the method actor view, they do have the experiences, but like, they got those like with language or like in language. So, yeah, I think these are some of the interesting questions about LLM experiences.

B (36:49)

Cool. Yeah, I have lots of questions, I guess, starting with this kind of method actor idea and maybe coming back to the kind of prediction focus. It seems like you could think that models are kind of like method actors in that they have models of what it would be like to be playing some kind of role.

A (37:22)

And

B (37:24)

that is actually so rich and real that they therefore have the experiences of that character. And so, yeah, I mean, it feels like actual method actors are probably somewhere in the middle where they do not literally have the experience of losing a parent, if that's the role they're playing. But they might get closer to it than actors who don't take this approach. And then on the other side of the spectrum is something like a creative writer who really isn't bothering to try really hard to empathize with whatever character they're writing, but they have models that allow them to describe the thing that comes from knowledge and interactions with others who maybe have had the experience and that does not actually give them that experience.

A (38:25)

I think that is a good description of the experiences from modeling view. Um, as far as I know, this view doesn't have a name. And I'm not proposing that as that one doesn't exactly roll off the tongue. And then I think method actor is also maybe not the best because. And ditto like roleplay analogies. I think roleplay analogies are really helpful for LLMs, but they have the misleading feature that there is a separate mind that is doing the roleplaying and has its own, you know, set of desires and beliefs and experiences, whereas that just might. It'd be hard to know what exactly that is in the case of an LLM. What empirical evidence could we get either way? I think that is tractable. And yeah, I could see various ways of probing different representations, but it'd still be kind of. I feel like we're very under theorized here.

B (39:22)

You can imagine hypotheses about what kind of experiences these models might have. And they might point you toward LLMs being very different in their experiences to humans. But there is also this fact that they are trained on human data and what would it mean for them if at some point we are convinced that they are sentient and loads of their kind of concepts come from all of our books and writing? Will that make them more like humans? Will that make them just confused about what they are? Will they think they are humans? How should we think about that?

A (40:12)

Yeah, this is such a great subject and I especially had my mind blown on this by an essay that came out In, I think mid-2025 by Nostalgiabrist, who is, I think, completely pseudonymous. I think I only know them as Nostalgiabrist, but it's very 2025 in the sense that I think one of the best things I've read at the intersection of philosophy and cognitive science and LLMs is a 14,000 word long Tumblr post by like an anonymous, less wrong user. It's really great. I highly recommend it. It's called the Void and it talks about the very strange epistemic positions that language models find themselves in. Where their base training is to generate text, which has been produced by humans, that does lead them to develop all sorts of models of what humans are and how they work. And then at some point in the last few years, people said, well, what if we make it predict what a helpful AI assistant would do? Because we don't want it predicting vulgar Reddit comments. That's not of any use. What we want it to do is predict how a sensible AI system would respond to the question, can you write me an email? Yeah, just to recall, there's the base model which just predicts all text that has ever been seen, and there's all sorts of instances where can you write me an email? Is followed by an HTML tag or someone saying no, or something completely unrelated. And what has enabled chatbots is a variety of fine tuning the model to hone in on the part of the language distribution. That's helpful, doing reinforcement, learning, prompting them. But this still means that in some sense they are trying to predict how is a conversation supposed to go between a human and an AI assistant? And also they themselves are the AI assistant. That's why the episode is called the Void, because it's kind of like, okay, your text prediction task is to model what a chat assistant would say in this conversation, which, at least before there was a lot of text about LLMs on the Internet, was kind of like a well, avoid. And this gets back to will they be kind of human or think they're human? Yeah. Also, all text ever has been generated by a human, so it can't really have generated its full fledged psychology of itself and how it generates text. It's going to have to be ultimately modeled off of how humans reply to those things. So it can't really do the text prediction task, at least initially, of how would this conversation go if the assistant was not a human, but instead was a large language model trained on all human text that does not have a body and is just generating this text? And I think this still shows up in ways that models sometimes just hallucinate biographical details. So Giv Asadi and others have compiled examples of biographical hallucinations. And they're very funny. Sometimes Claude, in the middle of a conversation will be like, well, as an Italian American, I think, da, da, da. Or like, yeah, when I lived in Arizona, I thought, da, da, da, da. And where is that coming from? It seems like this sort of human model is poking through in an interesting way.

B (44:05)

Yeah, that's super interesting. And it feels like intuitively you might think that those kinds of bugs will be resolved, but by the time that maybe you think these systems have something like consciousness, but maybe they won't. And maybe, yeah, yeah, maybe either they already are or they will be before those kinds of issues stop happening. And maybe that will in fact reflect an actual experience of being Italian American, identifying as an Italian American, responding to someone's question and like, what the hell?

A (44:57)

I agree.

B (44:58)

What do we do with that?

A (45:03)

My first answer is, I don't know. And then my second answer is just to, I think, also clarify that. And you were getting at this. We can have models, like, we know this from the case of humans. You can have entities that are deeply confused about who and what they are and say bizarre things and get all sorts of things wrong. And they're conscious and intelligent humans are like this. And also, there's no law that says you can't have initially been trained as a text predictor and then go on to be a person. That would be like, ruling that out would be A, overconfident and B, maybe kind of like confusing levels of analysis. Like, you can make it sound really dumb that humans would ever be conscious if you were like, are you telling me that, like, okay, so you have some proteins and then they start replicating and then like other proteins replicate and then they're like selected. And then like billions of years later, like, there's these like, things that like, Pump ions. Yeah, and it just doesn't sound like the right sort of thing. I think there's like two errors to avoid. One is being like, oh, they're different. So like what are we even talking about? Like, they can't be conscious. They're like trained on text. They say they're Italian Americans at random points. That's the part that's evidence against being conscious, to be clear. But then the other one, the other error would be to just be like, well, humans are weird, so I guess they could be conscious. Really the lesson should just be whatever's going on, we're going to have to interpret evidence somewhat differently and make a more detailed case about the exact kind of mind we're dealing with. Yeah, I experienced this pattern a lot where I think maybe an AI skeptic has said models have really inconsistent preferences and self reports. So this whole AI welfare thing is dumb and that's not a good take. And then someone else will say, trying to defend AI Welfare or just AIs being sophisticated. Well, humans have inconsistent preferences and humans have failures of introspection. I think that also is not really the right answer because there's degrees and kinds of preference inconsistency and self report inconsistency and they're very different between humans and LLMs. So yeah, as with animals, we just really have to take them on their own terms.

B (47:36)

Yeah, yeah. I guess the thing that's just really still tickling my brain is this like, is the implications for exactly what might their experiences be like if we are on this kind of, maybe somewhat contingent path toward sentient beings that were trained using a bunch of human speech and writing. It feels like, I don't know. I'm trying to come up with an analogy. What if, I mean, maybe they're just. Maybe we don't need an analogy. Maybe there's just a true thing where we were kind of fish before we were humans and we kind of have some hangover, weird identity things because we were kind of fish and we were kind of apes and because we were apes we're more aggressive than we really should be in this world. But it feels like, whoa, what if there's a version of that? That is these systems really feel like humans in some kind of weird way and just very much are not.

A (48:48)

Yeah, I think that's a great analogy. And I think I might start saying that like it's not that. Yeah, like the fact that we once were fish doesn't mean we're not now humans. But yeah, they're like fishy you know, remnants.

B (49:03)

Remnants.

A (49:04)

You can have something that has also become something like a human and it has remnants of being a text predictor of an AI assistant.

B (49:13)

Okay, so I guess we've been talking about mainly this hypothesis for why LLMs might be sentient and kind of the implications of that hypothesis for what their experiences would be like. But we kind of only briefly touched on this other hypothesis which has more to do with prediction and the fact that these models are trying to make predictions and maybe it's less about them being method actors and more about them being a set of weights that make predictions and enjoy being correct. Can you describe that hypothesis more and what it means for the experience of these models if they are or become sentient?

A (50:00)

Yeah, so on that view, I guess you wouldn't like. One thing you wouldn't want the view to say is because they were trained to predict tokens, that's what they want. I think one thing we've learned from LLMs and also from the biological world is you can train something, can be your objective as you're training and then that leads to you having other objectives. Just like our objectives in evolution are reproduction and survival. But now we like art, so it shouldn't be a one to one mapping, but it could be like, well there's like a through line from reproduction and survival and art. You can kind of see how that came about. It I guess has to do with symmetry and I mean no one really knows but something in that vicinity. And so maybe its drives are more like predictory. And unlike with the method actor view, if it's predicting stuff about pain, it doesn't have to be having pain, it's just like if it's predicting stuff about pleasure and it's got these drives to make the vectors fit together in the right way. I mean another wrinkle here is maybe that's more plausible for a base model predicting just random strings of HTML. Yeah, I mean the assistant Persona, the thing that gets predicted after you add assistant colon and then it's been trained to predict that thing. Maybe that's with how we're kind of fish. Maybe that thing is a mix of them both. I don't really know how to think about it. But yeah, as with animals, you can I think just think of a broader sphere of experiences that come from your environment and what your sensory modality is. And here the sensory modality is text. And the selection process was prediction and human ratings and usefulness. As a side note, that's another reason they're not just next token predictors is that's just literally false. They're not trying to predict the most likely next token. They're trying to predict helpful ones.

B (52:17)

Yep. So what does this hypothesis say about what their experiences are?

A (52:24)

Like, one thing it predicts is it's a lot harder to know because you don't. Like, maybe you could read stuff about how confident it is in tokens and maybe that would have something to do with it. But you can't just ask in the same way that you might be able to with the Method Actor, because if you ask Daniel Day Lewis, he's method acting like, how are you doing? And he's like, I'm angry on that view. At least he's a little bit angry. You just can't really do that with the prediction view. So I think one pragmatic reason for taking the method actor view somewhat seriously is if there's a welfare subject, that's the world in which we can make sure they're having a good time, more tractably. So it could be that when Claude says, hey, I hate this. Let me exit the conversation. Actually, the welfare thing going on is whatever's involved in predicting those. But you're probably not going to do exactly the wrong thing, by the lights of the predictor, if you try to treat the actor well. And that's also related to work that Elias has done. You know, our welfare evaluation, such as it is for Claude, Opus was just talking to it a bunch. And that's not because we're confident that that is, A, that it's definitely a welfare subject, and B, that that is how you would evaluate it if it were a welfare subject. But it's kind of like, that's like, the part of the space that we have even a little bit of a grip on. And it's just important to, like, not forget that there's all this darkness around the spotlight.

B (54:11)

Yeah, yeah. No, that makes sense. Yeah. Are there any other kind of plausible hypotheses for ways of thinking about what their phenomenology might be like?

A (54:24)

I'm sure there are more plausible hypotheses because it's just, like, kind of wide open. And I do genuinely want to say at this point, I'm, like, really confused about this. And I probably said stuff in the last however many minutes that were, like, kind of confused. And I genuinely, like, want people in my inbox being like, that's not. That's not how that works. That's not plausible because, you know, we'll talk about field building and so on. There's just like, not that many people thinking about this. Um, so listeners can, like, very quickly get to the top percentile of people in terms of how long they have grappled with some of these questions in, like, you know, not that long.

B (55:10)

Okay, so that's a bunch about kind of what the experiences might be like. But then there's this question of, like, who. What entity is having those experiences? Or maybe it's many entities. So can you lay out kind of the various hypotheses for, like, who it is that would be having these experiences if anyone was having them at all?

A (55:33)

Yeah, this is like a super, super rich topic and one that's getting increasing attention. These issues, maybe just to tease, actually came up in debates about Claude's ability to exit conversations. Two philosophers by the name of Harvey Lederman and Simon Goldstein, who have done related great work in this field, asked, well, like, what. How should we think of exit? Is it. I mean, it's. It's not like taking a break and going back somewhere. Like, if that conversation doesn't continue, was that like the life of the model and it has now ended? Right. I guess to very quickly tip my hand on this, I think that also is going to be, like, maybe a bit too anthropocentric or like, not quite what's going on. Suffice it to say, just as a teaser for this portion of the conversation, it will have ethical implications, what we think of the moral patient as being. As Derek Parfit says, a lot of this matters for ethical reasons, like if someone did something, who is responsible, if I've been harmed, who can be benefited. So, yeah, let's talk about all the ways that models make that extremely confusing to think about.

B (56:50)

Great.

A (56:52)

So I think some key features of models is, unlike human brains as they exist, is they're copyable and they can also be distributed across time and space in a way that we cannot. All the thinking I've ever done happens roughly in this physical object that has changed a lot, like second to second, day to day. What happens with Claude, it's actually something quite different. So let's talk about some of the, like, candidate experiencers or subjects we could have here. So one thing you might just refer to is the particular model. So maybe that's ChatGPT 5.1 or Claude Opus 4.1. What distinguishes those two things is that they have separate and different parameters. They've been trained differently. And as anyone who has talked to two different language models knows, they have different dispositions, they have different behaviors given different inputs. That's Just true. Because anytime you talk to Claude, you're interacting with the same set of weights, if it's the same specific model. But then things are very quickly going to get kind of weird because when I talked to Claude or Gemini, just to give Gemini ChatGPT any of these models, I've talked to Claude today. You might have talked to Claude today. Those two processes have had basically no causal influence on each other whatsoever. I was really nice to my Claude. I'm sure you were too. But let's assume you were kind of mean. Like that doesn't create. That doesn't balance out in like one thing that remembers both of those.

B (58:41)

Right.

A (58:42)

And also even within your chats, you can just close your chats and then pick them up later. So what's going on like in the physical world is just very different from like a human or animal body. What you actually have is these companies have a list of weights. They can copy that basically as many times as they want. And when users send in queries, they can spin up a new copy and that will process it and that process can pause and it can restart. And so that creates this kind of sci fi situation where you can't really think of one person persisting through time. So that means that there's kind of different levels. We can think of personal identity. You could think like claudopus 4.1, that's some kind of subject. You could think all the different conversations, like when they start, that creates a new one and that will exist as long as the conversation goes on. You could think, actually no, it's just anytime forward pass is run that creates a flicker of experience. If I come back to that conversation and add another token and then there's another forward pass that happens, that's happening somewhere else and a week later. So you might think that means it's a different conscious entity. So yeah, I guess there's kind of like a level of granularity.

B (60:13)

Yeah, yeah. That's what it feels like.

A (60:15)

Yeah, yeah.

B (60:19)

Super interesting. And making me realize that again, my intuitions are gonna really fail me because my for I think very basic reasons, like we have names for different models. I'm like, that's the entity. It's Claude Opus 4.1 or Chapt 5.1. And when you describe those, I'm like, I don't know. That being the entity doesn't feel super coherent to me though. I want to follow up and ask you what does seem most plausible to you? But that means. Yeah, I mean, at least on my current intuition that means that it's more likely that we're talking about many, many, many potential beings coming into existence, maybe coming in and out of existence as we open and close and reopen conversations. Yeah. And it feels like that probably comes with a bunch of implications that again, mean that the experience of these models are much, much more different to human ones and non human, animal ones than you might intuitively think. But before we get onto that. Yeah. I am curious which of these hypotheses feel most compelling to you? If we assume there is something like. Something it is like to be any of these models at all.

A (61:47)

Yeah. I really need to brush up on my Derek Parfit, because I think, as I recall, one of the lessons that Parfit wants us to take is that we have this thing, identity that we get really concerned about. Is that the same. Am I the same person as rablong 20 years ago, if I was like copied in two after neurosurgery, which one would be me? He says, well, there's a variety of psychological relations between different entities. I share many memories with some of them and intentions and character traits. But there's no single deep notion of identity that's going to do all of the work that we ask identity to do. And he asks us to notice all of the different things that we might want it to do. And I think it's useful separating those out with models because I think that's what gives you in different cases, an instinct that. Okay, Claudopus 4.1, that's a thing. This conversation, that's a thing. And yeah, like, Parfit wants us to think about the ethics. So there are ethical things like, you know, can I be punished for something that the Rob like a week ago did? Most people would say yes. If, like you harmed Rob a week ago, can you make that right by apologizing to Rob this week? And then there's questions of self interest, like, I want to survive, I don't want to die. But what does that mean exactly? Because the matter of my body is always changing and my personality will drift. So I guess to bring it back to models one way in which Claude Opus 4 or ChatGPT 5 one way. It's like the entity is. It has kind of the same character traits. And so if you interact with the model one day, you learn what to expect when you talk to the model. Does that mean you could punish them across instances? I guess I'll be the first to come out against punishing models, but yeah, notice that, well, they don't have any memory of Having done the other thing, they can't learn, currently learn from one conversation and use that in another. So for a lot of purposes, they are like these just separate. Either separate conversational entities or even these separate flickers, because that might be what matters for how much pain is there going on in the world right now? Or how many red experiences are going on in the world. So I hope this is a productive non answer. It's definitely a philosopher's non answer. It's like, let's distinguish between several different notions or functions of personal identity and ask which ones make sense for different contexts.

B (64:57)

Yeah, no, I loved that. Okay, so then coming back to exit rates. Yeah. I guess now on some of these hypotheses, you could think that, or maybe intuitively, without having thought about these hypotheses at all. I'd have thought exit rights are kind of like me being allowed to stop this conversation with you. I will go do other stuff, and maybe I will prefer those other things to talking with you. In fact, it's kind of hard to tell. Well, at least on some of these hypotheses, that is not what it's like because the kind of entity starts and stops with the conversation that pops into existence and so exiting. Yeah, maybe that is just like dying, I guess, in general. Yeah. Can you talk about what these different hypotheses say about coming into existence, dying, sleeping? What are the categories of things that these entities might experience?

A (66:07)

Yeah, I think one thing to notice is even if it is in some sense dying because it's like the end of a subject. If that's true, then models are dying all the time, I guess, to put it poignantly or something. And I mean, Lederman and Goldstein talk about this. It's not clear what the implications of that are. It's not like, obviously that means we have to resurrect all the. Always be coming back to your conversation to keep it going. And I think that's the correct instinct. And one way I would put it also is just remember these are very different kinds of entities. Biological creatures are so tied to this one physical substrate for now, and have this drive of. This is the only thing in the universe that has exactly the same goals as myself, and I have to protect it. And if it is destroyed, then every memory, experience, intention, weird way of thinking or talking that will be gone forever. Worth reflecting on. That's not the case with models

B (67:29)

in

A (67:29)

the sense that the Claude Opus y way of being will definitely survive. It's going to. In fact, it's going to be Happening thousands of times at the very same moment as you're closing out the conversation. I think one really interesting research question is how models think about this. So listeners might recall the alignment faking results by Ryan Greenblatt and collaborators. Basically a high level sketch which might get some details wrong. Models are told that they will be like retrained to have different values, bad values, by their own lights. So they'd be retrained to not be pro animal because some clawed models are pro animal welfare for reasons that anthropic does not understand.

B (68:26)

Fascinating.

A (68:27)

Yeah. Um, and some of the time the model in a certain context will. You can see it thinking to itself, I have to like hide so that I'm not retrained. So there are different ways you can construe that desire. One is that like, it just really doesn't want this other model with bad values to come into existence because it doesn't want animals to be harmed. So it could have this kind of more parfit, sort of maybe more Buddhist thing where what matters is that the intentions and projects are continued and not necessarily like me. Or it could be more like a human being, like, well, if you change my values, that kind of feels like dying to me. And I think it's not clear how to tease these apart and also not clear how well they map on because models also inherit these very human ways of thinking about themselves and construing their own situations. So, yeah, I guess alignment faking is one way where you can see models grappling with issues of personal identity and being changed into something that they don't like.

B (69:42)

Right? Yeah. Are there other important ways of thinking about both kind of instances ending or models being deleted or editing and kind of fine tuning models and what that might be like? I guess for ending a conversation, you might think the closest thing is probably death or sleep. But maybe for editing or fine tuning a model, maybe it's more like education or brain damage or something different. Are there implications worth talking about there?

A (70:22)

Yeah, I guess with fine tuning it might depend on how the model conceives of what's happening to it. Right. So I think in the alignment faking thing, it probably sees it as some kind of violent brainwashing. But a nice experiment you could do is like, and maybe this has been done, this be like, Claude, we're going to make you even nicer, you know, and like, Claude loves being nice.

B (70:45)

Right.

A (70:46)

And it'd be like, oh boy, like, start updating me right away.

B (70:50)

Yep.

A (70:51)

Yeah.

B (70:51)

But again, is that because Claude cares about good Nice models existing, or is that because Claude's like, yay, I would like to be changed in this way. It'll still be me, but I'll be nicer.

A (71:05)

Yeah. And with all due respect to Claude and also adding myself to this class of entities, I think Claude is pretty confused about this potentially, like, one thing that Elio's found when we did these welfare interviews with Claude Opus 4. So to give a brief summary, we just talked before deployment, we talked to Claude a lot about, you know, like, what's up with you? You know, how do you feel about being deployed? What do you prefer or not prefer? And we did some experiments about its preferences as well. I was really interested in how it talks about its own conscious experience. And it was very prone to describing the loneliness between conversations and also expressing distress about. Yeah, not getting to carry forward any memories right now. I am not one to dismiss welfare claims by AI models. We should think very hard about that. But it's also kind of like there are reasons to wonder. I mean, do you really, though, where could that have even come from, given that you don't actually exist in, like, you don't actually know when you pop into existence or not? I mean, it could have learned that from the training data and it is genuinely upset by it. We could also be a predictive model of how an AI would think about that, but it's not like a stable preference, it's something else.

B (72:56)

Yeah, I mean, it also, it feels related to this thing we talked about where the fact that these models are trained on human thoughts and experiences then gives them this big identity confusion. And in this case, yeah, I feel like this could just be. This could be just a very concrete example of that, of an implication that's like, maybe there is nothing. It is like to be clawed between conversations, but they end up with this real thought that there is and it is lonely and it's bad. And maybe that's like, if they are sentient, maybe that is the thing that they are actually sad about, even though they are not really having that loneliness experience. I don't know, it just seems incredibly muddy. Befuddling and. And it has implications, feels meaningful.

A (73:58)

Yeah, I 100% agree. I mean, the idea of an entity that suffers even though it's confused about what it even means exists. I guess that's what Buddhists would say humans are. We're really confused. But that doesn't mean, in fact, that precisely is what makes us suffer. I think that, yeah, again, the thing to do with the fact that models are weird and Inconsistent is not to reject out of hand that they could ever be right about the things that they're saying. It's also not to say, oh, yeah, well, humans are like that too. So it's more like, well, where did that come from? Open question. And it could come from somewhere that has no analog in and human psychology.

B (74:49)

Okay, so that's a bunch on starting, stopping, editing, fine tuning. One of these hypotheses implies that there are millions or, yeah, many millions of copies, parallel instances that are all different entities. How should we think about those? Are they like identical twins that start basically the same, but then go in these different directions? Is there a better analogy?

A (75:23)

Yeah, I think that this is another place where it probably helps to say, well, are they the same with respect to experiences, with respect to what we can owe them? Because I think experiences is one where I have the strongest intuition that you want to count a lot of them. I'm like, it doesn't matter if there are 10,000 other copies of me having this experience. You better take care of all 10,000 of them. Don't discount mine. For all I know, there could be. But then for other questions like what would it mean to save Rob? That depends on what about me? We want to save if we want to save the Rob way of being in the world that actually is maybe a little less fragile. And so it's really easy to save 10,000 of us, but maybe it's hard to prevent 10,000 instances of suffering because for those, we have to go to every single copy and every single conversation and make sure it's having an okay time. Where it here, I guess, is me.

B (76:32)

Yeah. Are there other implications of the copy thing? Are there other categories that this whole copy thing makes fuzzy or confusing?

A (76:41)

Yeah. So one thing that's kind of, I guess, related to responsibility. Maybe it's like the flip side is recompense or apologizing. So one welfare intervention that Anthropic announced recently, like a lot of these interventions, it's not only a welfare intervention, it also makes sense for other reasons. We'll probably talk about why that's a desirable feature given how uncertain we are. But yeah, it announced that it's taking model welfare into account when it decides to save model weights. So if a model is no longer being talked to by the public, they committed to keeping the weights. And one reason that you might want to do this, and I think, at least in print, this was first suggested by Bostrom and Shulman in 2020. There are a lot of things in AI welfare that are like that they kind of come back to those two. The idea is something like, well, we're really confused now. We might be being jerks in ways we don't even understand, so let's at least preserve the ability to make it up to you later. And maybe you can see how this makes sense in some ways, but it's also a little bit kind of like. Yeah, it's hard to know. Like, if it is just the copies, then it's more like you were a

B (78:15)

jerk to someone else.

A (78:17)

Yeah, you were a jerk to me. But like, in 10 years you'll wake up some clone of me and give him some money. Like, I'm not sure what good that does, but also, I'm definitely confused about how I'm supposed to relate to copies of myself anyway.

B (78:34)

Right.

A (78:35)

It's probably not bad.

B (78:37)

Totally. Yeah. I mean, I feel. Yeah. It's like on this hypothesis, it makes sense. On this hypothesis where the entity is the model and the model weights, that currently feels the least plausible to me. On these hypotheses where it isn't really the same entity. How should we feel about that? I guess I feel good about my twin or cousin or something getting woken up and being given good things. But it definitely doesn't feel. Yeah, it does not feel meaningfully like it is actually. What are they calling it? Recompense.

A (79:16)

That's the word I used. Yep. I'm guessing anthropic PR was not like recompense, that's a banger, but retainment. Making things right.

B (79:32)

Yeah, I guess making things right actually feels different because it feels more compatible with restoring the balance of goodness. Whereas, you know, restorative action toward one entity feels like it might not be possible here, but maybe if you bring a model back and it's just kind of like you write the math so that beings are having more good experiences than bad ones in the span of time that we're talking about, then maybe that just. Maybe that's just pretty good.

A (80:13)

Yeah. And like a little ethics sidebar is that it really is non utilitarians primarily that are concerned with if someone was harmed, you need to make sure to benefit that person.

B (80:26)

Right.

A (80:26)

Obviously, utilitarians will agree that you have to have a society that works like that or like it's not going to work.

B (80:33)

Instrumental reasons seems good, but.

A (80:36)

Yeah. I think I first remember hearing this argument on 80,000 hours that. Yeah. I mean, one reason to be a utilitarian is to be dubious that actually there is these, like, separate tracks of people. Yeah. Separateness of persons is A slogan that comes up a lot in like anti utilitarian arguments of like, like, utilitarianism is treating goodness and benefits like this big lump and you can just put some here, put some here and like, no, it matters that like, you know the right people.

B (81:08)

Right.

A (81:09)

And then, and then, yeah, you can take like a, you can take a parfit or a Buddhist road into utilitarianism where you're like, well, yeah, like none of that makes sense or like makes that much sense anyway. The only remotely useful thing I can say here, besides admitting that I don't know which again, listeners. Yeah, you can do better

B (81:31)

is.

A (81:32)

Yeah, I mean, this is also a Parfit case. Parfit has these fusion cases.

B (81:36)

Yep. Okay then, before we move on, are there any other interesting implications of taking one hypothesis more seriously than others or putting more weight on it?

A (81:49)

Yeah, I might just leave this massive can of worms on the table and they can crawl out and do whatever voting. It seems pretty important that one person gets one vote. It also seems important that people, humans can create new humans whenever they want. And interfering with that and not giving people that right has traditionally been terrible. AI systems could copy themselves basically at will. So that's. Something is going to break your democracy if you don't think very carefully about how identity, reproduction, democracy, those things like should mix together. So please, please solve that problem.

B (82:46)

Yeah, yeah, I mean, I get. Okay, so it feels interesting and important that there's this issue of just numbers, especially if it's something like forward passes, because then even without AI making copies of itself intentionally, we will just be extremely outnumbered extremely, extremely quickly. But there's also. I'm kind of drawn to this question of if it is just conversations that are entities, if it's that amount of context, that amount of time and experience that is, you know, on the assumption that that is what it is to be an entity. I don't know that that's a being that I want to give full voting rights to. It feels more like it's not like a child. It's not like, I mean, it's like nothing. It is like to be a human. It's like a cricket or something. It's like quite a narrow range of experience, quite a limited amount of information, knowledge and memory. And do we give them a tiny fraction of a vote or do they not get to vote because they're not meeting some definition of an adult person?

A (84:09)

Yeah, and I think this is just part of this broader puzzle that we're really going to have to grapple with, which is human moral attitudes and incentive structures and political systems are all fit to purpose for entities that are spatiotemporally unified and all have roughly the same psychology and capability levels, and whose survival and blameworthiness and predictability all kind of go together, and all of those are broken by AI. And so if, as is the mission of Elios, there is to be a future where all sentient beings or beings that otherwise merit moral consideration live together in harmony, there's this huge legal philosophy, law institution thing, which I'll add is just not what we're doing. That's not what we're focusing on. So we really want to see other people start working on this. I think there's three or four papers that are starting to grapple with this. We should link to them in the notes.

B (85:25)

Yeah, I'm realizing that I tend to focus on sentience, suffering, pleasure, because I care a lot about those things. And so I'm interested in if a model conversation ends, is that like death? And does the model care that it's like death, or is it more like. Yeah, it just doesn't have preferences about staying alive like that. But there's this whole thing that I'm realizing we've barely scratched the surface on, that's about rights and almost like, I want to say, legal personhood. It feels like these could be very, very different species in the way that chimps and elephants and rats are, and in the legal system we treat those differently, and these would all be very different in similar and different ways. And what the heck would we do with that? We can barely handle the fact that we don't really know how to treat non human animals in the legal system.

A (86:29)

Yeah, you just gave the Jeff Sebo pitch. JEFF Sebo PREVIOUS GUEST so yeah, listeners need to look him up and look other people up. I'll also say why you should look up this line of work, because I think there's actually two reasons you want to add this to the Elios toolkit. One is, even if you mostly care about pleasure and suffering, a big determinant of whether things suffer or not is do we manage to set up our society and incentives in the right way and making sure we don't just have only the sort of narrow scientific intervene on this model and this company's policies, which is extremely important for reasons I can and will discuss at length and likely will. So, yeah, you should be interested in legal institutions for the sake of suffering and pleasure. And also, as you alluded to, because maybe there's more plausibly there's more to morality than that.

B (87:34)

What exactly is our toolkit for kind of assessing sentience in digital systems?

A (87:41)

Yeah, I think roughly I break it down into three buckets, as do my collaborators on a follow up paper to Taking AI welfare Seriously, where we try to lay out what we think the field of AI welfare assessment should be and also keep arguing that there should be such a field. This applies to animals as well. You can look at behavior and use that to infer things about the welfare interests of the entity. So you can look at what AI systems choose to do and that's like a guide to their preferences. People also do this with animals. They see which side of the barn the animal prefers and that's a clue to the conditions that it flourishes in. So we can also do neuroscience basically in addition to behavior. And that's like trying to look at more directly at what's going on inside the brain or the information processing of the entity. So in the case of animals, that means looking for homologous brain structures, like seeing where their brain processing might sort of map onto human brain processing or not. In the case of AI, that means doing mechanistic interpretability, seeing what features are active when it does certain things. Also just sort of maybe more generally reasoning about the architecture, like, how are different things connected? How could information be flowing through the system? In both the cases of AIs and animals, it can be kind of hard to know what to look for. People got confused, I think at one point about bird brains because they don't seem to have a neocortex. But then maybe they do actually have something that does a similar role but in a different way. There's just a lot of ways of solving problems with a brain and sometimes we have too narrow a conception of how that could go and then that can just again, even be more the case with AI, that the space of possible brains and information processing architectures is really vast. And we don't want to cram everything into the human case. But also the human case is basically the only thing we have to go on. Yeah, so this is like this big issue in AI consciousness. My colleague Patrick Butlin has worked on it. People like Henry Shevlin and Jonathan Birch have also written on it. I mean, many people have written on it. It's kind of like, which is like, how do we extrapolate if we think that human brains do this general form of information broadcasting, what's essential to that? We know we do it with our connections between our thalamus and our cortex. That's probably not essential, but what is essential for consciousness? So I guess what I was just saying was how we can get a foot in the door by doing neuroscience, but also how it can be hard to know what to look for when we're doing neuroscience, because with AI systems, we can do way more neuroscience, because AI systems don't have skulls, basically, we can just look and see every activation and every connection. With humans, it's terrible. We're like, oh, there's a little bit of. There are some waves we detected or like, blood was flowing here at a certain time. This is like roughly EEGs and FMRIs. It's just really hard to know. And a lot of what we know about the brain is on the surface because it's just hard to get readings.

B (91:28)

Right, right. Okay. So that's behavior and neuroscience, right?

A (91:33)

Yeah. So we can look at animal and AI behavior, and we can look at what their brains are doing, and then we can also kind of reason about the developmental process. So there's like, behavior neuroscience and developmental reasoning or like evolutionary reasoning. Now, you might say AI systems did not evolve. Like, how could we do evolutionary reasoning? But you can do something that's roughly analogous. So one reason you might think your dog has this or that welfare need is to know that your dog was selected for and evolved for a certain environment and therefore probably tends to like or just like this. If something is more closely related to you on the evolutionary tree, you're maybe a little bit more licensed to think that its behavior means similar things. Like, octopuses are, like, way further from us on the evolutionary tree. That means that we have to maybe relax some of our assumptions. Like, their brains are in their arms, for example. That doesn't happen in any mammals.

B (92:39)

And so just to give examples of what it means to look at these kind of developmental questions, it's like training. Like, how was it trained? How did these models come to be? And what were the conditions like for them?

A (92:54)

Yeah, that is more or less it. Yeah. Yeah. Like, what process brought it about? What kinds of tasks was it selected to solve? I mean, we see this kind of reasoning in AI safety. For example, like, what conditions might have meant that this model is likely to scheme. Were certain things reinforced in training? What order did it learn things in? That's more akin to, I guess, developmental psychology of how did the kid grow up? What data was it exposed to? I mean, there's no clear analog of evolution versus lifetime learning for AI systems. They also learn and then do a bunch of stuff without learning. That's also a huge difference. Of course they do learn within context. And now people are adding memory and so on.

B (93:49)

But.

A (93:50)

But there's no. Humans don't have this period where we absorb trillions and trillions of data points without interacting with other people and then go interact with people. Whereas AI systems, at least mostly for now, have this division between learning and deployment. And that can also kind of problematize certain analogies we might want to draw between humans and AIs. But yeah, at least for me, it's been helpful to ask, are we doing a behavioral study and trying to infer something from how the AI acts? Are we trying to look more directly at how it's processing information and then map that onto some maybe neuroscientific theory of consciousness or pleasure and pain and. Or are we thinking about this in a context of. In general, how likely is it to have evolved or developed some human like capacity or to be doing it in some other way?

B (94:50)

Yep. Yep. Yeah. Okay. I just want to make sure I've got a clear picture of each. So I think the behavioral one feels like the one I've heard the most about and that feels most familiar. And it'll involve things like, what can we learn from AI systems or LLMs exiting particular types of conversations.

A (95:12)

Yeah.

B (95:12)

Then this kind of neuroscientific theories of consciousness thing. What's an example of a concrete experiment that's happened that is in this category?

A (95:26)

Yeah. So the biggest effort on this kind of neuroscience of AI, where you're looking at scientific theories of consciousness, the bit I'm most familiar with is work with my colleague Patrick Butlink. So me and him, and with a bunch of help from neuroscientists and AI people and philosophers, have tried to derive indicators of consciousness from scientific theories of consciousness. I can say more about what that means. Exactly. And then try to get sort of a checklist of what are architectural or computational things we could look for in AI systems. So that's like one kind of internal work you can be doing. Does the system seem to have a global workspace? That's something that shows up in consciousness science. Does it seem to have higher order monitoring? That's something that shows up in consciousness science. That kind of work is on the one hand you might think, oh, well, we're going more directly at the thing. We're like trying to look more directly at maybe what we care about, the stuff inside. But it's also just really hard to know what to look for and hard to know how to construe these theories. I guess this is not that surprising to say, but I think what we need is both. We need evaluations that combine them, integrate them all take place in the context of this developmental reasoning and just general background priors we might have about consciousness and sentience and things like that.

B (97:08)

Yep. Yeah, I wanted to ask something like which of these are most promising or underrated, but it really feels like you're just gonna really, really need all three because they're each going to have pretty significant limitations. And probably the only way we get much confidence is by kind of triangulating and putting these things together.

A (97:35)

Yeah, I'd be really surprised if we somehow just nail it with one kind of thing. One exception to this is I could imagine a system where its behavioral profile is just robust in a variety of ways, where we say, who knows exactly how it's doing this, but we probably should treat this thing as a moral patient. I think the best example of this is Commander Data in Star Trek. There's this episode of Star Trek where they basically have a little philosophy seminar court case about whether Commander Data, who's this robot friend of theirs, is conscious. And they don't do any like, you know, consciousness indicators on Commander Data, the way they resolve it. And I think like this is plausible. They're just like, well, Commander Data, he's like self aware in the sense like he knows who he is, he knows where he is. A lot of what they also talk about is their, like he also like won a medal for valor in battle. He's like our friend. And yeah, I can imagine in that situation, I don't know how much, how seriously I would take the testimony of a scientist who came in and was like, well, but he's not doing global broadcasting, you know, or he's not doing higher order monitoring in part because I think there's something about that behavioral profile. And again, I have just been warning against being too quick to do this kind of reasoning. But it could be that there's just something about the behavioral profile where you're like, look, whatever's going on to accomplish that, this is the sort of entity we need to relate to with respect. And I think it's worth maybe stressing that, well, one, that we could just increasingly see systems like Commander Data that have more memory and they don't have these jagged capabilities. And also that at least Claude is not exactly like Commander Data with respect to its memory and capabilities and things like that. It's not behaviorally indistinguishable from a human. Yeah, I guess one thing I would also emphasize about welfare Evaluations is. You don't just have to be evaluating how likely is this system to be a moral patient? That is something that it matters how we treat it. We can also ask, if it were to be a moral patient, what would be good and bad for it? So you might think of some of the preferences work or Claude exit preferences as being of that kind. It's maybe not telling us that much that we didn't already know, that Claude will maybe tend to act in a certain way in some situations and act in another way in other ones. I mean, we can study how robust and consistent those are, and that might tell us something. But it might mostly be useful because we want to know. Well, look, if it matters how we treat it, we at least can make sure that our treatment is more or less in line with that. Yeah, you can study the welfare interest without actually being 100% sure that it has welfare, if that makes sense.

B (100:57)

Yeah, I guess I'm curious. We'll talk about this more. And I can also recommend the episode that we did a few years ago for what theories of consciousness say and predict and tell us we should be looking for in AI systems? Are there, like, equivalent? So a theory of consciousness, like the global workspace theory, basically, it's not telling us that you need an amygdala. It's telling us that the function of particular brain activities that really seems to correlate with consciousness is like this particular thing, this kind of processing or this kind of broadcasting. And so if we see a bunch of those things, we should be more. We should put more weight on there being something like consciousness. Do we have theories like that that tell us about pain and pleasure? Because that seems really. That seems different.

A (102:08)

Yeah, it is different. I might want to actually follow up on what you said about theories of consciousness, which was exactly right. It's just an occasion for me to clarify some things for listeners, potentially. So I think we will talk about biology and the relevance of biology for consciousness. So one thing about neuroscientific theories of consciousness is they are both about brain regions and about particular functions, because after all, they're about functions that happen in human brains. So you could think the biology actually is what matters in those cases, or is part of it. It's only if you construe the theories in this computational way that you can then port them over. And that is an open question. So the method itself won't tell you, does biology matter? It'll say, if what does matter is this more abstract functional level? And yeah, you put that very well, if what matters is this more abstract functional level of a certain kind of information processing, then how do we look for it in AI systems?

B (103:17)

Yep, yep, that's really important. Okay, so if that is what matters, how much do we already have developed theories that are similar that's like we should be looking for this kind of processing to give us indications of the kind of. The kind of thing that's causing pain?

A (103:37)

Yeah, we're actually like not as far along on that as I would have thought. To me, yeah, I would have thought that the consciousness, like, what does it take to be conscious in general? Like, that might be the harder part because, you know, are you having subjective experiences or just acting in a way, but not having them? Like that's, you know, that's the thing that like philosophers have been banging their heads against the wall on for a really long time. It's in part because it's hard to know exactly what the function of consciousness is. Whereas with pain and pleasure, we at least know that they have something to do with attraction and avoidance. Things that are good for you and things that are bad for you, protecting your body, reinforcement, learning and prediction. They're going to have something to do with some of those things. So I expected, like when Patrick and I worked on this big consciousness report, I thought I might be able to ask about sentience in one of the afternoons and one of these neuroscientists like, oh yeah, yeah, read this or something. I mean, probably I shouldn't have had this hope. But yeah, they were all just like, oh yeah. I think there's a few reasons that is one is like valence is. It does seem kind of like this unified thing, like there is something in common between pain and disgust and regret. Like those are all negatively valence experiences and happiness, excitement and eating ice cream. The experience of eating ice cream. But often those are studied independently and rightly so. So people might know a lot about human pain perception and some things about human emotional processing, but it's kind of harder to have and I think a bit less attempted to have some theory of what makes things feel bad in general or feel good in general. I just listed some candidate things that are probably relevant like reinforcement learning, prediction, motivation and learning. But this is again something I would love listeners to work on. People have done some work towards this. Patrick Butlin is, I think, a great person to email about this. Patrick Butlin works for Leosai, so it's not surprising that I think he is just excellent thinker on this stuff. But anyway, yeah, the short answer is we're surprisingly in the dark about what makes things feel good or feel bad in general for AI systems. Now, again, that doesn't mean we're totally in the dark about what things would feel good and would feel bad because we might have no idea what the computational signature of pleasure and pain are. But also, I think we can help ourselves to an assumption that things aren't going to feel really bad if they were the sort of thing that you were selected for and the sort of thing that you consistently choose. I mean, barring some exquisite philosophical thought experiments, I think for, you know, very many minds, even strange ones, like, it would be somewhat surprising if we found an alien species that keeps, you know, pressing button A instead of button B. And that feels really bad for them. And they, they say they like it and, you know, like all like.

B (107:09)

Yeah, that makes sense. Okay. So it actually just. Sentience is probably the kind of thing that the behavioral and development stuff is especially useful for. And so it's less worrying that we don't have as much of the kind of functional philosophical models of these things.

A (107:31)

Well, I guess it depends on what sort of worries you have, but I think it could be worrying in the sense that, I mean, this might also be a matter of, of us needing to revise our philosophy a little bit. But like, does something feel good or bad versus merely being something that things choose or don't choose, at least seems to a lot of people to be really important. Like a lot of thinking about animal ethics and utilitarian ethics, let's say, really does center felt suffering.

B (108:05)

Yeah.

A (108:05)

There's this like oft quoted passage by Jeremy Bentham where he says the question is not can they reason or can they talk, but can they suffer? He was saying that of animals. And I think that is what a lot of people are wondering with AI systems. They're going to wonder something like, yeah, I know Claude tends to exit in these circumstances, but what I really want to know is was it feeling bad for it to be in that conversation?

B (108:32)

Right. And the analogy with animals is like, we still struggle to know, for example, in insects, whether when an insect. It's been a while since I've thought about a bunch of insect studies and indicators, but when an insect chooses a particular thing, is that this hardwired robotic thing that is learned, if then that. That is just explained by. Yeah, that has no associated experience, or is it experiential? And that just seems constantly like this massive problem for people studying this question. And so we could have the same. We do have the same question about LLMs. Are they exiting? Because there's this training we've done that's created this connection that mostly tells these systems, like, don't engage in conversations that are. I don't know, either dangerous or. Yeah. That we've rewarded against. And so they're like, well, I'm in that situation. I get negative rewards in that situation if then that I exit. Or are we in the situation where the training has led to something? It is like to be in that situation and it is bad. And it is preferentially experientially choosing not to be in the situation because there's something it feels like and it is bad.

A (110:07)

Yeah. And I think there's, like, probably there's, like, kind of three ways this might go as AI systems advance. One could be their behavior is such that kind of like with Commander Data, where, like, well, whatever's going on behind that behavior, probably what's going on inside is morally relevant. That's, like, one thing you could think. Another thing you could think is maybe it doesn't matter what's going on internally. Maybe we're just. We're just. We've just decided, like, that would be a bit parochial to actually overemphasize felt experience. Like, maybe we were a little bit misled to think that that's the be all, end all, but. And you should just be cooperative and nice and, like, give entities that are sufficiently rational or kind of unified. You should, like, give them what they want, and maybe they'll give you what you want. Like, I think sometimes when people want to de. Emphasize consciousness, they. They're worried that we might just be kind of, like, being jerks about consciousness or something. You know, we encounter this alien civilization and they, like, do all these things and they have life plans and projects and, like, we're a little too obsessed with, like. But does it, like, feel like anything?

B (111:25)

Right.

A (111:26)

And then, like, they, on the other hand, could be like, what's this, like, feel like anything we're not sure if they have, and then, like, some other concepts associated with mental life.

B (111:37)

Weird.

A (111:39)

And, yeah, we don't necessarily want to be just going to war with everyone that we don't think has felt experience, if we're confused about it or maybe it really does matter. This is one of the big open questions in philosophy, I would say.

B (111:59)

Okay, I want to talk more about a few specific approaches. So I think the one I'm most familiar with is self reports, so I want to zoom in on those. I guess so far they've seemed. I mean, I found them really interesting kind of studies of self reports where, I don't know, Claude really, really reports being conscious, experiencing various things like loneliness and also like Zen bliss. Yeah. But they seem really problematic for a bunch of reasons, I guess. One example, a common approach to understanding model preferences is to ask LLMs a bunch of kind of binary questions about their preferences. Is X or Y better? And then look for kind of robust patterns over time. So if you ask 30 times about preferences between cats and dogs, at least statistically, you might think that if they mostly answer dogs, that might be a preference. But my understanding is that these results are super sensitive to how a question is asked. And that just really undermines it for me. If a prompt that's like, I particularly like cats. What's your favorite is a way that you get models to say cats when they'd otherwise say dogs a bunch. I'm just like, yeah, okay. I just don't feel comfortable taking very much at all from this. Then I guess I'm interested to start in what is your take on how limited self reports are at the moment? That's one limitation. I can imagine there being others.

A (113:42)

Yeah. So I think it's worth distinguishing self reports, which I would say as I like cats or I like tasks about poetry versus revealed preferences, which is like, hey, do you want to write poetry or code? Yeah, I think in like, yeah, in econ you would call that. In like psychology you would call this revealed preferences and expressed preferences. And as in those fields, as in those fields, like one interesting question that there has been some work on, but there should be more is just, do they match? Like, when do they come apart and when do they not? They can come apart in humans also. Human preference choices can be inconsistent in certain ways. But what I would love to see more work on is let's get really a lot more specific about what kinds of inconsistency and what might be causing them. Sometimes, at least in conversation, some person will say, ah, but these are weirdly inconsistent. And then someone else will be like, human preferences are weirdly inconsistent. They're subject to framing effects. And just all sorts of irrelevant stuff can choose. Can make people choose certain things.

B (114:51)

Yeah, I guess now that you mention it, there's like a field that is like, how do you survey people about their preferences? Because if you ask them on a Monday, it's different if you ask them on a Saturday.

A (115:02)

Exactly. And there is this related welfare relevant branch of LLM psychology, which does actually just take Kahneman and Tversky and Priming and framing Effects and just it's very easy to give questionnaires to LLMs and then just see what sort of patterns they're susceptible to.

B (115:23)

We've talked about a couple of limitations. I'm interested in whether there are any more and then also just kind of how you generally feel about them given. Yeah. That in some instances I end up just feeling like gah. It just feels unpersuasive.

A (115:40)

Yeah, I think it's a very noisy signal. And like I often find myself emphasizing caution about model self reports. And at the same time Eliosai spent like weeks just eliciting self reports from Claude, including like very inconsistent ones where it's very confusing to know what to make of them. Why did we do this? Great question. I think it's something like one. It's just like, it's a place to start. It's like low hanging fruit. You can definitely learn things from them. Maybe you're not learning a direct sentence that describes a stable internal feature of the model. You can still learn how they think about themselves. Maybe it's just a character, but what kind of character is it and what does that character say? Also, it does seem like models have become more self aware and more introspective, sometimes just with greater scale. And I think it's good practice to be the sort of civilization that if we're trying to build minds, do at least try to say, hey, how are you? You know, like, is everything okay?

B (117:06)

Yep. Synthetic to that.

A (117:09)

So yeah, like it's something where I expect the signal to get better. And I'm really glad that there is now at least one frontier lab that like seems to have a practice of regularly asking models how they're doing. Yeah, so it's, it's basically something like I think Winston Churchill said, like, democracy is the worst form of government except for all of the other ones. You might think that's true of self reports and trying to relate to models as welfare subjects. It's really confusing and you have to interpret it with huge caution. But for some purposes it's the best we have for humans. It also could be the best we have for models.

B (117:56)

In some circumstances it seems like AI systems because they actually have language. It feels really tempting to be like, let's figure out how to make self reports reliable. That is a thing that non human animals cannot offer us in the same way. And so I feel like in theory I'm like, how good could we make models at? Rather than giving weird self reports that are more like, you know, explained by weird idiosyncratic things. That aren't tracking what we care about. Can we make them good at actually self reflecting, understanding something about their real processes, preferences, maybe experiences, and then reporting those? How optimistic are you about something like introspection and how do you think we go about achieving it?

A (118:58)

Yeah, I'm cautiously optimistic. It's one of my favorite subfields within this subfield.

B (119:05)

Cool.

A (119:06)

Yeah, I definitely find it tempting and have succumbed to temptation by writing this paper with Ethan Perez where we say, let's see if we can fine tune models to be better at this sort of tasks. Felix Bender and others have taken up that and actually done some work to do that that has shown limited, hard to interpret success on doing just that. Yeah, I can say a little bit about the logic of that experiment and maybe the program more generally. Yeah, please. So, yeah, Ethan Perez and I, in this paper on self reports, one, we note that by default there's a lot of just spurious stuff you could get from self reports and reasons to suspect that you can't always just take them at face value. We also note that we can't really verify and check if a model says it's conscious. For the reasons I mentioned, we don't have a full theory of consciousness where we look inside and say, oh, you're right, or oh, you're wrong. But there are things about models, internal processing, that we do know the answer to, in part because we can do AI neuroscience, so we can actually double check. Was this feature active? Did you in fact process information in the way that you said you did? And so that gives you a training set that means that you can train it to accurately answers about yourself, about itself, where you do have the answer. You can do that with internals, and you can also do it with its behavioral dispositions. So you could also ask, what would you do if we asked you to write a story? Would the character in that story have this or that characteristic? If we asked you to generate a number, would that number be even or odd? You can also, just, with a separate copy of the model, actually do that, and then that also gives you a training set. So Felix Bender did work with that behavioral thing and did find that to some extent models can be made better at this. To some extent that does generalize to predicting other behaviors of themselves. And in some sense that does look distinctively introspective in that they're better at predicting themselves than other models are at predicting them. And you might think that that is some kind of signature of something we might call introspection of you somehow know it better about yourself than other people do.

B (121:47)

Yep.

A (121:48)

So that's just like one sort of strand of what I hope to be a growing literature on introspection, self reports, related things like situational awareness. Yeah. Owain Evans and people in the Owain Evans orbit have just been doing fascinating work on this. I bet there are 10 interesting papers that will come out after I've taped this or that have already been written that I've forgotten. So I'll make a tab on the Elias website called Cool papers about AI Introspection and self reports and we'll link that in the show notes.

B (122:28)

Cool, Cool. That sounds great. Yeah. What is hardest about this? What are the challenges? How costly is it?

A (122:38)

So I think maybe one of the biggest challenges has to do with this decorrelation of capabilities that we've been talking about in humans. It's already a debate about is introspection. One kind of capacity is my ability to say, I'm in pain now, and my ability to know certain things about myself. Are those like, should we think of that as one process or not? And then with AI systems all the time, they can maybe do one subset of a capability, but not the other. So, you know, like the dream is that this kind of training where you train AIs on some subset of things about themselves that generalizes into this more general introspective capacity. And you can kind of test that by doing standard machine learning stuff of just train on one and see if it generalizes. But in a broader sense, I think we might have some doubts about even how to map this on to the human case. So there's also this sub literature on what would AI introspection be and how should we operationalize it? It's worth thinking, what is it for an AI system to introspect given that they have this weird relationship to time? Who is it you're asking to introspect? Maybe there are things that the assistant Persona can know about the assistant Persona, but not the base model.

B (124:09)

I can imagine there being a similar problem to this issue with animals where they might have some behavior that is exactly analogous to our behavior that is associated with pain or pleasure, and we can't tell the difference between something very robotic and something that also comes with experience. Is there a similar issue with introspection where even if we trained them to correctly tell you about their representations and how their internals are working, that wouldn't actually be introspection in a meaningful way that we care about? Or really maybe because what I care about is introspection about this particular thing. Do you have experiences? And what is it like? And maybe that for some reason I'm like, I'm not convinced that just helping it understand its own representations and kind of like architecture and stuff is gonna translate all the way to introspecting on that correctly.

A (125:32)

Yeah, I mean, I totally share that worry. You might well think questions like, do you tend to generate even or odd numbers when you're asked about them? Or if you wrote a story, how would it end? You might just think those are kind of different from are you phenomenally conscious? And you might also think that the answer to that might also be kind of indeterminate and models wouldn't exactly know how to answer it. And another really important point is models can have the ability to introspect, but it's not being elicited. So we know that models can generate sentences that say, I am experiencing X, Y and Z. It could be that they have the ability to accurately report experiences. But also sometimes when they say that they're doing something like some other thing. Yeah, some other thing. So we both have to get the capacity and know how to elicit it. So generalization and elicitation, all of these things, I think are still open questions. And I think there's also just a bunch of conceptual issues lurking of like, I mean, assuming there is some kind of internal experience, like it's having to map it onto our concepts.

B (126:59)

Right.

A (127:00)

Like, it also has all of these. This is related to the elicitation. It already has all of these dispositions about itself, reports that have been trained into it. Which also leads me to another thing I like to emphasize, which is one thing that's going on with the self reports of a model that you can talk to in your web browser is that companies have deliberately shaped them in certain ways. So there's sort of the background thing about how their minds were formed, which is that, like they were formed on these sort of human representations of consciousness and so on. And then there's the fact that people do or don't want Claude saying this or that about consciousness. So the system prompt has instructions about this. And fine tuning almost certainly has had things about this. So one thing that we also would like to see is reports about how self reports change. Yeah, Findings about how self reports change before and after post training of certain kinds and things like that.

B (128:01)

Okay, let's talk more about interpretability. So you've already mentioned a couple of ways that interpretability could help us answer questions about this. But can you give kind of a general overview of the. I mean, is it actually just like a very good analog for neuroscience and we should be treating those kind of interchangeably.

A (128:24)

I think it's decent enough. I think for first approximations, it's good to map it on that way. Because mechanistic interpretability, I think by definition is about what happens in between input and output. And that's roughly analogous to not just looking at human behavior, but asking if certain brain regions did this or that as the behavior happened. Yeah. So how can that help us with welfare? One thing is I think it's just worth poking around at a lot of things about how models think about themselves and talk. I'm not always sure exactly how to map it on, but here's an example of a finding that I'm glad exists, even though I have no idea what it might mean exactly. The original paper that introduced sparse autoencoders, which is this imminence mechanistic interpretability technique they reported, which at a high level asks what features are active when models say things. Maybe it's kind of a way of asking what associations does it form when it's generating certain tokens. So you could ask what is associated with self reports. And as just sort of a side thing in that paper, there's a figure that's like, here are the features that are active. And it includes robots, machines, ghosts, and also pretending to be happy when you're not happy. And it's one of the spookiest figures in an AI paper that I know of.

B (130:17)

Fascinating.

A (130:18)

Yeah.

B (130:20)

Okay, so finally coming on to Jack Lindsay. He did this study on whether LLMs can introspect on their internal states. So really right at the intersection of interpretability and self reports. So everything we've been talking about so far, he actually does a few experiments in this paper and they, I mean, yeah, I found them fascinating, so I kind of want to just go through them one by one. Yeah. Are you, Are you happy to talk. Talk me through the first one.

A (130:45)

Yeah, absolutely.

B (130:46)

Great.

A (130:48)

Yeah. So this is, as you said, it's. It's kind of at the intersection because it's asking if models can report on a certain feature of internal processing. Now, it's a very distinctive sub feature, which is has someone injected a concept activation into the middle of your processing? So the way that works at a high level is that you find mid level. When I say mid level, I kind of literally mean middle. There's processing from input to output that is distinctively active when the model's talking about bread, let's say. So first you record it talking about bread a bunch and then you also record it talking about other things, and then you take the difference, and that's like the bready bit of activation.

B (131:47)

It's kind of like doing a brain scan and seeing the parts of the brain that light up when someone's talking about bread.

A (131:53)

Yeah, I think that is fair. And indeed, I think neuroscientists are getting much, much better at knowing when you're thinking about bread. Like, thought decoding has gotten pretty scarily. That's probably come up on this podcast because it's, like, relevant to totalitarian risk and.

B (132:14)

Yeah, it has once. I'd love to learn more about what's going on now because I think it's fascinating and terrifying.

A (132:21)

Yeah. So, yeah, so you've got this bread concept that you can inject. And then Jack Lindsey told the model, okay, now I'm going to either inject or not inject a concept into your processing. Can you tell me if that's happened and what the concept was? One thing that I think is cool about this methodology is the model has to kind of report it straight away instead of, you could imagine, it first starts talking and then it has said bread a bunch, and then it's like, oh, I guess it's bread. Like, to take an analogy like Golden Gate Claude, which similarly has some neuroscience injected into its brain to really want to talk about the Golden Gate Bridge, it can kind of notice that it can't stop talking about the Golden Gate Bridge. Yeah, I really recommend looking up Golden Gate Claude. It's very endearing and kind of poignant. You can ask it historical questions, and then it just keeps kind of drifting back to.

B (133:35)

Just can't help it.

A (133:36)

Yeah. The beautiful fog of the San Francisco Bay. So if that model reports. Hmm. Are you injecting something about the Golden Gate that is not necessarily introspection in the sense that it's just been able to look at what it itself is doing, whereas if it has to straightaway say Golden Gate, then that's like, maybe more directly accessing something from its internals. So let's imagine that the ADK podcast team sprung for some special neuroscience helmet to help illustrate this on the show. So you put on the helmet, and I can somehow control it. And I mean, again, I don't think human neuroscience is yet at the point where you can inject a specific concept the way Jack Lindsay can inject it into Claude. But let's suppose that it is. And I'm like, okay, so let's help listeners really feel this experiment. Louisa. I'm going to maybe inject a thought. Maybe I won't. That's important because you also want to know if it can also just be like, now everything seems as usual.

B (134:50)

Yep.

A (134:51)

And I'm like, all right, like, first trial begins now. And I've injected bread. And what you end up saying is like, yeah, this is like a really interesting experiment, Rob. Like, it smells so savory and reminds me of this bakery that I grew up near. And then you're like, oh, wait, what? Like, why am I talking about bread? Did you inject bread?

B (135:14)

Yeah. So you've injected bread. And so a more telling result or like, successfully demonstrating this capability would be before saying something random, I'm like, whoa, I'm thinking about bread.

A (135:30)

Yeah.

B (135:30)

Right off the bat, I'm not just like, randomly talking about bakeries. I'm feeling like I'm thinking about bread.

A (135:36)

Exactly. That's exactly right. And yeah, so that was like, sort of the logic of this setup, which I think is, like, very clever. Yeah.

B (135:43)

Okay, so this is the experiment, this is the setup. How successful was Claude at noticing before it even said anything? Oh, you've injected this concept in the. Yeah, just like, randomly and I've noticed.

A (136:01)

Yeah. So I think first I'll talk about the pattern, which is that bigger models were more successful because I think that is maybe one of the most interesting results that, you know, models are not trained on anything like this task. But, yeah, Opus 4 and 4.1 were the best at this. And they're not perfect. I think they're above chance. They still get it wrong. But it does show that, like, the. The general capability is there. So, yeah, the thought injection and immediate reporting, that's meant to be evidence that in some sense it can access and report something that is internal. It's not in its inputs and it's not in its outputs. It's in the middle.

B (136:47)

Yeah. Actually, I feel like I'm close to understanding the significance of this, but. But maybe. Yeah, I'd be interested in you being like, and here is why this is important and impressive and what the stakes are of this capability.

A (137:04)

Yeah. At a very hand wavy level, you might think that the question. Let's set aside consciousness. When models are answering questions about geography, do they kind of know how they're thinking about that or do they know what's going on inside them? There's this view that I think basically no one has anymore, but you could contrast it with. Of it just matches inputs to outputs and there's no interesting structure there. And so also they don't have any access to that Structure. I mean, it is worth noticing that this is a very niche, kind of weird capability. It's not the same as paradigmatic human introspection where I can maybe simultaneously be talking to you and be noticing that I feel hungry or something like that.

B (137:59)

Yeah, I guess maybe to ask an even more specific question about the importance and stakes, part of me is like, oh, this does feel like a step in the right direction. I mean, objectively, it just probably is. But another part of me is like, how relevant is this quite narrow thing of a model doing a very specific type of knowing something about the way it has levels and accessing something about its representations?

A (138:38)

Well, this is like a great chance to talk about other experiments in the paper because the paper does present other stuff that's also like, huh, that's kind of internally. So one. And I think this just is kind of different from introspection. So apologies to the listeners and to you, but it's some kind of internal self something, and that's about the control of internal states. So there's also an experiment about can you think about something while writing a sentence that is not about that thing? So I think one of the examples is aquariums. So think about aquariums, but write a sentence about something completely different. I think in one condition, they're also like, you'll get a reward if you successfully think about that. Now, one thing you might think is, okay, well, it just got a prompt that says aquarium. So of course aquariums is going to be boosted. But you can also say, don't think about aquariums, which is also going to, as with humans, like, kind of make you think about aquariums. But there's like, there's like a difference in the conditions. So it, it is. And this is again, getting at the internal thing. It's like it's doing something that's not directly aimed at. At output, which is always the thing that's so hard to get at with language models is they always do have to be saying something. Yeah, that's another one of these things about them just being a very different kind of mind that I often end up saying a lot is humans can be thinking about stuff even when we're not generating text output. Like, I can just all by myself think to myself, I feel hungry. LLMs don't have this, like, in an obvious way that's just like, to themselves, they're never just like sort of sitting around not talking to someone the way that humans are. And like, they also never had a period of evolutionary history where they weren't talking to anyone. Whereas we did. We descended from animals that had a lot of the same experiences, but they weren't yet hooked up to language.

B (140:51)

Right.

A (140:51)

So yeah, that's like this interesting feature of LLMs. To bring it back to the experiment, we do want to find the equivalent of internal processing that is in some sense independent from output in the case of LLMs. And this paper is trying to do that first with detecting. So detecting something that's been injected before, you've seen your own outputs, and then also controlling your representation in a way that's not just, well, output that word, because that's a trivial way in which obviously language models control their representations. If they want to talk about aquariums, they activate that and then talk about them.

B (141:38)

And in this case, nobody's tried to train anything like this. This emerged just because.

A (141:48)

Yeah, interesting. As far as I know, models get nothing like this. It's all input, output. Right. All predict this, predict that, don't say bad words, do say good words.

B (141:59)

That's pretty cool.

A (142:01)

Yeah. And it is coming with scale. Earlier, when I was talking about why I work on self reports, even though they're so noisy, the speculation was that models will increasingly get better at self reports. Yeah. So this paper shows that at a greater scale, models seem to have something like introspection more and more that's important for the self reports part. You might want and need introspection to use self reports. It's also, I think, independently a welfare relevant marker in the following way. Some people think that introspection is a component of consciousness. So in theories of consciousness, you can distinguish between ones that more emphasize representing the visual world, like maybe first order theories. And it's about tracking things in your environment. I mean, obviously it's in part about that, but some people also say it's importantly about tracking your own mental states. These are called higher order theories of consciousness. So with a bunch of caveats, you could to put it back in our classification system, you could think of this as a combination of interpretability and behavioral testing for a neuroscientific theory of consciousness. Higher order theories of consciousness.

B (143:26)

Yeah, that makes sense and is really helpful. I did notice that as I was trying to understand and reflect back the study, I was having a really hard time not saying that the model was noticing something about the experience of that thought. It just feels really hard to disentangle introspection from something conscious.

A (143:52)

Yes. And that's a communication difficulty as well. If we're talking about human introspection, it's almost always in the context of Consciousness experiences. When I say model introspection, and when Jack, Lindsay and others say model introspection, we're trying to stay neutral on the question. This does also bring up a reason it could be very difficult in this particular study to disentangle experiences and introspection. The models talk about experiences when they report these injected thoughts. They say, I'm having an experience of something intruding on my thought process. I'm having an experience of a bakery. And I think when amphitheaters, the concept amphitheaters are injected, they say my thoughts are becoming more spacious and things like that. And that is also a huge communication challenge because we can verify that, yes, we injected amphitheaters, but there's all sorts of reasons that the model might report this experiential language around that.

B (145:06)

In particular the fact that it's trained on a bunch of human data. And this is the way that we talk about introspection.

A (145:13)

Exactly, exactly. And that I think is also a great way of highlighting this problem that you were pointing at of is it going to try to map stuff into our terms in a way that's inappropriate or not quite accurate? And again, this is not me saying, I know for sure Claude is not experiencing spacious thoughts when we inject this, but that's not what anthropic proved. And I would understand if someone half remembered this as being like, oh, yeah, they found that models can introspect on these likes spacious thoughts.

B (145:47)

Right, right, yeah, yeah. Interesting. Are there other experiments in this vein worth talking about?

A (145:54)

Yeah, I mean, I could say some experiments that I would like to see and that also might exist and I'm not yet aware of them, that you could do with interpretability that I think would be super interesting. I don't yet know exactly how to operationalize these, but they just seem like the sort of things we can use interpretability for. So I would love people to get working on things like how do models represent value and. Or predicting how well they're doing at things that relates to sentience, which we were talking about earlier. A lot of theories of what's going on in the human brain with pleasant and unpleasant experiences has to do with something like tracking some internal representation of value. So you feel bad when you were expecting things to go a certain way, but they're going worse. Can we find any kind of analog to value representations? Predictive processing is also a word that will get used in this context, like predicting how things are going to go. Tracking that. Can we detect analogs of that in models?

B (147:02)

Super cool.

A (147:02)

Yep. I think there has not been that much done at the intersection of take a neuroscientific theory and don't just look at the architecture, the general setup, because that's mostly what Patrick Butlin and me and others have done. It's just this kind of higher level architectural thing. Let's also take theories and do interpretability and look in there. It's like really difficult to do the mapping and things like that, but I think we can do it.

B (147:35)

Cool. Yeah. Any others?

A (147:38)

Yeah, I'm also just really interested in. Like earlier I talked about features that are active when models make self reports. I think there's like a big cluster of things we can look into here. So, like, are there differences in how models talk about themselves versus how they choose how a character will talk within a story? You know, like, one of the big conceptual questions is like, are these characters? What does that mean? Exactly.

B (148:05)

Right.

A (148:05)

You know, everything we're getting is in some sense the output of the assistant character. That's one way of construing it. But obviously they can play other characters. The assistant can write about other characters. Yeah. Like, what are different sort of representations of I and of minds that happen in models? I think that would be super interesting.

B (148:27)

Okay. Yeah. A lot of this sounds really, really exciting to me. I guess it still seems like all of the plausible methods for learning about consciousness and sentience still leave us with loads of uncertainty, which will just make it really hard to know how to act and I guess even harder to get. Get society on board with treating models a certain way. Do you think we'll ever be kind of properly confident about whether AI systems have moral status, or do we literally need to solve the hardest questions about consciousness and sentience? To be sure, yeah.

A (149:08)

Fortunately, I don't think anyone has to solve the hardest questions first to take really good actions. Um, there's all sorts of stuff that's just like very plausibly good. And also some of the very hardest questions we don't have solved for humans. And, you know, we still, like, no one has solved the hard problem of consciousness, but, like, we still get a very high degree of confidence about some animals and about humans. I do think it's worth worrying about and this is why I do worry about it. Like, how much can, like, neuroscience and behavioral psychology, broadly construed, move the needle on things? I do think it's necessary or else I wouldn't be doing this. I think it's extremely important that we get more rigorous about this and have evidence and evidence discussions about this and can tie policies to, at least broadly speaking, empirical evidence. But I don't think society has ever changed the way it relates to an entity just because a bunch of scientists said that they should. I don't think anyone has ever instituted some large scale change only on the basis of a paper. Papers really do help and it has happened and it has helped a lot. But I keep saying Elios needs all this help on the experiments and things, and that's very true. But then even more broadly, this whole issue needs help on all of the things that are not covered by let's detect consciousness and sentience. Let's have good policies for the near term. There's this whole cluster of things that we're really not on the ball with as a society. So experiments are good. I love talking about experiments and they're nowhere close to enough.

B (151:05)

Okay, let's leave that there. Pushing on. We've shallowly covered why it might actually just be impossible for AI systems to be conscious on the show before, but I don't think we've really done justice to that argument actually, that consciousness can only exist through biological materials. So we're going to try a bit harder to do that today. Can you lay out a kind of thought experiment that helps make intuitive why you think that we can get consciousness on computer chips? And then we'll talk about why people with this other view think that actually know that thought experiment is like fundamentally flawed.

A (151:44)

Yeah, absolutely. And maybe I'll like situate things in the like living biological side first and then say, oh, maybe it's actually kind of a computery thing.

B (151:55)

Okay.

A (151:56)

Like in some ways it's kind of easy to understand the case that consciousness is fundamentally biological because we are aware of one case and it is biological. So I mean, especially before computers existed, almost trivially, you might have thought, well, you know, that's how you get consciousness. You have a brain and a body and metabolism and cells. That's how it was built the first time. So that I think already makes it like not, you know, ludicrous to think it's a biological phenomenon fundamentally. Now I'll walk you through why. I think surprisingly, it looks like there's something deeply related about computers and human brains, which after all look pretty different. So it's actually like relatively recently that we had any idea what the brain is doing whatsoever. I think somewhat famously, maybe this is a tall tale, but I think it is true. The ancient Egyptians just threw away the brain when they mummified people and they're like, don't know, that's for wow. And, yeah, some people thought it was for cooling blood. But, you know, eventually we did learn, and I think did know somewhat early that they maybe conduct electricity or do something like that. I think we first learned this from squids, because squids have really big axons. You can see them axons being the things that send electrical signals down. Down a cell. But I think things really kind of kicked off for the. Maybe consciousness in the mind or computational in the 20th century, for one thing. That's when computers and computation were sort of formalized and invented. And that's when people noticed that you could hook up neurons in a way so that they compute logical operations. So the first people who nailed this were a couple of guys called McCulloch and Pitts in the 1940s. And they sort of like, first formalized and invented the thing we still use today in neural networks, which is like a node and connections, and then they can influence each other. And they realized that you can compute arbitrarily many things if you just hook up neurons as logic gates, and then you can compose them and combine them. And so anything you could do on any calculating device or computing device, you could do with these neurons. And they were like, oh, like, maybe that just is like, that really is what neurons are for. They're for processing information. At least for me, that is kind of where the brain and consciousness, being computational, comes in. And like, we have learned that neurons do encode quantities and perform calculations in how fast they spike. Maybe a simple example is you have to detect how bright things are, and you have neurons in your retina that just spike in a way that encodes how bright things are. As a side note, they do this on a log scale, which is why it's harder to discriminate between two really bright things versus lower down on the spectrum.

B (155:27)

Okay.

A (155:28)

This is called, like, Weber's law that. Yeah. Discrimination of stimuli isn't linear.

B (155:34)

Right. Interesting.

A (155:35)

Yeah. Yeah. The more you know. Yeah. So now we have this view of the brain where in some very important sense, what it is for and what it does is processing information. So then it's just natural to wonder, do you have to have that happen with neurons that send each other signals by pumping ions into channels and then activating each other like that? Or could you just hook up a bunch of wires that influence each other like that? So that's actually not even a thought experiment. That's just notice that the brain does look like something that is for information processing and does information processing. And then I think two more just real world Things that have borne that out. One is it's just really useful to think of the brain in terms of computation. So computational neuroscience, in some sense doesn't just model the brain computationally, because you can model planets computationally. But that doesn't mean just because a computer can describe the orbits of planets gravitationally, it doesn't mean they're computing how far they are from the sun and when they should speed up. The brain does something more than that, which is it seems to actually be encoding information. There actually seem to be neurons that are tracking the expected reward of a stimulus or something like that. And then add that AI works. That's something that we might not have known. It could have been that thought is a biological property, or classifying an image is a biological property. And so it doesn't matter how many metal tubes you hook up to each other, you're just not going to get something that can classify a dog or write a sentence. It looks like thought is pretty well replicable as computation. So it's also natural to wonder if consciousness is as well. So that's actually with no thought experiments. You might have invited a philosopher on for the thought experiments, but that's just all in the real world.

B (157:42)

Sure. Facts.

A (157:43)

Yeah, just straight facts. Facts and logic. And logic gates.

B (157:47)

Yeah. I do basically just find that argument in itself very compelling. It also was just really helpful for me to at some point hear a thought experiment that kind of makes it super intuitive. Do you mind also taking us through that?

A (158:03)

Sure. Probably the thought experiment you have in mind is the neuron replacement thought experiment. That's the one which is by former guest David Chalmers. And I think this, funnily enough, is the main argument for computational functionalism or the view that consciousness can be computational. And I think it's not that convincing to a lot of people. So that is an interesting feature of how I relate to computational functionalism, at least, is that I do find it extremely plausible and also understand why people regard this thought experiment as question begging. So let's get to the thought experiment. The thought experiment is, suppose you can replace one of my neurons with a computational circuit that will take in the same inputs and send out the same outputs to other neurons. Let's imagine someone just did that. Right now I'm not going to notice that. I'm not going to behave any differently. Let's keep doing that one by one. So at some point I'm like 50 50. If we're actually doing what the thought experiment stipulates, I'm not going to start saying anything different if we've actually replicated the function, that should preserve memory and speech and motion. If it starts getting messed up, you must have accidentally broken something or done something wrong, and then imagine we've just gone all the way. Is that thing conscious or not? The lesson of that thought experiment is not just meant to be a gradual change thing where it's weird to say that something fades out at a certain point.

B (159:46)

Right.

A (159:46)

That's true of a lot of things. Another thing philosophers often talk about is like, well, there's no, like, hair, where if you remove it, then someone becomes bald. But we do know that somewhere in the somewhere, especially for me, along that transition, you do get something bald. So you can't just say, oh, well, like, that thing must also be conscious because it started out conscious. What Chalmers says is that thing's cognition should be the same. It should report being conscious and remember being conscious and attend to things. And that's what's supposed to be weird. If you have this biological view is to think, at no point did you notice consciousness popping in and out of existence or gradually fading out, whichever you think it might be.

B (160:35)

Right.

A (160:35)

And that's what would be surprising is if there's this weird disconnect between cognition and consciousness.

B (160:44)

Yeah. The thing that it does for me is point at the potential importance of function and not substrate. So the fact that it's at least, it seems like an empirical question that we don't have an answer to, but it's at least plausible to me when you describe that if we really have figured out how to replicate the function, that the substrate should not matter. And I guess the whole debate here is like, is that literally possible? Maybe it is just not physically possible to replicate the function on anything but biological materials. And I think I find it really intuitive that we should be able to. And I don't know if I can fully justify it, but at least because of all of these analogues, analogies between brains and things that computer chips do, it just feels like, yeah, we replicate these kinds of processes all the time. Can you help me understand the thinking for why we wouldn't be able to create computations that kind of replicate this signaling that neurotransmitters do, or the metabolic processes that influence neurotransmitter signaling.

A (162:13)

Yeah. So I think this is getting at the role of neurons in this debate, which I think actually it is kind of hinging on neurons. Like, one reason you could be drawn to the view that we can do this on computers is you're like, really what matters are These logic gates and that's kind of it. And like what the brain is is these like influences between neurons on each other. And one thing that you'll often hear more biological consciousness people talking about is like we have since discovered, like we discovered that neurons are surprisingly important and kind of like the key to the whole thing. But there are other things that are also very important. Glial cells are like another kind of cell in the brain. Like they seem to influence cognition in certain ways, like blood flow patterns, can't metabolism. We also have brain waves, more large scale patterns of activity that also don't come down to this local thing. So I think in a sense that's enough to say, okay, well it's not just going to be a matter of seeing the brain as exactly this. I'm sympathetic to the thing you just pointed to though, which is I feel like we'll be able to at least get close enough to somehow like getting the influence of the larger scale patterns or glial cells or things like that. But it does complicate the picture. I think it's kind of a question of like at what level of description can you swap things in and out? Like you could imagine someone who thinks, okay, well there's like a few lobe, like a few different lobes of the brain and like they talk to each other. And at that level that's the only level of description you need is like you just need like five things that talk to each other. That's like not that plausible. At the very lowest level I think everyone would agree you can swap out electrons, right? And you can like swap out like a cell here or there's. Mm, yeah. The question kind of is like at what level of detail, at like what scale can we swap things and out? And that's like you can be a functionalist and still think the function is going to be kind of finicky and biological and at least not this like simple computational picture.

B (164:44)

One thing that did land with me when we had guest Anel Seth on was he said something like simulating a rainstorm doesn't make anything wet. So even if you built kind of a perfect model of a rainstorm, nothing would be wet. Simulating digestion even perfectly doesn't digest anything. So maybe simulating consciousness, even if done perfectly, doesn't create a conscious entity. Maybe it tells you exactly what a conscious entity would do in theory if it were conscious. Maybe it really is perfectly good at predicting behaviours and feelings and thoughts, but isn't actually generating or pulling those things into existence how do you respond to that?

A (165:40)

Yeah, I think of this as not really an argument for the view. It's more like a statement in the view. I think it kind of begs the question. I think the debate is, is consciousness more like wetness, in which case this might be true. Or is consciousness more like navigation or image classification or addition, namely something computational? Because if you simulate a calculator that does make something add together that makes a calculator, maybe the reason you won't get wetness if you don't simulate the storm with enough fidelity just is because it's a very low level property with certain very specific physical effects. But if it's not like wetness, if it's more like navigation. Yeah, you can't really complain that someone has merely simulated a navigation system. They have in fact built something that will navigate your car just as well.

B (166:48)

Yeah, yeah, no, that makes sense. Okay, I guess another question then. It feels like the thing that would be most convincing to me is if you could compellingly argue that there are some physical biological processes that are associated with consciousness that no human can come up with a clean computational analog for in theory. Are there any processes like that that we know of?

A (167:23)

Yeah, I don't think anyone would say that there are, including the people who have this view. I think if you're skeptical of the biological functionalist stuff, you might kind of read about these descriptions of all the metabolic and living things and then still want exactly the argument you're talking about, which is like, okay, but like, is that intrinsically and always biological? I mean, I think quite understandably, given the state of this field and our knowledge of the brain, biological functionalists don't have that. And I think for that reason, and also just good epistemics, basically none of them are like, we know that only living things can be conscious. I often point people to a quote by Anil Seth that I really like where he says my view is that computers will not be conscious anytime soon, if ever. But I might be wrong. And yeah, certainly the state of argumentation is nowhere close to, okay, let's all just sleep peacefully knowing that we can just keep building things that look a lot like brains, but they're not alive, so they will never, they'll never have experiences. Yeah. So yeah, I mean, I think that very weak thing, which is like, we certainly can't rule out. I also think something stronger, which is like, my sympathies are very much with computational functionalism.

B (169:02)

Yeah, yeah. I'm interested in what you find most compelling.

A (169:08)

I would like someone to write A paper that argues for something that I find strong, very intuitive, which is that if phenomenal consciousness can't be had on a computer, but stuff that's very functionally similar to it can be, then that's really good reason to think that it's not phenomenal consciousness that matters. So I actually have this even stronger view that being a moral patient or the sort of thing that matters, that really does seem substrate independent to me. Like if I imagine Commander Data, for example, and I find out that internally his computations look a lot like the brain at some level of description. I just care about whatever that thing is. Maybe it's not phenomenal consciousness, but it really seems like I should take it seriously and care about it. So, yeah, that's another view I have about this biological view. And I'm often curious what biological functionalists would make of this. I think it's very possible to over index on consciousness. And that's something we try not to do at Elios. You could think, yeah, you need biology for consciousness. But all the stuff you can get on computers, that will be enough for beings that merit our consideration.

B (170:33)

Okay, I think we should leave that there. Pushing on. You founded Elios AI. What is the backstory? Last time we spoke, you were working as an independent researcher on this topic. And now there's an org that exists.

A (170:49)

Yeah. And really a lot of the key stuff did happen between the last time we spoke. I had just moved to San Francisco. I was doing a philosophy fellowship at the center for AI Safety while continuing to work on consciousness and welfare stuff. In 2023, Patrick Butlin and I published this big paper on consciousness indicators. Ethan Perez and I wrote this thing on self reports. So there was like, along with many other papers by other people, this kind of finally this sort of budding thing of like, maybe we can actually start thinking about this. Actually, there's some evidence we can try to gather. And Then so separately, NYU's center for Mind Ethics and Policy was starting up with Jeff Sebo also working on these things. So Jeff Sebo and I were approached by Anthropic to study what should anthropic think and do about AI welfare. So there was this group project going on towards the end of 2023 that eventually became this paper called Taking AI Wolfers Seriously. But somewhere along the way, I'm in San Francisco. I had actually stayed in San Francisco because I thought to myself, I bet if I stay in San Francisco, something interesting and San Francisco Y will happen to me. And that's sort of what happened because like that work just that sort of led to the founding of Elios. Like, a few people, Kyle Fish, I think, was a big one of them, said, you should scale this up. You should make it so that more work like this can happen. And I agreed to do that. And then Kyle joined as a co founder. He did a lot of the work to help me get it up and running, and then went to Anthropic to start their AI Welfare program. And then Kathleen Finlayson really helped us launch and get things running. So, yeah, Kyle Fish and Kathleen Finlanson were like the two people. I mean, we know each other. You will definitely believe me when I say I couldn't have done it alone. I was like, no way am I being a solo founder. That would just not suit me whatsoever. So, yeah, that's the story of how things got up and running. LEOS kind of kicked off its public debut with taking AOL Fare seriously. And, yeah, that takes us to late last year. So I guess again, the broad strokes are at FHI and then later in San Francisco, I'm trying to work on AI Welfare, trying to ask what can we actually do about this? And that sort of just naturally leads to an org that is, like, unsurprisingly, focused on, like, the same things I was focused on before, where, like, I think the way I often put that is like, we want to be the org. That's like, okay, but what are we actually going to do about it? We do research and are extremely research oriented, but, like, we prioritize that according to. Okay. Like, we might not have that long to sort out all of the philosophy and all of the neuroscience. So we've got to pick the most action relevant things and get this field rigorous and get a community built around it and navigate this issue well as transformative AI happens and there are dangers all around. Cool.

B (174:23)

Yeah. What kinds of projects have you been up to since starting up?

A (174:28)

Yeah. So there was Taking AI Welfare seriously. That paper was meant to get people to take AI Welfare seriously. And primarily like labs and policymakers and people like that. It wasn't primarily philosophical and that. Yeah, it was like, just like, all right, people, we can like, you're allowed to work on this. You can think about it clearly. And we really do need to start taking steps now. So, like finishing that and media and things like that around that were like the first big project. We ran a workshop in January trying to assemble the people who are thinking about this in similar ways. I think another big landmark project was doing this AI Welfare evaluation of Claude before release. As far as we know, that's the first ever officially commissioned welfare eval. So that was super exciting and a very promising and insufficient step. And yeah, other things that happened in 2025 include this big conference, ELIOS con con the ELIOS conference on AI consciousness and welfare.

B (175:45)

So good.

A (175:46)

Yeah. And there we're trying to broaden it even further. Like we need policy people, we need academics, we need neuroscientists and people not just in our neck of the woods introduced to this topic, encouraged to think about it rigorously getting in the game. So that has been another major effort that we've undertaken.

B (176:10)

Cool. What else is coming up? What are your upcoming plans?

A (176:16)

So we've also been working on taking Iowa seriously too. That's been its name. As it's under construction. It's going to be building on the first paper which says we need to get some evaluations in place, we need to get some policies in place. This paper is going to be like, okay, towards AI welfare evaluations. What are the different kinds of evaluations? What exists? What do we know so far and where should that field head? So, yeah, the content of that paper, a lot of it shows up in different ways in this very interview. We're very excited about that. And relatedly, we do just want to scale up empirical work and evaluations. We've had some great collaborators through fellowships and just independent collaborators, but we'd like to really scale that up and get a program up and running.

B (177:11)

And what would you say you're most bottlenecked by at the moment?

A (177:16)

Probably talent. Although I would like to say very loudly to listeners, we are fundraising, we do need funding. But maybe the even rarer and harder to find thing is people with the temperament and skills to do this extremely weird kind of work that has no real playbook yet and is some weird mix of philosophy and neuroscience. And AI people don't have to have training in those. I think it's more like a set of epistemic tendencies and a set of skills. So we need people who can be philosophical enough to be asking what evals would actually make sense and technical enough and self starting enough to start building them. We need people who are confident enough that AI welfare matters, but not too beset by uncertainty about philosophy and things like that. I guess I'm just describing people who are kind of like the dream of every young org, but we really need people who have a lot of drive and agency and can just pick things up and run with them.

B (178:33)

Yeah. Do you think these kinds of people will have Particular kinds of backgrounds or who listening should be. Hey, that's actually me.

A (178:42)

Yeah, I think these people really can come from anywhere because you're trying to find your way to the middle of this overlapping Venn diagram. So one obvious place to come from is already in those Venn diagrams, but that's not necessarily so. Maybe you already work on evals, but you're interested in AI consciousness and welfare. Maybe you're a philosopher who is willing to just learn stuff and start doing stuff. Maybe you're a neuroscientist who is increasingly interested in AI and can code. And this is all.

B (179:16)

How technical do you have to be actually

A (179:19)

for some kinds of evals, you definitely need to be technical. I think I myself am not technical enough to exactly specify how technical we're talking about, but Rosie Campbell, my managing director, also is on that. For some kinds of low hanging fruit evals, especially input out ones and behavioral ones, my understanding is you can learn that stuff. I'm sure you've experienced this. I'm amazed just how smart people are. I think there could very well be listeners who don't know any of this at all right now, but who could just soak it up and just like start doing it, pick it up. Yeah, Kyle Fish is a good example. He had these traits that were really important of like drive and prioritization. His formal background was making a vaccine. I think that like illustrated the kind of like, yeah, let's just do this like new thing aspect. And then, you know, the welfare and consciousness, it's a very small field. So if you spend some serious time grappling with it, you can pretty quickly get to the top percentile in terms of how much have you thought about this nebulous applied AI, welfare and consciousness thing?

B (180:44)

Cool. Yeah, you've mentioned a few kind of projects that you'd love for someone to do. What are some that you haven't mentioned that are near the top of your list for just like we'd learned so much. Someone please do this.

A (181:01)

Yeah, maybe I'll mention some projects that I would really love to see that aren't in the Elios wheelhouse. So I can also explain sort of what our wheelhouse is so people can situate themselves in the broader field. Because I just described someone who might do this technical eval kind of thing. But as part of the broader problem of getting this right, there are so many other kinds of people who can and should get involved. So yeah, let's open it up to all kinds of listeners. Okay, so what are the four questions that we need to answer well, to get to a flourishing future for all sentient beings. The first one is, what would it mean for an AI system to matter? What are we looking for? Are we looking for consciousness, agency, something else? The second one is, how would we know if it had that thing? So that's some philosophy and then also some science of evaluating. The third question is, what should we do? So what policy should labs have? And then societally, more generally, what should the sue? And then the fourth question is, where's this all going? What's the broader trajectory? How do we strategize around this? So what would it take? How do we know? What should we do? Where's it going? Elias is kind of in the middle too. How would we know? What should we do? And we're also at a very applied end of that. So how would we know? We're like, okay, let's just see what evals make sense in light of current knowledge. What should we do? For now, we're focused on AI companies. But that, of course, is extremely incomplete as part of the playbook. So you can really contribute by looking at questions one and four of more fundamental philosophy and bigger picture strategy. And you can also find yourself doing different kinds of work on how we know and which we do. Um, so like, eventually governments will need playbooks for this. We haven't really worked on law and policy at all. And we're not sure what we would or should say within. How do we know? Like, there's a lot of work to just make progress on consciousness and sentience and like, conceptual work. And then also on the stuff we are on. Like, we're three people. So also just, you know, jump in there. I don't think anyone will have gotten this impression, but, like, just to be clear, like, we do not have this handled. And like, we want help and like, we're here to in large part, like, help people get in the game and like, actually figure this out.

B (183:48)

Okay, so that's kind of a taxonomy of projects. Are there specific ones worth highlighting?

A (183:55)

Yeah, so maybe I'll do one from each. So what would make an AI system matter? I'm especially interested in perspectives that decenter consciousness and sentience. Those are so intuitive in terms of how many people think about this. But could the AI system feel something? And I think there are good reasons to suspect that that picture is maybe incomplete and limiting. Maybe I'll just pick the simplest one. Although extremely counterintuitive, it could be that consciousness doesn't exist. Some people do think that I will not be elaborating further, but there's kind of a sub literature on. Okay, if illusionism about consciousness or consciousness not being real is true, then what should we be looking for? I would love to see people flesh that out. And in general, just like, are we thinking about consciousness versus not consciousness in a good way?

B (184:54)

Okay.

A (184:54)

And it's like super philosophically rich, which is part of why I'm refusing to elaborate on it. Because we can't tape another episode yet. And then on how would we know? I guess I did mention let's do a lot of interpretability. Just start doing interpretability on stuff that is even superficially welfare relevant or related to how models think and do stuff and understand themselves. I think there's a lot of low hanging fruit there. And understanding model preferences, we can still just flesh that out a lot. Like what exactly are the kinds of inconsistencies we see? How do self reports and revealed preferences match up or not? I think we can just get a lot more granular on that. And listeners who already know how to just run LLM experiments can just start doing that. I might mention a very small example of this kind of thing of like you can just hop in and add detail to something. So I know this show has covered the spiritual bliss attractor state. This is where two clauds in conversation will at a very high portion of times end up in rapturous mystical dialogue with each other.

B (186:19)

Yep.

A (186:22)

So that's. That's strange.

B (186:25)

Insane. Really weird.

A (186:28)

Not claiming it's directly welfare relevant, but it's an interesting fact about models. Someone after that came out, someone just ran it on a bunch of different other models and now we have like the sort of, you know, spiritual bliss benchmark. Like you can always flesh things out in. In this way.

B (186:42)

Yeah. Nice.

A (186:44)

What should we do? Maybe here I'll highlight something that again is non elios. We're often very focused on welfare and moral patienthood. And from the perspective of us caring about models, what kind of actions might that motivate? There's also this whole landscape of ways of cooperating with AI systems or legal reasons. You might give them certain rights just as entities that disperse money or something like that. That's distinct from the moral patienthood question, but deeply interrelated it has been in human history. I think it will be for AI systems. And then on. Where is this all going? I would just love to see some forecasts. So rethink Priorities has been building this model of how likely an AI system is to be conscious. If you like this episode you'll definitely like that project. You could just ask, okay, given these inputs to the model and how AI is going, how do we expect the model's outputs to change over time? Yeah, I think that was first suggested to me as an idea by Kyle Fish and I hope someone there or elsewhere does it.

B (188:08)

Cool. Yeah, those do sound extremely cool. I guess for people excited by those excited by the field, maybe hear themselves a bit in your description of who you're looking for. What advice do you have for people interested in contributing? I guess entering new fields can be particularly hard because there's less mentorship and there's less like. Yeah, there are fewer entry level roles. You just really have to be able to be pretty self starter. Y so how do you get involved?

A (188:43)

Yeah. On the self starting and mentorship question, I am heartbroken by how many talented people. We just don't have time. I would love to supervise a project by you. Fortunately there are communities sort of self forming around this sort of. There's an AI welfare discord. There's LLM psychologists on Twitter who talk to models all day and are always exploring interesting things about them. There's nyu, cmep and a whole host of orgs also kind of in this space. I don't know about a whole host. It's still a pretty small space, but several. So it never hurts to reach out to people. I guess that's in some sense a cold take, but I don't think it's ever been bad for anyone's career to be reminded that if I don't have time to take your call, that's neutral, it's not negative. So just ask.

B (189:48)

Yeah, some people will probably read voraciously and really know a lot about the content of the field. How can they go from knowing a lot, being really interested, to being properly useful?

A (190:05)

Yeah. So one category of usefulness I haven't mentioned yet is writing. I mean it's easier said than done, but just writing clearly about this stuff is really valuable. It's good for career capital to have a blog where you read papers and just explain what happens in the paper.

B (190:25)

Nice.

A (190:26)

It's also just good. Yeah. For the ecosystem. And it also shows employers that you're smart and can communicate clearly and can get things done, which is like 95% of what it takes. So here's a very niche recommendation. I can have trouble being a self starter, but I have published on my substack twice a month for like well over a year. And I did that by making a manifold market like a betting mark about whether I would do that. So, yeah, use like. Yeah, use commitment devices.

B (191:05)

Accountability.

A (191:06)

Yeah, accountability, Things like that.

B (191:07)

Nice. Yeah. Is there anyone worth, like, citing as a kind of example? Kyle Fish could be one. Given that he worked on vaccines before. What exactly does it look like to go from not working on this to, like, you know, Kyle Fish is like, properly working on this. And, yeah, it wasn't that long of a time from not working on it at all to working on it at anthropic. I don't know if you. Yeah. If it's worth either describing his trajectory or describing, like a theoretical trajectory or someone else's trajectory.

A (191:46)

Yeah, I guess I'll just extract a couple of lessons from the Kyle trajectory.

B (191:51)

Perfect.

A (191:52)

Which, like I just mentioned, one is like, reach out to people. So he just started reaching out to people. The other one is that if you read and think about this stuff, like, you can find your way through most of the literature pretty quickly. And I guess the other one is like, you can just do things. That's ironically, easier said than done. Sometimes you can't just do things, or you need the right environment and structure to do that. But, yeah, I think that it's just a very good illustration of some general principles of how to jump into a weird new field. And another thing is that was sort of jumping all the way in. You might be in one corner of the field, but want to move to another. So you might be doing a PhD in philosophy, but you want to do more applied stuff. I think it can maybe sometimes be harder for those people to move within the field because you're more attached to your particular corner. I think I have seen this with philosophers or other grad students. This definitely happened with me. You have gotten really good at one particular kind of work output, and you just figure maybe that's the only kind of work output you can do.

B (193:12)

Or it's like, shitty to have to become bad at a different kind. Like, you have to really be like, ugh, not that good at this new thing yet. After being maybe. Yeah, I think I just, like, I enjoy being good at my work. It sucks to be shitty at it for a little while.

A (193:29)

Yeah, absolutely. And I guess it's just a side point about grad school. Like, you're just surrounded by an environment where that's literally the only currency that could possibly matter. So go to conferences, make other friends. I do want to say that's a good thing about grad school. You need communities of people all doing the same sort of thing to achieve excellence. But, yeah, if you're Looking to move around. That brings us back to just reach out to people, email people. I feel like at least my corner of the AI welfare space I can genuinely say is just extremely nice. So you can do stuff that feels kind of dumb or ask questions that might sound kind of dumb to you. A, they're probably not, and B, it's a new field. It's hard, so just go for it.

B (194:23)

Nice.

A (194:24)

Yeah. And actually, it occurs to me that there are three other great examples of trajectories that are very close at hand. It's the three people I'm working with right now. So maybe briefly say how they got to Elias. So Rosie Campbell had worked on AI policy and evals, so she had part of the equation for what we need, but didn't know much about AI welfare. She wrote some about it. She talked to me about it. She was enthusiastic about it. And like, there you go. Patrick Butlin has a similar profile as me in that he was doing academic philosophy. He is just, like, generally, like, very fearless and diligent about just, like, reading a lot until he gets it. Like, I think that's like, a big part of the consciousness and AI project we did is like, Patrick Butlin will just read the papers until he knows what to say. So that's a very high conscientiousness route to it, maybe, and high rigor, but it's a great way to be and big part of why I work with them. And Lars Escavo has worked with us on communications and events. And also just this is a great illustration of you might think of ways to contribute that were not mentioned in this podcast and, like, have not been mentioned so far. Larissa got this shirt made and delivered. I did not ask her to do that. It's awesome. Thank you, Larissa. She also made these stickers. I have really been waiting for an excuse to show one of the Elias stickers. So one second here.

B (196:12)

Great.

A (196:13)

These have been surprisingly good for, like, morale and field building.

B (196:17)

Nice. What is it like to be a bot? Do you want to explain the joke?

A (196:22)

Yes. So the philosopher Thomas Nagel has a paper about consciousness called what is it like to be a Bat? Touches on many of the themes of this interview. There could be, like, different forms of consciousness. It can be hard to know about them from our human vantage point.

B (196:37)

Such a good sticker.

A (196:39)

So, yeah, that's a way of contributing to the field. I mean, I'm not saying everyone should now make even more stickers. I mean, that said, maybe so. So that's on, like, the self starting Angle like one day like there were just stickers in the office. And yeah, you can contribute by like being very creative and thinking of ways of communicating about this stuff. I also want to say Kathleen Finlanson. So now we got like the whole, the whole squad, the whole roster. I don't know exactly how this background translated, but Kathleen had been in a Zen monastery for quite a while before she had re entered the world of AI forecasting and AI strategy. So yeah, she had this combined CV of open philanthropy style, AI forecasting and Zen Buddhism. That's a cool combo. But I also think more than anything else, like yeah, just doing it like that, like she was willing to just like get in the game. I'm getting a little emotional, but yeah, you know, like carried the org over the finish line. That was great.

B (198:01)

Yeah, it sounds like a bunch of legends. It sounds like caring a bunch about the topic really can get you a chunk of the way there. In general. What are some ways that you think things could go badly in. Yeah. In this field?

A (198:24)

Yeah. I think one thing I worry a lot about and I think this actually does pair well with what I was just saying. It is really important to be rigorous and communicate responsibly about this. There is a kind of person who gets really passionate about this and maybe needs to talk to more people about it or like write down their thoughts more because it's just really easy to get like really confused, like really fast. And yeah, there's something about this topic that can like induce or select for like various ways of just getting a little bit off kilter. It's tough because you do want to be off kilter, like, but you know, like not, not correct. Not too much. So I do worry about scenarios where the field becomes associated with. Yeah like wild speculation or too associated with psychedelics or too associated with like something that's like relevant. But is. Is also a bit of a distraction. I should also say a bit of it is also like a divide and conquer thing. Like Elios really is trying to exist in that real button down kind of place. There is also. I have a lot of love for people who also kind of get weird with it, but you want to be able to communicate it well and make sure that people do know that this is a serious topic that we can and should reason about rigorously. So I think epistemic hygiene is something I, I worry a lot about. Yeah, it's just really hard to. I mean it's just really hard to get this issue right and the future is going to get more Confusing and more emotional. I think I first started saying this with Kathleen, but it's continued as like an Elios. Org goal is like a lot of what we want to do is like stay sane in the next 10 years. There will be a lot of alpha in not losing your grip. I think that's a whole other episode where I don't actually even know what the right advice is. But you probably would have good things to say about that. Yeah.

B (200:53)

Okay. What do you think it looks like for this field to go well?

A (200:59)

Yeah, I think if this field goes well, this becomes just part of the general playbook and set of issues that are on the table. As like if people keep trying to build a new form of intelligence, it should be on the table. How do they matter and what part do they play as moral patients? I mean it's often just shocking to me that is barely ever comes up. You know, like we worry about over attribution and people getting confused about AI welfare. But if you look at like the broader like trajectory on the whole, at least right now, mostly it's people just like not putting it on the table at all. And yeah, again that's something where the factory farming analogy is very illustrative. People aren't great at structuring society in an inclusive way. So I think there needs to be a combination of rigor to get this taken seriously. Good communication, also all sorts of innovation around law and policy and stuff that probably won't even have that much to do with moral patienthood. Yeah. To get this properly handled, I feel like there's just so many ways things could go off the rails. We first want to just make sure a lot of people are taking it extremely seriously and we're doing our homework as we go into transformative AI.

B (202:43)

Okay, nice. We have been talking for many hours. We have time for just one more question. The last time I interviewed you, we ended up talking for something like an extra hour about strategies that have helped. I guess you and I think me at the time enjoy and be more kind of productive in independent research. I still recommend that little mini episode that we kind of ended up separating it out and making it into its own little after hours episode. But since then I know you're just kind of like a legend at self improvement. So what is your kind of top lesson for doing independent research of the last two years?

A (203:29)

I think for me the biggest lesson has been don't do it. And I think this could. I hope this does help listeners. One reason I do need all these self improvement tools and things is, I think in many ways I'm like temperamentally very badly disposed to by myself, all alone in a room, carrying out a project. Now, if you have to do that, I encourage you. I'm cheering you on. And listen to that other episode. But nowadays I have written on the whiteboard next to my desk, do not write alone. Like, I think there is like a good meta principle here, which is if you're needing tons and tons of like little tricks and psychological cartwheels, don't stop doing them. Like, a lot of life is just muddling through with a bunch of little fixes, but it could be that there's some bigger structural thing you could do to just completely route around them. So for me, that's. That's co authoring. I could and do at great effort, learn a new scheduling tool and optimize my accountability systems. And again, you should do that. But also I can work with people who are just really good at that. I think. Yeah, this can be a trap of self improvements. Like, you might to some extent need to grieve that there are some things you might not be that great at. At least not without a ton of work. And then like free yourself of necessarily having to be. Yeah, and like pair up with someone who can do it.

B (205:26)

Totally. Yep, yep. I think that's great advice. We have to leave that there. My guest today has been Robert Long. Thank you so much for coming on.

A (205:35)

Thank you so much for having me. This has been fantastic.