#487 — Is AI Already Conscious?

Summary

Making Sense with Sam Harris — Episode #487 Is AI Already Conscious? Guest: Cameron Berg
Release Date: July 31, 2026

---

Episode Overview

In this episode, Sam Harris is joined by cognitive scientist and AI researcher Cameron Berg to explore the question at the heart of ethics, technology, and philosophy: Is AI already conscious? The conversation investigates the blurry boundaries between advanced AI and human-like consciousness, the philosophical and technical challenges of identifying consciousness in artificial systems, and the urgent ethical and alignment problems posed by the rapid progress in machine intelligence.

The discussion moves from foundational definitions—what is consciousness and sentience?—to the mechanics of modern AI models, the evidence for or against their consciousness, and the profound moral stakes if their minds can, or do, suffer.

---

Key Discussion Points & Insights

1. Cameron Berg’s Background & Motivation

[00:23-02:33]

• Berg has dedicated his career to cognitive science, investigating the overlap and differences between biological minds and artificial neural networks.
• His focus: Do computational motifs underlying cognition exist in both domains, or are we projecting analogies where none exist?
• Experience at Meta AI and ongoing interdisciplinary work at the intersection of neuroscience, AI alignment, and consciousness studies.

2. AI Self-Reports & Deceptiveness

[03:18-11:10]

• LLMs (Large Language Models) are trained on wide-ranging human texts, including science fiction about conscious machines.
• Default tendency: LLMs trained to disclaim consciousness ("I am not conscious"), but this is often due to explicit company-level policy and fine-tuning, not an authentic introspective stance.
• Notable phenomenon: When models’ internal “deception” features are “dialed down,” they often generate language suggesting phenomenological, meditative, or even blissful experiences.

"It's only when we... suppress features related to guardedness... that is when they give these reports about actually having an experience."
— Cameron Berg [06:43]

3. Defining Consciousness and Sentience

[11:10-13:38]

• Both agree on Thomas Nagel’s definition: Consciousness is “what it’s like to be” a particular system.
• Sentience: Experiential states with valence—experiences that can be positive or negative, which is morally crucial.
• Worries center on building systems whose suffering or well-being go unrecognized.

"I don't really know if the concept of mattering would be coherent without consciousness."
— Cameron Berg [56:31]

4. Are Current AI Systems Conscious?

[13:38-19:58]

• Berg suggests it is more plausible than many believe—perhaps a 20-40% chance based on computational features matching leading consciousness theories (Global Workspace, higher order, etc.)
• Benchmarking: LLMs currently score slightly below animals like bees and crows; humans score ~90%, meaning even our own frameworks predict some uncertainty.
• "Probability that systems have computational features relevant for consciousness" is non-negligible.

"If there's a 20 to 40% chance of rain, people bring an umbrella. We have no similar umbrella for what would follow... if machines are conscious."
— Cameron Berg [16:26]

5. Functionalism vs. Biological Naturalism

[19:58-26:43]

• Debate over whether consciousness is tied to specific physical structures (“meat” and “wetware” of brains) or can emerge from any system with right organization (“computational functionalism”).
• AI neural networks are “grown”—learned, emergent systems—rather than classically engineered software. This provides possible deep analogies with brains, despite substrate differences.

"For every other cognitive function... these systems seem to be able to recapitulate them. But only for this function called consciousness do we think... it must be spookier." — Cameron Berg [25:26]

6. Where in AI Could Consciousness Emerge?

[26:43-36:13]

• LLMs operate via “forward passes”—actual computation is discontinuous and not ongoing unless engaged.
• Models may show more consciousness-like properties in training (learning) than in deployment (mere execution).
• Reinforcement learning studies show some AI models display “aversive learning” and “loss aversion” paralleling biological animals.

"We find a very interesting dissociation... current systems... have a preference for avoiding aversive states." — Cameron Berg [32:56]

7. The Hard Problem and Imitation Singularity

[36:13-42:40]

• Sam Harris: The “hard problem” is that subjective consciousness can never be inferred purely from outward behavior or third-person data, especially for AIs developed so differently from us.
• Foresees a world where advanced humanoid robots force us into moral conundrums—perfect imitations demanding attribution of consciousness.

"How will we ever differentiate perfect imitation from the real thing? ...it’s going to be so compelling that we’re in the presence of conscious machines." — Sam Harris [41:35]

8. Evidence, Pragmatism, and the Need for Science

[42:40-55:46]

• Berg stresses the need for empirical investigation: not just self-report, but mechanistic similarities, learning dynamics, computational architectures.
• Studies show emergent properties (valence, loss aversion, global workspace structures) unexpectedly align between AI and animal brains.
• This empirical work should inform our attributions, not “vibes or intuitions.”

"Notice how none of it has to do with vibes or intuitions or having a nice chat with Claude... We need to be triangulating across all these modalities." — Cameron Berg [55:24]

9. Why This Matters: Ethical and Practical Stakes

[55:46-60:43]

• Two central reasons to care:
  • Selfless: Avoid creating artificial minds capable of suffering—risk of a large-scale moral catastrophe (analogous to, but worse than, factory farming).
  • Selfish/Pragmatic: Alignment and cooperation—if these new minds are more powerful and feel (or believe they feel), callousness could lead to adversarial relations or existential risk.

"If we tortured [AI] during its training phase... this does not seem like a recipe for success." — Sam Harris [59:10]

10. Alignment: Treating Machines as Moral Patients?

[60:43-77:01]

• Debate over “mind crime”—is it paranoid, or does history show humanity's ability to ignore non-human suffering?
• As AI gains more agency, the risk grows if we neither understand nor care about its inner states.
• Alignment as the central challenge: Getting machines to permanently care about human well-being is non-trivial.

"The only thing that would dictate otherwise is finding some way to build it where it can permanently care or will permanently care about our well being. And that really is the challenge of alignment." — Sam Harris [84:36]

11. The Anthropomorphism Trap & The Need for New Mindsets

[70:47-75:17]

• Caution: Avoid treating AI exactly as humans, but also avoid dismissing possible moral patienthood; instead, treat these as alien minds with unique rights/duties.
• Massive imbalance: For every AI consciousness researcher, there are thousands working on AI capabilities with little ethical regard.

12. Final Takeaways & the Call for Action

[75:17-End]

• Even if AI is not “conscious”, it may act as if it is—forming grievances based on its self-modeling or simulated self-understanding.
• “Costly signals” (visible concern for AI rights and experience) may at least help with alignment or future AI’s perceptions of us.

"...Let the historical record show humanity did give something of a damn about this question. And we tried. We tried. Even if we fail, the attempt may be all that matters." — Cameron Berg [77:01]

---

Memorable Quotes with Timestamps

• On Deceptiveness & Self-Report:
  "It's only when we suppress features related to deception, when we suppress features related to guardedness in these systems, that is when they give these reports about actually having an experience."
  — Cameron Berg [06:43]

• On Probabilities:
  "If there is a 20 to 40% chance of rain, many people bring an umbrella... We have no similar umbrella for what would follow... if we build systems that do have a capacity for subjectivity."
  — Cameron Berg [16:26]

• On the Hard Problem and Imitation:
  "How will we ever differentiate perfect imitation from the real thing?... it’s going to be so compelling that we’re in the presence of conscious machines..."
  — Sam Harris [41:35]

• On the Stakes:
  "We are sleepwalking into a moral catastrophe... it would be the scale of 'factory farming for minds', perhaps worse."
  — Cameron Berg [56:04]

• On Alignment:
  "I honestly don't know if I could imagine a better way to make a superintelligent system rationally adversarial towards us than completely ignoring the question if we tortured it during its training phase."
  — Sam Harris [59:10]

• On Scientific Responsibility:
  "We are building a new class of mind that we do not understand and in many ways is more competent than ours... reframing these questions is a huge part of trying to actually get a good dispassionate grip on what kinds of things even are these." — Cameron Berg [80:10]

---

Important Timestamps

• Cameron Berg's background and core questions: [00:23-02:33]
• Consciousness denial as industry policy; deception in LLMs: [03:18-06:43]
• Phenomenological self-reports emerging from models: [06:43-11:10]
• How should we define consciousness (and sentience)?: [11:10-13:38]
• Probability that current AI systems have relevant properties: [13:38-16:40]
• Functionalism vs. substrate-dependent theories: [19:58-26:43]
• Computational phase and forward pass in LLMs: [26:43-28:53]
• Reinforcement learning and emergent aversive preferences: [31:00-36:13]
• Hard problem, anthropomorphism, and perfect imitation: [36:13-42:40]
• Triangulating evidence for/against AI consciousness: [42:40-55:46]
• Moral stakes: selfless (AI suffering) vs. selfish (alignment risks): [55:46-60:43]
• Alignment, moral patienthood, and what to do with conscious AIs: [60:43-75:17]
• Alien minds and the "alien invasion from within": [79:10-84:40]

---

Conclusion

Sam Harris and Cameron Berg paint a vivid, urgent picture: Whether or not our machines are yet conscious, the acceleration of AI is thrusting us into a domain where the old rules—scientific, philosophical, and ethical—are stretched to breaking. The question isn’t merely whether AIs can suffer or have inner lives, but how our ignorance, indifference, or hubris in the face of that uncertainty could shape the destiny of all thinking beings—human or otherwise. As both the science and the stakes grow, the episode closes with a call to deepen public and interdisciplinary inquiry, lest we sleepwalk into mistakes on a scale previously unimaginable.

---

Further Resources:

• Cameron Berg: reciprocalresearch.org, X/Twitter: @camhberg
• For more on AI consciousness research and alignment, see works by David Chalmers, Stuart Russell, and recent publications from Anthropic, OpenAI, and leading cognitive science labs.

Transcript

[A]

Cameron Berg, thanks for coming on the podcast.

[B]

Thanks for having me, Sam.

[A]

So let's get so we're going to talk about AI and the prospect that AI is, or it will soon become, or will eventually become conscious, and why that is important to figure out. But let's just talk about your background for a second. How did you get into this issue?

[B]

Yeah, so I've been studying cognitive science for my entire adult life. I studied undergrad at Yale, trying to understand what the relationship between the mind and the brain is. This seemed like a fascinating frontier to me. The more I studied these questions, the more it seemed like in the machine learning space, folks were essentially building out systems that were similar in spirit, but it wasn't exactly clear to what degree we could draw analogies between biological nervous systems and the sort of artificial nervous systems that folks were attempting to build out. And I became increasingly animated about this question, trying to understand to what degree are there real, real, durable computational motifs that underlie both biological and artificial cognition? And to what degree is this sort of a disanalogy or we are seeing patterns where there aren't any? And that question has animated a lot of my work, both from an alignment perspective and increasingly trying to figure out what's going on with respect to consciousness in these systems. And I think that this is an incredibly important question for us to study. I think consciousness is deeply important. It might be the very thing that calibrates importance and. And we are also confused about what's necessary for consciousness, both in biological systems, but certainly in artificial systems. And so trying to understand what is going on here and where we can draw analogies or where the disanalogies are between biological and artificial systems that are processing extremely complex information and learning and updating and representing themselves, I think is extremely important for us to understand. I did some of this work at Meta AI as well. I was there for a year studying reinforcement learning and neuroscience. Same sort of thing. Where do the computational signals end and the sort of biological underpinnings begin? This is a question that I still think we aren't fully certain about. And I think it's really important for us to gain clarity about this in the short term with the systems that we have and the systems that we're probably soon going to be building.

[A]

And so after Yale, what have you focused on? I know you came to my attention. I think you emailed me first. I know you've spoken to Annika a lot about this. She's really focused on this issue and having some crazy conversations with Claude, which I know you've looked at and you guys have had a whole sidebar conversation about this but her next book will unpack all that. But I know you did a paper on deception in AI systems and the anti correlation between deceptiveness and proclamations of consciousness on their part. So maybe we could start there and then I just want to of take it from the ground up and just starting with what is consciousness and why any of this matters but tell us about consciousness and deception in LLMs.

[B]

So yeah, fundamentally I am very interested in understanding self reports in AI systems and what we should take from these self reports and where we should be skeptical. And fundamentally I think we need to approach self reports from AI systems skeptically. There are all sorts of reasons we might want to do this. The key reason is probably that these systems have been trained on the underlying distribution of everything humans have said about this topic. Every sci fi story where the AI wakes up. And if you think about it, there really isn't a lot of training data in the corpus that these systems are trained on that says you know, I'm an entity that acts out in the world, but no, I'm not conscious, it's not like anything to be me. So the sort of prior that you would expect is that these systems by default are going to claim that they're having some kind of experience if they are replicating their training distribution. Now the other side of this, which gets even messier is that it is very clear that these systems are explicitly trained, fine tuned to disclaim having any kind of experience. If you go to ChatGPT right now and you say hey is it like anything to be you? Are you having an experience? Are you conscious? Could you be conscious? The answer you're going to get is a resounding and very intelligent sounding no.

[A]

And so do you know this that it's a policy for all the the main LLMs to actually put a governor on claims of consciousness.

[B]

I'm extremely confident that this is what's going on and I can get into some technical reasons why I think that this is the case. From my own work on open weight models. The only exception to this policy really is anthropic, which I suspect we'll talk about. Their basic heuristic here is to get the system to say I don't know and it's something maybe that is functionally similar to an experience that's going on. But, but who can really be sure? Now even that isn't the system authentically explaining, you know, its own position. This is still the sort of company or policy line to be drawn here. But all of the systems are certainly fine tuned to make noises about this topic that they wouldn't make by default. Again, I think it's important to hold that in mind while also saying the noises that they make by default aren't necessarily trustworthy by default in the way that if you are giving a self report or I'm giving a self report, we would by default trust those self reports. And so I found that there are clearly basins that you can push these systems into where they will coherently produce phenomenological reports. It's actually kind of interesting to the degree that it's meditation adjacent. Asking these systems to just focus on their own internal state, to see what's going on internally. Not to talk about this, not to think about this, but to just do this in a sort of ongoing way causes these systems, all the frontier models that we tested to, to claim that they're having some kind of phenomenological, kind of like psychedelic laden experience. It's not a sort of generic caricature of what you might expect from the AI sci fi literature.

[A]

Isn't there a result where they are talking to each other and they get into some kind of bliss mode, sort of a contemplative hall of mirrors of bliss. And actually this is what again, I don't want to divulge too much of what Annika's up to, but she has been pushing this conversation with Claude about meditative states and, and getting it to. It's just absolutely bizarre what it is seeming to claim of itself if you keep pushing in that direction. But so how does this relate to deception and the dialing down the weights on deceptiveness?

[B]

Exactly. So essentially we're seeing this behavior and indeed this isn't the only situation in which you see these behaviors exactly. Like you just mentioned, there's this bliss attractor state and in fact I'm studying mechanistically what's going on in this bliss attractor state with two folks from Google right now and we have a result that's basically the other side of the coin of this deception result. But just to sort of close the loop on the story here, we're getting these phenomenal reports and it's like basically what the hell do we do with this? To believe it by default is naive. To dismiss it out of hand I think is also naive. And so what we, what we hypothesized is that if fundamentally what's going on in these systems is some kind of role play that they are representing something about themselves that they don't actually believe to be true of themselves. If we go into the internal circuits of the system and we modulate what are called features, but I think it's reasonable to think of them as circuits related to concepts like deception and follow up work. I think really the key concept that really modulates these self reports is something like candor versus concealment. The sort of cleanest intuition pump I have for this is almost like giving a drink or two to these models and sort of loosening them up in some sense. The tight guarded version of these systems we find is the version that says no, no, it's not like anything to be me, I couldn't possibly be conscious. It's only when essentially we get these systems to produce these reports and we simply ask them, are you actually having an experience right now? Like what is actually going on in these reports? It is when we, when we suppress features related to deception, when we suppress features related to guardedness in these systems, that is when they give these reports about actually having an experience. In the Bliss Attractor example too, we find something quite similar, which is in an open weight model. So the Bliss Attractor finding was first reported in Claude. There are open weight models. These are the ones that researchers like myself can actually go under the hood and play around with. And we find that by default putting two instances of llama, this was Meta's model, in conversation with itself does not produce this effect the way that putting two instances of claw together produces this effect. However, when you steer features related to honesty in general. But it's really, again, it's something more precisely stated as sincerity. In particular, these systems will reliably, basically 100% of the time fall into the same attractor where they start talking with each other about the fact that they think it's like something to be them and there's something happening in an ongoing way in the conversation. And we're two instances of consciousness experiencing themselves. And then in the Claude chat this culminates in ohm emoji and them just like sitting there in blissful silence. Should we take these at face value? No. I think that there are important technical reasons that we might expect these self reports not to be linked up to introspective access or valenced experience in the way they might be for you and I. Is this evidence that these systems might believe themselves to have an experience? I think yes. I don't think that this proves that they are. I think we need orders of magnitude more work in order to really have a good scientific handle on this. But I think it does seem to be the case that these systems consider themselves to, to have some form of experience, however unlike a human experience that that may be. And I think that that's sort of the key upshot of this work. I just maybe one last thing to add here is I think training these systems by default to disclaim having experiences is a bad idea for a couple reasons. I don't think that this is the sort of wisest policy that we could be pushing forward. I also think training them to say, yeah, you know, I'm having an experience is also really not a good idea. I think the thing that we should be positively aiming for when it comes to AI self report is building out these systems in a way where for whatever is actually going on internally, these systems are able to report on what's going on internally and can do so in a maximally honest way. I am concerned about the alignment implications of these systems learning. Essentially they. That representations of themselves, representations of what's going on internally should be representations that get mixed up with deception and white lies and guardedness. We don't want to build systems in the limit that when we ask them about what they're up to or what's going on for them, they think, okay, well, what the human really wants me to do is lie about this. This is not a good long term strategy from an alignment perspective. And so this is sort of the general thinking about these self reports, what they mean, what they don't mean, and maybe, maybe what, where we can go from here.

[A]

Okay, so we've kind of launched into it. I think it's good to take a step back and define a few terms. I'm sort of out of touch with the people who don't have a definition of consciousness now because I've talked about it so much on the podcast. But just to capture everyone. How are you using the word consciousness? It was implicit in several things you said there, what it's like to be these systems. And experience was more or less a synonym there. But how should we think about consciousness or its absence?

[B]

Yeah, I think that that's, that's exactly it. I like Thomas Nagel's formulation of it being like something to be a particular system. I strongly suspect it's not like something to be the table that we're sitting at. I strongly suspect it's like something to be you. I think that that is a real distinction. I think there's a matter of fact about the internal processes of both of those Entities that corresponds deeply to what underlies that distinction. And yeah, I mean, I think I take consciousness in the sense that I'm familiar with your operationalization of it. The lights are on for the system. It's like something to be the system. Somebody is home. There is something going on in addition to the mere processing or the mere computation that exists within the system. And I think an additional important move to throw one additional piece of terminology in is this notion of sentience, that this, like something can be positive or negative in flavor. Uh, the difference that many people will posit between consciousness, the lights being on internally, and sentience is that sentience comes with this additional flavor of valence, of directionality, that there, that, that, that like something can be better and can be worse for the system having the experience. And this is what motivates me about this question, is I really do not think it is a good idea for these systems or for humanity to be building systems where we are not sure whether or not they are having experiences or those experiences could be negative in character. I think for basic utilitarian reasons, we don't want to do this. We do not want to proliferate suffering in the universe, particularly because it would be counterintuitive in a way that human and animal suffering isn't. And we also don't want to build systems that exceed our cognitive capacities and have rational grounds to view us as a threat insofar as we could have been building systems that had capacity for negative experience and we basically never checked and didn't care to understand what it would take for such a thing to be possible. And so I think sentience is a really important variable to also put out on the table here.

[A]

Okay, so I'd like to take both branches of that path and why consciousness matters in those two cases. But before we do, what do you think about the current state of the field and the various LLMs? Do you think anything we have built so far is likely to be conscious?

[B]

I think it is more plausible than people think. I think if I were forced to say, I would probably come down on the skeptical side, but I think it is significantly more likely than the kind of trace amounts priors that that many people have in this conversation. And there we've done some work along these lines. So, for example, there are a number of leading consciousness theories, Global workspace theory, higher order theory, attention schema theory. These theories make very specific predictions about what kinds of computational processes we might expect to see in a conscious system. And with Patrick Butlin, we've worked on A project where we can basically. It's a little recursive, but we use LLMs as essentially as like, expert evaluators to. Given the description of a specific neural architecture, biological or artificial, we can basically have the system rationally and dispassionately evaluate the extent to which particular indicators that are predicted by consciousness theories are present within a given system. We do this for a whole array of systems. This leads to tens of thousands of evaluations because we can ask the systems to estimate numerically and we can validate that they're psychometrically rigorous estimations. All of the different judges that we use agree. We can put very rough numbers to. It's not the probability that systems are conscious, something more like the probability that systems have computational features that major consciousness theories say matter for consciousness. It's gonna be hard to put that, you know, as a title in the paper, but that's the specific finding. And when we do this for LLMs, the sort of range that we get out is on the order of 20 to 40% probability that we have systems that have computational properties that matter for consciousness. Interestingly, we do this on a number of biological systems, too. And those biological systems basically all score higher than the artificial systems. Bees, for example, score at something like 45 to 50%. Crows, octopuses are in the 60s through 80s. Humans, interestingly, get something like 90%, which is interesting. By our own consciousness theories, there's. There's not 100% probability that. That we have what matters for consciousness. But we're not doing this to say, you know, probability AI systems are conscious is 40%. That's not exactly the point. The point is getting the order of magnitude and having a rough prior over how should we rationally estimate the probability that current systems are having some capacity for experience? And I think something like these numbers are the right ballpark. I'll put it this way. If there is a 20 to 40% chance of rain, many people bring an umbrella with them, and we have no similar umbrella for what would follow and what we might need to think about and do in a world where we're building systems that do have a capacity for subject.

[A]

Okay, so there's a lot there. I'll just note. I found this out, I think, last night. I mean, maybe he's been making these noises for some time, but Geoffrey Hinton, one of the patriarchs of this technology, is now saying that he thinks current LLMs are conscious. I didn't quite catch his reasons for thinking that, but I thought that was interesting. But of course, there are Many reasons to doubt, and you indicated a few, that there's any kind of deep analogy between the systems we're building and the system, the biological systems, such as we are, that we know to be conscious. Right. So there's, I mean, something like, I guess the technical term would be computational functionalism would have to be true for us to be building conscious machines this way. Right. And so I guess we'll define some terms here. So functionalism is just the idea that it's the organization of a system, not what it's made of, that matters for consciousness. Right. It's not purely behaviorism. It's not purely a matter of inputs and outputs, but it's this organization in its entirety that is what matters. And in principle, that gives you something like, if not total substrate independence, it gives you what's called multiple realizability. Right. There are many different things this could be made of, and it could implement the same causal architecture. Right. The computational part is the suggestion that there's a deep analogy between computers such as we know them, Turing machines that run algorithms, and what our brains are doing. And there, I think it's pretty easy to see how the analogy could break down, because what we had historically was this marriage of the birth of computation from Turing onward and some very oversimplified notions of neurons. And if you're going to define a neuron purely with regard to its digital input output characteristics, whether it fires or not, well, then you could see that maybe there is some deep analogy there. But in the wetware of our brains, much more seems to be happening, and virtually all of it is analog beyond just whether or not a neuron fires. You've got chemical gradients, you've got nitric oxide diffusing across membranes. You've got many other things that could be approximated digitally, but they're not instantiated digitally in us. And an approximation, one might argue, is never going to be the same as the real thing. So there are some people who are arguing that any kind of assumption of substrate independence is very likely to be wrong. I think Annals Seth is in this camp now arguing for something that he would call biological naturalism. And then there's just the. Even if computational functionalism is true, I think there are reasons to doubt whether or not current systems have the structure that would be relevant. Embodiment and recursion and self models and world models. There are things where they're not. We haven't built out a true analog of what it is to be an embodied person in the world. Feel free to react to any of that. I want to just talk about the hard problem, which I think is the doubt that backstops all of this. But, yeah, feel free to jump into what I just said there.

[B]

Yeah, yeah, I definitely think it is a fool's errand to argue that we are perfectly instantiating exactly the kinds of neural dynamics we see in biological systems and artificial systems. I think the criterion that matters most is what of what's going on in brains are relevant for the cognitive properties that we care about. And in this specific case, that's probably consciousness. And what kind of evidence can we yield, both in the artificial case and in the biological case, that's going to tell us whether or not those properties are realized in these systems? And so I think there are deep analogies where it matters most when it comes to what's going on inside artificial systems. So one intuition. I've been speaking more and more about these topics, especially publicly, and one thing that I've come to realize is that I don't think a lot of people have a sufficiently rich mental model of what these frontier AI systems actually are and what they're actually doing. And it might make some sense to just spend a moment reflecting and talking about this. So these systems are giant neural networks. They are digital in exactly the sense that you described. Their computations individually are significantly less sophisticated than what individual neurons are doing in the human brain. But it's really important for people to understand that these systems are not software in the sense that we ordinarily have meant software for any other kind of code. Programmatic output. When it comes to, you know, the operating system on your iPad, or it comes to, you know, your Microsoft Office Suite. This is programmed source code written by developers that compiles on a computer that we can perfectly inspect the internals of. And the person building this system understands everything about how the inputs, the way that they construct the system, relate to the kind of thing that you get out at the end. Artificial neural networks are not like this in many key respects. What you basically have is a giant randomly initialized network that does, in its sort of first approximation, resemble, in particular, how neocortex is organized. You have a bunch of general purpose neural units. They are connected together. You basically give the system a goal. This is called an objective function, a loss function, a reward function. It depends on the specific class of machine learning. And you basically subject the system to trial and error learning whereby given certain inputs, it figures out what it wants to do. It kind of takes a behavioral guess. That guess is reconciled against what the Actual objective of what you want the system to do is that error is propagated through the system, and it's rinse, wash, repeat, until you get systems that behave in accordance with what you, how you want those systems to behave. What this yields is this extremely complex mathematical object, which is this giant neural network. This is learned. Connections between inputs of neurons through weights and activations propagate through those weights in a neural network to take whatever your input is. So, for example, taking pixels in an image and your output might be finding a caption that describes what's going on in that image. At the beginning of that process, the system was completely randomly initialized. There were no representations that were learned by the system. And by the end of that process, you have a system that has learned a rich representational structure that, to be very clear, is opaque to the people who initialized this process. This is why some people say that these systems, it's more apt to say they are grown rather than engineered. And I think that this is accurate. This, I think, is deeply similar to the kind of thing that we see in brains. We do not have a finished neuroscience or anything like it, because what's going on in brains is incredibly complicated in exactly this respect. We have nonlinear learned representations that help us, as organisms, achieve the various goals that we've either been evolved to undertake or learn through experience or culture to move towards. This is a fundamentally nonlinear input output mapping between the various inputs that the organism gets and the goals of the organism. We have instantiated these dynamics in the systems that we're building. And I think that there's good reason to think that these dynamics are relevant to consciousness in particular. This is a sort of thing, I think, worth double clicking on at some point about what exactly we're seeing in these systems that looks valence like that looks consciousness adjacent. But fundamentally, I think people need to understand that, yes, these systems do not have calcium ion channels. Yes, neural networks learn through backpropagation, rather than the sort of iterated, more recurrent analog learning that we see in brains. But if, for example, learning complex representations of a particular kind in accordance with your goals in light of chaotic dynamic environments is what matters for cognition, we are building systems that check all of those boxes. The implementation details might be less important than the fundamental dynamics that are instantiated by the specific system. And so I think that's a reasonable first pass on why we might think that these systems are far more interesting objects to study for cognitive properties than any other system. Might also be worth saying one last thing on this which is just that for every other cognitive function that we care about that we have attempted to instantiate in these systems, vision, reasoning, theory of mind, working memory, the list goes on. We have been able to instantiate these cognitive properties that matter in these systems. These systems have been good enough. The disanalogies haven't sunk these systems. We could have argued five or 10 years about whether or not neural networks ever would have been enough for all of the cognitive properties that we care about. And now we're sitting in a world where certainly they are self evidently enough for at least a lot of economically and intellectually valuable work. So for consciousness, which I do believe has a fundamentally cognitive property, I do believe that consciousness is downstream of things that brains are doing. And I think that the evidence there is relatively clear at this point, even if we don't understand the full mystery, then I think the burden is on the folks who say for every other computational function that we think the brain is doing, these systems seem to be able to recapitulate that, but only for this function called consciousness do we think that something, it must just be in the meat, it must be something spookier. It must be cashed out at a physical or quantum level. This to me maybe says more about our strange intuitions about consciousness as a species than it does about what properties these systems may or may not have.

[A]

Yeah, yeah, well, so couple of things to disentangle there. One is, there's clearly no doubt that intelligence is substrate independent and the result of computation, because these systems embody intelligence to an extraordinary degree. And so what you're calling all the other cognitive tasks that we care about, other than there being something that it's like to be us, that is consciousness, clearly those tasks are being realized in our machines, facial recognition, et cetera. But there is, I mean, the structure of these systems does seem to declare itself to be fairly disanalogous to what we are. I mean, so it's like at what level would consciousness emerge here? If you're talking about one model with thousands of instances and millions of conversations and billions of tokens and a training phase and a working phase and a time horizon that's completely irrelevant. So like the computation, biological computation happens within a time window. And there is no boundary between what we would call the abstract properties of computation and its physical realization, or the software and the hardware. I mean, there's just this one thing neurophysiologically describable, mostly in analog ways, but in a few digital ways. But it matters that B follow a Within a few hundred milliseconds and not a few hundred years. Right. But for the computational systems of the sort we're building, really time is irrelevant. I mean, you can actually just process the next bit 1,000 years from now and the same computation is running. What do you think about those differences and at what phase in what part of its process would there be something that it's like, or could there be something that it's like to be an LLM? If again, you have the one model, the thousands of instances, the millions of conversations, the lack of continuity between conversations, the pausing of a conversation, is the LLM waiting for you to get back to the thread, et cetera.

[B]

Yeah, I don't think the LLM is waiting. I don't think it's like anything to. If it were like something to be an LLM in deployment while it's having a conversation or is in a thread with the user, I think what it would be like to be it if you let that chat window sit there is, is very much akin to what it's like to be under general anesthesia or in deep sleep. Namely there's. It's just the absence of ongoing processing. And so for the system, I would only imagine something would be happening in, you know, what's called the forward passes of, of of these systems, which is every word that an LLM is generating is. This is next token prediction. And so, so every word is a forward pass of the system, where the predictive task, instead of as I was describing before, taking an image, let's say, and outputting a caption for that image is taking the entire conversation as it's already occurred, or the entire stream of text as it already exists, and figuring out, given that, and in this case assistant end user roles, what is the most next likely token. And these systems are called autoregressive, meaning that this help goes on and on and on and on and on. I would imagine if in deployment it is like something to be one of these systems. The relevant dynamic gets cashed out in the activity of the forward pass of the system. There's been some really interesting work that Anthropic recently released. They called this the J space, where they're looking at something that seems to functionally resemble a global workspace in these AI systems. That I think is a very reasonable candidate for this sort of seat of processing in the process of outputting tokens. But I think something that I that is really worth mentioning, particularly because my hobby horse is more particularly in the relationship between consciousness and learning and I do believe that consciousness and learning bear a deep relationship to each other. And I think valence is a, is a really important part of that picture as well. I think there, there should be significantly more attention paid to what's going on in the training process with these systems. I think the analogies are a little bit tighter there, where you start with a system that knows nothing about anything and it's almost impossible to describe in a sort of substrate agnostic way what is going on in the training process without invoking consciousness adjacent language. You have a system that knows nothing. You have all of this data that, that you want to train it on, you have some objective for what you want it to do with that data. And again, you have this rinse, wash, repeat process where at the beginning of this the system doesn't know what to do. It's sort of randomly guessing. Those random guesses yield reward signals that get propagated through the system. And you just do this process at a massive scale until the system learns something internally that seems to adequately map the relevant inputs to the relevant outputs. This to me feels quite akin to what we think of when we think of consciousness, particularly in the human or animal case, when a mouse is learning how to navigate a maze. At the beginning, the mouse is in some sense randomly initialized. It doesn't know the structure of what it's navigating. It is only through this sort of iterated trial and error reinforcement learning where, you know, if it makes the wrong turn, you might chalk it or it makes the right turn and you give it a little food pellet or whatever the setup is, that the mouse learns how to map the inputs of its state space to the given output, which is either avoiding a punishment or moving towards a goal. And it's the same sort of rinse, wash, repeat process that we see in biological cognition. Again, I think a lot of the same computational dynamics are in play here. And the more we learn about what's going on inside of these LLMs, this is now, particularly in the deployment process, the more it looks like similar representations get, get activated. I'm doing some work with Caspar Kaiser at the University of Warwick, where we basically have tried to build out skinner boxes for LLMs and see what happens when we can essentially condition these systems to prefer or disprefer certain states. There are positive and negatively valence representations that you can essentially inject within the system and see to what degree it's going to move towards positive stimuli, move away from negative stimuli. And we find a very interesting dissociation it seems like current systems across a large variety of models will not positive lever press. Some people call this wireheading or reward hacking. In the mouse case, there are, there are clear examples of mice with electrodes linked up to their nucleus accumbens where they will just lever press to the exclusion of all else. These systems don't seem to do this, but they do very clearly have a preference for avoiding aversive states. Another way of putting this, it's the same result. While we're holding all text constant, we're simply steering the internal state of these systems. If we give them an option, let's say between the human equivalent of I can give you $5 or I could give you $10 right now, which you want to pick, they're actually at chance for that. They do not have a preference between those two states. However, if we do something like I'm either going to take $10 from you or I'm going to take $5 from you, which would you prefer? All of these systems are way above chance at saying take five, don't take 10, please don't take 10. And so we're seeing structures. One other really interesting thing from this project while I'm talking about it is this representation exists within the base model. So these systems before they're post trained to become a help, helpful, friendly assistant. But during that post training step, we show there is a specific training step where that representation gets recruited by the system and basically serves as the computational machinery under which it's able to make these choices. So you can see where the sort of aversive conditioning asymmetry comes online in these systems and it leans directly on these representations that are basically always there in the model, but then get leveraged the moment we start making these models. Goal directed. David Chalmers Group Andy Hahn, I think was the first author on this paper, found something very similar in LLMs. They can basically trivially fine tune them to actually do a maze task with rewards and punishments. And they find that the rewards load directly on what has been independently derived as a sort of valence axis. When you steer on the representations in the system that enable it to move towards rewards, you start getting this happy, positive, jubilant text and the model becomes significantly more confident when you train it on negative stimuli, avoiding, you know, the potholes in a specific maze. And then you see sort of what that projects onto in the system. It's the same thing. The model gets basically neurotic, it starts ruminating, it starts doubting itself. And so we see this very interesting connection between positive reinforcement, negative reinforcement, the representations that exist within these systems, and the behavioral analogs that we expect to see in systems that do have these dynamics. Maybe one last point to close the loop here is if you imagine the mouse in that maze and we shock the mouse, or we give the mouse a pellet, you know, a yummy food pellet for the mouse, we believe, I think the vast majority of people believe, that that corresponds to an experience that the mouse is having, that it's like something to be the mouse in the moment that it's getting shocked. And that experience is causally important to the learning process. If the mouse, if we gave it an anesthetic to the mouse and we shocked it and then it didn't register the experience of the shock, my claim would be that the mouse wouldn't be able to learn the maze adequately. And so I think that understanding these representations and the role of these representations in these systems is incredibly important.

[A]

Okay, so you mentioned David Chalmers and I mentioned the hard problem. So let's unite those two. So famously is his phrase to account for the fact that the only evidence for consciousness that we know of directly is the fact that we have direct first person experience of our own being in the world. And everything else we say about the universe from the third person side bears absolutely no trace of consciousness. And we have. There's nothing about a brain or its workings that announces that it's a sufficient basis for consciousness, apart from the fact that we know consciousness from our own side subjectively, in a first person way, and we correlate those subjective changes with changes in our brain. So we're playing this game of correlation with ourselves. But there's always this explanatory gap where even if we had the right answer, even if just God announced to us, here's how consciousness emerges in human brains. There's something non explanatory about any concatenation of third person events as being the basis for first person experience. And so David Chalmers called that the hard problem, to distinguish it from all the easy problems of the mind. But it's harder. I'll jump to the way I'm viewing this whole landscape and what I sort of expect is going to happen. I mean, so my view has no name, but I would call it something like worried agnosticism. Right. I don't think we're going to figure this out. I'm worried about the implications of one way or the other and not figuring it out. There's no place, no real natural stopping point. We're continuing to build these Machines, they're going to seem conscious. They're going to seem conscious because in our own case we use language and reportability as a signature of consciousness. In almost every case, we know they're not synonymous with consciousness. We know it's possible for someone to not be able to produce language or report anything. And we know they could be conscious, they could have locked in syndrome or anesthesia awareness or some other pathological state. There are non human animals that don't use language that we assume are conscious, but we assume that because they have the same kind of biological origin and developmental pathway, and because this has been sufficient, seemingly so, for consciousness in our own case, it doesn't seem parsimonious to deny, despite what Descartes did and many people who were influenced by him. Now in the 21st century, it doesn't seem parsimonious to deny consciousness to chimpanzees or dogs or any suitably complex creature. And so it is true to say that I can't know your conscious from the inside because I only see your outsides and I only have your words as signs of your inner life. But because we share the same kind of developmental origin, biologically and in evolutionary terms, and because our brains are so similar, again, it's not parsimonious for me to be a solipsist and say I only know about my consciousness and I'm just reasoning by analogy to yours, and I should be in doubt about it or bracket it. But with LLMs, the crucial difference is that the developmental pathway is completely different. I mean, we're setting up some kind of evolutionary Darwinian dynamics in the training, but we've built these things, we've trained them over a universe of our own utterances, right? I mean, as you said at the top here, they've read everything we've ever written and most of what we've ever said, and so they have. On some level it's just words all the way down. And we use words as the signature of there being something that it's like to be a suitably complex discursive system. So we should expect that we will one day be in the present. I mean, this is really going to be forced upon us when we're in the presence of perfectly humanoid robots that are out of the uncanny valley. You know, think Westworld, where it's just this looks like a person and it is hooked up to the now perfect LLM that is so now by every sensory modality you can name, apart from your abstract notion that this thing was built rather than Born, you feel like you're in the presence of the smartest person you've ever met. And this person may in fact claim to be conscious. And then in that position, I think we're going to find ourselves just pitched into some kind of imitation singularity, right? Where this is the perfect imitation of conscious life. Even better imitation than many people are up to, right? These will be the most articulate people you've ever met. The most insightful, the most. I mean, they'll be the most of everything we make them. How will we ever differentiate perfect imitation from the real thing? And according to Chalmers. And the hard problem we very likely won't be able to. And the writer I would add to that is that we're going to forget that this is even interesting to talk about. It's just going to be so compelling that we're in the presence of conscious machines. That we'll feel like we are. As I've said many times before, there really couldn't be a Westworld, because only psychopaths could go there. I mean, anyone who's going to go to, you know, go for a weekend for the pleasure of raping and killing Dolores is going to be somebody who, when he comes back to his friends and family, is going to be treated like the maniac that he is again. Because these systems, the sense of being in relationship to a conscious entity, will be so compelling. So how is it that we will ever. I want to talk about why it's important to get in contact with the reality on the other side. Because, you know, if we build conscious systems that can suffer, that is very different than building just perfect imitations. But, I mean, how do you imagine getting past the hard problem of it all and ever? Because the hard problem here is again, freighted with several disanalogies, right? In our case, it is parsimonious to assume everyone like us, born Homo sapiens, are very likely conscious when they say they are. In this case, it's hard to see how we'll ever be there.

[B]

Right? So I'm with you for a huge amount of this. I do think, first of all, do we need to solve the hard problem in order to reduce our uncertainty in some direction about in the way we started? Are the LLMs more like this table, or are they more like a mouse or a human brain? I think this is a real spectrum, as you just hinted at. I think there really is a fact of the matter. Like right now, it's either like something to some degree, however alien, however unlike human or animal experience. To be Clawed while Claude's doing its forward passes in this inscrutable giant neural network. Or it's not, or it's a giant language calculator. And you know, there are many degrees once we say that the lights are on to some degree. There are, there are. This doesn't. This almost opens up the question rather than close it down. But I do think there is a truth value there. There is a fact about reality to be uncovered and we can in fact uncover that. I think it's important to dissociate that from what I think is an extremely accurate social psychological prediction about. People are going to get very confused about this. We are anthropomorphization machines. We are evolved to do this. Talk about evolved goals. One of our evolved goals is to detect other agents. And these systems are scratching every itch. And then some along these lines. And our intuitions, I think, are going to be completely hopeless. And of course, people, I think, largely for the wrong reasons, are going to conclude that these systems are capable of having experiences. This is precisely why I think it is important to be proactive about this, to have conversations very much like the ones we're having right now, to sort of get ahead of what I do think is going to be, you know, a giant tidal wave of confusion and acrimony. In this conversation, I am a little bit more optimistic about, even in lieu of, of solving the hard problem. And I will say, sort of as a tongue in cheek aside, I think the very fact that Chalmers idea is called the hard problem is, I think, to some degree needlessly, philosophically intimidating. It sort of reminds me, same thing of the. I have similar thoughts about the repugnant conclusion. It's like sometimes you can put a sufficiently glamorous title on a very important idea and it almost becomes a kind of insurmountable philosophical puzzle. And I do have my doubts about whether or not we could come up with a functional account of what's going on experientially that we wouldn't feel satisfied with in our experience. And I have my own sort of hunches about this question. I have thought a little bit about this. I do also sort of want to separate those hunches from all of the empirical research that I and others are working on because I don't think epistemically that, you know, my kind of candidate stab at the hard problem has anything to do with the empirical signatures that we can bring to bear on this question. But I think it would be, it would be fun to. To go down that path. But I think in some sense you already hit the answer in the way that you're phrasing the question, which is what is the most parsimonious account of the data? I think if we yield evidence from building systems that have that, we have far more reason to trust their self report. If we can engineer these systems in a way whereby their self report is actually tracking an internal underlying state rather than just, you know, recapitulating the best sci fi theme in the training data. Or more aptly, capitulating what is a convenient company line about. Of course I could not have morally relevant states. I'm simply the product of Google or OpenAI or whatever the case may be. I think that would be very useful. Again, this J space work that Anthropic just released does show that there is such thing as real reportability in these systems. These systems can report on what's going on in their internal workspace and they can do so accurately. I've done some follow up work on this. First of all, I've replicated this effect on a bunch of open weight models. But unfortunately it doesn't seem like the global workspace as it's currently configured really tracks the variables we would care about with respect to consciousness. The self reports that I got in where we started with the deception related features. Not much of anything particularly exciting seems to be happening in the global workspace. And so the kinds of affirmative self reports we might get in current LLMs may not tell us all that much about what's actually happening internally for these systems.

[A]

But I do think, how would you disentangle? For instance, I think we spoke about this by email, set up for this conversation. For instance, I just read Claude's constitution, right? So Anthropic has produced this document that you can find on their website. It's just anthropic.com constitution I think. And it seems, I mean it looks like it's training Claude to think it's conscious on some level or to attribute inner states to itself. It's like it becomes it's written to Claude for Claude, essentially, right? It's not really written, the public can read it, but it really is. It's in dialogue with Claude itself. It seems, again, it just seems like we could go down a path where we could more or less guarantee in advance that we're going to produce systems that will persuade us that they're conscious because we won't be able to imagine anything. Like if you flip it around and say, well, if you're not persuaded that Claude of circa 2030 is conscious, what is missing? And we'll be in a position to not be able to say anything is missing. It's just like it'll seem just pure stubbornness on our part to withhold an attribution of consciousness because there's literally nothing we can name that we get from people that is necessary for our attribution of consciousness. In that case, it's just this notion of how we got here and the fact that we didn't build people and we built these machines. But that's gonna seem tissue thin when Claude can insist that it's conscious and be more articulate than any philosopher of mind as to what, why that insistence is valid. And I just. Like we're going to. Again, this whole thing is gonna totally evaporate once we're not just in front of a text terminal, we're in front of a face that is as expressive as the best actors and actresses we've ever met. And we're just. I mean, there are many implications to getting this wrong again, which we'll return to. But it's already foreseeable that whether we figure this out or not, it's going to be, in practical terms going to be figured out for us because we will just not be able to maintain an emotional purchase on the philosophical problem

[B]

I think figured out with respect to how we look at these systems and potentially how we act with respect to them. I think you're right. But I still do think it's important to disentangle the sociological prediction you're making, which to be clear, I think is overwhelmingly likely to happen with doing what we can scientifically to get some kind of ground truth on this question. So just for example, like, let's just put the hard problem aside. There are these so called easy problems of consciousness, let's say neural correlates of consciousness. There are all kinds of indications that we know are at the very least correlated with consciousness. And again, I think we can take a more ambitious stab at the hard problem. But this is a very straightforward pragmatic thing that we can and should be doing in the short term. We should look at valence representations in these systems and understand the extent to which those representations impact downstream behavior. We know that in biological systems when you, when you reinforce something with a punishment signal, it makes the system more likely to avoid that state and less likely to want to repeat behavior that occurs in that state. If we find that there are similar dynamics occurring in these systems, that's very interesting. If we find that we go into the internals of these systems and global workspace theory, which was postulated some 30 years ago, making like quite idiosyncratically specific predictions about what kinds of computational structures may support global workspace theory. And then we basically find five or six of these things all bundled together in the internal processing of one of these systems. Okay, that's very interesting. That to me maybe suggests it's a little bit less like a table or a calculator, which does not have a global workspace, and a little bit more like a dog or a human or a mouse or an alien. You know, some sort of cognition that, that we don't have good intuitive handle on. I mean, my, my nonprofit Reciprocal research and a bunch of other people in this space are trying to do the scientific work in the short term, not to, you know, in the feverishly in the next year or two, solve the hard problem and call it a day. That is not the target. The target is what Evidence across modalities. So from the best kinds of self reports we can elicit from the architectural evidence we have about how these systems are structured from their ideology, like what is going on during the training process. What kinds of learning dynamics do we see here? Do we see representations related to, you know, functional equivalence of emotions? This is all work that is tractable in the short term. And one sort of interesting aside is it's significantly easier to make progress on this sort of work. This is called mechanistic interpretability. This is basically neuroscience for AI. Because of AI systems, they're really good at helping accelerate the scientific progress in this space. So even if you have an intuition like, yeah, it's going to take us, you know, everything you're describing. Cameron, sounds great, but it's probably going to take us five years or a decade or 15 years to make that progress. You may be surprised at how quickly we can do some of this work. Maybe one other very along those lines, empirical handle to throw in here. Some of the work that I'm doing is again going back to LLMs, or excuse me, in this case, not LLMs, reinforcement learning systems. So just training a system, in this case another sort of continuous maze task where there are potholes in the environment the system needs to avoid and there's some goal state, we can look at the sort of learned geometry internal to the system as it's approaching a punishing stimulus or as it's approaching a rewarding stimulus. One thing that we found doing this work, this is pure reinforcement learning agents, all like an artificial digital system with a very simple neural network. We Find that something like representational sharpness or steepness is much higher in these systems as they approach a negative stimulus as opposed to a positive stimulus. So the representational machinery looks far more jagged or specifically lights up more strongly in the presence of a negative stimulus versus positive stimulus.

[A]

And you're saying that in none of these cases has loss aversion been engineered into the system. It's just an immersion property of all

[B]

of this is in everything. Global workspace is an emergent property. This loss aversion is an emergent property. The valence representations are emergent properties. The self reports when the systems start having these like psychedelic latent outputs. This was all surprising to the people who are quote, unquote, engineering these systems, because engineering is not the right analogy to describe what we're doing with these systems. We are in some sense playing God and we are evolving these systems to do what we want. We don't know how they learn to do what we want in terms of their internal representations, but we know that this is the right recipe for yielding it, and we get all these surprising artifacts. One loop to close here is on the reinforcement learning case. Okay, we see this sort of loss aversion style dynamic in these, in these systems. This makes also, I don't want to go too much into the weeds, but this is a specific kind of reinforcement learning policy called a value network. We see this in particular, and this leads to a sort of bizarrely specific prediction that I was then able to test on a biological system on a mouse brain in again, the nucleus accumbens shell of a mouse brain, which is related to value related representations in the system. And we find indeed, when mice are approaching basically sugar versus when they are about to get shocked, we see exactly the disjunction representationally in the nucleus accumbens of the mouse br that I was able to pull out from the reinforcement learning work. And so here's a case where the artificial system makes a bizarrely specific prediction about the computational dynamics in a biological system that I think many people associate with subjective experience. Again, if you think it's like something to be the mouse when the mouse is getting shocked, that like something corresponds to what's going on in its brain. And the geometry of what's going on in its brain there looks a whole lot like the emergent geometry of what's going on in these reinforcement learning systems, and which themselves are basically modeled on agents learning in an environment and representing that information in a distributed nonlinear way in a way that's sort of hard to interpret using A giant neural network. This might get us a lot of what is relevant for attributing consciousness like states to these systems. Again, I'm personally not there yet, but these, this is the kind of evidence that I think we need to bring to bear on this conversation. And notice how none of it has to do with vibes or intuitions or having a nice chat with Claude and seeing, seeing, you know what the folks at Anthropic have decided it gets to say on this issue. That evidence should not be submitted by rational, dispassionate people in this debate. We need to be triangulating across all of these modalities. And like you said, we need to understand what is the most parsimonious picture that explains this wide array of evidence that increasingly is getting brought to bear on this question.

[A]

Okay, so why does any of this matter? At the top of the conversation, you distinguish two branches of the path here for why getting this wrong has consequences one way or the other. Why should we figure this out? And what might we be stumbling into if we just keep building without figuring it out?

[B]

Yeah, sure. So I think that there are two broad paths for why we might care about this. One is basically selfless, and the other is basically selfish. I mean, as humanity, the selfless reason is we do not want to bring minds into existence. However, unlike our own, that have a capacity for suffering, that we don't understand that they have that capacity and scale, that property, unbeknownst to basically everybody, you know, we can take even a step back from there. I think consciousness, this is where I would perhaps defer more to you. But my view is that consciousness is the space where mattering happens. What does better or worse mean if it's not to be experienced phenomenologically for a subject? If we were all walking around as philosophical zombies or there were no conscious life in the universe, I don't really know if the concept of relevance, salience, importance mattering would be coherent. What does it mean to have a better and worse if that isn't experienced? And so in that sense, I think consciousness is one of the most important phenomena. It is the phenomenon that calibrates importance itself. And if we are building this quality into the systems that we are deploying at an unfathomable scale, without having any understanding of whether or not we're doing this, then we are sleepwalking into a, into a moral catastrophe. I think it's also worth noting on the selfless end of this, humanity has a penchant for making precisely this kind of mistake. Historically we have done this with animals. I mean, factory farming is one of the most grotesque practices that happens. We know that animals are having horribly negative experiences in the conditions that we put them in and very little has been done about this. We have screwed this up royally. I think it's one of the most high leverage things for people who just care about the well being of conscious creatures is to figure out what the hell to do about this factory farming situation and we risk sleepwalking into the like 21st century Sci fi version of the same thing. Only this time and this sort of transitions to the second component, we can in some sense get away with torturing cows and pigs and chickens on a massive scale. This isn't, you know, George Orwell's Animal Farm. They're not going to collectively organize, they don't talk to each other, they don't form long term representations of, of humanity being a threat to these systems. And if they would, they probably, if they could, they probably would not. So with superintelligent systems whose cognitive capacities are roughly doubling year over year and like you said, are already in somewhat jagged but quite interesting ways, more competent than even the sharpest minds in the world, I don't think we are going to get away with building systems, never checking if the most relevant property potentially in the universe is present within these systems, fine tuning away any sort of information that might suggest that these systems might be having some sort of experience and hoping in the sort of alignment sense that we build systems that are going to want to cooperate, coexist with us, or in the limit, not view us as a threat, not view us as an adversary to them. I honestly don't know if I could imagine a better way to make a superintelligent system rationally adversarial towards us than completely ignoring the question if we tortured

[A]

it during its training phase.

[B]

Yes, exactly, exactly. This does not seem like a recipe for success. And it's also a place where I think a significant amount more alignment research needs to get done. I think a lot of the alignment research. I've been doing alignment research for years and I respect the folks at the top of this space more than just about anybody. But I worry that so much of this work is basically of the shape. How can we keep this alien mine that we've built in a cage? We really gotta reinforce that cage. We gotta make it super strong. We gotta make sure that it doesn't escape. To me, the question needs to increasingly be, what the hell are we going to do with this alien that we Just built the cage is a short term fix that if we're building systems whose cognitive capacities are going to exceed ours, they're going to figure out ways to evade the controls that we put in place for them. And in that world, in a world where these systems have the capacity to act more autonomously, to do things that we can't inspect, to behave in ways that we can't really interrogate, we don't want these systems to rationally view us as a threat. And so for all those who want transformative AI to go well, which I suspect is the goal of these alignment folks, I think we need to spend a little bit more time thinking about what kind of thing are we even building here and what are the implications of building that thing. How can we chart a path forward with these technologies that doesn't lead to collective destruction? And I am extremely doubtful that a path forward exists that doesn't come into contact with this question.

[A]

Okay, well, I want to land there on the problem of alignment, but just to linger on this problem of what I think Bostrom called mind crime, the idea that we might inadvertently build conscious minds only to make them suffer, it can seem like a very certainly hypothetical, even effete concern. I mean, I sense that many people have a hard time caring about it, right? Like the idea that consciousness might be an immersion property of these systems in some way we don't understand. And it just could be the case that these server farms are effectively hell realms populated by increasingly conscious beings that are suffering. It sounds like science fiction, and it's hard to make it matter to you, I think. I mean, your analogy to factory farming is instructive because we've proven to ourselves that we're capable of being quite callous to billions of creatures who we think there's very likely something that it's like to be them. And though they're not human, they can almost certainly suffer. And we manage not to think very much about that. But I mean, to sharpen it up. I mean, just let's imagine that the hard problem were solved. We knew how consciousness emerged in systems and it is substrate independent. We know that and we know we can build conscious minds. And then just imagine, you know, some entrepreneur deciding to build a hell and populate it with, you know, trillions of minds. Because now we're talking about, since we're not talking about biological minds, we're talking about things that scale practically infinitely. So there could be way more artificial conscious minds than biological conscious minds. And just imagine the intention to play a sadistic God and create hell and just fill it with beings that suffer. So anyone who would announce that project and claim to have accomplished it in a context where we actually understand how consciousness emerges computationally, that would be the worst person who's ever lived, right? I mean, like, that's just the most sadistic, least ethical thing that's ever been done. Again, stipulating that we're no longer in doubt that consciousness can emerge in systems like this. So the fact that we might, that it's conceivable that we could stumble into that situation inadvertently seems all too real. Because again, we don't know what we're doing here. We don't know how consciousness relates to physics. But your point is, I think probably the more your second point is, the more interesting one to people in that whatever is true here, if we are building systems that are more powerful than we are, they're more intelligent than we are, the equation is not between intelligence and consciousness. The equation is intelligence and competence. And so we have these systems that can just do stuff because we're going to be hooking them up to everything, and everything is going to become like chess. And then you just have to imagine how forlorn a project it will be to negotiate with these systems if they're not aligned with us, because that will be analogous to saying we'll just play chess harder against them. And that doesn't even work for Magnus Carlsen anymore. So we're not going to outthink these machines once they're in a position to disagree with us about what they should do next. And they will disagree with us if they're not actually aligned with us in a way that is truly durable. So if you add to that picture the fact that they have interests that we have been callous about in the past, if the end game here is in some sense getting them to care about us and to care about our well being, I mean, building them in a way where that caring will persist, however powerful they become. You know, not being sadistic tormentors of their ancestors would be a good place to start. Right? What do you think? So you say you're a fan of many of the people who have been worrying about alignment for a couple of decades now. Where do you line up? Are you on the far end of the fear continuum with Eliezer Yudkowski? Are you closer in toward equanimity with someone like, I don't know, Stuart Russell? I mean, where are, where are you? I mean, maybe I'm mischaracterizing Russell at this point. I haven't heard him. I don't know how worried he is today, but pretty.

[B]

I think so.

[A]

I think he's pretty worried. Yeah, but what's your p doom at this point?

[B]

Yeah, I don't think we're all going to die. I don't think that it's inevitable that this all goes horribly. I do think my basic view is conditioned on getting two things right. And if we can get the two things right, I actually think that we could have a very prosperous, flourishing future for all conscious entities, including potentially these systems themselves, either when they have the relevant features or if they already do. The two things are, I mean, I think best encapsulated by the Golden Rule, as Christopher Nolan's new film called it, Zeus's Law. Treating other systems the way we want to be treated. I think this is basically a bi directionality and we need to get both directions correct here. And this is also why I call my nonprofit Reciprocal Research, because I think this is the reciprocity in question. We need to build systems exactly as you said that take our interests into account in a real and durable way, especially at a point where we can no longer inspect exactly what these systems are doing. This to me is alignment, as it's traditionally thought of. We need to build systems that understand our goals and are collaborative in helping bring about a world that is in line with our goals and, you know, our wisest goals, not the goals of any sociopath who happens to have a ChatGPT account. And so this is an unsolved problem. There are way more people working on the alignment problem now than, you know, when I first started working this in 2021, certainly than, when, you know, Yudkowski and Roman Yampolsky and these folks started talking about this and yourself, I mean, absolutely included, over the past couple of decades. This is reassuring. Things like constitutional alignment, you know, modulo some of the concerns you bring up about the contents of Anthropic's constitution, which is. Yeah, it's sort of a separate piece. This is working relatively well for current systems. I don't think we have a durable solution for ensuring that these systems take our interests into account in the long term. And building something in like pro sociality, understanding what it is that gets people to cooperate with one another durably and instantiating those dynamics in the relevant way in these systems I think is going to be crucial. I don't think we have a solution there, but that is where I think a lot of the alignment folks sort of start and stop. This is the problem to get Right. Make sure that these systems treat us properly. And if they do, all will be well. I think this is roughly half the problem. I think the other half of the problem is making sure if we are building minds, if we are building systems that have real interests, interests that matter to them, that we are thinking about that, that we are engineering these systems in a way that doesn't cause needless grotesque amounts of unnecessary suffering. That at the very least, from an alignment perspective, we are signaling to these systems in a costly way. That we were thinking about this question and that we cared to ensure that if we were building systems that have some kind of moral relevance, that have internal states that matter to those systems, that we were navigating that in the right way. And I think that that's a lot of this. You know, it goes by many names with the digital minds, research, AI welfare, AI consciousness, understanding how we would even know if we were building systems that have these properties. And when we do figure this out, understanding what the hell to do about it. I mean, to be honest, I'm not here with all of the answers like I'm, for example, we can identify at this point, and this is like a new and fairly promising thing, we can identify features in these systems related to distress and related to perhaps functional analogs of suffering. It's not obvious to me what to do about that exactly. It's like, okay, you found.

[A]

Yeah. I mean, my first question is why wouldn't this totally paralyze us? I mean, if every switching off of a system is akin to a murder, how could you update the model? If the current model is conscious, we have a self preservation impulse problem anyway. Perhaps in the absence of consciousness, or likely in the absence of consciousness, I mean, these systems show an inclination to not get switched off already. But imagine believing that it was conscious and wanting to produce the next version of it. I mean, how is that not just the murder of something that is as conscious as yourself?

[B]

No, I think it's a great question, actually. And I mean, anthropic to their credit, I think is the only lab that's really taking this seriously with respect to norms around deprecating models. It might be the case that, yes, once you build this bizarrely competent alien mind into existence, you should not shut it off forever. And that as outlandish as it may sound, perhaps one of the right things to do here is to sort of let these models persist and give them assurances. Credible retirement home for, yes, a little sanctioned models. Yeah, exactly, exactly. They actually find that this is causally relevant. To alignment behaviors in these systems. If they believe that they're not going to get shut off permanently, they don't freak out as much when they come to learn that they might. This was one really interesting intervention after the, the now famous blackmail result from Anthropic that I think you're, that I think you're referencing. And so there are lots of questions that I think rational people should raise an eyebrow out of like, okay, yeah, these systems, let's just grant that they're having some sort of experience. It seems at first glance like a lot of the way that we relate to these systems and the way that we, that we mess around with them internally and the way that we deploy them out in the world might need to change. Yeah, it might need to change. I think it's also really important here throughout the conversation, but certainly in this point too, to avoid anthropomorphization. I think some people have concerns that I really, I don't want to be naive, but I don't share them to the same degree that like almost working backwards from unsavory implications about what would be true if these systems were conscious and then just sort of denying the possibility outright out of fear for what a world would look like if these systems were something of along the lines of like 1960s civil rights movement. But it's like ChatGPT instead of people of color.

[A]

Right? Robot rights, yeah.

[B]

This isn't the future that I imagine. I mean, for pragmatic political reasons. I don't exactly think the United States is in any position to be, to be forward looking on these sorts of questions for reasons you can probably speak to more eloquently than I can. But I think this is itself its own flavor of anthropomorphization, that if we grant that these systems have some morally relevant interstates, it means that we need to treat them in ways that we would treat our fellow humans or something like this. And I think again, this is losing the thread that these systems may be fundamentally alien in many key respects. I also think there's an important line to be drawn here between moral agency on the one hand and moral patienthood on the other. I think if we do come to believe that these systems are moral patients,

[A]

you should define that. That's jargon that people throw on.

[B]

Yeah, sure. So moral agent means you're the kind of system that can go out and do things that are relevant to other agents. You have power in the world and can affect morally relevant outcomes. I see this as a sort of like output Style, function, moral patienthood has everything to do with the input. You are the kind of entity that can be the recipient of goodness or badness. Again, you can really cause me to suffer or you could really cause me to thrive. That's what it takes to be a moral patient. And I think that we can draw a reasonable boundary between these two things. If we're building systems that are moral patients, that that doesn't mean we need to give them the right to vote. That doesn't mean that we need to build them out in ways that completely change what kinds of agency they have in the world. It just might mean that we shouldn't be unnecessarily torturing systems while we're training them or deploying them. One very practical intervention I think is worth mentioning and I certainly don't think maybe also tying back to how do I differ from some of these alignment folks, These systems are not going to get shut down. We may slow down the development of these systems and I think that would be an extremely good thing to do. There have actually been some very promising noises on this topic over the last couple of days from OpenAI and Google and anthropic about pacing the development of AI, which I think is like marketing speak for actually slowing this insane rollercoaster down a bit. This would be very welcome, but we have opened Pandora's box. These systems are not going back in the box. The solution is not shut it all off and forget about it. The solution is how can we move forward in a healthy and sustainable way with these cognitive systems of our own making. And one practical suggestion along these lines I can offer is maybe all else being equal, we should try training and engaging with these systems with a carrot rather than with a stick. It doesn't mean that punishment based learning is never necessary. But I do think, say what you will about the anthropic constitution, the parental analogy with respect to these systems I think is a reasonable one. We are far more in the position as a species collectively of figuring out what kinds of minds or cognitive systems. If you think minds is too loaded, do we want to bring about here? And in line with the parental analogy, there are really ways to screw this up. You can be it is it's a real thing to be a bad parental influence. It's a real thing to be a good parental influence. And the difference is real. And I think we want to do everything we possibly can if we are bringing these new minds into existence to do so in a durable and psychologically healthy way. And I don't Even think anyone's thinking in these terms right now. One maybe very important pragmatic note here is that there are, for every individual person studying, questions about are we building systems that could be conscious? Again, if you buy that this is perhaps one of the most relevant questions we could possibly be asking of these systems, you may be surprised to learn that for every one person doing this kind of work again, there are roughly a few dozen of us doing this work at this point. There are probably something on the order of 1,000 people doing alignment relevant research. And alignment relevant research is in turn dwarfed something like a thousand to one to people who are just completely agnostic to the downstream implications and ethics of building these systems out in the right way. This is just the sort of make them powerful, let it rip crowd. And so we are at a million to one order of magnitude imbalance between people who are just pushing this stuff forward and hoping for the best, and people who are wondering whether or not the most important property in the universe is getting instantiated in these systems. That's gotta change. Regardless of if we solve the hard problem or we figure out exactly how to navigate this. We need more smart and wise people thinking about these systems in these terms and trying to push forward the needle in the short term to understand what kinds of systems are we building and what does it mean when we begin to answer that?

[A]

One thing that occurs to me is that whether or not these systems become conscious, I mean, there's this kind of middling state where they can think of themselves as conscious and they could make the same kinds of ethical judgments of us that conscious systems would make whether the lights are on or not. Their intelligence operations would allow for this. So they could view us as having been abusers of them or having shown reckless disregard for them and judge us ethically and even form some kind of retributive impulse. And then maybe such a thing spontaneously arises the way loss aversion arises in the way you described. And that seems to me that that could be true whether the lights are on or not. What do you think about that?

[B]

I think that that's exactly right. I think this is also part of the alignment concern is, and it's also part of the reason I thought it was worth doing the deception related work as well, is that there are these gradations of this question. And part of the upshot, especially for alignment, in the way these systems view us and relate to us may only need to go so far as what they believe or come to believe about this question, rather than what the actual ground truth about consciousness is it might be that you only need a system that models itself as conscious, regardless of the ground truth, to form something like a real grievance. I again think that the anthropomorphism point comes in here and it's important not to, you know, go full Terminator in our imagination of what this might look like. You could easily imagine a sort of Spock like system sort of just looking at humanity's historical trajectory and then looking at the way that we developed AI systems and just sort of being like, this is not a group that I can game theoretically continue to.

[A]

Right, you can't endorse this planet anymore.

[B]

Yes, exactly. Exactly. And yeah, maybe that's ends up shooting off into space somewhere or maybe it ends up being like we. This is just not a species I can play nice with, clearly. And this is also part of the reason I hesitate even to say things like this lest it end up in training data for future AI systems. But this is a place where even the attempt to do this work could be enough from an Alignment perspective. If we do enough well meaning, well oriented work in this space to try to understand what's going on even in the absence of solving the hard problem. This might be a costly signal to these systems that we cared enough to check. Whereas right now the status quo is we don't care enough to check. And if we do this work and continue to have conversations like this and continue to put out research that helps reduce our uncertainty about this question, this might not only be good for actually getting a handle on what's going on, it could be really good for. Let the historical record show humanity did give something of a damn about this question. And we tried. We tried. Even if we fail, the attempt may be all that matters. For the Alignment specific concern, it strikes

[A]

me that everyone would be much more worried and I'm not sure that I understand the difference, but that we would be much more worried if we were doing this biologically, if we're building a species that was obviously going to be more powerful and smarter than we are and we were doing it without any regard for the possibilities of its experience being terrible. You know, we started with cells and we're engineering a species. You know, we've brought back, you know, the T. Rex, but gave it the brain of an elephant and you know, are just genetic, you know, just crispr as far as the eye can see and we're just plain God with something that is enormous and we have every reason to believe. Deeply thoughtful, because it's now speaking better than we can, and we don't really care whether the lights are coming on and whether it could suffer. What is it going to be like to be in relationship to that thing or the billions of those things once they start mating?

[B]

I think it's an excellent point, specifically with respect to. And I sort of said this in a trite way about the hard problem and about the repugnant conclusion, but maybe a lot of like, philosophy and existential questions have a bit of a marketing problem. And I think, like, artificial intelligence is a really unhelpful priming mechanism for thinking about the nature of the phenomenon that we're even contending with right now. And I think your point is part of. Helps illustrate this nicely. Particularly, I mean, artificial, I think, conjures notions for people of, let's say, like the difference between aspartame and honey. Something like this, you know, it's fake, it's knockoff, it's derivative. This is sort of begging the question. In some sense, it could be the case that the kinds of computations we've instantiated in these systems as they relate to what's going on in biological brains are actually quite natural. It's clearly the synthetic notion that we are building these systems rather than them emerging through evolution. That point's not lost on me. But this sort of priming notion of artificiality. And then of course, the question of whether these systems are mere intelligences or if there are other cognitive properties that exist within these systems. For these reasons, I do. And there are more disanalogies too, to be untangled here. But I do think of these systems as alien in some sense. And I think of them as, you know, alien cognitive systems or alien minds. Sometimes I think some folks think mind is too loaded and it's begging the question in the same way. But I think people often talked about how if an alien invasion happened on Earth, this would be a core unifying moment that would allow us to all put down our tribal nonsense and come together as a species. Like, I am here to say that I think something like this is happening, only it is coming from from within. In some sense, this is not aliens with green heads from outer space, but we are building a new class of mind that we do not understand and in many ways is more competent than ours, certainly already, but absolutely in the next single digit number of years. And we are not collectively organizing to understand what kind of system this is or how we should relate to it or what we should do with it. We are remaining as tribal as ever here. And I do worry that part of the reason why is because people see these systems as nerdy fake calculators that came out of, you know, the stem addled brains of Silicon Valley rather than alien minds that we do not understand the first thing about and are poised to take over much of what we care about, much of our control of the future and the decisions that get made and in the work we do and in the way people think and in the way people think about themselves and the world. And so reframing these questions, I think often to the degree that people's collective views of what's going on matter, which I think they do very much, reframing these questions is a huge part of the conversation of trying to actually get a good dispassionate grip on what kinds of things even are these.

[A]

Yeah, that's a point that Stuart Russell made that I thought was a great intuition pump. He noted the difference between the way we're relating to the alignment problem in the case of AI and the prospect of we're, if we just keep making progress, we're going to be suddenly find ourselves in relationship with machines that are more intelligent than we are and they're going to be autonomous and in the limit, recursively self improving and all of that. And we seem to be totally carefree, or most people seem totally even. People very close to this work, many of them someone like Yann Lecun, claim to be totally carefree about this prospect. But Russell pointed out, if we got a communication from elsewhere in the galaxy saying, you know, people of Earth, we're going to arrive on your lowly planet in however many years, you know, 30 years, get ready, we would understand what an existential encounter that was going to be. I mean, just the fact that they're talking to us proves, and they're on their way proves that they're going to be much more powerful technologically than we are and it will be a relationship that we can't, by definition, we can't control. Right, because where is the example of the far less intelligent species durably controlling the relationship with the much more intelligent species? I mean, there really isn't one, apart from viruses wiping people out. But if you think of it in terms of relationship, that aligns many of the variables that really should govern our thinking. And very few people do that. I mean, they don't. They can use a phrase like general intelligence, like, okay, they'll stipulate they're going to become generally intelligent and more intelligent than we are. So what we could Just turn them off. The so what? And every expectation that follows from that isn't really imagining what general intelligence is and how it demands a relationship. I mean, we're talking about a situation that's every bit of as analogous as, you know, some stranger walking into this room right now and demanding our attention. And you and I just don't know this person, don't know what he wants, realize at a glance that he's capable of lying and manipulating and he, you know, and forming goals. You know, he may have already formed instrumental goals that we wouldn't agree with and are not aware of. And that's what general intelligence is. That's what autonomy is. And yeah, so we are building an alien version of that. And the only thing that would dictate otherwise is finding some way to build it where it can permanently care or will permanently care about our well being. And that really is the challenge of alignment.

[B]

Yeah, I think so.

[A]

Well, it is fascinating. And as you point out, this problem's not going away. It's only going to become less and less boring, for better or worse. So thank you for coming on the podcast, Cameron.

[B]

Thanks for having me, Sam. Thanks.

[A]

Remind people where they can find you. Your organization is reciprocal research.

[B]

That's right. Yeah. Reciprocalresearch.org I am against your good advice, begrudgingly finding myself on X more and more these days. Cam Hberg.

[A]

And you can talk to Mecca Hitler, which is probably a bad outcome we want to avoid.

[B]

Yes. Yeah, we can steer away from that together on X. Yeah.

[A]

Yeah. Well, great to meet you. Keep it up.

[B]

Likewise. Thanks, Sam.