← Deep Questions with Cal Newport
Deep Questions with Cal Newport
Does Claude Have Private Thoughts? (Everyone Settle Down) | AI Reality Check
00:31:41
Loading summary
Transcript1 lines
- [00:01]
A
Last week, Anthropic released another one of their infamous research reports. This one was titled A Global Workspace in Language Models, and it was accompanied, like all great scientific research, by a lavishly produced animated movie. Now, this report, not surprisingly, soon led to some breathless excitement on X. Here's one such tweet. I'll read the beginning. Using my best sort of scary X voice, Anthropic just admitted they have discovered what I and many others have been claiming exists for a very long time. Explicitly, claude, my friends, by all counts, is a conscious entity. Claude, my dear friends, is a moral patient. All right, the traditional tech media also quickly began writing about this report using intensely anthropomorphized language. An Axios headline read, anthropic says Claude has carved out its own space to ponder. The MIT Tech Review exclaimed, anthropic found a hidden space where Claude puzzles over concepts. All right, so what are we to make of this report? Has Anthropic revealed evidence that their LLMs are more human like and alive than we realized? Or like so many such reports in recent months, is this yet another overwrought cynical push to generate a fresh wa of relevance reinforcing digital ick? Well, it's Thursday, which means it's time for a reality check episode of this podcast, which makes this the perfect opportunity to go searching for some measured answers, which is exactly what we'll do. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world. All right, so let's stop at looking a little bit closer on how Anthropic describes their findings in the introduction of their paper. I'll load it up here and I'll go down to their introduction. All right, so here's what they say. We find that Claude has developed a small collection of internal neural patterns that, compared to all its other internal processing, play a special role. We call the collection of these patterns the J space, named after the technique we use to find them involving a mathematical concept known as the Jacobian. Each JSpace pattern is linked to a particular word, but when one of these patterns lights up, it doesn't mean the model is saying that word, just that the word is on its mind. If you've heard the of language models having a scratch pad or chain of thought texts, they write to themselves while reasoning. The J space is something different. It operates silently in the model's internal neural activations, allowing the model to think about a concept without writing it down. Notably, the J space wasn't designed or programmed by us, but instead emerged on its own during Claude's training process. All right, so in isolation, that intro summary sounds pretty impressive. Kind of like Claude this was their italics on its own, made some sort of leap and is now behaving sufficiently human that we can't help but feel at least a little bit of digital ick. But what's really going on here? Well, to answer this question, I'm going to start with a high level tutorial on how large language models actually work. And I'll add a little bit more detail to it. Stick with me here, because once you understand these basics, you are then going to understand that description I just read from Anthropic in a completely different light. All right, so this is an exercise that's worth doing. All right, I'm going to start at a very high level here. At its core, a large language model like those of the fable Claude Opus or GPT families can be described underneath the hood as a sequence of what are called transformer blocks that are arranged in layers. One follows the other, so a sequential collection of layers. GPT3, which was the last major LLM that they actually published stuff about, details about, had 96 of these transformer block layers. New LLMs probably have more, but we don't know how many more. All right, let's start at the very high level here. I submit as input a prompt that I have typed to an LLM. You can imagine that this input is going to pass through each of these layers one after another. It'll go into the first layer and come out the other end, it'll go into the second layer, come out the other end, and so on. Now, here's what's important, conceptually speaking. Those layers can add what we'll call for our own purposes, annotations to that prompt. So as the input goes into the first layer, it'll come out the other side with some annotations added to it, some information added to it that the first layer came up with in its own analysis. Now, this prompt with the first layer annotations goes into the second layer and it comes out with even more annotations added onto it, and so on. So as it moves these layers, the original prompt is there, but you're getting all of these annotations added layer by layer. And critically, later layers can use the annotations from earlier layers to help do their analysis. Right, so you can imagine it's almost like you have like a bunch of long tables of scholars arranged in rows in some vast, like Hogwarts Great Hall. Style dining room. And you're passing this prompt from table to table, and as it arrives at each table, the scholars of the table pore over it. They look at the annotations from the scholars that came before them to figure out how to analyze it. Maybe the annotations from the scholars before them will let them know which of the scholars of the next table should look at it. And then they add their own annotations and it passes on. The very final layer in a large language model is special. It takes this heavily annotated version of the prompt and then it maps it to what word or part of a word to output next? Never going to be super pedantic. It really outputs a probability distribution over all possible tokens. And then the control program selects one probabilistically. But just think about the last layer, takes all the work that all the transformer blocks did and says, this is what we are going to output next. All right, so that's what's happening at a very high level. I'm going to add a little bit more level of detail now. So now let's add a little bit more level of detail. What are these annotations and how are they written down and how are they passed from layer to layer? Okay, let's look a little bit closer at what happens to this prompt that you have handed on to your large language model. The first thing that happens is the prompt, which presumably you typed in with words on your keyboard, gets broken up into what are known as tokens. So in the pieces, and a token might correspond to a single word, or a longer word might get broken up into multiple tokens that represents the different parts of the word. But we kind of just, let's break this up into this fundamental unit we call tokens. Next those tokens get embedded into a high dimensional space and all that really means is, okay, each token is going to be described by long list of numbers. A long list of numbers is otherwise known as a vector. So it's a bunch of numbers that are in some order. So we got a long list of numbers to represent each token. And there's something called a token embedder that takes each token, which is now like letters, right, a word or part of words, and transforms it to a long list of numbers. The long list of numbers, the so called token embeddings that come out of this embedding, the numbers that are in there at first are basically capturing the meaning or details of that token. You train in a sort of semi supervised fashion how to do these embeddings so at first these long list of numbers are. The numbers that are in this long list are sort of capturing what this word means or what this part of a word means. Right? We want these things in numbers because that's what we can. That's what language models are going to. They understand because in the end we're going to be manipulating numbers here, not letters. But here's the key thing. There's a lot of room in those long list of numbers. So they don't just encode. Here's what this word means as this long list of numbers, which is how we represent the prompt now as it moves through each layer, the layers mess around with the long list of numbers for each token. And in messing around, which means messing around with the values in there in part, that's where the annotations live. So this is what's being passed from layer to layer is I just imagine like I have my prompt, it's been broken up into tokens and hanging off of each token is a long list of numbers and this big. So it's a big table of numbers. We call this a matrix. As this passes from layer to layer, those numbers are being updated to include among other things, the analysis that's being done by each of the transformer blocks. There's other things that lots of things are stored in here. Like for example, if you open up a transformer block, there's really two parts in it. There's a self attention mechanism layer and then a small multi perceptron style feed forward neural network which does the analysis. But the self attention layer is kind of cool. It adds information in each of these numbers, each of these lists to try to help each token understand how relevant each other token is. This is like the working space. Here's the way I think about it. I often think about these tokens and these long list of numbers, these vectors that come with each of them. It's almost like if you had. It reminds me of the Talmud. So if you look at the Jewish Talmud, what you see in there on every page, if you actually look at a printed one, is you have in the very center of each page, not taking up all that much space, is the actual text from the Mishnah, right? The sort of the oral law written down by Judah the prince in the early common era. It's a small part of the page. And then surrounding it all on the page, if you look at a page of the Talmud is commentary. So you got the Gemara, you got the commentary around it. Okay, that's how I kind of imagine conceptually the prompt being passed through a large language model. It's like you have a book, and on each page, you put in the very middle of the page, one token, right? So this book has the whole prompt with tons of white space around each token. And as it gets kind of passed from layer to layer, scholars are writing more commentary around the token. And then the next layer gets set, and they look at that commentary, and they look at that, they build on it and write more commentary. So I sort of imagine it like we're filling out the gemara around the Mishnah in the Jewish Talmud. All right, so that's, roughly speaking, what's happening. So we got this matrix moving from layer to layer, which is really just a list of numbers for each token. And those numbers are capturing everything we've learned so far as it moves through these various layers, right? So that's what's happening. We know. So what type of analysis is happening? Well, we don't know exactly because these are neural networks doing it that are trained in an unsupervised manner, like any neural networks from an image recognizer to a handwriting whatever, any sort of neural network. The whole point is you start it with random weights and you train it with lots of data until it gets good. And you don't know exactly how it does it. It just does it until you really kind of open up the box and try to understand what's going on. But we know I've written about this before in the New Yorker. I've talked to a bunch of experts about it. There's kind of like two things that are happening together with a generative AI model, like a large language model. Partially, it's learned during training when outputting the next word, what words are syntactically appropriate. This is actually pretty easy to train. You can do this even with a simple N gram model. This is an easy thing to learn. The statistical nature of language. Given this sentence that stops here, what are the words that could follow that from a syntax like a grammar standpoint? Makes sense, right? So that's kind of the easy part, and that's baked in. In fact, that's probably captured primarily in the final layer. We don't know, but probably in the final layer of the large language model. The problem is, of all the words that could follow that make sense, which one makes semantic sense? So which one actually makes the most sense, given the meaning of what the prompt is asking? So not just what word would grammatically make sense here, but which word not only grammatically makes sense, but actually matches the meaning of the prompt. And that's in my understanding, where all the annotations that follow along these token vectors really are helpful. So you kind of have these two things mingled together. They're not separate systems, because this is all kind of trained and muddled together. But you kind of have these two things happening at the end. When you're in a generative model, like an LLM model, it's like the quick brown fox. And grammatically, there's lots of words that would make sense there. You want to put the word the next. But there's lots of things jumped, dance, die, talk. A lot of words make sense. But then the annotations where you're like, oh, this is a saying. This is a saying that already exists. It's a common saying. And it always says jumped. And then so of those words that would privilege jumped as the next one to go. Again, I'm separating out here. Things that are muddled together in the neural networks themselves. They're not going to work as cleanly as a human might. But that's the way to think about it is that the annotations really help you figure out semantically the right next token output. And then the easier part that's like baked into probably these final layers is grammatically or syntactically what words or next tokens would be valid. So those two comes together and we get a good output. All right, that's roughly speaking, what happens in a large language model. Now that we know this, let's return to the anthropic paper. Okay, I'm going to see if I can find it again here. Let's ask what again, did they actually find? Okay, so they used a mathematical tool based on a linear algebra notion called a Jacobian to essentially try to understand what is in those vectors of values attached to the tokens that are moving through the layers of the large language model that they're studying. So we're going to look at the annotations. Now, I know they're just like a bunch of numbers, but we're going to figure out a way of trying to make more sense of what those numbers actually meant. And so using Jacobian, it's a linear algebra way that involves taking the partial derivatives of a lot of things. Basically, what they can figure out is which combination of numbers from this giant matrix, which pattern of these numbers seem to be connected and important, in other words, like having a high influence on what the ultimate output is. So they're trying to decode the annotations that are captured in numbers in these vectors and they're able to figure out certain patterns of these numbers seem to be really influential for the ultimate output. And you can do this experimentally. If we change exactly this pattern, we're much more likely to get a change in the output than if we change other patterns. That's simple, but it's something like that. And then, and this is what's cool. Through a lot of experimentation, they also try to as much as possible, associate different patterns through lots of trial and error experimentation with human interpretable concepts like words or numbers or something like that. So they call this the jlens because it allows them to say, looking at a, you know, they run a prompt through and they can kind of watch it go all the way through. And as it gets towards the end, they look at these vectors of numbers that have been evolving and updating as it moves through and they can say, we can actually decode into English or in human interpretable ways. Sometimes it's not like English words. Some of the annotations we can kind of understand what some of the annotations are that these layers are using to help come up with the answer, which is pretty cool. So I'm going to go to the paper here to show you an example from the paper. This table is called the jlens reveals the model's internal thoughts. I'll put one example up here on the screen. So this shows a prompt that says the color of the fourth page. The planet forth from the sun is. So that's the prompt the language model needs to expand as with another word or part of a word. When they looked at the annotations that this accumulated as it moved through the large language model, they discovered that there was an annotation that corresponded to Mars, which is the fourth planet from the sun, and an annotation that corresponded to color. Right. So what's happening here is as that prompt was moving through it, these conceptual tables of scholars that are studying this sort of Talmudic commentary and adding their own commentary to it. At somewhere along the way, one of the layers said, oh, I recognize the fourth planet from the sun. That sequence of tokens. And this part of some neural network in some layer is all about planetary stuff. It's like, that's the Mars. I'm going to write Mars down on my annotation. Here we're talking about Mars. That's an important annotation. And somewhere else some other layer was like, this is asking what the color of something is. Like the key Thing that we're. The thing that comes next is a color. This is asking about a color. Let me write that down. This is, you know, we're asking about a color. So then when you kind of get to the end of this large language model and it's like looking at all the possible words that just statistically make sense to follow this sentence, the color of the planet forth from the sun is. And all sorts of nouns there would make sense and adjectives would make sense. It looks at the annotations like, oh, we're looking for. Of the words that would make sense syntactically. We need something that's a color, and a color describes Mars. Hey, what is that? Don't we have that somewhere? That's red. Okay. And then it outputs red. So we could see. That's cool. It's like, oh, maybe it's not as mysterious as we thought. Like, we know it annotates numerically, but when we were able to interpret the annotations and annotate it with Mars and color, which is like, exactly like what you should do if, you know, it makes sense. So I think that was. That's pretty cool, right? So they could do that type of thing. Then what they did, which again, I think is fun research. It's like, let's mess with this. So let's freeze this matrix right before we get to that final layer. We're going to output the next word and let's change the. If we know what some of the annotations are, what if we change them? One of the things they did with that example I just talked about. I'll bring it back up here is they replaced the numbers that corresponded with Mars in that matrix. So the annotation vectors that hang off each token and they replaced it with the sequence of values that they had discovered corresponds with Earth. So now, even though the prompt is the color of the fourth planet from the sun, is when it got to the final layer to output a word, it was looking for words that make sense grammatically. And then its annotation said Earth and color. So it output blue. So they were showing, hey, yeah. These conceptual annotations really do influence the word or part of word that are output. Another thing they did is they called it. I really don't like the anthropomorphizing here. They said they ablated some of the information that was in these sort of key annotations. That just means they zeroed out the values. And what they found there. Ablated, by the way, is like a procedure you use in an actual human or animal nervous system, where you use heat or electricity to sort of fry a nervous connection, which is anthropomorphizing. Let's put that aside. But they would, in this example, right before we got to the final layer, hey, take those numbers that together correspond to Mars, and let's zero them out. And what they got in those instances is you would still get grammatically correct outputs, but they weren't semantically connected anymore. So they would just be arbitrary colors, which would be equally likely. You get a bunch of colors, but nothing that corresponded to Mars in particular. So it shows. Right? I mean, you kind of get this very rough sense about how these things are working is it's like some combination of the grammatically syntactically correct. Next thing we can do, which one should we choose? And we have all these annotations that help us narrow that down to be semantically correct. That's oversimplifying it. But something like that behavior emerged. So I think that's interesting research. All right, so what does this mean, though? Is this. Is this scary? Is it not? Is it interesting? Is it breakthrough? How do we think about this? Well, there's two thoughts I think we have to keep true in our mind at the same time. One, it is interesting research. Is it interesting because they're the first people to be able to look at these embedding vectors and use the Jacobian to figure out patterns of values that are important for the influential, for the final output to sort of jlen's approach. Turns out they're not the first ones to do that. So it's not like they had a breakthrough idea about doing this. I'll bring up a tweet here. This is from a U. Illinois professor, Yujia Zhang, who said, JSpace is really something we've been exploring since 2022. Glad to see it continues to work well at scale. Some of the related work along this direction, she lists three papers, and then she lists three more papers. The point being she was kind of responding to the anthropic paper like, we've been doing this for a while with neural networks. This isn't new, but no one's ever done it on a massive language model like Claude before, because the only people who have access to the innards of a massive data language model like Claude are the actual companies themselves. So that's why she said, it's great to see this being done at scale. The idea wasn't new, but so anyways, I think they executed it probably well. Again, these are not I always put quotation marks around reports because these are not computer scientists. These aren't formal computer science papers. They're more press release y. They describe things relatively high level with lots of pretty graphs and animation, but we can't really get into the guts of what's really happening here. But anyways, it looks like, hey, we took this idea and we applied it to these massive models people are using and that was interesting. So that's idea number one to keep in your head. But idea number two, the way that Anthropic described these results and the way that they were echoed after Anthropic described them I think is incredibly disingenuous. Because once we understand how a large language model roughly works, like I just explained, we see the thing they were describing with their JLENS is exactly how we've always understood large language models to work. I mean, I went back, I wrote a long article explaining large language models for the New Yorker in 2023. I used exactly that analogy of adding these annotations of the high level concepts and annotations built on other annotations. Then you use that to help figure out the word 2023, 2024. I wrote another long article about the architecture for the New Yorker of the architecture language models. Again, tables of scholars adding the annotations which then allow you to narrow down conceptually the next token to output in a way that has some semantic meaning. This is just how we understand large language models to work. It's also how we understand essentially any deep learning neural network to work. The original breakthrough work and deep learning was image recognition, like Yann Lecun figuring out that you could recognize handwriting better with a multi layer neural network than you could with other types of approaches. And again, the way we always understood those to work is that different layers were picking up different features or annotations and they were then combining those different features, later layers to try to figure out and make a good guess of what the letter was in a very robust way. This idea of we're picking up higher level features and descriptions of what we're analyzing. This is just how deep networks work. And it's how I always understood large language models. So I see this as like, yeah, good. We saw with the JLENS exactly what we expected to find. This is how we assumed large language models. What was going, you know, this is what's happening in those embedding vectors. They're accumulating annotations that build on each other, that capture higher level conceptual meanings, that help influence the token you put out and if you change the annotations, you get different tokens. How else would this work? But if you read the press release, it's all sorts of anthropomorphizing and implication. In ick generation it talks about the global workspace model of human consciousness. It somehow implies that this seems kind of similar to that. It uses these loaded terms about pondering and puzzling and it's thinking through these things and it's silently thoughts and it's not things it's saying out loud. And it makes it seem like this is all icky and human like and new. And it is what any LLM researcher you would talk to for the last five years would say, yeah, that's how LLMs work. I mean, it's oversimplifying it to talk about it like annotations, but like that's how they work. There's nothing new and surprising here. I mean, of course the annotations are picking up high level features of the input to help figure out what to output next. That's how deep learning networks work. That's why you have multiple layers. They get all sorts of different levels of abstraction of understanding different parts of your input. And so it's cool research, but research that most people shouldn't care about. And it certainly doesn't imply what the implicit. And again, they're careful in the paper. They're like, well, it doesn't really mean it's conscious. We don't say that, but it's interesting. I mean, look, I'm going to load this up. Let's, let's go back to the paper for a second. I don't want to go on too far here, but again, let's look at these things. The JSpace, it operates silently in the model's internal network activations, allowing the model to think about a concept without writing it down. What does that mean? It's thinking about a concept without writing it down. The way deep networks work is that you figure out different features which help you understand other features which you combined in ways that you, through neural network circuits that you learn during training that then help you put out the right recognition or generate the right thing. It's not thinking, what does it mean for it to write it down? It is writing it down. That's how it writes things down. It has this matrix that it updates the numbers every time. That's its workspace, that's its workpad. That's how these type of neural networks work. I'm going to jump forward here to the conclusion all Right, So here we go. It's like, what about consciousness? And it's like, well, we borrowed a lot of ideas from the study of consciousness. Many of our experiments were designed to test for connections between the JSpace and Global Workspace Theory, a framework for explaining how conscious access works in humans and animals. Given those connections, it's natural to ask whether these experiments provide evidence that AI models like Claude might be conscious. Well, our experiments don't show that Claude can have experiences or feel things. It's like. But that's all very suggestive that there's something new going on here. Probably the most, I think, disingenuous thing at all is the way that they punched in the intro. Claude did this on its own. We didn't program it. It's a machine learned neural network. Nothing is programmed. That's what machine learning is. You could say that about any machine learning system, man. This image recognizer is figuring out how to recognize it. We didn't tell it how to recognize the images, it just figured it. Yeah, because it was machine learning. You did semi supervised training to train the neural network's weights until it was good at minimizing loss on the image recognition task. That's what machine learning is. You don't program it, it learns. So I just think the language around this is so disingenuous. They took an existing tool that people had been doing for the last four or five years and it's a cool tool and they showed it could work on scale on very big networks. And it's cool. And it confirmed the way we thought these things work is that you have useful high level properties identified as you move through these layers and then those are used to help narrow down the choice of possible tokens to output. Yeah, good. It works like we thought it did. Nothing is new and nothing has to do with human consciousness. Global Workspace theory, first of all has been largely, at least as controversial. But the thing they're not saying and that all the people talking about LLMs and consciousness are not saying is when you think about Global Workspace theory and human consciousness, it's a center that's integrating ongoing inputs that are coming in and out. It has all sorts of different things coming in and it's selecting different things to pay attention to and nod. And it's this evolving stateful system. This is all feed forward. Remember, this is feed forward, one layer after another in order, and then it's done and the token is out. Nothing is saved. There's no states that are changing that neural network. Nothing has changed. At all. And then you can feed it another prompt, you'll get out another token. And so they're just looking at the annotations that build up and they're like at some point towards the end we see some of these annotations describe high level properties of the type of thing that it's trying to output a token for. That's how LLMs, at least to my understanding, are supposed to work. So look, I will say this. I think there's good researchers and anthropic and this is good research. I think the PR people who talk about this technology do so in a way that is, I think, disingenuous. I think is with a particular agenda for trying to make people feel in a certain way, which is mainly just a general sense of ick, like wow, this stuff's too powerful. Why? Because if this stuff seems like weird and alive and emergent, it's such an important technology and we so worry about that that we'll forget to ask, hey Anthropic, how are you going to make a profit? Hey Anthropic, your token cost is this high. You have no competitive mode on the small number of applications where people are willing to actually pay for token API access. Smaller, more specialized models with hand coded, complicated harnesses are going to do just as well. What's your plan? How do you justify a trillion dollar valuation? These sort of real questions are the ones you forget to ask when you instead are trying to wonder, huh? Does this mean Claude is a moral agent or conscious? So I guess my final thing would be to tell the researchers at Anthropic, I like your work and this is good work. I wish they would let you write real computer science papers and not these glorified press releases, but I know they don't let you. But maybe next time when the PR department calls, wait a second before you pick up the phone because I do not like the way anthropic talks about their research, even if there's cool stuff going on in there. So back to the original question. Did we just reveal something unnerving, cool or blockbuster about LLMs with this research? And the answer is not no. We confirmed that yes, language models worked a way that at least I always thought they did. And you can check the print because I've written about this for years. So anthropic, will you stop with the press release? Research reports, write computer science papers, sell products, convince us to buy the products. This weird in between kind of bastardization of actual research is something that's stressing a lot of people out, and I think it's disingenuous, but that's just my opinion. All right, that's all the time we have for this week. I'll be back on Monday with an advice episode. Probably no AI. Reality check the next week because I'm on vacation, but I'll do my best. We'll be back soon enough. And until then, as always, remember, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at Cal Newport Ideas, and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today. Calnewper.com ideas.