Gregory McNiff (2:27)

Gaurav, I'll start with you. Why did you write the Emergent Mind? And who is the target reader?

Gaurav Suri (2:34)

So the history of this is a little bit about the history of my life and Jay's life. I, after initially learning mathematics, went and worked in industry for 20 years, but was always struck by the things that I observed my customers and my partners do, and always had this question about where are these decisions coming from? In midlife, I decided to formally pursue these questions with a PhD at Stanford, and that's where I met Jay. But before I met Jay, I was constantly trying to answer this question, which is, what are we? What are our thoughts? Where is it coming from? I knew the Cartesian inner soul wasn't a satisfying explanation for me, but I didn't have another satisfying explanation either. And it wasn't until I took a seminar with Jay that I got introduced to the neural network approach. And I think I had the hooks in for such an approach. And when Jay started describing one of the networks that we describe in the book, I thought, wow, this. This is the kind of thing I've been looking for. And that's when we started interacting with each other, first as me being a student of Jay, and then writing papers together and then being friends, taking walks and meals. And we wrote the book because these ideas are, I think, profound. And I have this deep conviction that science is not something that is just done with scientists or among scientists, that people who do science shouldn't only be talking to each other, that these are. If the ideas are powerful and if the ideas have beauty and grace, it's absolutely imperative to try to get them out in the world. And I felt a little bit like the theory of the Origin of Species for Darwin existed, but there wasn't a. A vehicle to get these really powerful ideas out there. And that's who we targeted, and that's why we wrote the book.

Gregory McNiff (4:50)

Excellent. Yeah, it really comes across. Like I said, you articulate the themes very well, and they're obviously complicated. Jay, I'll turn to you. Could you define this notion of emergence?

Jay McClellan (5:01)

So the basic idea of emergence is that there are properties that exist when lots of things interact with each other that don't exist in the individuals by themselves. And this is a fundamental thing about many, many phenomena in nature. So the properties of a water molecule do not exist in the. In the hydrogen atom, and the hydrogen atoms and the oxygen atoms that make up that atom. And furthermore, the experienced properties of water as a fluid that slips around and allows many things to dissolve in it only exists at the level of the interactions between all those separate water molecules. You don't have the properties of water with a single water molecule. Similarly, you wouldn't have the properties of the mind with a single neuron. The neuron itself is a cell. It's a. It's a very interesting and complicated cell because sends these electrical signals that, you know, are happening at a much faster rate than chemicals could perfuse through a medium. They travel down little wires and influence each other so that they can all act like a whole population, almost like a. Of people working together to make some collective outcome occur. And the basic idea of the emergent mind is that the mind is essentially composed of all of these little things which themselves cannot think, but which, when they work together, give rise to essentially our thought processes, our experiences, and then the outcomes and actions that we take based on those experiences.

Gregory McNiff (7:06)

Fascinating. Gaurav, in the beginning of the book, when you're describing emergence, you use the example of an ant colony. Could you talk briefly about that and how that reflects this concept of emergence?

Gaurav Suri (7:19)

Yeah, happy to. So the phenomenon is that if you imagine or see a train of ants that are going from their nest to a food source and back, just imagine a straight line. And further imagine, and I did this as a child, further imagine that you put a obstacle in their way, and the obstacle is arranged so that the ants have to go around it to get to the food source and back to their nest. But there's a short way around the obstacle and the long way. And if you do this, what you see in the first minute or so is exactly what you'd expect, which is half the ants roughly go one way and half the ants go the other way. But in a few minutes, something, to me, miraculous happens, which is most of the ants are going the short way. And the question is, how does this happen? And what we talk about in the book is ants do two simple things. They lay pheromone trails and they follow strong pheromone trails. So given a choice, an ant will follow the stronger pheromone trail compared to a weaker pheromone trail. Now, it's with these two properties, you can show that the shorter path is going to have a denser concentration of pheromone because it's more traveled by the ants. And new ants that have to make this decision about the long way and the short way are going to follow the greater concentration of pheromone on the shorter path. And this is an example where the ant colony has an intelligence to Solve a problem of going around an obstacle, even though each individual ant doesn't. And this is connects to Jay's water molecules and neurons. And each water molecule doesn't know about wetness or sliding on other layers of molecules. Each neuron doesn't know about thought the way the collective, the brain, does. And this intelligence in the ant colony is emerging. And what we've done in this example is trace how the individual properties of ants lead to the emergence in the ant colony. And that's what we do in the book, with neural networks, where the individual properties of neurons lead to the intelligence of the neural network.

Gregory McNiff (9:43)

Yeah, that's a fantastic answer. And I want to circle back to that because it does get into the evolutionary ideas that you talk about, as well as clearly the importance of neural network. But just defining, really, the second concept in the title, namely the mind. Gaurav, in the beginning of the book, you briefly touch on several concepts of the mind. Religious traditions and beliefs, beliefs and desires. Some even view the mind as software. Could you maybe talk briefly about those descriptions and then the description. And I'll maybe turn to Jay for this. The conception that you guys propose as the mind is arising in neural networks.

Gaurav Suri (10:23)

Yeah, happy to. Right, Greg, I think this is an eternal question. Who are we? What are we? So when we're asking this question, we're really asking about what kind of thing our mind is. And in the book, we talk about progress made by Descartes, Right. So Descartes, we say in the book, was walking in a park and he stepped on a stone which made a statue move. And he asked the gardener or someone, and it turned out there were hydraulic pipes between the step and the statue. And Descartes gets inspired and says, maybe this is the system for simple actions such as moving away from a fire. And he's wrong about that. We don't have tubules filled with fluid. But his insight was that maybe we can start to understand the mechanistic properties by the properties of the mind by a mechanistic process. And this was a move of genius, right? I mean, before Descartes, it wasn't even considered, systematically considered, that we could understand the mind the way we understand our stomach, mechanistically. And Descartes started us on that journey. Of course, Descartes didn't go far enough because he said, gee, there's this spirit matter that controls the higher aspects of the mind he located in the pineal gland and so on. But I think that was a glorious set of mistakes. And that was a common conception of the mind in some form or the other, some aspects have been captured in religious traditions, beliefs and desires of what the mind feels like. Right. So it often feels like, why did you decide to live in this house? Because I have a belief that this neighborhood has a good school district and so forth. So the belief and desire model really captures the experience of having a mind. It doesn't necessarily explicate where the mind comes from. And still other explanations have to do with a sort of a computer program that evolution is. Or something has specified that somehow the brain runs. But none of those were quite satisfying as I thought about them.

Gregory McNiff (12:36)

Excellent. And then, Jay, I'll turn to you. You define the mind as arising in a neural network. Could you expand on that?

Jay McClellan (12:46)

Yeah. So when I look at a beautiful painting and I experience, let's say, the water lilies floating in the pond, I have an experience that, you know, I marveled at when I went to art galleries, right. When I was an undergraduate. And it's like, okay, so that is my mental life, my mental experience. Where does it come from? Rob is asking that question. Right. So what if. And there was a book actually written by Francis Crick and Christoph Koch called the Astonishing Hypothesis. Right. What if all of that was the consequence of neurons influencing each other, full stop. Right. So it's like a collection of physical things. The neurons are physical cells, Right. And their connections are allow electrical impulses, which are also physical things, to occur. But this experience is something that comes out of this process. And it was the sort of realization that it was okay to say to myself, I'm going to try to do as much as I can to capture that, the properties of those experiences by building neural networks, things that are like real neurons, populations of real neurons. Importantly, there had to be a collection of them interacting with each other. I'm going to build a model that tries to capture aspects of what we know about those experiences as arising from the interactions of these little neuron like parts.

Gregory McNiff (14:59)

Yeah. And Gaurav, I want to, because it's such a key concept in the book you write, all our thoughts, experiences, actions are emergent consequences of neural activity in the brain. Could you maybe just touch on that as well?

Gaurav Suri (15:13)

Yeah. Isn't that a beautiful possibility that, oh, what we think is connected to these physical things that Jay is describing neurons, that it is their interactions that cause us to act and us to think. And, you know, I remember even where I was sitting, where we were talking about a bus coming at some person and the person getting out of the way. This happened before I Had met Jay and I had. This was the person who was describing these events. I had this, this image in my head about activity from the retina going to the brain, going to different neurons going to other neurons which signaled muscles which made muscles twitch. And I said, wow, you know, that. That seems like it could be a thing that I could get behind that. Oh, this is what the mind is doing when we get out of the way of a bus. Now, of course, if we ask each other, hey, why'd you get out of the way of the bus? We'd say the bus coming at me, and I didn't want to be crushed. And of course that's real. But there's a flip that this activation related account does. Right? The flip is that it's starting with the neuron and starting with the properties of these things that we know exist in the brain. And it's not reliant on the experience. It's not reliant on the story that we're telling ourselves about how we got out of the way. I'm not denying the story. And I think the story often accompanies the things that we do, but often the story may not be the cause of the things that we do. And this idea that it's a pattern of activation in a neural network that's controlling our thoughts and action and the mind is emergent is really exciting to behold.

Gregory McNiff (17:10)

No, absolutely. And I want to drill down now into that. Obviously, we're talking about the brain as a neural network and how it thinks and is responsible for our emotions and actions. There's some terminology I was just hoping either of you could define. There's about six or seven terms I'll start. The first is spikes or active potentials. Could you talk about that?

Jay McClellan (17:32)

Yeah. Let me actually take you back to the neuron itself. Okay. The neuron is a cell. When we think about cells, we usually think of things that are a little compact entities. But actually a neuron is an amazing, amazingly structured thing that has a cell body and then many, many branches. So it's like a whole tree. It's got this elaborate branching structure coming out of it, going up, as well as other branches going out around it so that it. It has this. It spreads out some of them. You know, the cell body itself is like tiny, tiny. But the branches can spread out to cover several millimeters of surface in. In your brain. And the neuron has another part to it, which is the axon that comes out of it. So the axon is a single little thin wire that comes out of the bottom of the cell body, but it, too, can branch in up to hundreds of thousands of branches. Okay? So each neuron is capable of receiving inputs from throughout its tree, throughout this dendritic tree, from many, many other neurons, Coming from the signals that come out their axons and propagate into the dendritic trees of other neurons. So that's the fundamental substrate of thought. And there are 86 billion of these neurons in your brain. That's a number that somebody came up with, Obviously not subject to revision, but it. There's just vast numbers of them. And each one of them, you know, does some fairly simple things. So now I'm ready to get to the question that you asked. Neurons, when they're in the natural milieu of the brain, tend to be fairly quiet. They especially, let's say, a neuron in the visual cortex, it usually is driven by external visual input, but when there is no input, it's not very active. So what that means is that every once in a while, it emits a little signal that through its axon, it's a little blip of activity that propagates out and then goes and perhaps influences the tendency of other neurons to emit these little blips as well. But it's kind of quiet and sitting there. If you present a visual input to that neuron, to the input, to the eye, Some neurons will respond, and we'll get to why those neurons are responding. But suppose a neuron respond is getting a lot of input from others. Well, it can excite it. Those inputs can have influences that tend to increase the firing rate of the neuron. And so it will, in turn, send signals that will increase the firing rates of other neurons. So we have this sort of baseline firing rate where the neuron is kind of like, quiet and not doing too much. And then we have inputs from other neurons which can excite them or inhibit them. And so when we think about the individual neuron, Neuroscientists think about it this way. It has a kind of electrical potential which is a. And, you know, a state of essentially its resting state, which is subject to perturbation by these inputs. And some of the inputs will release a chemical at the junction between the two neurons called a synapse, which tends to change the polarity of the neuron, Moving it towards the direction of firing more. But others of them have the opposite effect. They have a different chemical that's released at that synapse, which tends to cause the receiving neuron to be less likely to fire. So we have Excitatory and inhibitory influences coming from other neurons, which together determine how, you know, the tendency this neuron will have to emit an action potential. Now, a question that many people ask at this point is, okay, so if you're going to model how the brain give rise to cognition, shouldn't you be modeling all 86 billion of those neurons, each of which is emitting action potentials, according to this summation, over up to 100,000 inputs from other neurons? And the answer to that question is no. In order to make progress, we need to simplify. And so a neural network model of the mind is one that has taken the fundamental ideas out of this set of concepts and said, let's distill something we can get our head around a little bit more simply so that we can build a model of it in a computer program that runs on the computer that we have today and see if we can understand how these emergent consequences that we've been talking about can arise from the interactions that occur in our somewhat simplified neural network.

Gregory McNiff (24:00)

No, great answer, Jay. You're doing well, and I appreciate the simplified notion. You have a nice quote in the book from Borges about how cartographers created a map the entire size of the empire, which was not appreciated by future generations. So absolutely understand that. I want to ask a few more terms that you use in the book. Baseline, firing rate, excite, inhibit, and neurotransmitters. I think you touched on that.

Jay McClellan (24:28)

Okay. So, you know, like, when we build a neural network model, we. We make an assumption. We say, okay, we've got all these units. We usually call them units in our models because they're not real neurons. And we're going to make some assumptions, right? So we're going to specify that these neurons have a quantity that ranges. In many of our models, we have this fairly simple idea of this, where the quantity could range above zero. And when it's above zero, it means it's sending signals to others. When it's below zero, that means it's sort of been suppressed below its ability to send out and communicate. So the baseline activation of this unit in these simple models is usually put below zero, so that essentially, at rest, they're not communicating with each other at all. In reality, neurons always have a very low level of occasional spiking in the brain. But for the sake of simplifying and getting your head around it, it's been very useful. And many, many neuroscientists think this way. When the potential is low enough, it's essentially not able to send an action potential, and when it gets over some threshold, then it can emit a spike. So the spike is a little jolt of electrical activity that travels down the axon. But instead of having a system where there's many, many, many neurons, each of which are sending these tiny little jolts at random times or semi random times, we say, okay, well, let's imagine that one of our units is doing the work of many of these neurons. And so it's going to be sending out an activation signal which is essentially representing the firing rate of the sort of population of neurons that it's kind of a proxy for. And then, so we get this notion of a continuous signal that, you know, can be below zero, but once it's zero or above, it's a graded matter of degree, that is how strongly it's sending signals that are going to have a chance to influence others.

Gregory McNiff (27:08)

Last term, Jay, bidirectionality.

Jay McClellan (27:12)

Okay, so this was a super crucial point in the work that David Rumelhart and I did together. And that turns out to be central also to Hopfield's insights that led him to be a recipient of the Nobel Prize in physics for his contributions to understanding neural networks. And that is the idea that neurons tend to influence each other in both directions. So in Hopfield's beautifully simple model, even simpler than the ones I've been describing, he said, let's start with the assumption that if there's a connection from neuron A to neuron B, then the connection in the other direction will be exactly the same. Okay, so if A influences B, then B will influence A in the other direction. That's what we mean by bidirectionality. We don't necessarily think that the individual neurons are able to be bidirectionally connected point by point, but at the, at a larger level, in many, many models, it's been really useful to make that assumption to understand some of the collective and emergent properties that these systems can have.

Gregory McNiff (28:42)

Perfect.

Jay McClellan (28:43)

Yeah.

Gregory McNiff (28:44)

I was going to ask you about a quote relevant to what you just said. The influence of one neuron on another is always a matter of degree.

Jay McClellan (28:54)

So, yes, that is, you know, on my personal tombstone, it's going to say, he thought it was continuous. And why do I like to imagine that that's what it's going to say on my tombstone? You know, in, in psychology, in theories of many, many kinds, there are these rules that say if A, then B. And, you know, if you're going to do logic, then the rule has to always apply. Logic is about the absolute. You know, what follows if the rules are always followed and the whole essence, I think for us, at least when we were first doing neural network models, was to break out of that and say, you know what? We're going to imagine that it's always a matter of graded or continuous valued influence. And when you think about it in terms of these networks that we've been talking about, these populations of neurons, actually one neuron out of 86 billion can't really make all that much of a difference, Right. There's always a population involved. So that one neuron's individual influence is, is a matter of a contribution to the influences of many, many neurons working simultaneously. And people will say, well, what about particular neurons in very small animals or particular connections that happen to be much stronger than others? And yes, there are triggers, connections that are much stronger than others sometimes, but even then, you know, there's a modulatory, modulating influence from so many other factors. So as a starting place for thinking, the idea is that the neuron, one neuron's activation is a matter of degree, and the extent to which it influences the next one is also a matter of degree. So there's two quantities there. We're going to get to those in a minute. There's the activation of the neuron and the connection, the strength of its connection to the next neuron. Both of those are matters of degree. So when you multiply them, you've still got a matter of degree.

Gregory McNiff (31:43)

Yeah, that's fascinating. And you have some nice examples. And I absolutely want to get to activation and the connections. But before we do that, Gaurav, I'd like to turn back to you, and this is a two part question. Could you briefly talk about our understanding of neural networks from an evolutionary perspective, starting in the oceans and how they came to create electric fields, and then maybe some of the key individuals responsible for our understanding of them. And here I'm talking about, like, Gogli and Ramoni Cajole. I'm saying his name, right?

Gaurav Suri (32:17)

Yeah. Happy to, Greg. So the, the summary takeaway from what Jay said earlier is that in a neural network, we are defining something called units that stand in for population of neurons. And these units are activated, which stands in for these little bolts of electricity that each neuron has, right? So we've got a unit. If it's going bing, bing, bing, and if you convert the electricity to sound, you can actually hear the crackle. You can convert the electricity of a single neuron. This is a very tangible thing, right? It's real flow of charged particles. And so that corresponds to the activation of a. Of a unit. And if one unit is active and it's connected to the other unit, then it influences the activation of that other unit. And this is the sense in which these are networks. Now, just as activation can go fast or slow, bing, bing, bing, bing, or bing, bing, bing. And connections can be one neuron may be completely not influencing the other neuron. It may be highly exciting the other neuron. It may be inhibiting the other neuron, and that depends on the synapse between neurons. So the question arises, well, how did neurons get this way? And why are they signaling each other with electricity? And how is this different connections happening? And what does it mean for one neuron to influence the other neuron, more or less? And here I think the evolutionary perspective is useful, because the general point of all emergent systems, Greg, not just the emergent mind, is that there's interaction between simple processing units. In a neural network. The interaction has to do with electric fields, electric charge. Why electric charge? Why not something else? Well, one candidate answer is that life originated in the ocean. And the ocean is filled with charged particles. And life often features membranes through which these charged particles permeate, and indeed, the selective permeation of certain charged particles across membranes, and then the restoration of equilibrium causes these action potentials. Right? And this was a click for me, like, oh, yeah. I mean, we have a lot of sodium, we have a lot of potassium in the ocean. And what are these cells going to use to communicate with each other? Well, these charged particles, and that is the root of activation, that's this train of charged particles coming in and out of membranes is what is causing this wave of electricity which we are calling the action potential. The other part is the synapse, which are these gaps, tiny gaps between neurons, much smaller and much more difficult to see than the neuron itself. Some neurons are very large and very easy to see, but the gaps are tiny. And this had led to a debate between are there gaps at all? And there were two central individuals here, and I find their story really interesting. Golgi, who is an Italian neuroscientist, had a way of staining neurons and had made a lot of progress in helping us understand what the dendritic tree and the axon might look like. And Cajal story was he was interested in art. Ramon y Cajal, he was interested in art, but his parents, his father, wanted him to be a doctor. He, being a good son, became a doctor. But then he drew pictures of neurons. He, too, was interested in these staining techniques. But it was both Golgi and Cajal had profound contributions. The synapse, though, is most often attributed. The discovery of the synapse is most often attributed to Cajal. Both men got the Nobel Prize and it was Cajal who discovered that, oh, there is this synapse and there are receptors and vesicles in sending and receiving neurons that control the amount of neurotransmitter, this chemical that goes from the sending neuron to the receiving neuron. And the strength of that sending and receiving varies. And that's what in a neural network, we refer to as a connection strength. Right. So activation, meaning the electricity in neurons and populations of neurons is varying and the connections between populations of neurons is varying based on the sending vesicles and receiving vesicles. It's all happening at the synapse which was discovered by Ramon y Cajal.

Gregory McNiff (37:20)

Yeah. Excellent. And I just want to read you have a brief quote from Ramon y Cajal.

Gaurav Suri (37:25)

Quote.

Gregory McNiff (37:25)

All great work is the fruit of patience and perseverance combined with Tanash's concentration on a subject over a period of months or years. Just like that quote. And I should say there's some wonderful. All the illustrations in the book are fantastic. Gaurav and Jay are doing a great job in terms of the audio description. But the book does provide some very nice charts.

Gaurav Suri (37:46)

I do want to say here, Greg, that Ramon y Cajal himself has some beautiful illustrations and I'd encourage your listening audience to just go online and look them out. They're awe inspiring.

Gregory McNiff (37:58)

Absolutely. And you mentioned that in the book. I agree with you. They are beautiful. Jay, I want to turn back to you. Gaurav has sort of set us up now to discuss connections and activations. I'm going to read one or two quotes in the book and this seems like such a key theme. Clearly in the first half you write, the. More I'm sorry. Connections contain the knowledge in the neural network, AKA they are the currency of the neural network. Again, activation flows are influenced by the current set of connections weight, which changes in the connections weight change and are guided by the information carried by activation flows. This iterative process allows neural networks, both biological and artificial, to adapt and improve performance over time. And finally, propagation of activation in a neural network is determined by its connections. Could you help us understand these key terms?

Jay McClellan (38:51)

Sure. So, you know, Rav has set us up beautifully, as he said, for this. These activation signals are things that we think of as electrical. And these electrical signals come and go very quickly. They actually come and grow much faster in Our artificial hardware that we can build today with these incredibly fast switching circuits. But in the brain, they're still relatively quick, right? A visual stimulus will evoke a response that will, you know, peak shortly after the onset of the stimulus, maybe a third of a second later. And then, you know, if the stimulus then goes away, it'll fade out and be gone by the end of the second. So. So we. This is the, you know, the experience I have from the presentation of an input is this produced by this wave of activity through the system, the activations themselves. But as Gaurav said, the ability of a neuron to become activated by a stimulus, like especially a neuron in your brain, depends on connections from other neurons, because the neurons in your brain aren't directly getting the visual stimulus that hits your eye. And in fact, there's several stages of neurons and synapses, even within the eyeball, before the signal starts going through this set of axons, these fibers coming out of the eye, transmitting this information to the. To what we think of as our brain. Some people think of the eye as an extension of the brain, and in some sense it is, but very specialized one, right? So anyway, for a cell to become activated in the brain, it has to be activated because other cells were activated by the stimulus, and they, in turn, activated the target by virtue of connections onto the target. And so if it weren't for those connections onto the target neuron, the target neuron would not be activated. Right? That's. That's the fundamental idea at stake here. A really important thing about connections is that they, you know, they persist in the absence of stimulation. So once they're set up, they are available for later stimulation to produce an effect. Okay? So we think of experience as shaping the connections so that later inputs can rely on those connections to influence what we see, what we experience. But the residue of the experiences in the connections. It's not like we're not maintaining these electrical patterns of activation. We're only maintaining the connections between the neurons that participated in those electrical patterns of activation.

Gregory McNiff (42:02)

And, Jay, that's an excellent analogy. In the book, you use the example of waterfalls or water mainly to describe this connection. Could you briefly explain why you.

Gaurav Suri (42:13)

I guess, yeah.

Jay McClellan (42:15)

So I was thinking about this idea that it's all continuous and a matter of degree. One time, when I was actually walking in Yosemite national park, and there's this beautiful walk along a stream that flows, I think, a couple of thousand feet from a meadow above down to the valley floor below. And as you walk along the stream you see these pools of water that are getting flows into them from other little pools of water. And the amount of flow from any one of these pools to another one is sort of determined by the width of the channel from one pool to the next pool.

Marshall Po (43:06)

Right.

Jay McClellan (43:07)

And that's kind of like the connections, and then the water is like the activation throwing through these channels. So the more the water has flowed through a particular channel, the wider that channel might end up being, and the more strong that flow will be. And so, you know, past flows determine the pathways through this stream, but the current water coming down through it is then channeled by those past effects. And of, you know, the water is evanescent. It can come and grow after a rainfall, but the channels remain there to channel the water in future occasions.

Gregory McNiff (43:57)

Yeah, seems like a very helpful analogy. Jay, you discuss your concept of the Interactive Activation and Competition Network. The IAC would you suggest is helpful in understanding memory, but also other aspects of the human mind. It's obviously tied to your primary research. Could you talk briefly about that?

Jay McClellan (44:17)

Yeah. So we've talked about the idea that neurons can influence each other because they have bidirectional connections between them and Dave Rumelhart and I. And I just want to say Dave Rumelhart really was a brilliant scientist, sort of who played this incredibly essential role in my own scientific development, but also in laying the foundations of what the work that he and I did together, as well as pointing the way, in fact, inventing the algorithm that became the center of AI. But he and I worked together and developed this model that captured this very, very central idea. So I'm going to ask you to think about a drastic oversimplification. And again, Rumalhart deserves the credit for saying, let's keep it simple so we can all understand it. Okay? So that's really at the heart of all of this, where we have neurons for individual letters that might spell words and neurons for possible words that they might spell. Okay, so I'm going to really simplify this down to a very, very simple world in which we have four different letters. I might need five. So let me. Let me count here. We have the. Yeah. Okay, so I. We have the letters T, H, E, and then we have C and A, and we're going to need T twice. So that's five letters. So the word the can be spelled by T, H and E, and the word cat can be spelled by C, A and T. Okay, so if I present something that's ambiguous, halfway between an A and an H, and the input will activate The A and the H sort of equally. And we think of letters as, like, somewhat competing with each other. So if these two have equal partial activation coming in from the bottom, they're going to be like, oh, you know, you want to be on, and I do too. So we should compromise. We should find an intermediate and not be really sure. Well, but we know that if this letter is surrounded by a T and an E, then it would make the word the. So how could we let that influence what's going on in our neural network? Well, we can make it so the T activates a unit for the word the, and the H and the E do two, and then they're reciprocally or bidirectionally connected back from the word to the letter. Okay. So that when that partial activation of A and H comes in between the T and the E, the word the will be much more strongly activated than the word cat, because there's no C around anywhere. Right. You know, so this will allow the network to settle into a state where the word the is more activated than the word cat and the letters T, H, and E are activated and the letter A isn't activated because the A got more activation from the. The feeding back down and it inhibited the A and suppressed it. So that's the idea of interactive activation. Crucial to this idea is that the interpretation that we made of an ambiguous input depended on the context in which it occurred. And that was the goal of this work, was to show how, if we had graded activations like partial activation of these two possible letters, we could let the context influence the decision about what the outcome of this process was going to be. And Rumalhart and I spent about a year sort of in each other's office working these ideas out. And 1977 into 80. 90. Yeah, somewhere in there. Around 1978, mostly, I think. So let's just check. Have I gotten this idea across here of how these units, by virtue of their interactive or bi directional activation of each other, have allowed a network to settle into an interpretation of, you know, a whole word, you know, in a way that wouldn't have been possible for the individual units by themselves to do that? Right. Because by themselves they only maybe have their direct bottom up input or something like that. But by virtue of their interconnections with each other, they can work together to create this system that settles into a stable state.

Gregory McNiff (50:04)

Fascinating. Gaurav, I want to turn back to you and we touched on memory, but could you talk about how important context is to memory and then maybe describe the kuleshove effect.

Gaurav Suri (50:17)

Yeah, I'm happy to. I just want to say that what Jay's describing is really context, right? So if you have an ambiguous letter which kind of looks like an H, kind of looks like an A, but within the. That's an H, and within cat, that's an A, that is a context effect. That's. That's exactly what we mean by context. And I want to say another. Another note about this network. For me, it was a transformative experience as I traced the activations from units to other units, and I saw how the network would settle to one answer or the other. And what stops a lot of people from seeing the majesty of the neural network view of the mind is that they often require computer simulations. And one of the things we've done in the book is we've sort of just said, let's not worry about computer. Let's actually trace the activation from one unit to the other unit so we can capture for the reader how these effects are happening. And, you know, Jay talked about context effects in the realm of letters and words. Of course, context effects are everywhere. Jay and David Rumelhart selected that as a example of this general phenomenon that context effects are everywhere, and context effects are central to how we remember things. So if you have something at the tip of your tongue and you can't quite say the word, bring it out. You know you have it there, but you can't say it out. What's the best thing? Think about the context in which you've encountered that idea. Right? And because memory is this interactive set of activations, doing that will invariably bring that word to mind, even though you're not directly thinking of whoever it is you're trying to think of. So there's a really interesting effect which also illustrates the power of context, and that's called the Kuleshov effect. And Kuleshov was this Russian filmmaker, and we have these pictures in the book, but what he did was he took a picture of a rather, I would say, neutrally express, neutral expression on a. On an actor. And it was the same expression, but when it was placed next to a coffin bearing the body of a child, the same expression looked unmistakably sad to many people and was placed next to a bowl of soup. The same expression looked hungry to a bunch of people. And when it was faced next to a woman reclining on a couch, that seemed to have a lustful expression. And of course, the magic of this is that it's the same photo, and we are bringing the context in which the photo is placed to postulate or perceive, even a expression, a particular expression of a different emotion. And in this sense, context is defining our reaction to the same. It's very similar to the H and the A, Right? It's the same letter, but the context defines it. And in the Kuleshov effects is the same expression, but the context next to the photograph is describing. Describing a reaction. And to me, this is. It seems like. Why am I on about context? This is a central feature of cognition. We perceive things, we understand things in their context. And I like to think that it's all context. That ultimately, setting aside whether it's all context or not, the argument we're making is that our understanding of the world depends, or something in the world depends on the context in which we encounter it. And how that happens is very well modeled within these neural networks. Because context units are there in the neural network, right? They're influencing what activation, what pattern of activation flows in the neural network. They are participants, just like the thing, the person's picture or the letter H, A is a thing. The things around it are context, and they're influencing each other. And from this influence, our understanding emerges.

Gregory McNiff (55:07)

Yeah, those are great points, Gaurav. And I want to read you two quotes that I think support what you're saying, but I want to ask you on a follow up about them, the mind doesn't always work in accordance with how we think it should. And then a later quote along the same lines, quote, Often our network behaves as if they are following rules, just as if our minds operate as if they are implementing hard coded rules, but they do so with. Actually, I'm sorry, but they do so without actually using these rules. Could you talk a little bit about that? Is the mind imperfect? Do we. Do we sort of have. And by we, I mean the public, have a incorrect or not fully complete understanding of the mind. Can you touch on that for a bit?

Gaurav Suri (55:54)

Yes. These are two central quotes from the book. I'm so happy you selected these quotes. We are, I think, storytellers about coming up with constructs that explain the things that we do. Right. So if there's a beautiful experiment, this was done in the 70s, where these two scientists, Nisbet and Wilson, they took identical stockings and they ask people to pick the stocking they like best. And many people pick the last one, and there's very good reasons. People often pay more attention to the first and the last, and they will often select the last item on a list. But when asked, why did you pick these Stockings. People talked about the thread con or the color or the hue or it was softer or something. And they were very convinced that they picked that last stocking not because it was the last in the sequence, but because it was the last, because there was some property of the stocking itself that made them prefer it. Now this is a story, right? In this case, it's a disprovable story about why we do the things that we do. But the central notion of the neural network view is that there are these interacting activations that produce our actions and our thought. And that same substrate also produces these accounts that we come up with of why we do the things that we do. Sometimes those accounts have a lot to do with the underlying activations, and sometimes we don't. But Jay's idea about this being graded and continuous is very different from an idea in which it's a computing engine of logic. Logic is not continuous. Logic is continuous. Logic is discrete. If this, then that every single time. And what we're talking about here with these accounts and this imperfection of the mind, imperfection seems like a derogatory term, right? The mind is continuous. The mind is making these interactions and is constructing a concept or constructing a perception for us that we react to. That's context rich. It's the stories that we come up with are often logic based stories, because logic or increasing value. Choosing a taco versus a burrito because I get more value from a taco. That's a really. It's a story that we often tell ourselves. And sometimes that story has a lot to do with the underlying activations. But we don't have access to the operations of the minds. We have access to the stories that we tell ourselves about the operations of the mind. And sometimes those stories are right and the mind works according to those stories. But sometimes, and often it doesn't. And if we see structure and pattern, it's because there is structure and pattern in the input that's coming into the mind. And the rules that we see of the mind are the statistical regularities in the input. They're not rules of logic in our mind. Of course, maybe we'll talk about this later, but the mind is also capable of building systems in which we follow rules of logic. But the point in these two quotes is that we often have accounts of the mind and those accounts may or may not have to do with the underlying reality. And that our accounts are often dependent on this logic like formal system. And that need not be the case.

Marshall Po (59:30)

It's okay not to be perfect with Finances. Experian is your big financial friend and here to help.

Jay McClellan (59:37)

Did you know you can get matched.

Marshall Po (59:39)

With credit cards on the app? Some cards are labeled no Ding decline, which means if you're not approved, they won't hurt your credit scores. Download the Experian app for free today. Applying for no Ding decline cards won't hurt your credit scores. If you aren't initially approved, initial approval will result in a hard inquiry which may impact your credit scores.

Jay McClellan (59:58)

Experian.

Gregory McNiff (60:00)

This episode is brought to you by LifeLock. It's Cybersecurity Awareness month and Lifelock has.

Jay McClellan (60:06)

Tips to protect your identity.

Gregory McNiff (60:08)

Use strong passwords, set up multi factor authentication, report phishing and update the software on your devices. And for comprehensive identity protection, let LifeLock.

Jay McClellan (60:16)

Alert you to suspicious uses of your personal information. Lifelock also fixes identity theft, guaranteed or your money back.

Gregory McNiff (60:23)

Stay smart, safe and protected with a 30 day free trial@lifelock.com Podcast terms apply. Fantastic. And that's a great setup. Excuse me too, a question I want to turn back to you, Jay, again. In the book you write, our minds result from activation based processes in our brains and not from feelings of pleasure or pain that often accompany these activation patterns. That surprised me, and I think that would surprise a lot of people, that we know why we're doing what we're doing and it's driven by some form of pleasure or pain. But you suggest. No, the models, it's different, and there are other factors that we may not appreciate. Could you, could you talk about that?

Jay McClellan (61:04)

Yeah. So the conviction that it's important for people to realize this idea is actually one that has been reinforced for me by the work of a man who studies addiction. And he thinks of addiction as being a sort of a extreme form of a natural process. The natural process involves things that cause one pattern of activation to produce another pattern of activation, which might be the action of doing something. Ultimately the act of reaching for that glass and guzzling another half a glass of beer in a single swallow. Okay. On the one hand, and the sort of experience that you have of what happened after you ingested that, which is what the standard theory says is why you did it. Okay. This impulse, the tendency to pick up that glass of beer and swallow the beer, is something that we can understand as happening through connections in a physical system that's wired up in a certain way to produce that effect. And, you know, I. There was a period in my life when I found myself in that very situation. Very frequently I would be like sitting there with this empty glass of wine and then some waiter would come and fill it up, and then it would be empty again, and then it would be filled, and then it would be empty again. And I was like, what's happening here? And I'm like, I'm not feeling good anymore. This isn't great. You know, I, like, can't even get my thoughts to come together in a coherent way. So something. Some process was occurring inside of me that just subconsciously caused me to pick up that glass and pour it into my throat, you know, So I had a very palpable, experiential kind of basis for kind of agreeing with this guy's theory, which also he's done a huge amount of careful experimental work to validate, right? And one of the things that you can show is that there's a part of the brain where you could destroy it, right? You could put an electrode in there and burn it out. Something we don't do in humans, but, you know, doing them with care. Under conditions that minimize the pain that the animal will experience. When you do that, we can perform these lesions and find out what their consequences are. And in this case, he found that there were. It turns out that rats love ice cream. You know, that everybody loves ice cream. Well, rats love ice cream, too. They will work really hard to get the ice cream. And if you give them ice cream, they'll sort of go, ah. You know, it's like they. You could see them sort of relax and sort of almost smile. And he uses those, and he measures physiological things that go along with that, which he thinks of as the liking response to that ice cream. The thing is, this little lesion that we made in the brain can take a rat that will work really hard to get ice cream and turn it into a rat that will never do anything to get any reward at all. It'll just sit there, even though if you put the ice cream in its mouth, it'll still go, ah, right? So that liking, that experiential pleasure that comes from the ice cream is not itself enough. The other system that controls the impulses is what actually gets the rat to the ice cream so that it can. So evolution has built particular structures into our neural networks that allow us to have the. That cause us to have these impulses that make it so that we don't die of starvation or thirst. But when we talk about our minds are imperfect, they often, you know, have deleterious consequences. And especially in addiction, the deleterious consequences are that these circuits run away. They get hijacked. The connections in them get Strengthened in such a way that the presentation of the stimulus causes an excessively strong impulse. Yet the pleasurable or liking aspects may no longer be even occurring at all, or you're definitely going to have that horrible hangover after you've drunk down that, you know, whole pint of whiskey that you just bought right at the corner store. So it's, it's. I find this research sort of really palpably connecting this notion that what impels what, what underlies the action that we actually take may be very, very different than the, you know, the consequence that actually ensues upon having undertaken that action. Yes. In. In that when things are working together in a constructive way, the impulses align with the subsequent liking. You know, the wanting and the liking are aligned, but they can get dissociated. And I think that this happens so too easily. Certainly addictive substances are, you know, contribute to this, but there are other sorts of impulses that. And other sorts of addictions that are much more purely behavioral, like addictions to engaging in sexual activities that can be entrenched, or addictions to gambling, which is, you know, the outcome of gambling is purely symbolic, but it's a whole pile of chips or dollars in the bank. Right. It's not that direct, but it, it. These things are things that we can work to understand as scientists and that if we do understand them better, they will enable us to be more able to figure out how to help people avoid getting stuck in those traps, helping ourselves learn to tell the waiter to take the wine glass away, you know, so you won't even be there for you to pick it up and guzzle it down. And so this is one of the reasons why I was actually so impelled to want to write this book with Rob. I, you know, that the ability to sort of communicate that there can be things that cause behavior and also cause experiential emotional reactions that are set up by experience, but that are not necessarily always constructive.

Gregory McNiff (69:05)

Yeah, that is fascinating. I'll tell you, just an aside. I don't think I'll look at drinking Coca Cola the same after reading your book. You have some nice case studies about that. I do want to move on to this notion of distributed representation. So far, we've been talking about single unit representations of neurons called localists. What's so powerful about distributed representations in terms of helping us process the complexity of the world? Gaurav, Maybe, if you want to fill that one.

Gaurav Suri (69:40)

Yeah, yeah, happy to do that. So so far we've been talking about a unit for Coke or a Unit for the letter H. Right. Or slash A. And we've been thinking of units corresponding to things in the world that we think about, and we imagine that each thing that we think about is represented by a unit. Now, this way of thinking has advantages because it can get us started in building neural networks, and it can sort of show us the power of the idea that activation is the currency of a neural network. And indeed, I don't think if someone had tried to teach me distributed representations to start with, I would have ever gotten into neural networks. I think this way of initially thinking that, oh, gee, one unit per idea is a really important step in understanding neural networks.

Jay McClellan (70:41)

And it was important for me and David as well. We started with that, too.

Gaurav Suri (70:45)

Yeah. But it turns out that there are some fundamental problems with having one unit representation a one thing in the world. The alternative idea is to have a pattern of activation, right? So let's say we have five units, okay? And one pattern is high, medium, medium, low, high. That's one pattern. Another pattern is high, high, high, medium, low. Right. So these are patterns of activation. And of course, I'm only doing high, medium, low. But they're not discrete. They can be anything. But you can imagine you can have these patterns that are corresponding or representing to ideas in the world rather than one unit. The problem with having one unit is, let's say I have a unit for Coke, Greg, and that unit somehow gets destroyed. Does that mean that everything related to Coke is gone? So that's one problem. Another problem is that it seems to me that the brain should represent Coke and Pepsi in somewhat similar ways compared to, say, representing a Coke and a hummingbird. Right? And the idea of patterns is that similar things can have similar patterns. And this turns out to be extraordinarily important because how we understand how we make concepts in the world depends on us generalizing from instances. So the pattern, the concept of three, the number three. Right. So I've seen three trees, and my mom's pointed out three ducks, and I've seen three people by the store. And you can sort of imagine that this concept of three is arising from these concrete experiences involving the quantity three. And the distributed representations allow, with their intersections, this concept to arise. And that is their power. And they're really central to artificial intelligence. And they're also central to why we do the things that we do. Right? So if. If I'm drinking a lot of red wine and a Merlot pattern and a syrup pattern, and its effects on me are similar, right? So even in very simple organisms, it's not necessarily one neuron, for one thing, but many neurons activity activation across many neurons to represent that thing. That's a distributed. That's called a distributed representation.

Gregory McNiff (73:31)

Perfect. I have two follow ups there, Gaurav. One is, you note that distributed representations can also point a way to capture the hierarchical relationships between categories. Briefly, what do you mean by that? And then I want to talk about error correction learnings, which seems so important for both our mind and how we approach AI or how AI is modeled.

Gaurav Suri (73:52)

Yeah. So hierarchical ideas. Right. So let's say I see a dog, and I have a representation for a. For a dog, and then I see a cat. And initially, by the way, kids will call these animals dogs, and then they. Then they respond to the differences and they. They make this separation. But I start to get these patterns for things like dogs and cats and horses and. And these, These enable me, if I'm exposed to all of them together, I get a category of animals starting to emerge that are quite different from these other things that are starting to emerge, like, oh, those roses and those trees. Those things seem to have different patterns. And maybe I can't distinguish yet between a tuna fish and a salmon, but I can have this idea of fish. And it's not a logical structure. If it has a tail or if it has fins, it's a fish. If it has a tail, it's an animal. It's a clustering of properties that are going off and going together that allows these categories to form. So the categories of furniture or the categories of animals or categories of plants, these are formed by the common characteristics of our individual instances, and that's enabled by distributed representations. Right. So this is what I meant by concept. It would be very hard if we couldn't sort of see patterns to be similar or different from each other, to come up with concepts and make hierarchies and taxonomies in the world, which we think are logical. Right. So all animals move. Right. We think that there's a hierarchical logic structure, but the point of the neural network and distributed representation in particular, is that that logic structure emerges.

Gregory McNiff (75:49)

Yeah, no, that is fantastic description. Moving to error correction, Gaurav, I just want to read you the quote here, and maybe you could expand on it. The key takeaway is that error correction learning provides a viable way to produce distributed representations in a neural network that are similar across items belonging to the same class and different items belonging to different classes. Distributed representations will able a lot of progress related to the nature of understanding. Good for humans and AI. Could you just briefly Talk on why error correction learning is so important.

Gaurav Suri (76:23)

Yeah. So if we have localist representations, it's very possible to imagine simple rules that make connections between the letter, the shape of the letter H and the sound of the letter H. Right. So I can connect these things. I can conceptually understand what these connections are. I can maybe make local rules that if both these things are active together, if I see you and I see your name, I associate the name Greg with your face. And one can imagine these kind of rules. If one is thinking in a localist way, when I was thinking distributed way, one is confronted with the problem of having a population of neurons connect to another population of neurons that are representing these ideas. Now, this is, in general, a learning problem. Making connections or changing connections is a learning problem. How does a neural network learn? Error correction is one way for neural networks to learn. I want to emphasize this. It's one way. It's not necessarily the way that only occurs in humans. It certainly dominates what occurs in AI. But it's one way for these representations to emerge, for us to learn the properties that make these representations. And the idea of error correction is you give a network an input and you look at its output, and you know what the output should be. You know the correct answer, and you start with some random small connections that initially give you kind of random answers all over the place. But what you do every time is you guide the difference between the actual answer and the current answer to adjust the connections so that the network is more likely to give the correct answer. Right.

Gregory McNiff (78:26)

That.

Gaurav Suri (78:27)

That is the essence of error correction learning. And it's this idea that I, for a set of things, I know the right answer, and I can look at what the output of the network is. For a certain input for which I know the right expected output, I can measure the difference between the expected output and the network's output, and I can use that error to change the connection weights so it makes the network more likely to give me the correct output. And doing so is learning, right, Because I'm changing the connections.

Gregory McNiff (79:03)

Yeah, that's fantastic. And that's a nice segue. I want to move to AI and I, Jay, I want to turn this back to you because a recurring theme, and candidly, a sentence that just stood out in your book is this notion of, quote, we learn at the level of thought. What does that mean?

Jay McClellan (79:21)

In order to do so, I'm going to talk about a man whose name was Frank Rosenblatt, who was a psychologist, but he had this idea about error correcting learning as A way to program a computer to assign the correct label to images, the correct labels to images. And you know, he simplified this in a way that allowed him to understand it using initially physical hardware that implemented the units and the connections between them. But finally he got access to a computer in the late 50s where he could, he could actually run a computer simulation of this idea. But his basic idea was, hey, look, if I have a pattern of activation that represents a particular image on, on an input like a, like a, in a photograph, right? So you, you've got, you can imagine breaking the photograph down into some number of little pixel like elements, just dots, right? Is this dot black or white? And Rosenblatt's model becomes the, that the digit classification neural networks of the, you know, 20 years later that people began to study where you asked the question, if I've got a digitized image of somebody's handwritten character, can I say what character that is? You can't write rules that tell you the answer to that question, right? There are fuzzy boundaries between these letter categories that if you just tried to write rules for classifying actual handwritten digits, you'd maybe get 80% correct. But that's not very good, right? You can't reliably recognize what somebody wrote on a check or a zip code they wrote on an envelope from that kind of rule. So Rosenblatt's idea was fundamental to this, right? So you have this input, in Rosenblatt's case, he mapped it through some fixed pre wired intermediate level neurons that had wires established by him in advance, and then he allowed those to connect to output neurons representing the different possible characters, right? And so he could say, okay, I'm going to present this little bitmap of a particular character, like a seven, and I'm going to tell the network that its output should be 7 and not any of the other possible digits. And you present the input, you use the existing connections to produce this intermediate level of activation. Then you use the connections forward from there to produce the input to all the different possible categories. And if you were going to score the network's behavior, you would say, which one is more activated? Did it get the right answer by activating the correct one? But it's actually a matter of degree, right? All these different ones are getting these inputs. If you say this should have been a seven, then you can use the error correcting learning rule that Rosenblatt described to adjust the connections from that intermediate layer to the output to make the correct answer more likely. And that was something that Rosenblatt had this insight into. In the 1950s, he thought he could get a long way with this, and ultimately he was right. But the computing resources available to him in the 1950s were nine orders of magnitude less powerful than the ones we have today. Right. Computing power increases a thousandfold every 20 years. That's my sort of condensation of Moore's Law. So 60 years later, we have a thousand times, a thousand times, a thousand faster computing. We could do so much more computing. And. But Rosenblatt, you know, it was a very simple idea. He could actually start to explore it. So it wasn't until 2012 that we got to the point where these networks could be deep enough, they could have enough layers that we could use Rosenblatt's fundamental idea to train the neural network so that it could learn not only. And this was the incredible beauty of Rumalhart's contribution to artificial intelligence, we could not only learn how to adjust the connection weights from that last layer of preformed neurons to the output, we could actually learn the connections coming from the input through all those intermediate layers.

Gaurav Suri (84:29)

So.

Jay McClellan (84:29)

That not only would the network learn to map patterns to the output, but it would learn to map inputs to the intermediate layer that would be useful for getting the right output. Okay, but notice what's going on here. We're training connections between units, each of which is a tiny component of a thought or a representation. You've got a bitmap pattern of zeros and ones representing which pixels are black and white in the input. Then you've got graded connection weights to a first layer, and then another layer and then another, and then an output layer that Rumelhart's extension of Rosenblatt's algorithm was able to train so as to allow people like Yann Lecun and others to, you know, get very high accuracy in classifying handwritten digits. Right. But by 2012, Hinton and his students were able to get that extensions of that idea to classify images in ways that nobody else could achieve. And, you know, handwritten digits are one thing, but recognizing that this actual photograph has a cat rather than a dog, you know, that's a hard problem. And it was. Suddenly there was a step increment in the ability to do that in 2012 that, you know, launched the AI wave I think of today, or it marked at least a very important milestone in this. And, you know, the graphics processing unit that Jensen Huang sort of insisted on inventing was part of what allowed that to happen. Right. And it's so central to everything that's going on and extensions of his hardware that are allowing all of this AI to occur. But this. This process is happening at the level of, how do I adjust all the connections throughout this very, very deep network so as to make it a tiny bit more likely that I'm going to call this a cat than a dog, you know, And. And in order to train these models, you have to present massive numbers of examples that cover the space of possibilities so thoroughly that there can't be any. The network can't cheat. Right. The networks are. These artificial neural networks are incredibly likely to cheat, find shortcuts any way they can, exploit some subtle correlation in the data that isn't really what you want them to get is. So engineering is of these things is all involved in, like, making sure that the patterns that they're trained on sort of are capturing the correct statistics and not having misleading statistics, because it's only those statistics that are driving these connections.

Gregory McNiff (87:32)

Yeah, no, that's a fantastic explanation, Gaurav. I want to go back to you to build on that. Jay's talked about Rosenblatt, Rummel, Hart, Donald Hebb. Another key individual is Jeffrey Elman and his, I think, discovery or conception of hidden networks. And in the book you write, modern deep learning networks can sometimes have as many as a thousand hidden layers, and each layer might have hundreds or thousands of units. It seems like that was another real advancement with this notion of feedforward connections. Could you talk about why hidden layers are so important to neural networks?

Gaurav Suri (88:11)

Yeah, I think Jay should talk about Jeffrey Elman, but I'm happy to talk about why hidden layers are so important. Imagine a world that we only have an input layer and an output layer. This kind of world will allow us to make relationships, input output relationships that are linear. Right. And in the book, we have this example of wines and, say, two categories of red wine and two categories of white wine. And we can sort of make a connection about, oh, these types of wines, say, red wines, are more expensive or less expensive. Now, it may be, for example, that the relationships that we want to capture are more complex than these straightforward linear relationships, such as what characterizes, say, height and weight. It may be that we have things like a cabernet, if it's a plummy is really expensive, but if it's apricot, it's not. And a white wine that's plummy is really. Nobody wants that. But apricot, that goes. So this is a second order relation. Relationship. And what we show in the book is that you can't capture these relationships unless you have These hidden layers, because they allow for second order variables between the input, second order relationships between the input and the output to be captured, which would be impossible to capture in just a straightforward network. Now, interestingly, biologically, there is a very parallel set of layers that the brain uses. So when we process vision, it goes through these successive layers that are integrating features from diff. From earlier layers, which are simpler. And we may have the first layer detecting very simple features of things like straight edges and maybe darkness and lightness or a particular pattern. But as multiple simple layers are aggregated in the next hidden layer, we're able to now represent more complex things in our visual field. And so we literally describe the visual system as having V1 and V2 and V3, et cetera, which do in some sense correspond to this notion of hidden layer big picture. The hidden layers allow us to capture complexity, infinite complexity. If the structure is there, you can theoretically capture it with having hidden layers. And more hidden layers allow ones to capture a greater level of complexity. But that first hidden layer is particularly important because for me, it easily shows that, oh, there's some relationships that I just couldn't capture in a two layer network, and you can in a hidden layer. Jeffrey Elman was Jay's colleague, so I think Jay should talk about him.

Jay McClellan (91:40)

Jay, the question then arises, well, okay, exactly how does that work? If you've got this multilayer network, you've had inputs, you propagate activations forward through connections to one layer and then from there forward to the next, and so on through tens or even hundreds of layers in many neural networks, how do you decide how to change the connections from, let's say, the input to the very first layer of that network so that it's doing a better job of encoding that input and making it so that the output will be what you want? And this is the problem that's solved by the backpropagation algorithm that Rumelhart invented with input and some seminal kind of focusing thoughts from Jeff Hinson and with Jeff, then building on that and using it to really show how it could be used in AI. So the key idea is that the way Rumalhart thought of it is that, you know, we know directly at the output layer that we wanted that seven to be activated and it wasn't as strongly activated as it might have been. So we can say, okay, well, I'm going to strengthen the connection from all the units at the intermediate layer that are strongly activated now to make it so that they activate seven and I'm going to make it so that those strongly activated ones are less likely to activate other things, because those are the wrong answers. So you're going to use the error on the output side and the activation on the sending side as, like how to adjust that connection length. So if the sending unit's active, then it's contributing a lot. So we're going to make an adjustment to that unit's output, and that'll be very helpful. The flip side of this is that if the neuron already has a strong connection to the output, then that would be a good neuron candidate to change how strongly it's activated to help solve the problem. Okay, so what his key idea was was let's use the existing values of the connection weights to figure out which sending units we should think get responsibility for this error. And so, for example, if two units are in an intermediate layer and one is more strongly activated by the current input than the other, let's say they're equally activated. I'm sorry, so we got two inputs that are equally activated, but one has already got a strong connection that will help reduce the error. Then we could say, oh, let's encourage that unit to be even more strongly activated because that will allow it to exploit the connection that it already has. So that's the essential idea. And what Romelhart was able to do was to write a little piece of computer code that would iteratively apply this idea going backwards through a network whose activations were produced by going forward through the network. So activation flows forward through the network, and in Roma Heart's way of thinking, the error signals flow backwards through the network. And at each layer, you have like, okay, the. These units are the ones that are going to help reduce the error if I make them more active, which sending units have the activity that will support that? And then you essentially have this simple rule of, okay, at each layer, we make the adjustment that we make proportional to the error at the output times the activation at the sender. And that is, he called it the generalized error correcting learning rule that applies to all the levels of the network. And it essentially works by sending error signals backward through connection weights and activation signals forward and then adjusting the connections by taking both of those things into account. So, you know, in 2012, Hinson and his students used this, Rumelhart's algorithm, together with a few innovations in some of the details of the neural networks and the GPU that Jensen Huang had invented and built this system that, you know, effectively showed a dramatic improvement in the ability to classify images.

Gregory McNiff (97:10)

Yeah, just to clarify, Jay Would you say back propagation? It enables systems to learn by how it processes. Depends on what it has previously learned. Or said another way, it's a recursive system that keeps iterating to achieve the correct answer. Yeah, element to the back propagation.

Jay McClellan (97:31)

So, so this is really very central to my way of thinking about how neural networks learn. And I think I. I think it's extremely useful in thinking about how we learn. So I've been motivated to think about this in it, given my interest in how humans learn. Because I've been struck over the years by a very important actual phenomenon in child development, which is that, you know, when children are of the right age or have they reached the right stage of maybe prior experience setting them up, they're able to learn new things quickly, but before that they can't learn this stuff at all. And the same thing applies to me when I'm learning a new domain or when I'm trying to teach students about neural networks or high school students about trigonometry. If they don't have the prior way of representing the information that puts them in a place where they're ready to learn the next bit, they're not going to be able to benefit from the conversation. And so this is something that we are able to study in artificial neural networks. So if you take an artificial neural network, even one with just one input layer, one hidden layer, and one output layer, and you initialize that with small random connection weights, the connection weights don't propagate activation signals forward very well, and they also don't propagate air signals backward very well. So what happens is that at first, learning doesn't really seem to be happening at all. You can go through, you have a training set of input output things that you want the network to learn, and it exhibits these long plateaus where nothing seems to be happening at all. In fact, when Rumelhart first tried to get the algorithm to work, it didn't seem to be making progress. And that was because it was on one of these long plateaus. And so he said, oh, maybe it doesn't work, I have to. He came back next time he got a slightly faster computer, and he let it run for longer and it finally converged. But the thing is that, as we were saying before, the error signals that tell the input units how to change their connections to the hidden layer are propagated backward to the hidden layer by the connection weights from the hidden layer to the output. So if those connections from the hidden layer to the output are small and random, they don't send a very strong signal. And if the input weights are very small and random, they don't give it anything to work with. And so it's only as these things begin to build up that they begin to be able to benefit from both the forward propagation and the backward propagation. And so there's this incredible reciprocal process whereby as the adjustments start to take hold, they make the subsequent adjustments have a greater impact. And you get this acceleration of learning, and you get these stages of developmental transitions that are. Have struck, you know, child development researchers like, oh, at a certain moment, you know, you can. Children seem to suddenly know something they didn't know at an earlier age. And we can. We can begin to capture how that can happen in this sort of like earlier, they're just building this stuff up gradually in the background. And then once it's reached that stage, few additional experiences can push them over the edge and result in this new ability to learn.

Gregory McNiff (101:36)

That's a great point, Jay. And is that related? You have a great chart. It's figure eight, three in your book. It's an inflection point representing scaling improvement. And you write, over the course of human evolution, the brain areas which have improved the most are those that sit between sensory input and motor output systems, consistent with the view that at least in part, a relative advantage is simply a matter of scaling up. Throughout this entire conversation, we've talked about, is it linear stair step? What's going on there? I mean, like you just said, is it just, you know, one day the light. The light turns on and all of a sudden we get it? I mean, that chart was really dramatic. I mean, it's like almost parabolic on the x axis there. What, what's going on with the human brain that it really just dramatically improves like that?

Jay McClellan (102:24)

You know, it's. It's something that we should be humble about and not say we fully understand. But there really is this very stark distinction between two kinds of ideas about what differentiates the human mind from other. The minds of other animals. And one view, you know, and I think Chomsky is very strongly associated with this, is that there was a very special thing that happened by accident some hundred or maybe 150,000 years ago. It was a random mutation that made it so that the mind could engage in symbolic thought. And that's what separates the human from the non human animal. That is Chomsky's sort of oversimplification perhaps of Chomsky's view, but that's kind of the way it comes across. And something that I'VE always felt, well, I think it's continuous. So let's think about an alternative way of thinking. So instead, a other idea is that the difference between humans and our closest animal relatives isn't this moment of insight, but it's something that separates our closest relatives from others and so on. And that it's this matter of degree, of the expansion of the network that sits between the immediate sensory input and the exact details of the motor output that manipulate your finger to press a key or to grab a particular raisin off of a tree or something. And, you know, allows for this greater complexity. And once you, once you have this kind of thing, there's these exponential kinds of effects that you're talking about are things that just, they emerge. I guess it's a way of putting it from what happens when you increase the scale. So there's some fairly simple versions of this that perhaps can be communicated quickly and intuitively. But, you know, one, one way of thinking about it is that if something depends on eight things, each of which all start out as very, very small, then all eight of them need to build up to a point where they. Because if any one of them is zero, it all comes to zero, right? So if to be able to do something requires the product of eight graded things, you know, any one of them could be the weight link in the chain. And so the, the more complex, the slower the process will tend to be. And this was a huge frustration for people. Like, they believed that deeper networks were important, but they didn't have the computing yet to get there before 2012, you know, because, because those complexities couldn't be captured in the simpler networks. So it turns out that there are some ways of ameliorating those complexities. So they're not as dramatically different as that. But the view is now gaining traction and it's always been there. It's always been an alternative. There are evolutionary and comparative neuroscientists who have been thinking about this for many, many years, and where the key difference between our most capable biological organisms, namely humans and our predecessors, is really the scale that allows for greater complexity to emerge.

Gregory McNiff (106:52)

That's a great answer. Gaurav. I want to go back to you. In this notion of large language models, you write, just as human thought is an emergent consequence of a distributed interactive activation process in a biological neural network, and LLM capabilities are an emergent consequence of a distributed interactive process within an artificial neural network, as you note, these LLMs are achieving and surpassing human abilities and exhibit human like thinking qualities. What Are they? And why are they so powerful?

Gaurav Suri (107:25)

Yeah, so there's a very basic things to say to start with, which is large language models are neural networks, right? And they are, they have units and they have connections between units. It's easy to lose track of it because when people talk about hallucinations and so forth, they think it's some all knowing repository of facts. It's not a repository of facts, it's a neural network. Now it's a particular kind of neural network. So we've been talking about neural networks where you give it some input and it comes up with an output. So you can give it a handwritten three and it says, oh, this is the number three. That's an example of an input output relationship that's learned by a neural network. Now what large language models are in the business of doing is predicting the next word from a prior context, right? So if, if I say the sentence the boy jumped into the. There's a variety of next words that are possible and some words are very unlikely. So the word like broccoli is, is extremely unlikely and the word pool is much more likely. So what the large language model is trying to do is it's trying to come up with what are the most probable outputs given my input. Now it turns out that there's a very nice labeled set of data which are basically all the words that humans wrote on the Internet, for example, that can be used to make this input, to have large language models learn these input output relationships. And what you do is you take a sentence and you have one part of the sentence as the input and you have the large language model predict the next word, which you already know, right? And so you can, you can therefore use error correction methods to start training the large language models. Now there are some complexities that come up that are really worth underlining. First, if I say to watch tv, Greg sat on the couch, the word couch should be roughly equally as likely as sofa. And you know what? Distributed representations do that work for us because the distributed representation of the word sofa is not that different from the distributed representation of the word couch. So there's that really important piece that is distributed representations are provided in this context. They're called embeddings. So they're being provided by how these things learn about words. The second technicality is that many words, most words in the English language have multiple meanings, right? So the word bark, for example, could refer to the bark of a tree or the sound of a dog, right? Now let me, let's. So this is a Sentence that we have in our book, it says, upon hearing the doorbell, Amy's Labrador began to bark. Now, the question is, is this bark here referring to the bark of a dog or the bark of a tree? Well, the context of the sentence, and this connects to the context idea that we were talking about earlier. The context is suggesting it's a dog. Well, what is the context? Well, doorbell is kind of the context. It's someone is hearing the doorbell. Amy's dog. We don't usually talk about Amy's or Labrador. Amy. It could be a province in Canada, but could also be a type of dog. And it began to bark. So what kinds of barks do dogs have? If we've sorted out that we're talking about the bark of a dog and not the bark of a tree, that context makes it much more likely that the next word is going to be loudly rather than the bark fell off the tree or something. So the initial problem is, oh, how do I predict the next word? One of the complicating problems is that the particular word to use is dependent on the context that we find ourselves in. Now, there's a innovation that large language models do. This is called the attention mechanism. And what. What it uses is it finds words that are most likely to tell us the sense in which we mean bark in our example. So in. In this sentence, the words doorbell and the words Labrador strongly bias us towards thinking of the bark as the sound that the dog makes rather than the bark of a tree. And this innovation goes beyond error correction, Right? So this is about having a mechanism that allows us to focus on the words that disambiguate the meaning of the word that we're concerned with right now. And this innovation was actually what we go into in the book is it's not a conceptually, it's not a complicated innovation. It involves the representation of questions that words might ask of other words. These are called query vectors. And a vector is best thought of akin to a pattern of activation. And a key vector is it's providing information to incoming questions, right? So the word doorbell might say yes to questions about houses and doors. The word Labrador might say yes to questions about Canada or to the Atlantic or to dog breeds. Now, meantime, the query vector is saying things like, hey, anything about ringing, because I have doorbells here, anything about ringing, anything about visitors, anything about dog barking. So there is a query vector that's shopping for answers. And this is done by pattern matching to disambiguate which sense of bark do I mean? And these two innovations the main engine being error correction. Learning across several layers and this attention mechanism were pretty foundational in large language models. But I want to emphasize that today's large language model, what I've described here, are sort of the core of foundation models. But large language models have a lot of human and machine training that accompanies them. So there's a lot of human intervention in making the abilities of a large language model possible. There's a large language model that recently did really well at the Math Olympia. When I was a kid, I was thrilled to get a score of 33 on this same test or a very similar test. And this large language model got 80 something, I think. And I looked at the question, I went back and looked at the questions, and the questions were not trivial. Right. But it turns out that there's a lot of specific data that was fed by a lot of trained mathematicians for these large language models to do the things that they do. Yes, their properties are emergent. This connects with what Jay was talking about earlier, that if you have smaller models with smaller number of parameters or connections, they're not going to show the same level of cognition thought that smaller models do. And it's a hockey stick. This is what you were talking about, Greg. But, yes, there are emergent things, but there's also a lot of reinforcement. This is the right kind of answer that make today's large language model possible. And we can talk about what kinds of things they do well versus what they don't. But at their core, their neural networks learn to predict next word with an attention mechanism and a lot of human training.

Gregory McNiff (116:15)

That's an awesome definition. You nailed it, Gaurav. Just one quick follow up. You do talk about Thorndike's notion of reinforcement learning. How does that play into the LLMs?

Gaurav Suri (116:25)

Yeah. Right. So we get into that to describe something called reinforcement learning. And Thorndike, the experiments he did were he put a cat in a sort of a cage, and the cat could take various actions like thrash around and press this and move that, and eventually, initially, maybe just by accident, the cat would get out of the cage. Right. And you would think that if it was completely accident, the next time or the fifth time or the tenth time should take equally long, but it doesn't. There's a reduction in the time it takes the cat to get out. And so one idea is that the actions that are producing a desired outcome get reinforced. And this is core to this notion of reinforcement learning. And reinforcement learning is used in large language models both to train the kind of responses that human modelers want the LLM to give and also to use machine data to kind of train the responses that are expected. But I want to emphasize the core of these large language models is still error correction, still backpropagation. It's supplemented with reinforcement learning. Thorndike's contribution was that it made us think about how reward can really lead to learning. Rewards and punishments could really lead to learning. And some algorithmic aspects of that are captured in modern reinforcement learning.

Gregory McNiff (118:04)

Perfect. The last part of the book, you talk about how a better understanding of neural networks can give us a better understanding of our mind and even improve our mindset. You talk about the notion of kindness, Jay. Can you maybe expand on that? How can us understanding neural networks better improve our mindset and notion of how the mind works?

Jay McClellan (118:29)

So I think the first point to make here is that one that we touched on already, which is that we don't have full visibility on what it is that is leading to our reactions, our perceptions, our emotional responses to things and the actions that we take. And neither do we have full visibility on the factors that are leading to the actions and responses of others. And, you know, so society, tradition and principles and precepts that were taught through religious practices and other cultural inventions.

Gaurav Suri (119:21)

Are.

Jay McClellan (119:22)

Things that we use to cast judgment on each other and ourselves all the time. And what I feel, you know, is. Is missing from that. And Rob and I deeply share this, is this notion that there really is a multiplicity of influences that simultaneously can conspire to give rise to whatever happens in a given moment. When a person has an emotional outburst, you know, how should we think about that? Should we attribute that to their inherent sort of impulsivity, or should we perhaps wonder if there was something that set that up? And, you know, I have to say that when I think about differences between politicians in the world today, I think, oh, I wonder what kind of an upbringing that particular politician might have had that had that led them to respond in the way that they do to these situations that's so different from the way that other politician responds to those situations. So I actually deeply strive to avoid passing judgment on the individual. The behavior could be reprehensible and could require us to intervene because as a society, it's undesirable. But the individual's tendency to produce that behavior is the consequence of all of their prior experience that gave rise to all of the connection weights in all of their neural networks, together with the factors in play in the current moment and the recent, you know, situation that they've been in and our way forward as a society is to ask, you know, how can we do more to create the situations in which our reactions are constructive, both for ourselves and for others, and to make it so that we give everybody the best opportunity to actualize their better nature as opposed to, you know, placing them in situations that are intrinsically polarizing and stress based, kind of inducing of reactions that are defensive and therefore potentially aggressive, as opposed to trusting and therefore constructive. And it's through this process of knowing that we're not, you know, the product of some dictum that thou shalt not kill, but that we are. We are striving to find ways of living constructively with each other and that we will benefit from knowing as much as we can about the actual biology of our brains as we try to figure out how to make it so that we can encourage the most constructive responses and avoid channeling things towards more destructive tendencies. So this is what I think this notion of kindness, reframing and so on is about. But of course, it's also, and this is something that I want to pick up on in relation to our language models. You know, we shape each other's behavior through how we encourage certain kinds of attitudes and reactions. We not only model the reactions and so we learn by imitation, but we also reinforce them by saying, oh, what a good boy you are to think that the right thing to do is to kill the infidels that are, you know, as opposed to, you know, have a more constructive response to the fact that their environment was different and they don't believe what you do because they were raised differently. So I hope I'm conveying what I think is the essence of this, which is that by understanding what evolution has provided us with and how experience works together with that to shape who we turn out to be, we will be best at. And by working together collectively so we know what each other's needs are, we'll be in the best position to shape our own world with each other. But we are like our neural networks, and they are like us in the sense that we need oversight, we need guidance, we need acculturation, and we need the community to come together and provide the context in which we and our artificial neural networks will be nurtured together towards these better outcomes that we seek.

Gregory McNiff (125:07)

It's a wonderful vision. I feel like certainly this book has made a contribution to us. Getting there. Last question, and I'll ask you both. And Gaurav, I'll start with you. You argue throughout the book that neural networks have the ability to Think logically, pursue goals, and even potentially demonstrate consciousness. I realize consciousness is still in the evolving category in terms of how we understand it, but it seems like it's only a matter of time before an AI program is indistinguishable from a human. And in the last chapter, you give a number of ways in which our biology could help augment and improve AI. Is there anything you think AI won't be able to do in terms of how we think, how we feel, how we process?

Gaurav Suri (125:52)

So I resonate, Greg, that there is no reason to think that an AI system would be distinguishable. Like this is the essence of the Turing Test, right? So Turing said if you want to know that a machine can think or not, one of the tests devised he has proposed was have a conversation with it without knowing whether the responses are coming from a human or from a machine. And if it's statistically, you can't really tell a machine from a human, then, well, you got to acknowledge that the intelligence exists in the machine. And large language models, of course, have made giant strides in passing this kind of test. Now, I want to build for a second on what Jay was saying, which is these machines are intelligences amongst us, right? So we've now got to live with these machines. And I think it's really helpful to think about intelligence as a process in the universe. Photosynthesis is a process. Water going downhill is a process. Intelligence, the emergence of intelligence that we're arguing is a process that we can mechanistically understand. And the consequences of that are my process and Jay's process and Greg, your process share, because of our common biological heritage, share a lot of commonalities. Neural networks are also processes. I mean, artificial neural networks are also processes. Artificial systems are also processes that share some aspects with us and don't share others. What can't they do? Well, what they're doing right now is enabled by mechanisms very different from mechanisms that enable us. We learn from orders of magnitude less data than they do. We use orders of magnitude less energy than they do. We use learning algorithms that seem to not be reliant on this propagation of error. The same channels that propagate activation forward are not the ones that are propagating error backwards, suggesting that backpropagation in its classical form is not working in the human sphere. We have goals, right? An artificial system doesn't have autonomous goals. Our goals come from our bodies. And these are all deep, essential differences. In some domains, it seems like large language models and human cognition are very interestingly similar and often Even indistinguishable in some domains, large language models are exceeding or poised to exceed human cognition. And there are also fundamental differences. But the one that I want to underline here is consciousness. What Jay and I are arguing in this book is, well, look how far we've come without assuming pleasure or pain, right? We didn't. Pleasure and pain are conscious experiences. We haven't assumed this conscious facility to look at something with a tail and say, oh, that's an animal. We have suggested that these emerge from activations. Now it's undoubted that consciousness also. Well, we also have consciousness. That's not debatable. It's possible that it emerges from these same activations. I think that's an interesting rich view. Now, it may require a biological substrate. We don't know. We don't exactly know how consciousness comes about. The contribution that we're making is that many aspects of the mind can be understood without resorting to consciousness. The role that consciousness plays, whether consciousness requires a biological substrate or could it emerge in non biological substrates? These are open questions and I hesitate to predict anything about them. I do think that in many spheres of the mind, there is no principled reason why artificial systems won't meet and exceed human cognition. The challenges for us to put in guards, safeguards, just as we put for each other, put those on our machines so that these machines can enable our journey as a species to thrive in the coming decades.

Gregory McNiff (130:39)

Yeah, no, I think that was a really thoughtful response and you articulate it well here and in the book. Jay, I want to give you the last word. Clearly you've thought about these issues. How do you see AI developing? Are biology supporting that development? And at some point are they going to be. Is AI going to be indistinguishable from human intelligence thought?

Jay McClellan (131:01)

I think that AI is essentially like any other technology. Airplanes have not been designed to be exactly like birds. They've been designed to serve purposes for people which are very different from the purposes of the birds. And you know, the birds are these autonomous things that have emerged in nature and obey certain physical principles that have been exploited in the design of airplanes, but used for. For very different purposes. And of course, when we design an airplane, we design it within limits. We try to ensure that all kinds of safety mechanisms are in place because of the potential catastrophes that occur if one of these great big monstrous airplanes goes awry. And, you know, I think we should understand this distinction, right, that AI is designed for human purposes and may not be brain like in all ways, but it may well exploit a lot of similar features to what we exploit ourselves. And I think one of the very central issues here for me is the issue of goals, which Rob alluded to. And I would like to end what I have to say here on that note. I feel like evolution provided biological organisms with mechanisms that allow them to organize their behavior in service of goals that are internal to themselves. These goals are not, in my view, strictly determined by biological needs. They're actually, in humans especially, very strongly influenced by social needs. And that, you know, the need for the joy that comes from acceptance, social acceptance, and the despair that comes from rejection are also fundamental. And they're part of what evolution provided us with. And so, you know, we have to understand our own responses to situations as being responses that are based on goals and needs that evolution prepared us for, and then experience further shaped. And, you know, the, the notion of autonomous goals that aren't themselves innately pre specified is, is, is huge. And I don't fully understand it myself, but, you know, when, when a leader like Mahatma Gandhi realizes that, you know, the goal is to create a society where those who've come from these alternative traditions can live together in harmony, you know, this is not an intrinsically biological goal, right? And so yet it's something that emerged in the human mind and is something that, you know, I myself would love to see us constructively figure out how to encourage further in, in ourselves and others. And this is a place where the role of society and of our collective engagement with each other in shaping how we all think in such a way as to move together in the most harmonious, harmonious possible way is really the crux of the matter. AI systems are going to need to have a certain sense of autonomous goal directedness in order to be the most effective at meeting the needs that we want them to serve for us. But at the same time, like ourselves, these goal directed systems can easily be hijacked by, for purposes other than those that are necessarily socially constructive. And this is where we, we need to be as informed as we can and come together as a community to help us progress together in a world where not only our machines, but also ourselves are sort of part of a collective interactive activation process where we're all shaping each other all the time in the direction of more constructive and useful productive outcomes.

Gregory McNiff (136:01)

That's a great analogy and a great closing answer, Jay. Like I said, I think the book that you and Gaurav have here is really a great step in achieving that vision. That concludes our interview. Again, the book is the Emergent How Intelligence Arises in People and Machines by Gurav suri and Jay McClellan. To the listeners, I can't stress how thoughtful this book is. Believe it or not, we really only touched on some of the key themes in this conversation, but it is really one of those rare books that will enlighten your understanding of the subject matter, but also yourself. Guravanjay, thank you so much for your time and thank you for writing such a thought for provoking book.

Gaurav Suri (136:39)

I really enjoyed the conversation. Greg, thank you so much.

Jay McClellan (136:43)

Thank you. Your engagement is extremely valuable to us and to the communication of our ideas. I really appreciate.

Gregory McNiff (136:53)

Likewise. Thank you both.