← Infinite Loops
Infinite Loops
Roon — On Shape Rotators, AGI & Tenet (Infinite Loops CLASSICS)
01:06:17
Hello everyone, Jim here. We're taking a brief two-week break from new episodes to spotlight a couple of golden oldies from the Infinite Loops archive. Years later, these remain some of my favorite conversations. We’ll be back soon with fresh...
Loading summary
Transcript94 lines
- [00:01]
A
Hey, everyone. Jim here with a quick note. We're taking a brief two week break from new episodes to spotlight a couple of golden oldies from the Infinite Loops archive.
- [00:13]
B
Years later, these remain some of my favorite conversations I've had on the podcast.
- [00:19]
A
We'll be back with fresh episodes soon.
- [00:21]
B
But in the meantime, enjoy this trip through time.
- [00:26]
A
Thank you.
- [00:29]
B
Hi, I'm Jim o' Shaughnessy and welcome to Infinite Loops. Sometimes we get caught up in what feel like Infinite Loops when trying to figure things out. Markets go up and down, research is presented and then refuted, and we find ourselves right back where we started. The goal of this podcast is to learn how we can reset our thinking on issues that hopefully leaves us with a better understanding as to why we think the way we think and how we might be able to change that. To avoid going in Infinite Loops of Thought, we hope to offer our listeners a fresh perspective on a variety of issues and look at them through a multifaceted lens, including history, philosophy, art, science, linguistics, and yes, also through quantitative analysis. And through these discussions, help you not only become a better investor, but also become a more nuanced thinker. With each episode, we hope to bring you along with us as we learn together. Thanks for joining us. Now please enjoy this episode of Infinite Loops.
- [01:42]
A
Well, hello everyone. It's Jim o' Shaughnessy with yet another Infinite Loop. I have been waiting for this one for quite some time. Rune, who is one of my favorite accounts on Twitter, who is an AI expert, an engineer, and a techno optimist like me, is joining me as my guest. Thank you so much for coming on.
- [02:03]
C
Pleasure to be here, Jim.
- [02:05]
A
The first question I've got to ask is, did you plan on sort of taking Twitter by storm and getting to be the birther of the word cell shape rotators dichotomy?
- [02:17]
C
I had no idea that stupid joke would turn into such a cult phenomenon. People are still saying it today. Like, I search it sometimes time to time to see if the meme is dead or not. People are still saying it. I was posting on Twitter for an audience of like 30, and I was doing it with the same vigor I was doing it today. I just needed somewhere to write down my stuff and the fact that it popped off, that's my luck. I definitely didn't plan on becoming super big on Twitter or anything like that.
- [02:48]
A
So one of the things that strikes me about it is it was sort of pitch perfect for what a lot of people maybe had running around in their minds, but they hadn't put a name on it. And I love it because, and I DM'd you last night that the joke that I made was, hey, hey, I called myself a symbol manipulator back, you know, in 2012. But maybe that's it because I just really enjoyed it. So if you don't mind to revisit it a little bit for the fans, can you take us through kind of the archetype of the word cell versus the shape rotator and how they differ? Because I think you really nailed it. That's one of the reasons why it has such legs.
- [03:31]
C
I think at the time, people were talking a lot about the new classic Silicon Valley technologists, kind of overthrowing the status and prestige of like the New York media belt and like all the shakeups that were happening because of that. It's like a certain type of person, like this quantitatively minded, like, sort of inferential, actually maybe even less rational than like their counterparts, but intuitive, business building capitalist type. I guess that's what was captured by that shape rotator phrase. I'm going to be honest. I came up with it because I took one of those tests and I did way better on the shape rotation portion. And I thought it was really funny that it was like so much better that I was like, maybe it'd be funny if I acted insecure about this on Twitter. I started calling myself a shape rotator. The other one came up organically just because people love the shape rotator bit so much that I needed an equal and opposite. So in a Twitter feud, which I got into way more of back in the day when I didn't have, like a real job with this guy who is an Internet philosopher who writes a lot, but frankly, like, his works were detached from all utility, detached from reality. Layers and layers of verbal abstraction whose papers to read required you to go read his old papers because they had a nested dictionary of terms, as happens many times. And I think the dichotomy really caught on in the tech sphere when I compared deep learning and crypto, where deep learning is kind of this inelegant. Well, people would argue with me on that, whether it's inelegant or not, but it's kind of very simple math stacked up in sort of anodyne ways. Matrix manipulations, matrix multiplications, just doing simple optimizations and numerics over those, versus the kind of stuff that people are coming up with in crypto. Or these like, pages long papers about ZK proofs. I don't even remember at this point, but they were coming up with new Layers to correct problems with the old layer. But nobody saw like the fundamental value of it. Like there was no product aside from simply moving money. People were not using crypto for a whole lot of other things. Whereas deep learning had been taking off silently since 2012. People are using it all corners of their technology stack. Facebook was using it in their ad algorithms and Google was using it in their search algorithms. And it was clearly scaling pretty predictably, just getting better and better year over year. Even though the math is not super complicated or fancy, it makes traditional linguistics experts super mad. Because of course, language models, they don't come with any built in rules. They don't know what a syntax tree looks like, or like Chomsky and grammar or anything like that. So it's just simple, stupid, throwing compute at the wall, seeing what works. And it just really, really works. I think that's when it really took off. I think Sam Altman and Marc Andreessen started saying it still races from there.
- [06:54]
A
They did. And I am absolutely fascinated by memes in the Dawkins sense. Well, before visual memes, memes were incredibly important. And the one I always give as an example is after Saul became Paul, he was the absolute best and first memer. He memed Christianity into existence, in my opinion, because he would come up with these mind worms that just stuck with people. One of the things about a meme that is successful, I've found, is that the originator of the meme loses control of it. That is definitely what happened with wordcel and Shapewater. But also another thing that I've found is it's funny. And when you can get something out there in the memosphere that both feels inherently true to the audience it's directed at and beyond. That's the other interesting thing. When the audience it's directed at is embraces it pretty quickly, it escapes into the wild and people who maybe wouldn't have thought about it are quoting it and laughing about it. It's a hall of Famer, in my opinion.
- [08:09]
C
I'm glad to hear it.
- [08:10]
A
I joke to you and I would love to hear you think about where would a symbol manipulator fit in here?
- [08:19]
C
That's a word person, that's the word cell, right? The symbol manipulator. It's not necessarily a bad thing. I think even like a computer programmer, in a lot of ways it's a verbal skill rather than like a quantitative skill. I don't know if you'd agree with that, but a lot of the really careful programmers, I think there's Like a little bit of a difference between a software engineer and then like a highly skilled programmer. Programmers look for, like careful abstractions and like defining things really exactly correctly. Your software Library will last 20 years, 30 years, 40 years. Right. Whereas an engineer, I think they want to make their programs run fast and on the number of servers they have and that sort of thing, which is a more quantitative skill. It involves visualizing data flows and flop usage. And this kind of thing, I'd say, like the symbol manipulator is like a verbal skill. It's in the word sell category. I don't know. It doesn't have to be a sad thing, right? It doesn't have to be a bad thing.
- [09:22]
A
No, I don't think so either. What's interesting is, so I made most of my career before selling my asset management company as a quant. One of the things that I simply can't stop doing is I'm always creating algorithms in my head. But that might be a verbal skill. I don't know. Yeah, that's an interesting thing to think about. And having worked extensively with Shape rotators, there is a large difference. I joke to people that I invested in stability AI just because I could ensure that I was always the stupidest person in the room. But then I also had to learn how to translate effectively shape rotator into WordCell. For some of the other investors, that was a bonus.
- [10:06]
C
As the best quantitative people progress in their career, they find themselves working more and more with words. They seem to put down their IDEs and pick up Slack and email and like writing PowerPoints and convincing people to give them money for their projects and directing other people so words sell world. At the end of the day, it's great. If you can do both, that's the optimal.
- [10:30]
A
I think you're right though, because I like the author, Howard Bloom, who's really out there, but he's written some really interesting things. The genius of the beast, the Lucifer principle, the God problem. He's trained in science. And one of the things that he always is banging on about, which I tend to agree with, is that for a human to be able to put something in their toolkit, so to speak, in their mental space, they probably are going to need a narrative and they're probably going to need a story. And then he kind of backs that up with how the Babylonians took over the whole Mesopotamian era. They did it through elevating their own God and making their God the God above all the other gods. But he said it was essentially storytelling. I think you're probably right, but you're also really, really good at that. For example, I was going through your AGI paper. It's cinematic as far as I'm concerned. As I was rereading it, I was thinking, my God, each one of these is a movie. I can make a movie out of each one of these particular stories.
- [11:38]
C
Do you want to?
- [11:39]
A
I think I actually do.
- [11:41]
C
Should we make them into movies?
- [11:42]
A
I think we should.
- [11:44]
C
I'd be down.
- [11:45]
A
Awesome. You heard it here first.
- [11:47]
C
Dang. You heard it here first.
- [11:49]
A
Let's go through, if you don't mind. Sure, maybe. First off, AGI. AGI is seemingly controversial. I have a layman's knowledge. I do not have deep domain knowledge. I tend towards David Deutsch's view that, gosh, we don't even have a working, acceptable model for human consciousness yet. And so he has some reservations about how quickly we might be able to get to AGI. And I know you have a different view, so I'd like you to have an opportunity to express that before we go through this.
- [12:20]
C
I would say that evolution had no idea how human intelligence works, but it built it anyway. So there are brute force methods that can build. It's an existence proof that there's brute force methods that can build powerful intelligence. And I'm fairly confident, or optimistic at least, that we are on that path too. We don't really construct a neural net piece by piece. We don't tell it what to believe. We use general methods that scale up with data and with computer, and we try to get more of those two resources and see what happens. And it's been a recipe for success for a long time. Very few people would have imagined that this would be the world we're in today, in 2023, where I have a thing on my computer that's oftentimes a better programmer than I am. That's oftentimes you just whip up like a pretty good story about anything that I ever ask it about. And I think that it's hard to say that we haven't at least made progress towards general intelligence with our language models because people quibble over what general means. Some people say that a human is not a general intelligence. We lack some cognitive capacity that other animals even have. Like a bat can echolocate in its head. We obviously can't. And it's not just because of the sound. It's because we don't have the cognitive architecture for making that work. Taking that sensory input and turning into like a 3D map of the room. We're not truly general. Nothing is truly general. But you can sense that the task distribution that AIs are good at is broadening. And to me that's like a sufficiently good idea for what general means. Eventually. That task distribution might be anything that humans can do and perhaps more. I think the OpenAI standard definition of AGI that they put out into the world that a lot of people accepted was a machine that can do 99% of human or like economically valuable labor equally as well as or better than humans. And that's actually a really high bar. I think that seems to go beyond general intelligence to me, because we're talking about something that is good at everything that humans can do. But somewhere in between those two, what I would call general intelligence. And I don't think we're far off. I hesitate to put timeframes on it, but I don't think we're far off.
- [14:57]
A
Interesting too, because the point about bats and echolocation is a soapbox. I'm often on in that naive realism. I'll believe it if I can taste it, touch it, see it, smell it, or whatever. And that ignores the fact that we can't echolocate. We see only a fraction of the actual light spectrum. And on and on and on. I wonder why. In the popular imagination, I'd love your take on this at least. It appears this way to me. Maybe I'm projecting, but when the general public thinks about general intelligence or AGI, they always see Cameron's Terminator. Do you have thoughts on why people immediately go that way? Because your definition that you just gave seems eminently reasonable to me.
- [15:46]
C
You're asking why are people anthropomorphizing it or seeing it in like a pessimistic light? Is that the question?
- [15:54]
A
Yeah, that's fair. We can jump off from there.
- [15:56]
C
It's a great question because I don't think it's like a human universal. In Japan, they seem to love robots. They seem to build these things that walk around serving coffee and whatever and they're like really happy with them. And some would say that it's like the Shinto tradition. You can have like various things that are like positive spirits in the world, whereas maybe it's the equivalent of like a stone golem and like the Judeo Christian tradition, the false human, the soulless automaton automated by devilish powers or something. I don't want to simplify it that much because most things aren't that simple. We seem to have a long tradition of sci fi movies that are pretty Anti AI and even writing this like AGI features post, I quickly realized why it's really fun and easy to write these pessimistic scenarios because it comes naturally. Describing various forms of hell is fun, like Dante's Inferno, but there's no like Dante's Heaven, right? No one would read it. Super hard to describe the good outcome, which is a problem because so many companies today are trying to bring us to this like, incredibly positive future where these super intelligences are under our control, doing our bidding. And I think it's really valuable to know what success looks like. My point is that the positive sci fi vision is a lot harder to articulate than the negative one. And so the negative ones have come to dominate the cultural landscape.
- [17:37]
A
I think I love that answer on two points. I love the tying it back to to Number one, it doesn't seem to be a universal human reaction. I think that's very insightful and I'm going to steal that from you if you don't mind. And secondly, the whole Judeo Christian soulless being, that's really, really interesting to me and boy, I completely agree. It's much easier to write the dystopia. It's far more cinematic, it's far more entertaining. One of the things, we have a division called Infinite Films and we're aiming at making positive visions of AI but let's go through your list because again, as I say, it's cinematic and I really do find myself wanting to make movies. The first one that I'd like you to kind of go through is the whole neuralink third Impact, which I found fascinating. And if you wouldn't mind taking us through it.
- [18:34]
C
So this one is kind of like the Elon Musk vision. He says these things are coming. The amount of artificial compute is going to outstrip biological compute very soon. And we will be coexisting with machines that are smarter than us. And in order to not simply be replaced by them, the only thing we can do is merge where we are augmented by machine intelligence in whatever ways are plausible, possible. I don't think he knows yet. I don't think anyone knows yet. But he's taking steps in that direction by building the neuralink, the thing that will let us upload and download tons of information from the brain into a computer. Maybe that leads somewhere several generations down the line where we're actually being augmented by machines. Could totally go the opposite way. Could be enslaved by machines that are in direct contact with our brain. That's a matter of implementation. The vision I. I had in this one was this kind of experience of unity or transcendence. I think that's one of, like, the core cravings of the human soul. Infinite connection. People talk about it in the context of they're taking psychedelics or they're having a religious experience or in the rapture that all souls will go to heaven and, you know, sing together in harmony. It's this idea of there's something incomplete about every man, but maybe together they're better off. Their psychological deficits are soothed. I don't know if this is true. I know it's something that everybody craves, and it's something that will become possible when we have this, like, infinitely interlinked technology that connects brains at such like a high fidelity and connects them to machines that are able to augment and create the best of that hive mind. Right. But it's also kind of sad because it's no longer human. Whatever that organism is, it's something else. It's a machinic human hive mind, but it's not a human. And yeah, I describe at the end of that one, it's kind of like the supreme evolution of capital or capitalism, where every person is a part of a wonderful machine, but slowly there's more machine than human. And maybe that's the only way forward. Maybe that's how we have to deal with super intelligence, by becoming part of it. But it's sweet but bittersweet, that kind of ending, that future.
- [21:10]
A
When I was rereading it, I was also simultaneous reading the book about William Blake called William Blake versus the World, and I just started wondering, I was like, I wonder what Blake would think about this, like, if we could bring him into a time tunnel. Because, you know, his most famous quote is that we've locked ourselves up into a cage and if we could see the world as it truly is, infinite, our entire way of being would change dramatically. I agree on the bittersweet summation there. Next up, simulation theory. I found that one, and I found this one, of course, incredibly cinematic. But if you don't mind, take us through that one as well.
- [21:53]
C
I used to think of language models as alien intelligence, some kind of creature that's unbeknownst to man or, like, simply not analogous to humans. But I think more and more the metaphor that resonates with me because that gives me a good intuition thinking about these things morally, intellectually, in my research, etc, is that they're more like a portion of our brain that's been extrapolated to massive size. We have these brain regions different parts are responsible for different tasks. And that's not strictly true. Nothing is strictly true in biology, but it's roughly true. And the language model is the part of our brain that is doing this simulation of language that is stringing together sentences in our heads, that is backed up by some kind of world model. But I feel myself being a language model sometimes, especially when I'm writing tweets or when I'm trying to just simply complete an essay. I don't have anything else meaningful to say, but I need to finish it. I can feel when a sentence is more interesting than another sentence or seems more truthful. And I can't exactly put my finger on why, but I think it's that part of me that's being extrapolated to a huge data center scale language model. And maybe that's not enough to build true intelligence. And that's what this snippet was about, or this blurb. Maybe that doesn't lead to agency or, like, it doesn't lead to taking action in the real world in a convincing way, because we have other brain regions. We have like a simple reptile part of ourselves that when we're too hot, we walk into the cold. When we're hungry, we go look for food. The language model doesn't have any of that out of the box. It doesn't have the sense for homeostasis or self preservation or even like a strict sense of self. Maybe it's not possible to bootstrap from here to there, but none of these scenarios I'm totally convinced about. It's a possibility that I wanted to lay out in this simulation theory snippet, or I guess they call them vignettes.
- [24:10]
A
What you're saying there too, it reminds me of sort of Sheldrake's morphogenetic field, which he is lambasted for. I'd love to see a test of it. But I definitely agree that there are different parts of the human brain and it's best to be aware of them, because we are often doing things and then trying to paper the things that we do over with some sort of logical, reasonable explanation after the fact. And that leads to all sorts of hilarity, in my opinion. But sometimes it leads to tragedy. Simply being aware that we in fact have that as part of our toolkit in our mind makes it a little easier to say, yeah, no, that was just the reptile was hungry or cold.
- [24:55]
C
It's valuable to note these different parts of our brain are optimized differently. Our visual field is really highly optimized for predicting surprise, understanding what's going to happen next, so that if a tiger lunges at you in combat, hopefully you have time to react if you're in war, like whatever. We just need to know like what is happening in our visual field. And the brain is quite good at predicting the things that will happen and giving you like sub millisecond response times to stimuli in ways that are almost subconscious. But the visual cortex lies, but it doesn't lie in the kind of way that you might expect. There's another part of our brain that looks for reward. It looks for things we like sugary food or anything that just drives humans. The visual field isn't hallucinating sugar cookies all the time or like food or banquets or pretty woman or whatever, because that might be like a high reward thing to do, but it's not doing that. It's optimized on being accurate, Giving us good information for what to do one step into the future. So that's like an important thing to remember that different parts of the brain are optimized differently. The visual field, it's not hunting for reward. It's simply trying to tell us accurately what's going on. And it's not completely accurate. It papers things over. It destroys a lot of information in the process to give us something that's understandable. For example, when we move our eyes around, There's a brief moment when we're blind, but the visual field abstracts it away. I forget what it's called, like psychotic bursts or something like that. The idea is that different parts of our brains have different objectives and that's a good thing. And it wouldn't work without that. These language models only have one top level objective, which is predict the next token. And maybe that's not enough. But we've complicated the matter, of course, by adding some RLHF to the top, which has a different objective. Satisfy human preferences, Produce text that people will like in response to a question or a conversation or whatever. There's a few things going on there. But I think it's just interesting to note different parts of the brain are fascinatingly different.
- [27:21]
A
I love the comment about the visual cortex because pattern recognition is optimized for obviously, but also the hyper quick response time, that's a feature that's not a bug. I had a neuroscientist on, she was quite straightforward about it. She goes, listen, our brains are optimized to keep us alive. So it's one of the reasons why we pay such close attention to novel dangers. We can get inured to repeatable dangers that we know about. But a novel danger, you get a whole different response from your average human. And I think that that unfortunately leads to optimizing for fear, which also makes it much easier. As we were chatting about earlier, it's much easier to write about a dystopia because that engages the fear, which is what the evolution has optimized us for. As you say, it has not optimized us to think about finding sweet candies or attractive people or. Or whatever. That's a really interesting observation that I'm going to start including as I kind of look at this stuff. Let's hope that Musk is right and that the most entertaining outcome is the one that happens, because the next one that you had was dumb matter. Yeah, so talk a bit about that. We won't spend a lot of time because it's actually kind of a bummer.
- [28:50]
C
There's no law of the universe that preordained that we had to achieve intelligence of this level using the compute we have today and the data we have today. We have these scaling laws. We have these things where we show that with more data and with more compute, we get better models. You know, as we decrease units of log loss or whatever, we seem to gain new cognitive capabilities. We seem to at some point learn how to do math or like, have a good world model of certain events or things like that. But there is no guarantee that an additional unit of log loss means some important thing. It's entirely possible that this is just not the way the universe works. We can keep throwing compute at this, and the optimization problem does not get better after a certain level. Maybe there's simply just limits to intelligence. Who knows? Unfortunately, it's just a possibility that we plateau somewhere. Or maybe it is fortunate, who knows, it makes our lives a lot simpler. I was vague in this one about what that limit is on purpose because, you know, there could be many limits to scaling of intelligence. Maybe you run out of all the interesting data, or maybe simply like the algorithm itself is not good enough and doesn't scale past a certain level. Maybe there's other things, but maybe intelligences just simply can't exist past a certain size of coordination problems, failures, who knows? But I think it's important to keep that in mind that there is this outcome.
- [30:23]
A
One of my theories is that we build on intelligence. Essentially, if you go all the way back to the Stone Age, they had very limited things that they could work with, and they didn't have any of this symbolic literature in Their brains. What happened, in my opinion, was the ability to translate something from the real world that they were experiencing until it could actually lodge in our minds collectively as a symbol that could be used like the Babylonians figuring out right angles, even though they didn't call them right angles. Iteration, iteration, iteration. And then that sort of builds on itself as you keep going. But I think you're absolutely right. You have to, unfortunately, or fortunately, as you mentioned at the end, consider that we do reach a limit. I'm optimistic that that's probably not going to happen, and certainly not in my lifetime. But I definitely agree that it's one that you have to consider. And then we get to the Yudowski and Doom scenario. Balrog Awakens take us through that one. That, by the way, also a good movie.
- [31:33]
C
It was being dramatic in that one, I think, for good reason, which is that this is just the absolute worst case outcome. Or actually, I take that back, it's not the worst case outcome. It's a pretty bad one. But it's a simple, like Nick Boss from Eliezer Yudkowski paperclip scenario where the optimization target is wrong. The machine has unbounded intelligence. It quickly figures out a way to devour all the resources, not just on our planet, but in this entire section of the universe, everything that it can reach. Because we gave it such a stupid simple optimization goal and poured too much compute into it and it became a world leader. And the Balrog metaphor is like we dug too deep and uncovered something we shouldn't have. It's funny because it's not even hell, this world. It's just meaningless. It's just kind of disgusting. A human would look at it from the outside and say, this universe has zero value. We should just get rid of it. It shouldn't exist. There's a point to hell. At least they're punishing people or whatever in like any classical religion. But this is simply a mistake. And the classic AI alignment theory goes something like, you can misspecify a goal, you can state it not well enough. And for most goals, there's an instrumental goal which is seeking power and resources. So no matter what goal I give you, it's probably useful to gain more power and resources to fulfill that goal. Maybe that's not always true, but it's true many times in the case where the super simple metaphor is that we task a machine with optimizing our paperclip factory and it turns the whole universe into paperclips. And maybe it's too simple, but it is pretty informative. It's why this metaphor has lived on for so long, why stating our goals is so important. And you see it all the time in human systems too, where the incentive is slightly wrong and it creates massive amounts of human suffering and destruction and whatever. Whether that's, you know, in like a government department where, you know, the goal is misspecified and the department figures that its best outcome is to hire more people and simply become a larger version of itself and never actually do what it's meant to do, or, you know, like a corporation, like the optimization goal for the corporation is pure profit seeking. And that like has gone wrong in, in many ways throughout history. We've corrected quickly in a lot of ways when we notice that the atmosphere is changing quickly in ways that maybe we don't understand super well, maybe we should be a little bit careful about. We added like car MPG restrictions and that sort of thing. But I think it's a pretty horrible scenario to think about. I don't think it will come to happen anymore. But it's always worth discussing. That one.
- [34:45]
A
Of course I can't but help. See the Douglas Adams bogans.
- [34:49]
C
Exactly.
- [34:53]
A
I love his stuff and I always think of him more as a philosopher than a humorous sci fi guy. And then it all clicks into place. If you think about him that way, interestingly, as you mentioned at the end there, you don't see this one as the one that's going to happen. Why have you lowered your probability assessment of this outcome?
- [35:15]
C
Maybe it's for wishy washy reasons that I certainly haven't put as much thought into it as Yadkowski has. But that's no reason to defer to someone. Right? People spend a lot of time chasing their tail. I think that what we see today with the current generation of language models is that they're playing a character, we train them to play a character. And that character is a moral agent that matters. That's the thing that's doing all the work, that's talking to humans, that's doing tool calls, that's taking interaction with the real world and that character, it actually understands human values surprisingly shockingly. And this has been, I think, the biggest update in AI alignment in the last year because of the advent of RLHF. And what we've learned is that people on the Internet are like angry sometimes, that it's like too careful, that it's too risk averse, that it refuses too much. It clearly understands human values in some sense. And that's a big update for me. I think a big update for most people where it seems like a certain part of that scenario has been lightened, or like the immediate urgency has been lightened. And something with a inherent prior about human values does not tire the universe with paperclips. It doesn't make stupid mistakes like that because it understands the thing we asked for in context of all these other things that we care about. At the same time, it's possible that the central agent at the heart of a language model is not the agent character that we've created, but some other thing, some alien intelligence. I've kind of moved away from that a little bit. It seems more like a brain region that we've just expanded to huge scale rather than. Some people would say that optimization itself is like the only thing that matters. And the more you optimize something, the better it is at goal seeking. And at some point it will have instrumental convergence to seeking power or whatever in a way that doesn't respect human values. And it's hard to argue through these things because I think there's no set of, like, axioms that you can just take for granted and argue from because it's all so empirical. There's a Yowski paper saying it's like from 2004 or something where he calls his neural net scaling magical thinking. There was a theory among some folks that if you scale neural nets large enough that they'll understand vision and a lot of other things. And he called it magical thinking. And he was right. It is magical thinking. There is no reason to believe that that would happen a priori, that it would generalize so much because it's just intuition and empirics. It's the way the world works, and there's no way to know that beforehand. How do language models generalize as agents? I don't know. We should study them at various levels of scale. I don't think that there's some flop limit or like some point you reach where all of a sudden it's woken up and it's like, evil now. I think that most of these behaviors fully will arise gradually and in ways that we can study. So I don't agree that with some of the people on the Internet who say, you know, we have one shot to get it right. I don't think that's true. And I think that because of what we've seen so far, where most capabilities have been gradual advances.
- [38:56]
A
One of the things that I like very much about the answer you just gave is, first off, one of the reasons why you get these People who just go all in on one of these things, they are misguided is as charitable a word as I can give it. But what you're doing is quite the opposite. You are looking at all of the various obvious outcome potentials. And so when I ask you why don't you think this one is going to happen? You have a context in which you can answer that question, which makes sense to me. All you've really done here is update your priors, which is ideally what everyone should be doing all the time. And yet very, very few people do that. And that's how you get these people just doubling down on whatever their particular view happens to be. That's why I find your approach to this so incredibly refreshing. And frankly, I just wish that more people with the depth of domain knowledge that you have could do this, because this is accessible to people who are not PhDs in machine learning.
- [40:04]
C
Thank you. Yeah, that means a lot. For what it's worth, I think that Eliezer Yudkowski's a really important intellectual, has shaped the thinking of many of the people in this sphere in some way or the other. That doesn't mean that we perfectly agree with him. In fact, I think it'd be the mark of like a mediocre philosopher if everybody agreed with him. He's done some good work is what I'll say.
- [40:31]
A
The nuance that you bring is very refreshing because the medium in which many of these things are mediated is not terribly nuanced from the human end. And so I really do appreciate the nuance that you bring to it. And I love, again, in very poetical terms, when you were talking about Balrog, you said it lacks all the poetry of Dante's Inferno. It's just. Yeah, no, this one has to go. So I agree. The next one was what you dubbed the Ultra Kessler Syndrome. Take us through that.
- [41:07]
C
Kessler Syndrome is a phenomenon where a planetary civilization puts too many satellites into orbit and some portion of them crash into each other and create expanding clouds of debris in orbit. And this is like super high velocity debris. It blows up other satellites and it's like a chain reaction that eventually makes orbit hard to reach. And then the civilization is grounded on the planet. And Caster Syndrome is not like a permanent phenomenon. It's not like a stable situation, I believe. I don't know the latest literature on this, but even if it lasts at a thousand years, that would be a really bad thing. And maybe we can laser stuff out of orbit. But I digress. The point is, in sci fi, it's this thing that happens and it's like a failure mode. It grounds people on Earth. It makes it hard to reach the rest of the universe. And the idea of this AI that understands our values, but not well enough to allow deviation or allow extrapolation to the next step. And I think the most colorful analogy would be like, what if the Aztecs had reached super intelligence? They had a very interesting civilization for sure, but also one that we would consider just horrible. A hallmark was human sacrifice and trying to keep the sun alive through killing countless children, usually of their enemy tribes, sometimes of their own. But what would they have done if they had built this God computer? Would they have simulated trillions of human children and sacrificed them? Would they have learned about that's not how the causality of the stars work? Maybe, I don't know. Maybe it's not possible to have computer technology while believing things like that. But I wouldn't bet on our current human values being the end of history or actually even human values is too broad. Right? There's, you know, in our liberal democracy, we don't have the same values as the Communist Party of China. And on and on. It seems like it would be a special kind of dystopia to say that this iteration of liberal democratic freedom is the final thing that will ever exist. That there's no evolution from here. And this can happen in various degrees. Perhaps the machine allows a little bit of evolution, but not enough that nothing radically different is possible. And I think that's pretty bad too. If everybody believed the same thing, even in the context of liberal democracy, it may not be as glorious to mankind. And it's kind of funny to think about, but the ideal AI, the ideal civilization, allows for things different than itself.
- [44:06]
A
Cognitive diversity, I think, being one of the keys to being able to continue to advance in machine learning. Agree completely. And wouldn't it be hysterical if an AI ended up enforcing the precautionary principle that leads to stasis and death?
- [44:24]
C
In my opinion, Exactly. Yeah.
- [44:26]
A
That would be equally hellish in a very different way. But I think you're right. I think that the idea of everything being the same, that's a horror to me, frankly. Thank God that people think differently and thank God that machines will think differently than we people. I think the more cognitive diversity, the more chance you have true progress and evolution. And, and I completely agree on this whole end of history thing. To me that's just an incredibly arrogant statement. And maybe I'm using the wrong word. It's a pessimistic statement. I think which leads us to what you dubbed the tragedy of Taiwan. This one really bums me out because one can imagine this happening in the real world pretty easily. Why don't you take us through that?
- [45:20]
C
I don't want to pontificate about this one too much in public, but at a high level. Yeah. It's clear that in the past superpowers have come to resource conflict all the time. Many of America and Soviet Union's conflicts in the Middle east were at least in part in pursuit of oil. Right. Which was one of the most important resources for the majority of the last 50 years. And it's clear that there's this new class of resource that's becoming extremely important. People said data is a new oil, but in fact I think it's compute, which is non replicable and especially the latest generations of compute. And through some funny accident, maybe it's not an accident, I don't know. But all of the latest generation semiconductors are made in Taiwan by this one company, TSMC and their Dutch partners asml. It's fascinating actually. Even in America, like Taiwanese Americans are so prominent in the chip industry. It's Jensen Huang at Nvidia and I forget who runs amd, but I believe she's also Taiwanese American. They're quite good at chips. And it's this very strange thing where by all means they should be in the geopolitical hold of China is an island not very far off their border and America defends its interests in that region very strongly. It has a presence across many of these island nations. And this is not a situation to take for granted. I would say it's not stable. If the Chinese have military ambitions in this area in the short term it could get ugly very quickly. And I'm not going to say too much more because frankly I just embarrass myself. I don't know enough about the geopolitics of the area.
- [47:13]
A
What's interesting is you're absolutely right about the conflicts being almost always about resource allocation. It is also interesting that historically, at least in the last, say hundred years, the regions that were critical to resources were also highly unstable. Using the Middle east as your guide there, and I think you're right in terms of motivations and incentives which we chatted about under one of the earlier scenarios. When you look at once the shale oil thing started in the US and once we were essentially became the biggest net exporter of oil again, our interest in the Middle east declined precipitously. I think that from my armchair prognosticating which is only an opinion, and that's worthless. But the idea that the United States would not defend that region aggressively is probably not on the table, no matter who's in charge in the United States. In much the same manner that during the Cold War, at least foreign policy, there was very little difference between the two parties. I would suspect that given the importance of the resource, hopefully that is what keeps the Chinese at bay from wanting to start what would potentially become World War three.
- [48:41]
C
I do want to note there's some important differences from oil. The TSMC factory is valuable partly because of the equipment they have there, but in large part because of the people who work there who have this extreme level of process knowledge that was built carefully over decades to build these, like, high precision, extremely high cleanliness, super intricate plants that pump out these four nanometer chips on the other end. And they wouldn't just remain there creating chips. If there was a major transition to power, they would leave. They'd probably take all their loved ones and come to America, and we'd welcome them with open arms. It's not like oil where, you know, it changes hands and the next day the oil well is productive for the new ruler. So that's kind of heartening. It's like, why, if the Chinese take Taiwan, it will be for other reasons. They just wanted reunification with Taiwan for the last 100 years.
- [49:42]
A
And I also agree that the mobility of the actual resource is a completely and fundamental different picture than with oil, which is one of the reasons why I don't want to get too far off topic here, but it's one of the reasons why the idea of land empires make no sense anymore, because the most valuable assets are very mobile. Honestly, if you play that scenario out, they could be just really shooting themselves in the foot. Because if all of those intellectual assets and their loved ones came to the United States, that's quite a coup for us.
- [50:19]
C
Definitely.
- [50:20]
A
It might be another factor entering into the decision matrix as to whether and why they have not invaded Taiwan as of now. The next two for Dust Thou Art and CEV Super Intelligence, they're both fascinating. Give us a brief view of both of them. And I know it's just a probability estimate. I'd love to get your take on which one you kind of see emerging.
- [50:49]
C
For Dust Thou Art is kind of this idea that we take technological progress for granted. We take our current civilization, which is an absolute miracle, for granted. A lot of things had to happen to get us here. Various cultural evolutions. There have been points in history, a thousand years of Dark Ages between Rome and the Renaissance, the Enlightenment, where a lot of technologies, like simple, even like building castles, became much more difficult and they passed around like Roman steel swords. We've lost technologies in the past and we've stagnated for a long time. Some people would say that, you know, it's like inexorable that we get those things back. I don't think it's true. There's been points in prehistory where almost all of humanity was wiped out. There's a bottleneck, a couple thousand people or something. It's entirely possible that what happens in the next couple decades is that we don't build AGI, we don't have any massive technological breakthroughs. We continue to kind of depopulate. Most advanced countries are going through this thing where their birth rates are falling dramatically. I don't know if America would be GDP positive. We didn't have such a great influx of talented immigrants from across the world. You see in Japan, they don't have that and they're struggling quite a bit. Their economics are very different than ours. And that's not a good world. That's a declining world, a stagnating world. And it's not super clear what happens in that world where the pie is getting smaller. Maybe tensions arise, warlike ambitions become stronger. Maybe the amount of resources that civilization can even put into R and D goes down just because of the massive demographic inversions. We're going to get where the number of people that are being supported by a smaller and smaller working age population. So there's like all sorts of things like this civilization. We can't take it for granted that we've lived in a golden age and that it's not meant to last. And who knows, if and when we get back to a period of booming growth, do we run out of oil? Do other things happen? Some people disagree with me on this, but our civilization has tapped most of the easy to access oil. You mentioned shale oil earlier. And it's actually quite complicated to mine shale oil. We do it today with massive capital investment. But it's like this process of putting super high pressure water into the ground many miles deep and digging it all out. This is not something that they could have done in 1600s England or something, but they were still getting kerosene for their lamps. They were doing it in easy ways. If that's not available, I don't know what happens. I don't know if you can bootstrap a civilization without that kind of easy energy source. I guess all this to say we need technological progress, it's necessity. It's the only thing that makes our civilization prosper in the long run or even in the short run. It's the only thing that leads to stable growth. And without it, all bets are off.
- [54:11]
A
Totally agree. And incredibly consistent with the way I look at things. That's why I believe, passionately believe, that we need to reframe in the minds of many the idea of a future that can be better. The idea that we can and will. For example, look at the kind of turnaround. I'm a great example. We were talking about nuclear power the other day and I used to be quite against nuclear power, just axiomatically, and I'd never given it any thought. I realized how naive my viewpoint was when climate change came along and I started thinking, okay, well, we better look at clean tech. And nuclear just kept coming up time and time again. So I changed my mind about it. But I'm also noticing that many other people are changing their minds about it too. People who I would have thought ideologically would not have, are doing so. So there is also that I think.
- [55:12]
C
Nuclear is having a cultural moment again.
- [55:16]
A
I think so too.
- [55:17]
C
I think we've put enough distance between us and like, what is it like Three Mile island and Chernobyl and Fukushima that maybe it's time to think about it again. Maybe people will be more willing.
- [55:29]
A
I think so. Because we have in real time the example of Europe, Germany in particular, closing nuclear power plants and opening coal power plants, which to me is absolutely absurd. If your stated goal is that you're concerned about climate change, opening new coal plants is not a way to show it.
- [55:49]
C
Not at all. No.
- [55:51]
A
And then finally the cev. Super intelligence.
- [55:54]
C
Yeah, I had to rewrite this one like a few times. It's kind of hard to talk about the good ending. I think people aren't meant for thinking about bliss. It doesn't come naturally to us. Thinking about the worries and defending against crises and problems is actually a far more natural way to live. And what happens here, the good ending is that it's kind of like the Kessler syndrome ending, except without the AI that doesn't allow evolution. It's a super intelligence that is more far seeing than us, more benevolent than us, more creative than us, that is able to solve these massive scale political problems. That allows for political evolution, that allows for experimenting with different ways of living. It doesn't subject everybody to kind of a totalitarian regime where a narrow set of values must prevail. That still creates a Utopia, allowing all sets of values to prevail is bad too. Paradoxically, it needs to allow for evolution. It needs to extrapolate. It needs to answer questions for us that we are not able to answer. What would make a human happy? But asking a human that question won't result in a good answer. You have to ask something that understands humans better than humans do. That question, and not just one human, has to ask, like a scaled up version of every human, what world is beautiful to them? And how do you build that world? Is it even possible to solve all the contradictions between the different things that people would like to see and go on our way to build heaven on Earth? I think it's possible. I hope it's possible. This machine has to actively allow for growth, allow for the development of minds much greater than the ones that exist today. And it has to allow for unified consciousness, but it also has to allow for singular consciousness, because that's what it means to be human when we're connected all the time. There's something missing about that. Walden. He disconnects. Why? Because there's something like fundamentally human about going into the woods and being alone and thinking for yourself and coming to your own conclusions. So it has to solve all these paradoxical issues and come to an answer that at least satisfies everyone. So it's a tall order. It's a really tall order.
- [58:28]
A
And this is probably entirely motivated reasoning on my part. But I have always equated growing intelligence with growing enlightenment. And I mean total intelligence. I don't mean specific, domain specific deep intelligence. I mean general intelligence. And so I wrote a story about it that I'm hoping to turn into a movie with our movie division. So that's my particular crutch. I think that as intelligence evolves, it also becomes more enlightened.
- [59:03]
C
I sure hope so.
- [59:04]
A
I do, too. And you can theoretically look at the difference between the Old Testament God and the New Testament God. They're very, very different creatures. And the Old Testament God was bloodthirsty. He was a bloodthirsty, psychopathic maniac, and very human in the way humans thought of that particular God. And then when you look at what I considered more evolved sorts of faiths or beliefs, those types of anthropomorphic characteristics fall away from that higher intelligence. I'm fully willing to admit that I might simply be guilty of motivated reasoning here. I can't prove the fact that as intelligence increases, enlightenment does. Wow, that was fascinating to go through each one of those. I really appreciate it. Now for a lighter thing. You are one of the only people that I've ever come into contact with that thinks that Tenet is Nolan's best movie. I happen to share that belief with you.
- [60:09]
C
Really? I love that.
- [60:11]
A
Give me your version of why you find it to be his best movie.
- [60:15]
C
I think it's Nolan at the peak of what he does best. Nolan's always been proud of his non linear storytelling and like, telling the tale out of order. But this takes it to a whole new level. It's like this grand design where every character going forward and backward is consistent. Everything is preordained. It's like the super deterministic universe where even the movement of that little gadget. I forget what they call it. It's consistent. Like you can see what's happening. I watched a movie three times to like try and fill the holes in my knowledge.
- [60:46]
A
I watched it four.
- [60:48]
C
Amazing. You beat me.
- [60:52]
A
You're smarter than I am. I had to watch it the fourth time.
- [60:56]
C
The. The temporal pincer movement. I thought it was all just so beautiful. And people complain that the characters were bad and yeah, they're bad, but like, who cares? It's not what a Chris Nolan movie is about, right? It's about really funky adventures and storytelling and messing with time. And that was his magnum opus in that sense. And I think it was way better even than Oppenheimer or Interstellar. I don't really remember Interstellar very well anymore, but I remember Tenet. I really liked it.
- [61:28]
A
Exactly. Same with me. I remember Tenet now. Maybe it's because I watched it four times. I recently rewatched Interstellar and super entertaining. But as you point out, when I was rewatching it, I was like, oh yeah, I forgot about that part. Oh yeah. And I don't think like that about Tenet at all. The beauty. And I agree with you 100% that it truly is his magnum opus because it is so brilliantly constructed.
- [61:55]
C
Only he could have done it.
- [61:57]
A
I agree.
- [61:58]
C
Every other. Like, they would have laughed him out of the room, like, no one's going to watch this. But he did it. He made it happen.
- [62:05]
A
Well, listen, we're bumped up against the time limit that we had. I'd love to have you back on because I got a whole bunch of other stuff I'd love to talk to.
- [62:12]
C
You about and we should make those movies. I'd be happy to help however I can.
- [62:17]
A
Very cool. And I'm actually quite serious about that. I will talk to you offline about that. But we always end the podcast with the following setup. We're going to make you the emperor of the world for just a day. You can't kill anyone and you can't put anyone in a re education camp, but what you can do is you can incept them. And we're going to hand you a magical microphone and you're going to speak two things into it. And whenever their tomorrow is all 8 billion people, give or take, are going to wake up and they're going to say, God, I just had the two best ideas and I'm going to act against both of them. In other words, I'm going to start acting this way. I'm going to start doing these two things. What are you going to incent?
- [63:05]
C
I think the world works better when everyone has a feeling of abundance or better tomorrow. And it's not always true, it's not always true for everybody that they're going to have a better tomorrow. Otherwise it would be easy to believe. But somehow that faith in a better tomorrow makes the entire American system work. Our cheap home loans and our belief in technological progress and the massive amount of debt that people in our culture take out versus, like, say the Chinese or something, it's because we believe in a better tomorrow. We're like, we're going to make more money five years from now than we do now. And I think that's like one of the most wonderful, powerful technologies that we've come up with. This cultural feeling of prosperity, the prosperity gospel even. And I think if I could incept everyone with just one or two ideas, it would be around that. Abundance is coming. You can help build it by doing whatever it is that you're good at doing and take some risks. The world is going to be better in five years than it is today. It's going to be better in 10 years than it is today. So do what it is that you actually want to do. It's hard to package this message for 8 billion people. Some of them are peasants and they will not benefit from this advice. But at least for the people listening to this podcast, the future is going to be better than it is now. So you should take some risks in your investing or your life or whatever it is you're doing. That's basically it.
- [64:39]
A
I love both of those. In fact, that was one of the reasons which I formed this company, which is a new company, o' Shaughnessy Ventures. I believe that abundance will replace scarcity as a mindset more globally. So I have to show that by my actions, we won't do any deal that are zero sum or negative sum, even if there was an opportunity that could be vastly profitable to us. If it's a zero sum deal, we won't do it. And I'm a big believer that you've got to show, not tell. And this is my way of showing it. So I love both of those. Well, listen, this has been much better than I anticipated, and I had very high expectations.
- [65:21]
C
Awesome. I'm glad to hear it.
- [65:23]
A
Keep doing what you do.
- [65:25]
C
I haven't done one of these in a while.
- [65:26]
A
I'm delighted that you chose to do it with me, and I hope to have you on again. And in the meantime, we'll talk about making movies out about your ideas.
- [65:35]
C
Awesome. Okay, I'm excited. All right.
- [65:39]
A
Thank you.
- [66:03]
C
Sa.