Dwarkesh (2:28)

Yeah. So it's actually changed a lot. I remember a year ago, this was after. I think it was after GPT4, somebody asked me, you know, do you use AI to help you with your research or prep? And I was like, not at all. It's completely useless in mid. It gives you these banal. You ask it like, what should I ask? So it's a professor and it'll give you these banal. Where did you grow up? What's your book about? Whatever. So initially it was terrible. I think recently the models have gotten to just the point where with these 4.0, or especially with the new cloud models, they're intelligent and interrogative and can consider the context which you provide to them. And so they're still not that good at what should I ask this person? Because obviously that's why I have a job, so that I can come up with questions. But for the research itself, where you're. For me at least, I try to ingest everything they've ever written, all the rebuttals to their ideas, all the other considerations. And there's often a lot involved, especially given there's many different fields I try to go deep into. The last interview I just did was with Dylan Patel, who writes Semianalysis. It's a publication about semiconductors and AI hardware and so on. So there's a bunch you have to learn. And I can go through my workflow, but it's incredibly useful to able to have this thing where I'm like, what's going on here? Can you help me explain this? And I guess one bigger thing I've been thinking about is ever since I interviewed Andy Matuschak, if your audience is familiar, he's the guy who talks a lot about how space, repetition and other tools can enhance our ability to learn and how the normal mode of learning, you're actually not picking up that much. If you pick up a random book and start reading, you're not getting that much out of it. And I really have found that to very much to be the case to the extent that if I'm just like casually reading a book, I think I'm basically wasting time or entertaining myself. And I have come up with a couple of different workflows and tools that help me really interrogate and make sure I've reinforced what I'm reading about or learning. And a tool like language model is very helpful because it gives you the content in another context and you can like, it makes it can quiz you if you want. So it's super helpful, that kind of stuff.

Dan (4:47)

That's really cool. I think we should start. I want to start back to front, like with the stuff you're using to read. Because I think the reading, all that reading is sort of the input, like one of the inputs to the interviews and then we'll get into interviews and I'm really excited for both. So. So let's start with. With using AI to read and to learn the.

Dwarkesh (5:06)

You know. So as I was talking about, like, one of the main things I think is important is if I'm studying a topic over the course of a few weeks, it's. Especially if it's a difficult topic, it's like new to me. It's incredibly important that I've. I'm not just casually reading. Because if you're just casually reading, it's like every day you reread the same key terms, the same concepts and you're. You start over from scratch. So one of the things I like to do, for example, I was recently interviewing Dylan, right? So if I go to his publication, send me analysis, there's just a ton of lingo and things you have to understand. So the new one was pretty interesting. It's talking about why nobody has built a huge trading cluster yet. And then first thing I do is just like, what are the key ideas and concepts I really need to understand? So I made myself a hugging face space. You honestly don't need to do anything like this. It's pretty simple to have Claude build you hugging face space. Or if you prefer, what it literally does is apply this prompt to everything I paste in. So you just copy paste that prompt into Claude yourself. But basically it just has. I copy pasted some of the things in Andy Matuszak's post about how to write good prompts. And I just asked Claude to make those prompts for me. Spaced repetition prompts. When I do this, hopefully in a few seconds we'll get something back. Initially, this will just give me some ideas of what are the key ideas here I need to understand. Super useful. I can even zoom in a little bit, so it's more helpful. So for the audience who is listening, it's given me a bunch of question answer pairs that consolidate the key things I need to understand about this post. About. We can go through the specifics here. I'm sure that the actual specifics of AI hardware will bore people, but a lot of the things where it's like, okay, if you don't get this, you've totally missed the boat here. And so you can start with something like this. I add it to my Space Repetition app. Or I can just look through this and I'm getting a sense of like, oh, okay. Here's what it would take to train a GPT 4 level model on 100,000 H100 cluster. What are the three main types of parallelism you need to use to train on a big cluster or whatever? And this is on for a technical post. On other kinds of posts, there might be different kinds of cars that come up for history, it might be a different kind of thing. For philosophy, it might be a different kind of thing. So this gives me a lay of the land.

Dan (7:24)

I love this. This is super interesting. Like, I feel like I can go in like a bunch of different directions, but where I want to start is like, how are you reading and when are you reading? So is this, like, are you using this specifically for. For reading that you're doing for the show, or are you just doing this for any reading that you're doing that you feel like is serious and you.

Dwarkesh (7:44)

Really want to learn both. So just this weekend I was reading, I forgot the author's name. But it's a book called Medieval Technology and Social Change. And it's about how different things that were developed through the last 1,000 to 500 years, technologies like the stirrups, how they affected society. And it's like, you can, it's entertaining. You can read it. And then one of the things is like, okay, did I really understand what's going on here with the relationship he's trying to elucidate. So afterwards, in fact, I have some clot chats where I was just going through while I was reading it. Let's see if this recollects.

Dan (8:23)

Do it. I want to know. I'm on the edge of my seat because I have this book, it's like sitting on the desk in front of me. And so I want to know what you got out of of it.

Dwarkesh (8:31)

Okay, so first I just asked it to make some base repetition prompts for me. First of all, I was just like, I read the chapter. I'm not sure I got it. So just explain to me the chapter about how he says that stirrups created feudalism. Like, what exactly was the connection here? So it's a much more condensed, like, here's what's going on here. Basically, if you understand this, it's a useful scaffold so that when you're reading the rest of the chapter, you understand where the pieces fit together.

Dan (8:57)

Have you Tried like one of the things that's, that I've tried with this is like, because sometimes it doesn't know. Especially for a book like that where it's like not that popular. Have you tried like one of the things I do is create a little cloud project and then upload the text if I can find it. Have you tried that?

Dwarkesh (9:14)

In fact, let me just. Claude AI projects. So if I go to, um. I literally just sound like I'm a. I'm a host of a podcast where I try to ask good questions. My upcoming guest is a geneticist. And I just upload the. I get the epub of the file, I convert the epub to a text using an online converter. I upload it to Project Knowledge. Then I. I've only just started prepping for this guest, but I'll just have a bunch of chats where I'm like, you know, how. How does he explain what groups made up modern Europeans? It has all the context in there that's. That ends up being incredibly useful. Like you were saying.

Dan (9:53)

Yeah, that's so cool. I, I love that. I love that feature. Okay, wait, let's go back to, let's go back to stirrups and. And this chat you're having with. With this book or about the book.

Dwarkesh (10:04)

Yeah. So, you know, it explains that the reason Stirrups should create a feudalism is because you needed. You needed a lot of land basically to support the kinds of people who become heavy cavalry, the knights. The knights need a lot of land in order to have the income to have, you know, like armor and lances and other kinds of equipment and to train themselves. But a knight is only possible if you have a stirrup against which you can brace yourself as you're attacking with a sword. Because otherwise you're just like a mongol who's shooting bows and arrows. So. But then, then there's a bunch of stuff that's confusing here. Like, why is it that expensive to have be a knight that you need to like completely confiscate church lands in order to subsidize this knight lifesty style? And then on these kinds of questions, the author is dead. But it's just like, I'm just like murky about it. I don't know what's going on. So I can just. These kinds of things. I can. The book didn't even talk about. Right. But I can always just continue the conversation with Claude and have it explain what's going on. And so this is just like a recreational reading that Claude ends up being super helpful with.

Dan (11:13)

I think that's really interesting. What do you think about books like this as a person who likes history? A lot of books that sort of single out like a specific thing like the stirrup and then are like, well, you can trace all this stuff to that, like one thing where it like makes so much sense. But then there are things like, I don't know, like Guns, Germs and Steel where like Jared diamond had that whole thesis about I can't remember the exact thing, but it's like people in warmer climates or I can't remember the exact thesis but it turned out to be like totally wrong. Yeah. How do you feel about things like that?

Dwarkesh (11:44)

Yeah. So my opinion on these kinds of books, there's, there's, I mean the sort of concise answer is like, yeah, there's ones that do it poorly, but just don't read the ones that do it poorly or something. There is a failure mode for public intellectuals where they initially start off with a discipline and they do some exemplary work there and then they write an initial broad book that's about how this idea explains a lot of the world and it does incredibly well. And now they're in like public intellectual mode. And now that the next book has to be like, here's my theory of everything and it's just not that satisfying. So I do worry about those kinds of things. But presumably the reason, I don't know, I'm not into like reading 500 page books about like just how this tear up physically worked. Like, what's the point of that? Right. I do want to understand the implications and maybe they're wrong, but I mean, what else are we trying to do here? Right? Do we just care about. Maybe you just intrinsically care about how the stirrup physically works. I will point out a couple of examples. So there's a lot of interesting topics where you really can't get at the heart of the matter without just considering the whole story. And in fact. So a couple of biographies especially stand out in this way, where if you look at Caro's biography of LBJ or Kotkin's biography of Stalin, it's basically a history of this 20th century or in the case of Kotkin, even before the 20th century. I think this Caro books on LBJ start off with the Comanche raids on frontier settlers in the mid 19th century or something, and it goes through rural life in Texas, why electrification was such a big deal. A whole bunch of other things right now. So it's basically the history of the 20th century, but it has a very specific point of view or specific locus, a character that's moving the story along. And I find those to be incredibly helpful in getting a full picture of what's going on in an era. There's a couple other books where they really aren't trying to write a theory of Everything. Like, I don't think Caro's trying to write about, like, what is the history of the 20th century, but they just can't help themselves. They feel like you really cannot understand the very specific topic I care about unless I tell you everything about everything. You know, like, Kotkin's story, biography of Stalin starts with the. Like, Bismarck's career as a military general and how that changed the way that different power is thought about colonialism and the need to modernize. And that's where it starts. Right? It's a biography of Stalin. So, yeah, I love those kinds of books.

Dan (14:17)

I think there's a very just deep point about the universe being interconnected there. But there's also a really interesting point for people who want to make stuff like make writing or make podcasts or whatever, because there's this deep fear that everyone has about being pigeonholed. And it's like, well, if I pick this really specific topic, I won't be able to, like, bring all of myself to it, or I won't. I won't be able to be, like, multifaceted. And it's like, no, no, no. If you just pick one guy, Lyndon Johnson, and really get deep into him, you have to explain everything else about the world in order to explain him. And I love that. And, like, as a creator myself, like, that's the thing that I think about when I'm like, oh, maybe I'm getting too narrow here. It's like, no, no, the narrow is actually good. You can find the entire universe in the narrow.

Dwarkesh (14:59)

Yes. Yeah, yeah, yeah. I couldn't have said it better.

Dan (15:01)

Yeah. Okay. So. So basically, what I'm. What I'm seeing right now is you're using Claude when you're reading books that you care about, you care about, like, learning from, and you're using it a little bit to, like, prepare your. Prepare the mind. Your mind for, like, what you're about to read, which I think is a particularly good for, like, difficult books or for, like, thinking through a particular argument before you, like, go through it. You're asking questions. You're asking questions. So you're kind of like, it's a reading companion. You're getting more out of the books. You read from that. But then you kind of take what you've read and throw it into this card generator.

Dwarkesh (15:37)

Yeah. So that's mostly it's just chatting with Claude and so let me see if I can find a better example. So I mean a lot of topics I just find I have a vague sense of what's happening, but I don't really get it. And it's super helpful to chat with Claude to make sure I'm on the right track. One was like Dylan has a couple of posts about why packing as a technology is super necessary for these advanced chips. I'm not trying to make this podcast all about AI hardware. It just happens to be the last podcast I did. So that's what you're getting. But then it's like a confusing. It's like a five series post about how advanced packing works and how you know what the technical specifications are. And I'm like, wait, step back. Why is this necessary? What's going on? All kinds of other questions about when there's questions about, I'm worried about I might get too deep in the weeds when I'm just explaining. Yeah, basically I'm just like, how do I think about the broader context of what's happening here? Because I really can't ask good questions unless I have a good mental model of um, what they're talking about. Like I really get where all this is fits together.

Dan (16:56)

Hmm, that makes sense. And so, and, and, and so Claude is kind of like the first thing you flip to when, when you, when you want to know that are you using it on mobile or using on desktop?

Dwarkesh (17:05)

Desktop.

Dan (17:06)

Okay, interesting. So you're doing most of your like reading and research stuff on desktop.

Dwarkesh (17:11)

Yeah, that's right.

Dan (17:12)

Hmm. And what do you think about like just Claude being really great right now and like chat. Like I assume your ChatGPT usage is lower than it used to be.

Dwarkesh (17:22)

Yeah, I think these things will keep getting better over time and I think just like getting in the practice of using these tools, I'll talk a little bit about how these tools relate to my post production process. Initially it was kind of useless, but I did spend a few weekends trying to write a few prompts and create a workflow. At the time it was basically useless. Now it's actually ended up being useful and I can use the same jupyter notebooks or whatever to get things done. So it is worth investing in getting even if they don't work perfectly now to get them part of your workflow so that as they keep getting better, you're Getting the returns from that.

Dan (17:59)

Yeah, that makes sense. So I want to just go back to the Anki card generator, the space repetition card generator. So as part of this, once you've done all the sort of conceptual, you know, clearing the ground conceptually for yourself to like kind of understand the, the basics of, of what a guest is talking about or an idea that you're interested in, then you're kind of, you're adding to your, your flashcards so that I, I guess so that it, you retain the information past even when you talk to that guest. Is that right?

Dwarkesh (18:30)

Yes, that's right. I mean, I, I, I think of the larger mission of the podcast is to cons. Why does a podcast get better over time? And it's because basically I'm getting smarter or learning more things. I'm reducing my ignorance around a bunch of topics. And so if I don't do that, I mean, I think about all the episodes I did before I interviewed Andy and started using space repetition and I just like really regret it because I talk to all these world experts in a ton of different domains. And to be honest, like, in many cases, I didn't take that much away. I vaguely remember some things and now that I use, I can walk you through the kinds of cards I make and the space separation tools I use, but it's like totally a game changer in terms of what I can retain. And in fact, I think it's not even about making sure I remember what I discussed in a previous episode or what I learned previously. It's more about future learning because I'm sure you've heard the saying about the, you know, learning compounds because you can use what you've learned in the past to learn future things because they all interconnect. Well, you can't do that if you basically forgotten most things you've learned in the past. So I, my learning has future other things has become much faster because I have cached all these different concepts and figures and facts. So. But future things are just like, I understand how everything sits together much more. It's not even about the past. It's really about future learning.

Dan (20:00)

Can we see your. I don't know what you use for space repetitions. Can we see your deck?

Dwarkesh (20:05)

Yeah. I will point out, by the way, as a side note, one use case of Claude that ended up actually being pretty useful. Sometimes you read obscure. I was reading Nick Land's selected writings about AI and his accelerationism, and I was like, what's going on? Genuinely, I'm like, what is his Argument. Basically, why does he think that the AI takeover and whatever thing he creates in the AfterMath will be Gu. Because he's a smart guy. I assume he has an interesting argument. So I upload the PDF of his selected writings. I just asked Claude, like, okay, so why does he think it's a good thing that AIs take over humans? And it offers a summary. Initially, this isn't necessarily that helpful because I kind of did read this in the essay. But what's helpful is that when you go through and I'm like. I respond like, I don't get it. What does he think is wrong with human society, that you have to erase it? And then he gives the explanation, and I'm like, I still don't get it. Like, what exactly you're talking about here? And then here's what I do with the podcast, right? I have the guest on, and I ask them, what do you mean here? I disagree. Here's a contradiction, whatever. And going through their writings with Claude and, like, have I actually found a sort of blind spot in their thinking? Or is this just me being confused about their ideas? It's super helpful.

Dan (21:18)

That's. That is really interesting. It's like, you can get. You can get down to a deeper level before you talk to them so that you can start there with them as opposed to, like, starting at the surface.

Dwarkesh (21:31)

Exactly.

Dan (21:32)

Which is really cool. I use that too for, like, for. For difficult books. Like, not necessarily for, like, interviewing the author of those books, but, like, for example, I interviewed Reid Hoffman, like, I don't know, a month or two ago, and I wanted to talk to him about the kind of, like, intersection between philosophy and AI. And he was like, he, like, almost became a philosophy professor, like, at Oxford. Wanted, like, was really deep into Wittgenstein. So, like, I read a bunch of Wittgenstein, which I hadn't read in a while, and I just used Claude for it. And it was like, so much better because I, like, I haven't taken a Wittgenstein class, or maybe I took one in college, like, a long time ago, but I've read him a lot. And just there are always those points in those kinds of books where you're like, I think I know what they're saying. But, like, I'd probably have to go to a grad graduate school and, like, get a master's in this to really know. And Claude is actually makes me be like, oh, I don't need that anymore. Any book I want to read, like, this I basically know. And it just helped me so much in that interview, because I could just ask Reid really deep Wittgenstein related questions and he could answer them.

Dwarkesh (22:37)

Yep, yep. I think that's totally legitimate. I think some people would be like, oh, you need to read it in the original. Blah, blah, blah. I think if you care about the ideas and you think the ideas are timeless and not. The ideas are not about the specific kind of prose that the original author used, but just generally, what is the essence and the gist of what's happening here? If you care about the ideas. And I think this is totally valid. I don't disagree with the people who are like, no, you need to read the specific syllables that Wittgenstein used.

Dan (23:05)

Yeah. I mean, I'm also just saying I have the book open and then I just take one of his statements and just throw it in there. And then it's like, here's what it means or whatever, which I think is really great. Um, okay, so you're gonna. You're gonna show us the. The. The Space Repetition card. So what. What app is this?

Dwarkesh (23:21)

This is Mochi. Interesting. It's like Anki. But I. This is the one I use.

Dan (23:28)

Why?

Dwarkesh (23:29)

So that. Actually, I don't have any cards today because I just went through them this morning. But let me give you a sense of. Let me give you a sense of what kinds of things I have. Right? So I have. If you go through history, recently, I've been planning on interviewing David Reich, who is a geneticist who explores human origins. And these are especially cases where, just reading the book, I'm like, I would have totally forgotten. He names all these different ancestral groups and how they combine and in what eras? When did the Yamuna people come through Europe? When did the Anatolian hunter gatherers wash over Eurasia? All these things that were just like, you read it in one year, it goes out the other one, unless you make cards for it. And so I made a ton of cards about this kind of stuff. There's examples of that here. It's especially useful for hardware and technical things here. I feel like if I don't make cards, I'm just constantly relearning the same things because I didn't learn the lingo in the right way first. It's not just about learning the terminology. It's about understanding the underlying concepts. Let me give you a good example of that. Maybe I'll sit back and I'll explain. Like, I go through these cards in the morning. If I. Maybe you can see what it kind of looks like. If I Do the cram cards thing, please. I can go through them. And right now I'm like, I remember this, right? I remember it's, this is the first one that came up randomly. But it's multi query attention to not have to use kv, huge KV values and then sharing KV values between layers and using local attention. And that's the answer. Now it seems sort of trivial right now because it's just like three things. But I would have totally forgotten about this if I hadn't made a card for this as soon as I read the blog post. And then it just like I've wasted my time in the future. If I'm learning about these technologies in a different context, I just don't have the connection to what was happening here to connect it to. Right. If I go to a different category, if I go cram cards, this is the, the white thing. I would have totally forgotten about it if I hadn't made these cards. Yeah, I'm just a big fan right now. I sort of, I've become a space repetition fanboy these days.

Dan (25:48)

How do you think about like the, the usefulness of space repetition in a world where like any of these questions is possibly, is like pretty much answerable, like with Claude, with like a, you know, one, one search?

Dwarkesh (25:59)

Yes. So I think it's about not necessarily remembering this information, but when a future thing comes in, you understand like the conceptual. In fact, let me give you a good example of this, right? So I remember sometimes I actually make cards about facts that I don't even understand in the moment, but in the future, the, as I learn more about the field, as, as a sort of territory becomes more clear, the things I said in the card make more sense to me. So if I, I was reading some of Colin Burns papers and so I made this card about like why Colin Burns thinks that alignment is a tractable problem or understanding what the model thinks is a tractable problem. And at the time I wrote things down about like, you know, features are in a linear space, what does that mean? Or like we can sort of see features in other sorts of categories. And at the time I'm like, I have no idea what this means, so I'm just going to write it down because I read the blog post and there was no point of reading the blog post if I'm not going to make the card. Later on, as I learned more about how the residual stream model of how attention works, what that is and so forth, this card made much more sense to me in the future. But I would have just Totally memory hold, or not even memory hold. I would totally forgotten this content which required future understanding if I hadn't made a card of it. And then when I see the card again in the future, I'm like, oh, oh. This is what Colin Burns meant. Now that I understand how attention works, this is what it means, right?

Dan (27:28)

Yeah, this is, this is really interesting to me. So I want to get into some of the, like the ways that you use AI for, for interview prep because I think we've mostly covered the reading stuff. But before we do that, I just want to like, understand like what is driving all of this. Like it feels like you are just consuming massive amounts of information and turning that into knowledge in your head. Like you have this sort of just overdrive of curiosity, which I actually resonate with a lot. Like I'm, I'm surrounded right now by books and I'm just sort of like curious for you. What, what that, what do you think that's about?

Dwarkesh (28:10)

I think I, I've, I, I, I really just want to know everything. Right. It's, I don't know how to express it. There's a beautif in a Will Durant book as he's turning 90, where he's writing a memoir basically of his main ideas called Fallen Leaves. And there's a passage on philosophy where he says, you know, as, as you get older, maybe with all the philosophy and history I've done, I can, I've reached some plateau of higher understanding and clearer insight or at least I've understood that such a thing is possible. And something like that just resonates with me. I don't know, I just, like, I find that idea really appealing. I'm nowhere close to it, but I just, hopefully in the years to come that'll just be a thing that I also really admire people I've had on the podcast who do have these self consistent and really deeply interrogated world models. You know, I've interviewed these guests and some of them people, a couple of names come to mind. People like Carl Schulman or Tyler Cowan or Bern Hobart. It feels like they've really read everything and you know, everything you know is a subset of what they know. And I just, I find them to be super compelling as thinkers. Of course there's many things they can still be wrong about. I don't, I'm not one of these people who buys like there's like a thing where you just know everything and you can never be wrong. Right. You always have blind spots. But the ability, their ability to, which you can see when you talk to them to connect anything you ask them about and they're like Claude six in the sense of like, you start talking about why has a fraction of finance as a percent of gdp. I remember asking Tyler this and he has a right off the cuff, just a super interesting answer that connects a bunch of different disciplines. You ask Carl about like how fast AI hardware could grow and just like done this sort of Fermi estimates on how fast algae bloom and how much solar power they consume and how many fab CSMC is making. Just like I find that sort of compression of the input they've ingested over their lives. And they can. Not only do they know that stuff, but they can really connect it in a really interesting and compelling novel way. I find it super compelling.

Dan (30:26)

And in terms of developing your own worldview, do you have that anywhere where it's like you're creating some sort of living document, or is it just all in your head? Like all the stuff that you're learning, Obviously you have the cards, but that feels more like dots in the space rather than the ways that they all connect and how you think about everything all together as a system.

Dwarkesh (30:47)

Hmm. I think I've been trying to do more of this recently, and now that I've sort of built up an underlying maybe vocabulary or understanding because of the podcast, it makes sense to do more of this, something I've been doing recently. Let me pull this up. I've only just started. Hopefully there'll be more by the time people are looking at this. But I've started writing riffs on different books or things I read. And if I go to, it's basically on my website. And so just like I can read a book and I have questions or I connect it with other things I've read. I remember, for example, when I was in Steven Pinker's Language Instinct, he was writing the book before the Fox P2 gene that can help explain human language was found. And so he has all these observations that are then later explained by the FOXP2 gene. And so I can just sort of the sort of connection that you're talking about, I can do by riffing on other people's ideas. I actually am curious. Do you have suggestions on what I should be doing? Maybe I should be writing more blog posts or what do you suggest I should do?

Dan (31:59)

Well, that's a good question. So. Well, before we get there, one of the things that this reminds me of, I think Claude is so good for reading old science books because it can tell you what's outdated and what's not. Um, I do that all the time and I, I love that. I love that little thing. But yeah, I mean, I think basically like developing a worldview is like, you have to just try, you know, and you try over and over and over again. And I, I do think like blog posts are really good for that. Especially like, you know, for me, like, I have to write every week and so I'm like forced to take a view on something. And generally if you're like intellectually honest, you like, want one post to like somewhat like agree with the last post and your audience will call you out if you're like just disagreeing with yourself all the time. So you're kind of developing a worldview that way. But for me, like right now I'm actually like my big thing this, this quarter is like, I, I just have these like, ideas that are simmering that are like sort of the relationship between like language models and like some deep philosophical questions that we've been like, talking about since like Plato, which is like the appearance, reality distinction and like, how do we know what's true and what's knowledge and all that kind of stuff. Like, I think there's a lot of overlap there and it requires like, it's going to be like a 10,000 word post or something like that. And so what I'm doing is I just have like a Claude project with like, I have all these like little notes and riffs and like, and stuff. And I'm just like going into Cloud and being like, hey, like, what's, what's the thread here? Like what's going on? Can you help me like figure out like, there's something in me that I have all these little ideas for, but I can't quite like put it into an argument that all makes sense. And I think just honestly, like sitting with that for like a couple months, I will, I will know what's in there. But there's something in there and yeah, it's cool.

Dwarkesh (33:45)

Do you, do you make a clawed project to like, here's some of the things I'm thinking, how do they connect? Or like, how do you keep track of those things as over those months?

Dan (33:52)

Exactly. I mean, I'll just show it to you. Let me just pull it up. So, okay, so if I go into Claude, I have a couple different projects. One project is seeing like a language model, which is the title of this big post, whatever it is. Another is Zen in the Art of Motor Scout Maintenance. So this is like a book that I'm reading as prep for this, this piece that I'm writing. I've read it a bunch of times before, but like now I'm like doing a little bit of a deeper read. And so I have like the, I have the, the whole book uploaded and then I can like ask questions. Then I have another one that I love called My Psychology, which has a bunch of like journal entries, goals I've set for myself over the years, and then also like things I've observed about my psychology or things I'm working on, like little aspects of myself that I'd like to grow or change. And so when I'm making decisions or thinking something through, I just go in there and it can reference all that stuff so it knows who I am, which is really cool. So in seeing like a language model, let me see if I could pull it up in the projects directory. So I have like, basically I have this one note in Apple Notes which is like every time I have a little thing come into my head, I'm like, I just, I just put it in there. Like, let me see if I can find it for you. I just throw it in here. And this is like huge and messy. And it's like different quotes from different books and like, just different ideas that like come to me off the top of my head as I'm like walking around. And I think that there's like a thread here in all of this stuff. They're all like. I can see how they're all related but. But like I can't quite pull it out. And so like what I've been doing is I just like throw it all in here. We have all the quotes and all the ideas and fragments. I have a little bit of a draft, like an intro. And then I have a chapter of a book by Richard Rorty that I think is really good called Pragmatism as Anti Authoritarianism. That kind of sparked this whole thing. I read a chapter of that book and then I was suddenly rereading a bunch of Plato and Aristotle and I was just down this huge rabbit hole. And so like what I did, for example, is I put all that stuff in here and I was like, hey, I have a bunch of notes and some fragments of text for a long 10,000 ish word piece I want to write called Seeing like a Language Model. But I need to understand what I actually think and make a bit of an outline before I get started. In order to do that, I need to understand the patterns of what I've been thinking and writing down and Reading about. Can you suggest some ways that you can help me do this? I want to get from where I am to an outline. You have access to some fragments, notes, and early, unfinished intro. And it just has a bunch of ideas like thematic analysis or argument mapping or chronological development. And I'm just sort of going down the rabbit hole with it, where it's like, I asked it to do concept clustering. So it's like one of the concepts that I'm playing with is the philosophical divide, Plato versus Aristotle, which I think is not quite right. It's actually Plato versus the Sophists, but it's close. Or the evolution of Western thought. It's like, how does Plato. Like, how does Plato ladder up into the rest of Western thought and into science and just the way the Western mind works? And then how do language models sort of, like, differ from that paradigm? So that's the basic thing that I'm trying for. Because the reason I ask this question is because I'm selfishly, like. I feel like a little bit I haven't done the big idea thing as much as I really want to, because I'm. I. I am writing every week. I am sort of, like, reacting to stuff, and so I want to be a little bit more thoughtful. Um, and this is. This is my, like, attempt to. To, like, to put all of it together into something that, like, makes sense.

Dwarkesh (37:37)

Yeah. You know, as you're going through this, this really actually makes me want to write more. Because now that you're talking about it, now that you asked the question, I'm like, yeah, I. I should be sort of consolidating the things I'm learning in a more comprehensive way and in a way that's also more useful and accessible to other people as well. Right. I, you know, I spend weeks, like, learning about some random. Not random, but, like, the things I care about. I'm about to prepare for Daniel Jorgen, the guy who wrote the prize. It's a history of oil, or a geneticist, you know, AI researcher or whatever. To the extent that I'm getting something out of these research processes, I should consolidate it in a way that's not evident in the podcast itself.

Dan (38:17)

Yeah, I mean, I selfishly want you to do that, because I'm curious what you think.

Dwarkesh (38:22)

Yeah, I appreciate that. You will send these out through your newsletter, right? Is that the main way? Yeah, yeah, yeah.

Dan (38:29)

This will go out through the newsletter. I might do something special for it. Like, you know, maybe we'll. Maybe we'll make a little mini site for it when it. When it launches, but that's sort of in the future. I have to actually. I have to actually write it first. So. Okay, so, so, so let's, I want to move on. I want to. I want to talk about the. How. How you use AI to. To sort of like do. Do the interview prep. So, so let's, let's move into that and then we can also. We can also maybe even like prep for an interview together.

Dwarkesh (38:57)

Yeah. Yep. Okay, let's do it. I honestly, the interview prep is like, it requires a lot of work, but fundamentally what's happening is not that complicated. Like I can just show you a document I might have made in the past, share my screen. So honestly, it literally is just like I'm going through. I come up with a bunch of questions and I sort of group them together in relevant categories or if I go to. If I was interviewing Dylan Patel. Sorry, this is not the right one. Just like a bunch of different. Yeah, just like a list of questions basically. It's not complicated, but the process of coming up with them is, you know, very research intensive. So we can go through like, if I'm guess I might only barely started preparing for them. We can go through the process preparing for them.

Dan (39:38)

Yeah. Can we? Can we. I just want to stop at those questions like again, selfishly, because I think it's really interesting. Like you have these like long lists of questions that are organized by theme. Are you like going down the list or are you sort of jumping around?

Dwarkesh (39:50)

Yeah, so it's really interesting because I come up with these list of questions, but it's like it really never ends up being. I ask question one and I ask question two and I ask question three. You know, I start off with an interesting question and if you listen to the interviews, hopefully it comes off more as a almost conversation because I spend so much time preparing that I have these questions basically memorized. And so the next one that is appropriate to their response, if they say something about memorization and LLMs, I'll have a question prepared about that or related to that and I'll just ask it next because that's what fits in together. And so I'll have a list and this is what I'll send them if they ask for it. But really it just sort of me off the cuff like here's a question I remember that was relevant to this in, in the actual interview.

Dan (40:41)

That makes sense. So like the, the point of the doc is like, it's almost like writing the doc is. Is the prep itself and it's you don't even necessarily need it in the interview. Like maybe you have it just in case. But yeah, that makes a lot of sense. Yeah.

Dwarkesh (40:55)

And then, yeah, we can even go through, let's see, I'm doing a couple of interviews in the future. David Reich and David Reich and Daniel Jurgen. So the one, the former is a geneticist about human origins. II wrote the prize, which is the famous book about the history of oil. Which one sounds more interesting to you? We can do that. Whichever one.

Dan (41:17)

I want to do the famous geneticists.

Dwarkesh (41:20)

So let's go to Claude. In fact, I do have his book uploaded as a project, so we can just use that.

Dan (41:24)

That's great. And so basically what we're going to do is we're going to watch you and I'll do it with you. We're going to prep for an interview with this guy. What's his name again?

Dwarkesh (41:35)

David Reich.

Dan (41:35)

David Reich. Okay, cool. Can we get a little bit of background on David Reich? We can even ask Claude because like I'm, you know, obviously I'm an, I'm a newbie to David Reich's work.

Dwarkesh (41:44)

He is a geneticist at Harvard. And over the last decade or so their research into how have human populations across the world been formed. Basically, like how do they. Who are the Europeans, what groups make them up, what ancient migrations and genocides and population replacements made them. Same with the Indians or Native Americans or Africans. It's completely changed. I mean they've basically sort of made many academic disciplines irrelevant because they actually have empirical data on like here's actually what historically happened. You guys are completely wrong about what you think, your theories of what happened. If you're familiar with Nat Freedman Sisuvi's Challenge, you have these burnt up squirrels, but with some advanced techniques you can get some useful information out of them. I feel it's in a similar vein. Obviously they're not the same kind of project, but it's a similar vein of once we develop the advanced mathematics or genetics or whatever to understand what's latent in the genome, we've just uncovered a ton of insight about what's been going on in human history. Basically. Sorry, I'm just getting nerd sniped and just going on riffs here. But one of the interesting things is you can see when one population replaces another. Whether it was just like, oh, we met and we were now intermingling in trading and whatever, or is it like we're committing genocide against you? And you can tell that because in the case where it's Genocide or population replacement. It will be that the male line of the population that is invading will overtake the male line of the existing population, but the female line will remain. So mitochondrial DNA only comes up in the female line. And you'll see the female line because they're getting the new men who are coming in, are taking them as wives or something. And then anyway, so you can just learn a lot about like, what kind of invasion was it? Did they like conquer or was it. Were they just like mingling or something? One of the many things you can.

Dan (43:44)

See from the DNA that's really interesting. Wait, and so, so this is like basically reexamining DNA evidence of like old settlements and like, basically. And he's uncovering new ways of being able to analyze the DNA. Like what's the, what's the, what's the new methods that they're using to like draw new conclusions from existing ev.

Dwarkesh (44:03)

One of them is just that, right? Like seeing how the Y chromosome and the mitochondrial DNA. Because you can just learn a lot about a population based on how the female versus male line is propagated, about what was the social structure like and so forth. Another is you can even tell the level of inequality in a society because if there's a lot. So for example, in India, one of the things that was super surprising is that the amount of endogamy, which is to say that the. A certain caste in a certain village would just like, not. There wouldn't be any sort of intermixing with another caste in a neighboring village. Like to the extent that's true of nowhere else in the world. And they were able to find this in India, where the amount of social stratification, you can see that in the genetic catalog over the last thousands of years or for thousands of years, these two neighboring castes haven't mixed with like 99% or something, which is like even from sort of infidelity or rape or something. You would expect there to be more than what actually ends up being the case. So you can understand modern culture in India based on what has happened over the last few thousand years.

Dan (45:13)

That's really interesting. So I want to, like, I feel like you're doing like such a good job of summarizing his main ideas. But I kind of want to. I kind of want you to do the same thing with, with Claude so we can see how you stack up versus Claude because obviously you've input his book into this project, so it has that as reference material. Can we ask it to just summarize a few of his Main ideas.

Dwarkesh (45:38)

Yeah, that's a great idea. Can you summarize and maybe the techniques he used to come up with.

Dan (45:48)

With this perfect new. So what you're writing is, can you summarize the main ideas from the book and the techniques used to come up with new insights? Cool. And one thing that's like, really cool about this is like, you've been able to do something like this with ChatGPT for a long time, but ChatGPT's context window isn't that long. And so it chops it up and like it's not gonna. It's not gonna really be able to summarize the entire thing because of that, you know, it has to like, find the right parts of the book and the embedding search in it is not very good and all that kind of stuff. And Claude, you can just throw a ton of stuff in the context window and that just makes a big difference. Okay, so it looks like we've got some answers.

Dwarkesh (46:31)

So it tells us that the ancient DNA has revolutionized understanding of human prehistory. And then we've learned that populations today are the result of multiple waves of migration and mixture. And then just like a bunch of other genetic stuff. Then it talks about the key techniques about whole genome sequencing and how they've enabled these sorts of new discoveries they've been making. Yeah, yeah. But anyway, so there's a bunch of interesting things about their research.

Dan (47:04)

Well, now I'm interested in like, okay, so the key techniques that it's using are whole genome sequencing of ancient DNA samples. So is whole genome genome sequencing like a new thing? That you can do it on ancient DNA samples? So it's saying, like, by improved extraction and sequencing technologies. Is that. That's like.

Dwarkesh (47:21)

Yeah, that is an interesting question. So we can even ask a lot of. Because I'm not sure how. How exactly do you sequence an ancient or, you know, a prehistoric.

Dan (47:40)

Genome?

Dwarkesh (47:42)

Like, can you do, like, how does that work? Right. Okay, so they grind the bone and they have then techniques to get the DNA out of that. Now another thing we can ask is like, one thing I'm curious about. Let's see. I don't really remember the chapter on Native American. So I could ask about what exactly happened with Native Americans. Here's one thing I'm curious about. Like, how would. Okay, so I don't even know if David Reich addresses this himself, but like, how would David Reich's theories help explain why civilization suddenly emerges so rapidly? And that too concurrently in the new and the old world after 10,000 BCE aka the end of the last ice Age. And then maybe I'll just ask Claude why. I think this is an interesting question. So this seems like a really remarkable coincidence, given how long humans have been around.

Dan (48:57)

That is interesting.

Dwarkesh (48:58)

Well, coincidence, given that humans. It'll correct my spelling. Given that humans have been around for hundreds of thousands of years.

Dan (49:17)

I didn't realize that we believe that it emerged at the same time in geographically disparate places. That's totally new to me. I thought it was just in Mesopotamia.

Dwarkesh (49:28)

It's actually. There's a really good book by Peter Jackson called the Great Divide, and it's one of the most interesting books I read. Just as a side note, it's about. Yeah, it's comparing the emergence of civilization in the New World versus your world. So in the old world. Sorry, in the New world, the coral is like a civilization in 3000 BCE and it's based on fishing and not on conventional agriculture like Mesopotamia. And he talks about how that changed the evolution of the culture in the New World versus the Old World. But anyways.

Dan (50:01)

That'S really interesting.

Dwarkesh (50:02)

Okay, so major population movements and mixtures.