Loading summary
Transcript38 lines
- [00:01]
Derek Thompson
Hey, it's Craig Horlebeck here to tell you that the NFL is back whether.
- [00:04]
Cal Newport
You like it or not.
- [00:05]
Derek Thompson
And we are covering all the latest news, trades, rankings and more on the Ringer Fantasy Football show with my two.
- [00:11]
Cal Newport
Co hosts who are both named Danny.
- [00:13]
Derek Thompson
Check the Ringer Fantasy Football show out.
- [00:14]
Cal Newport
On Spotify or on our new YouTube channel.
- [00:18]
Derek Thompson
This episode is brought to you by Workday. When you're a forward thinker, you don't just bring your A game, you bring your AI game. Workday is the AI platform that transforms the way you manage your people, money and agents so you can transform tomorrow Workday, moving business forever forward. This episode is brought to you by KPMG Making an Impact is how KPMG Helps make the Difference KPMG applies advanced tools and strategic thinking to convert data into actionable knowledge and deliver value by improving performance through transformation, modernizing processes with technology, harnessing the power of data, navigating complex MA transactions, and enhancing trust among stakeholders. Go to KPMG US Advisory to learn more. KPMG make the Difference Today AI and US Last week, the Bureau of Economic Analysis published its latest GDP report and it contained a startling detail. Spending on artificial intelligence added more to the US Economy last quarter than than consumer spending. That means the only reason this economy is growing right now is because of the boom in artificial intelligence spending on advanced ships, data centers and energy. I think this is one of those stats that's worth sitting inside for a few seconds. We talk a lot about what will happen if at some distant date, AI eats the economy of the future, dominates the economy of the future. But we don't need to wait for the future. AI is driving this economy right now. Today. This is already becoming an AI economy. If, for example, you wanted to know why is GDP growing rather than stalling, the answer is AI. If you want to know why the stock market is soaring rather than stalling, here again, the answer is AI. According to Michael Sembelist at JP Morgan, nearly two thirds of stock market returns in the last two years have come from a small number of AI related stocks like Microsoft and Amazon and Meta. I'm working on several shows right now on what an AI economy will do to work and productivity and our jobs. The effect of AI on the economy is a really interesting and important question to me. But I'm not just interested in how AI is changing the economy. I'm interested in how AI is changing us. Our habits, our behaviors, our patterns of thinking. Some commentators have analogized artificial intelligence to the railroad build out of the mid to late 19th century. Both of these are capital intensive technologies that could transform the way we do work. But railroads, in addition to just transforming the economy, also transformed a way of thinking about the world. The science writer James Glick has suggested that it was railroads, more than any other tech, that introduced the idea of time travel to our culture. Railroads made the introduction of time zones necessary and they scrambled people's sense that there existed a universal time for all human beings, Glick writes. Travelers riding in steam driven railroad trains looked out their windows onto a landscape where oxen plowed the fields as they had done in medieval times. Horses still hauled and harrowed, yet telegraph wires split the sky. This caused a new kind of confusion or dissociation. Call it temporal dissonance. Technology profoundly changes the way we think about the world around us. So how will AI change us? Several months ago we did a show here called the End of Reading. I had on the Atlantic staff writer Rose Horowitz and Nat Malkus of aei, where we looked at surveys finding that leisure reading has declined significantly in the 21st century. And according to Rose, many professors at elite colleges now say that their students have never in their life read a full book. All they've read in high school is just little snippets of books to prepare for the PSAT and satisfaction. We are losing now, even at the highest levels of education, the ability to sit with a piece of text, read it, and make connections within it. Soon after we did this episode, New York magazine published a viral cover story on the epidemic of cheating in college. This was a long piece with a very short tldr. Practically every student in college, and I would assume high school, is using ChatGPT to write all of their essays and also practically nobody. The teachers, professors know quite what to do about this. The ability to use AI to summon into existence any essay has exploded colleges and high schools ability to assess intelligence with take home essays. The students are just gonna cheat. In other words, we aren't just losing deep reading in the age of AI, students are sacrificing deep writing as well. Today's guest is Cal Newport. He's the author of several bestsellers on the way we work, including Deep Work. He's also a professor of computer science at Georgetown. I wanted to have Cal on the show because one of the questions I get the most by email or in talks in conversations with people about the news these days is if these tools, if AI reads faster than us, synthesizes better than us, remembers more than us, writes faster than us, what's our Place in the loop here. What skills should we value in the age of AI? And even more pointedly, what skills should we teach our children to value? How do we ride this train without getting run over by it? I'm Derek Thompson. This is Plain English, Cal Newport. Welcome to the show.
- [06:53]
Cal Newport
Happy to be here.
- [06:55]
Derek Thompson
I want to start with observations before we move into analysis. In your reporting, in your experience, have you seen major changes in the behavior of students in the age of AI?
- [07:06]
Cal Newport
It's shifting rapidly. Like in my own case, for example, last year I did a story for the New Yorker about how are students using AI in the context of writing. And the premise of the story was I looked over the shoulder virtually speaking of a couple students to actually watch how they were interacting with AI. And what I found in that story, like the thesis of that story, is that a lot of what students were doing when they were writing with AI, they were not outsourcing writing to the AI. They weren't saving time either. So it wasn't an efficiency play. My argument was they were having these almost parasocial back and forth interactions that were aimed at reducing the peak cognitive strain of the process of writing a paper. So it was this interesting sort of interactive back and forth. They would have, well, what about this? What do you think about that? Okay, I'm going to write this. What do you think about it? And, and my thesis was it was trying to make the peak difficulty of writing better when you look today. So like let's fast forward a year later. There's been multiple things that have reported on this more recently. And it seems like this parasocial relationship, this sort of I'm in this interaction with you to sort of take difficulty, the smooth over hard peaks has really made me metastasized. So there's a big new article out recently from the Guardian where reporter convinced, I think it was three students to basically let him have full access to their ChatGPT+ accounts so that he could see every single thing that they were saying over a period of whatever it was, a couple weeks, there's a huge transcript and he says their interactions with it was constant. It was just all day long. Not just about homework assignments, not just about a paper they were writing. It is what does it mean to be a human being? What do you think it means that this girl said this to me when we were walking by in the cafeteria? That it had become a sort of interactive back and forth partner. So there's this interesting thing I think happening among young people with these tools where it's Expanding beyond its role in. Pragmatically speaking, this is helping me do certain work which we can get into because it is, and I think it's making a big difference on how college actually operates. But it is immediately seeming to move over into the space where social media and other tools were as well. In this sort of diversion, distraction, tickle your brain type of context that it is. It's interesting, it gives you a release in the moment, it prevents you from being bored, it prevents you from having to feel some sort of negative strain. So it really expanded its footprint. It looks like it's starting to expand this footprint in students life more than I thought it would. So a lot of things are going on here.
- [09:38]
Derek Thompson
One thing I hear you saying is that while a lot of people think about AI as an economic technology, you're watching the way that it's already become a social technology. You talked about the parasocial relationship that people have with the large language models they're interacting with. They're talking to it like it's a professor, like it's a friend, like it's a research assistant. That's how I use, say, deep research. Let's ground this at the level of college. How has this changed the way students and professors operate?
- [10:09]
Cal Newport
So if we look at it just from a functional perspective, it's changed a lot of things. Like for example, during the pandemic, I teach mainly math type courses or theory type courses. During the pandemic, obviously you had to administer these exams remotely because there was no one in person at all. There was a sort of brief window after the pandemic where it was thought I was like, oh, this is convenient. Like why not keep doing it this way, right during exam period. This way we don't all have to stick around for five days instead of going home to take the exams. Let's keep doing this online. I was doing it synchronously online essentially post chatgpt. You can't do that, not for an introductory discrete mathematics course, because every single problem on that exam can probably be answered for you by ChatGPT. Because it's, it's not only is it not that it's not very complicated math, but as I learned when I've experimented with using ChatGPT to help write problem set problems for a lot of these problems, it turns out there's only so many good examples that are out there. There's only so many actual ways to test like a sophomore level undergraduate on doing strong induction proofs or something like this. There's really like four Good examples and you can obfuscate them, but that's that. So they could. You could put almost every problem in the chat GPT, right? So, okay, we can't do remote exams anymore. I think the same thing is happening. I picked this up in my reporting. Whenever you have lower stakes writing, there used to be a big thing where you would say, come on, have a response essay at the beginning of each class. This way, you know, I'm not going to look at it too carefully, but you have to do the reading. And because you had the right 500 words, like your thoughts or whatever, as one of the students told me on my reporting, oh, those type of essays are eminently chatgpt able. That was the phrase, because exactly that type of writing is very chatgptable. When it comes to large papers, what seems to be happening is we thought when this technology first came out, we being professors, that maybe students would be able to essentially recursively generate the entire paper from scratch. You can't ask ChatGPT, give me 10,000 words on Jung and the collective unconscious. But what you can do is say, give me an outline for an essay like that. Okay, let's look at section two of this outline. Break that down into the three subsections. Okay, let's look at this subsection here. Can you give me some drafts of text? So there was some fear that doing that you could basically have a whole essay produced. It would be like a custom version of those cheating websites that were around when we were growing up. Or you could pay for papers on the Internet. That's not working too well. Those papers aren't coherent and the tone isn't that great. So on long papers, it's more that students are using it for ideas. They're using it for. Give me a structure for this argument. It's less about the craft of creating the actual words than it seems to be the craft of actually critically thinking about what I want to say. So that is creating changes. I think for the humanities, there's been a move towards a lot more in class assessment. If it's a quantitative class, there's a move towards. Probably what we need to do is have a quiz in the first part of class once a month and weigh that more than we're going to weigh problem sets. There's more of a move for intro mathematics classes that the problem sets are basically for practice. Right. Like you should do these problems, that's how you're going to practice. But it's going to be less of the graded assessment. That temptation is too Hard that you could basically solve any one of these problems. So there is definitely a big shift that we have to deal with. It's maybe comparable to the. I think the consumer Internet caused a comparable size shift in higher academia. That. That introduced a huge amount of changes. I mean, I was a student and grad student. Just as that got really big, that introduced a lot of changes we had to adapt to. These are kind of similar, I think what's happening, but they're numerous as well.
- [13:48]
Derek Thompson
Cal, there are so many committees right now at colleges and universities that are trying to help professors and deans and students figure out how do we incorporate AI into a college education. And I was having a conversation with someone at a Washington, D.C. college about this question of what should schools do in the age of AI and we sort of cobbled together an idea and I'd love to just throw it at you and have you tell me if you think this is remotely plausible. So my wife just finished her PhD in clinical psychology. And I love the fact that the verb of the Last verb of PhD is you defend your dissertation. You don't write your dissertation, you defend it. And it made me think, should we turn more of college into a defense? Right. What's chatgptable is writing an essay about the Habsburg Empire. What's not chatgptable is defending your essay about the Habsburg Empire in front of your history 105 class on ancient European warfare. And I wondered if turning college into something that was more stand up, more presentational, not only are you graded for defending your paper, I'm also going to hand out grades or incorporate the variable of how good are your questions of the person presenting at the front of the class. Is it possible, do you think, or just totally fanciful to try to transform more classes into the equivalent of dissertation defenses?
- [15:18]
Cal Newport
Well, I mean, it's not only possible some universities do this. I mean, this is largely the Oxford model where what you do is have tutorial with your tutorial head and it's you, however many times a week, basically explaining and arguing to them about what you read and they ask you questions about it. And that's where most of your evaluation, you know, comes from. Tyler Cowan made a similar argument recently as well. They said, okay, colleges need to shift to become much more participatory and also much more of, yeah, I don't know if you use the word definitive, but this sort of like, yeah, oral and interactive in that way. There are a lot of tools. I mean, one way to think about it is traditionally a lot of college education Was in some sense like this, right? I mean, the blue book exam is this. It's not out loud, but you're filling in those composition books there in the exam room from scratch with nothing there. You're making a written argument where you have nothing to support you. I mean, that's how I took exams when I was coming up in things like mathematics and science. I mean, this is what written exams and quizzes are. It's like you have to basically demonstrate your knowledge of this, how you got the knowledge. Like we gave you help, we gave you lectures, we pointed you towards things. But like essentially you have to sit down now and answer a lot of questions. So I think we do have some of those tools. The big, the big questions. Here's the two things that the big questions are about is problem sets and papers, right? I think that's really the crux of it, right? So if you're in a technical class, problem sets, which means here is a set of problems you do at home and then you bring back in the class, right? So if you're not a quantitative major in college, this is like one of the core things you're evaluated on. These are tricky. These are really tricky in the age of ChatGPT. And it's a problem because problem sets can give you hard problems, so they serve a purpose, right? Like I'm a theoretical computer scientist and when I'm taking theory classes, right? Like I took theoretical computer science at MIT from the head of the math department there at the time, Mike Sipser had like written the definitive textbook on this. The problem sets is where you really got your chops. Because in an exam you only have so much time, the questions can only be so hard. But a problem set, they could give you one problem that could take you four days. And so it's hard. That is hard. How do we replace that? Papers you take home is that's the other hard thing as well, right? Like how do you replace that studious systematic effort of gathering information and organizing your thoughts that's lost when it's just in. I'm going to defend something in front of a tutorial, right? Because I have to understand it and I have to know rhetoric. But there's not that I'm in day five of trying to organize my thoughts on this, and now I'm finally seeing how it comes together. So I think that's. I don't know what to do about that. I don't know what to do about problems. But I think in general, what you're saying is. Yes, is what's happening? I mean, maybe it's not out loud, but I just. More assessment is moving towards. Let's think of it as like real time pedagogy demonstrations. Real time pedagogy demonstrations are becoming once again, much more important.
- [18:15]
Derek Thompson
The question that I get most in this space is people looking at the rapid advancements that AI is making at writing, at math, at biology, at every single discipline, and saying what's going to be left in 10 years? And the rubber hits the road with a question as specific as what should we teach our children in the age of AI? As someone who's really close to this space and who I believe has kids, how are you thinking about this?
- [18:43]
Cal Newport
Well, I would be wary at continuing to extrapolate up the curve. So I would be wary at extrapolating from GPT4O produces more fluent text in 3, 5 produced, and then continue to extrapolate and say, okay, so writing is not something that we'll be able to do in the Future. Or say GPT4O. They did well on the International Math Olympiad. So math is something that we're not able to do in the future. I like to be a little bit more grounded in this. What I'm interested in less right now from a journalistic perspective is what could be replaced by AI. I'm much more interested in what has been replaced because that will tell us a lot about what might be replaced in the future. Now here's the issue is the answer to that right now is it's pretty narrow and fractured. We are still very much in a moment right now of potential that we are seeing is going to be played out soon. We're still in an era of benchmarks. We're still in an era of. In these measures. This model is now doing better. On this graph of the frontier benchmark. We have this other benchmark on reasoning, but we don't yet have a lot of examples of. Here is a small economic sector that here's a skill you could build for and it is gone now. Not it could go away. So we kind of need to see those to understand what it looks like when AI moves into an economic sector in sort of a more fundamental way. So I don't know how to answer that question yet. I mean, outside of like some pretty niche things, I said let's, let's wait two years. I think I'd have a much clearer answer to give my kids right now. We don't have enough. It's like early in the Industrial Revolution, like we haven't replaced any of the looms yet. We just know there's these technologies coming, but we don't quite know what it's going to look like yet when a factory automates. Right. That's where I think we are in AI. And it's also unclear exactly how far that's going to go or how many of these fields are actually going to end up being taken off of the plate of possible economic pursuit. So I don't know how to answer that yet because there's not enough case studies to look at.
- [20:49]
Derek Thompson
Can I offer an answer that I've been thinking about? This is a general answer to the question of what should we teach our children? What should our children value? Might be even closer to what I'm trying to get at here. Was talking about this recently at a talk in exercise in weightlifting. There's this concept called time under tension. So you can do a bench press in three seconds, or you can do a bench press in 10 seconds, or you can do the same bench press in 20 seconds. And slow, slow, slow up, slow, slow, slow down. It's the same rep, but it's much harder. It's time under tension. I feel like we're in an age right now where young people are reading less. Book reading rates have really declined significantly, even at elite colleges. And now with AI, as you've been explaining, students can write less because Jack will always be game to do your homework. And I feel like if students aren't reading as much and they're not writing as much, where's the thinking coming from? The best ideas that I've come up with tend to come from me being able to sit with a group of thoughts that are far flung in far flung departments of my brain and having the patience to sit with them for a long period of time. And they cohere into something that's combinatorially new. And I think of that as kind of the cognitive equivalent of time under tension. Right. Without the capacity for long form reading or writing, I worry that we're just going to lose that. It'll just be gone. And so my answer to this question of what should we teach young people? What should they value academically? I want to. I would want my children to be masters of cognitive time under tension, a kind of academic patience that I think will pay dividends, whether they want to be a theoretical computer scientist or a novelist. How does the idea of time under tension sit with you as you think about some of the awkward conveniences of AI for students?
- [22:43]
Cal Newport
Well, I think this is a key shift. You're making here because I agree with you and I think it's a key shift in the way that people are thinking about AI Right? Because what I was responding to is what I think is a key way that people are thinking about AI right now, which is what are the things AI is good at that now I have to find different locations for me to go like, what is it becoming so good at? That I have to go in another lane where really the issue that matters right now is what is AI making me bad at? And I think that shift in perspective is a really important one. Now, I've been talking about this for about a decade because basically AI as I see it, is the latest of multiple heavyweight entrants into this prize fight against our ability to actually think. So it is the latest thing taken away. I think it was ubiquitous access to highly salient distractions. So what was delivered through the smartphone revolution, this was the final nail in the coffin, I think. Plus also streaming and highly available high quality content, but really the ubiquitous, pocket accessible, optimized entertainment, that was the final nail in the coffin of reading. And now we see AI is coming in and taking out this other thing we had left to put mine under tension, which was writing. And it's either taking that away or like we talked about, taking the time under tension required from writing and reducing it because it can make it this more interactive thing where you don't have to hold ideas in your head in a very difficult way. It gives you this sort of cognitive relief. So, yeah, it's a problem. And it's a problem that's in a bigger constellation of other things that are causing the same problem. Because I'm a big believer our current world and economy, it's a system symbolic economy. You have to be very good with systems thinking and symbolic ideas and thinking. That is since basically the rise of the Neolithic revolution, that has been the evolution we've demanded of our brain. Right? We have brains that were evolved in the Paleolithic. They're pragmatic brains for a very functional purpose, which is surviving in bands in a sort of hunter gatherer, forager, small band type of setting. We culturally changed our brains. This was a technology we figured out how to do is how to. It rewires our brains. How do we rewire our brains to be able to do system symbolic thinking? Reading was the real driver of this. This is a cultural evolution. This is not something our brain is wired for. Reading forces us to make new connections in our brain that did not exist. That you would not find these Connections in the average person from the Paleolithic. And it's what enabled all of this sort of system symbolic thought that Yuval Harari would talk about as being sort of key to the development of all of human history. Right. I mean, this is grandiose types of stuff. And then writing helps you take these circuits you build with reading and apply them. And so figure out how do I take these circuits and aim them at things that I actually care about. So we have this one, two punch reading. Writing literally changes our brain, right? It's like this serum we have to take in a superhero comic book to gain this superpower. And so I had been ringing this alarm bell probably all the way back to my book Deep work that it's something that we have to keep taking the serum for. We have to work on our brains to have this sort of arbitrary, constructed, culturally constructed type of brain that we need to thrive in today's world, and that we should be super wary when stuff takes that proverbial serum away. So the death of reading because of smartphones, I'll say the final nail in the coffin. I mean, that was on a downward trend for a while. And now, as you're pointing out, I'm 100% on board with this writing, which was the other part of this. AI is taking a lot of the hardness out of that. Yes, there is a, I think, a sort of like social impact here, a sort of personal flourishing impact here. This is where I think the economic impact we should care about first is, is what happens when we have a knowledge economy that depends on our brains. We make the brains worse. Like, I care more about that than what is going to be automated by these tools. That's like, at least just as important. So I'm with you on here. This sort of time under tension, as you say, is critical. And it's been under a sort of unprecedented attack over the last 13 to 15 years.
- [26:42]
Derek Thompson
In thinking about this cavalcade of heavyweight fighters, you just described, right, there was television, which is a distraction. Computers, a distraction. Smartphones, even better distraction. These screens were distinct from one another. It's not like the smartphone is just more tv. It's also interactive. It's a conversation in a way that watching television is not a conversation. How do you see AI not just being more smartphone, if you get my drift? How is this new distraction distinct if what we care about here is the preservation of our mental acuity?
- [27:20]
Cal Newport
The way I often think about it is reading builds these smarter circuits and production of content is what helps you actually use the circuits and get good at actually applying them. Right. So Marianne Wolf uses this term deep reading processes to talk about what actually happens in your brain when you do hard reading. But to learn how to apply those circuits, to make an original idea, to have original argument, you have to actually practice the production of complex content from scratch, which is like what writing has you do. What is AI really good at is automating production, right? So phones distracted us from consumption of things that could make us have better circuits. AI, the fear would be that it reduces the amount of time you have to spend producing original content. Ex nilo. You take that out of the equation too and things get even worse. So yeah, I think these are focusing on different things and you could look at television. You're right, that was something else. I mean, that really hit sociality, among other things. And so you're right, every one of these technologies is different. And so it's the reduction of production that is where I think AI is a problem from a cognitive perspective.
- [28:33]
Derek Thompson
This episode is brought to you by Zendesk, introducing the next generation of AI agents built to deliver resolutions for everyone with an easy setup that can be completed in minutes, not months. Zendesk AI agents resolve 30% of interactions in instantly, quickly giving your customers what they need. Loved by over 10,000 companies, Zendesk AI makes service teams more efficient, businesses run better and your customers happier. That's the Zendesk AI effect. Find out more@Zendesk.com hey, it's Ryan Reynolds here for Mint Mobile. Now I was looking for fun ways to tell you that Mint's offer of unlimited Premium Wireless for $15 a month back. So I thought it would be fun if we made $15 bills, but it turns out that's very illegal.
- [29:25]
Cal Newport
So there goes my big idea for the commercial.
- [29:27]
Derek Thompson
Give it a try@mintmobile.com Switch upfront payment of $45 for a three month plan equivalent to $15 per month required new customer offer for first three months only. Speed slow after 35 gigabytes of networks busy. Taxes and fees extra. See mintmobile.com there's been a couple studies here that I want to get your brain on. One published by the MIT Media Lab that got a lot of attention in the media, called your brain on ch. And I'm just going to tell people a little bit about this study before you and I talk about it. MIT Media Lab assigned a bunch of participants to several groups. Each group wrote an essay. Some wrote the essay on their own. Some used Google search, some used ChatGPT and the researchers hooked up an EEG to follow the writer's brain activity and then maybe interview them after they wrote these essays and basically found that of the three groups, ChatGPT users had the lowest brain engagement. Quote from the paper is they consistently underperformed at neural, linguistic and behavioral levels. And Maybe most significantly, ChatGPT users got lazier as the experiment went on. So that by the end of the experiment, they were basically just copy pasting essays with no possibility of recollecting what they had posted because they didn't even look at it before they posted it into the box. I saw a range of opinions about this. Some folks said it was kind of cheap, some folks said it was quite profound. What, if anything, do you take from this study?
- [30:46]
Cal Newport
I think there's critiques about the specific construction. Sure, right. In particular, one of the issues that these type of studies have is that when you're writing essays for a study for which you're getting like a $20 gift certificate, you don't care. So you're like, you just start. This happens a lot with these sort of distraction studies. You just start copying and pasting and stuff because also you kind of want to get out of there and it's boring and who cares? You're not getting a grade. It doesn't matter if they put like $500 on the line. If you have the best essay, that would be sort of more interesting. So, yeah, I think there are those critiques. But I think the reason why this study got so much play and I wrote about it, I talked about on my podcast, is that directionally it's correct. It just seems self evident that directionally this is correct. Like, okay, who really cares about exactly what brainwave they measured? Of course this is true. It's exactly what I observed when I did the more anthropological study for the New Yorker where I just looked over people's shoulders. This was exactly the same thing I observed is writing is very hard. I got into the brain science in that piece because you have to connect together these various parts of your brains using the deep reading circuits. And that feels bad. It's difficult. It's time under tension. And when you're going back and forth with ChatGPT, you can just reduce that tension. And what's hard, what creates the tension, is to try to connect together these different parts of your brains and hold that connection. And stasis, it's like, of course brain activity is going to go down if you're just measuring with an eeg, because that's exactly what people are of course doing when they're trying to write with this is make writing less hard and that means less brain activity. So I think directionally this is true. It's just like self evident that this is true. The details of the study are just, yeah, who cares? Like it's really not that, it's really not that interesting. But just talk to a student, right? Hey, what's it like when you write with ChatGPT? They're like, I don't know, it takes forever and I kind of glaze over. But it's not as hard like that's what they're doing. So I'm not surprised by this study.
- [32:38]
Derek Thompson
There is another study that I think really sits in interesting juxtaposition with the your brain on GPT study. And this was one of software developers. The AI research nonprofit METR conducted this study where they had a bunch of seasoned developers use an AI coding assistant. And then after they completed their coding task, the developers were surveyed, how much time do you think you saved? And they said, on average we think we saved about 20% of our time. Right? We got a five hour project done in four hours. But the study itself found that using AI did exactly the opposite. It increased task completion time by 20%. You thought it was a five hour task. It's actually a six hour task. What, if anything again does this study tell us? Because whereas the Your brain on ChatGPT says we're using these devices, we're using these technologies in order to cheat our assignments, this second study says, no, we think we're cheating, but actually the outcome is worse. It's making us less productive, not accelerating the degree to which we reach the end of the assignment. So how did you feel about this second study?
- [33:50]
Cal Newport
Well, this one created much more of a stir I think within the AI industry because if you look at the MIT Media Lab, like look, these are academics writing about paper writing, like they don't want AI to do well here. Meter actually has been, you know, it's this nonprofit research institute that a lot, most of the stuff they've produced has been touted by sort of AI companies. They do all these benchmark measurements and they have these graphs that people show of like, look at the rapid improvements in skills. So this was sort of someone from inside the hen house saying like, oh, I don't think we have as many eggs here as we think. So I think this, this landed more. It's also at the core of the current economic sort of engine of these companies. I mean we know this is one of the Use cases in the business setting. This is like one of the early use cases that seems to have traction. So they really care about is this making programmers more efficient? I think the message of that study is that it's very difficult to measure what we mean by productive. Right. So all the critiques of the study, they're find critiques. It's like, well, I don't know, what were these tasks? How do you measure how long are these the right type of tasks? What's going on? And the message I get out of it is yeah, it is hard to measure productivity and productivity gains but that that swings both ways and that if you look at a lot of the chatter that's happening about are we even going to have computer science majors three years from now, right. Like all program is going to go away and be automated. It's suffering from those same flaws. It's suffering from the flaws of. I don't know, this made something really easy for me. I guess you're right. Maybe I don't need to hire someone or I'm way more productive than I used to be. It's all sort of like a vibe and anecdote but it's really hard to put your finger on. Did this make you faster or not? Or what does faster mean? Or was this code going to require you to debug it five times longer than it would have been if you spent more time writing in the first place? I think that's what that emphasizes is that that type of work is complicated and we should be wary. I'm wary about both the hyper hypers and the hyper skeptics. If you talk to programmers, and this is my world, what do you hear from programmers? They say first of all you have to separate Vibe coding from serious software development. Yeah, it's really cool.
- [36:06]
Derek Thompson
And just slow down here to define Vibe coding from serious software development.
- [36:10]
Cal Newport
Right. So vibe coding is where I might not know much about computer programming. I want a quick prototype of something. I'm basically having an AI model produce all the code for me or with minimal interaction from me, right. So, so I want to put together a quick web based scheduling app. This is a real example from a friend of mine for like an auction I'm having at my kids school and I don't really know how to program JavaScript or whatever. I could just sort of make this thing without having to code and that's really cool. And that's vibe coding. Then you have serious software developers where what they're doing is not that what they're doing is Preventing themselves from having to go to Google. And Google Stack Overflow. So it's a giant bulletin board where people publish answers to questions, programming to figure out, hey, how do I call this library? Or what's the right format for a sort of virtual constructor for a C class. So it's a lot of like looking up information about how to write certain types of code. And instead of having to go over to Google and look it up, these AI tools can just find it for you and put it right there in your code. So it's like really useful. It's integrating, searching for stuff you need to look up to keep writing your code, you integrating that into the place where you write your code. And developers really like that. And then there's vibe coding, which is people who don't really program much can build prototypes. That's kind of what's going on right now. And those are both two really cool things. But we don't have a lot of evidence. It's okay, we're one step away from these programmers are gone and the next year these programmers are gone. That's not really the way people seem to talk about it. If they're, if they're not managers being quoted in like Wall Street Journal articles about AI where you want to seem very like cool and with it and very high tech, if you're just talk to actual programmers off the record, that seems to be what's going on.
- [37:57]
Derek Thompson
When I think about these two studies and I put them together, one takeaway that I have is that AI is really, really good at answering questions, but it's bad at telling you that you're asking the wrong questions. Right. The first study and so much of college AI use is. I know what question I want to ask, right? How long did the Habsburg Empire last? Like, who was Genghis Khan's grandson? Like these kind of questions AI is really, really good at, especially if the answers are in the distant past, because it's not as good at looking at more recent web browsing. But AI is never going to tell you, hey, you're totally on the wrong trajectory for writing this essay about Genghis Khan. Don't write about his grandchild, you have to write about his nephew. Like, AI is never going to tell you that. And the reason that asking good questions is really, really important for productivity is that I think in a way, productivity in many fields is about having good taste in questions, right? It's being a software developer and knowing exactly how to tackle the problem of whatever this particular part of the website broke and it's true for writing as well. I mean, how many? There's so many essays where I've spent hours and hours and hours trying to write the essay and failing and failing and failing and realizing, oh my God, I have the wrong lead anecdote. If I had this other anecdote that was just in this other note, the story would have flowed perfectly. And so I do think that one thing that we're beginning to recognize is this discrepancy between the facility with which AI will always answer our questions. And sometimes we conflate that feeling with productivity. And on the other hand, the fact that actual productivity is not just about answers, it's about being really, really careful about good taste in questions to ask. And AI just is not as good at doing that. We maybe aren't as good at prompting AI to do that, which might be one of the reasons why we haven't seen sudden massive productivity gains in the creation of a technology that is in fact really quite smart.
- [40:01]
Cal Newport
I mean, I think I agree with that. A source was telling me the other day that one of the reasons why you see so much focus in the announcements of AI capabilities in the last year or so, they're all focused on benchmarks. So why, for example, are we announcing, hey, our model can do really good at this high school, high level, high school math exam, as opposed to saying, our model can do really good at solving these technical problems that are insanely lucrative for this company. Why is it focusing on abstract benchmarks? And one of the things the source was saying is because, well, the actual. Exactly what you're saying, the actual problems you're tackling and trying to solve in real companies are, they're complicated and they're really bespoke to those companies because of taste. Like, how do I have taste about, like, what is the right lead for an Atlantic article that's going to be sort of like a 4000 word idea article that is super bespoke to like exactly that context, right? And it's not profitable to try to say, let's spend a lot of time building synthetic data sets and rl training them and tuning them to be like really good at this job that like 17 people have. And so this might be one of the things that is happening is that the taste required to do almost any non trivial job is pretty bespoke to the position. And it'd be very difficult to get a model to be good at exactly that unless you really put a lot of resources into, hey, we've got traces from Derek from the last five years of everything he's ever written, and we've really trained him. But why would that be profitable? That makes no sense, right? That's not worth the effort. So I think you're absolutely right there. It's not that you couldn't maybe build a system to do specific things you do in a way that's very helpful. It's just that it makes no financial sense for a company to build a system that does the things in your particular niche does for. So that might be one of the things that's going on. So I'm with you on that.
- [41:49]
Derek Thompson
There's this cliche that you write to figure out what you're thinking, and I think that's more or less true. But as I was listening to you talk, I thought, I also write to figure out what I'm not thinking. And that's just as important. I think I have a thesis that makes a lot of sense, and it makes sense in my head as I'm thinking about it really, really quickly and taking a walk. And then I sit down at my computer and I write out what I think I think. And I'm like, oh my God, this is gobbledygook. It makes absolutely no sense when you reckon with it in the form of a written sentence. And I think that this is a good example of, or another recapitulation of the idea that AI can be really helpful at being benchmark smart at providing excellent answers to well defined questions. But in most companies, it's the questions or the challenges themselves that aren't well defined. And we don't quite have an artificial intelligence technology to be facile with that. It looks like you're champing at the bit, so I'll let you jump in.
- [42:50]
Cal Newport
Well, yeah, I mean, I've said, what's the Turing Test we should care about in 2020? Whatever. I think it is an AI that could empty an email inbox. So if an AI could empty your email inbox, think about what you would have to master. You have to master a relatively bespoke sort of economic activity landscape. You would have to master interpersonal relationships that are subtle, trying to understand who is this person and who are they in this hierarchy. And if they're asking me to do this, what's the right tone to respond to them? But also it's going to require future prediction in a way that a feed forward static model can't really do. I wrote an article about this a couple years ago because you have to say, if I agree to do this, what is that going to generate in the future? How is that going to fit into my current workload and things I'm coming up. Okay. I think I need to defer from saying yes to this, but this person is in this position of authority, so I got to be pretty. You know, I have to demur in the way I do it so that it's socially appropriate. You can answer an email. Clear out my email inbox. Like, now we have AGI. Like, to me, that's AGI, and that's not at all what you're getting, I think, by. Yeah, instead having training a model on the entire Internet for text production. But I wanted to also just quickly point out. Go back to the thing you were saying before about what happens when you write and how that's where real thinking happens. Because I want to just touch briefly again on the art of thinking and how this is. It's supposed to be hard. Just like if you want to get in good shape, your muscles are supposed to feel uncomfortable. You're supposed to be lifting heavy things. One of the things I found in my research on that writing paper is that when you're actually putting words onto the page, I thought this was really cool. One of the parts of the brains that actually gets activated is the part of the brain that's used for spatial reasoning. So it's the same part of the brain that gets activated if you're trying to keep track of in your head where things are. So you can do this in MRI studies, but they have you trace patterns with your hand on a particular type of board. And you can see there's this part of your brain where it tries to keep track of things in physical space. Writing hijacks that part of your brain, and it hijacks that part of your brain to make a physically intuitive structure of the argument that you're writing. It uses something that is supposed to be used in your brain for keeping track of items in space. And part of what helps you figure out when you're writing if an idea makes sense is that that part of your brain is basically. Basically building an abstract structure. And this piece doesn't fit. It doesn't work. If these are the pieces that fit together to make this argument make sense. This is hanging out over here. And then this doesn't make sense that you're putting this piece here. Oh, I didn't really think it through. So it's almost like a physical metaphor is happening in your mind. I mentioned that example because this is the type of stuff you're working out when It's a blank screen, and you're putting sentences down. It's exactly the type of training you begin to lose when instead of going through that exercise, you. You have ChatGPT give you, like, a bad draft, and then you kind of look at it and are like, hey, can you tighten this up? Or are you sure that makes sense? You're not training that thing at all. And there's a dozen other things like that, a dozen other mechanisms like that going on. But, man, it's such an amazing art. What happens when the human brain does thinking? From the very earliest days that we had people doing philosophizing as an actual activity, go back to ancient Greece. What were they rhapsodizing about? Man, humans can do this type of complicated, symbolic thought. I mean, this is the whole, like, Nicomachean ethics. Aristotle ends up with. This is kind of what we're supposed to do. At humans, the main thing that separates us is, like, deep contemplation, structured thought. No other animal can do this. We are so good at it, and it takes a lot of practice, and it's so cool to get good at it. So it just made me think about that, that why do we want to. We should be wary at least about, like, taking those practice reps off the table. I love that.
- [46:38]
Derek Thompson
I also love the inbox zero test replacing the Turing Test. I think that's a brilliant idea. Just to pause strangely on Aristotle. Are you sometimes, like, just aghast that this guy with how many books could he have possibly had? Like, how many books were there in, like, 300 BCE Greece? There's no way that he had access to a fraction of the knowledge that we have today. And I remember reading him An Introduction to Philosophy, and it's like, you know, he's not right about everything. Like, he doesn't understand gravity. He's not Isaac Newton. But the ability of some of those ancient philosophers to be so voluminous and daring in their thinking and be such polymaths, I think it's astonishing. And it sometimes makes me think, was there a way of thinking that has been lost to antiquity? I am not one of these Bronze Age pervert kind of guys who's like, let's all go back to the Bronze Age or the Hellenistic Age. But I do sometimes wonder, is there a modern mind that could write Plato's Republic? Is there a modern mind that could write the complete works of Shakespeare, Aristotle? Because those do seem to be singular achievements that I'm not sure I see on any given day these days.
- [47:58]
Cal Newport
Yeah, I Think a lot of what's going on, if I had to guess is I'll be really reductive. Comfort with cognitive discomfort. Right? This is such a key part of trying to extract the best value out of whatever circuitry you happen to be bestowed about. I wrote an article last winter about my experience in the theory group, the theory of computation group at MIT where I did my doctoral training. And it was an article that was titled Learning how to Think. Like what I learned being in this environment. And it was one of these places. There's not very many of these left, but it's one of these environments where a tier one skill, the tier one skill that mattered was what everyone about is, what everyone cared about was like your ability to think. That's how you made your living. That's how you got ahead. This was the only thing that mattered, is how good of a thinker are you and what does it mean to be a good thinker? And they thought a lot about thinking and they cared a lot about thinking. And one of the things that came out of that environment was comfort with cognitive discomfort. So it feels uncomfortable when you're trying to hold things in your working memory. You're yoking together different parts of your brain and then you're holding that in stasis while you try to rotate something mentally in your mind or move yourself around an argument. It uses a lot of energy. It's strain. It's not what our brains were evolved to do because we're not evolved to be system symbolic thinkers. So we feel like a real feeling of resistance in it. And the great thinkers can sit in that and they can hold that and they can move with that. I think it was easier to be comfortable with cognitive discomfort 2,500 years ago because there wasn't that much else to do with your mind. I mean, like, life was like, pretty rough and boring. Right? It was equal. I mean, what could Aristotle do? He had the grove outside of Athens. And then, you know, you would. It was very peripatetic. You would walk around with, with other thinkers and you would think and you talk and it was like your very best entertainment. So like, why not like, let's, let's go and do it. There was like, very little else to do. The Pythagoreans would like, sit around and try to think so hard that they had mystical experiences and. And that was kind of it. Like that was their cult, you know, and so. Right. I think partially there's probably a lot of genius being left undiscovered because it has to be the combination of like your brain formed in some way that's capable of doing Aristotelian thought. Plus you develop the ability to take advantage of that, which meant you got really, really good at just sitting in difficulty, sitting in strain, sitting in what it takes to pull together new thoughts or to master something you didn't understand before. So I don't know that we're less capable from like a brain wiring perspective. I think it's a hundred x harder, however, to find those needles in a haystack. We should have a way. You're right. We should have way more Aristotle's because the population is like a thousand times larger than it was, you know, back in that time. And we have a knowledge economy and it's much easier to have time to think, but it's way more rare to see a thinker of that caliber. So that maybe that's what's going on. It's. We're. We're missing our hit rate at geniuses. Figuring out their geniuses is way worse than it probably was back then.
- [50:53]
Derek Thompson
Well, it's fun to think that people are familiar with obviously the concept of physical fitness. I never thought of a concept like cognitive fitness that would become more flabby as we get worse over time, especially throughout modernity, at working out our ability to engage in deep thought and deep work. Right. I mean, you've written about this a lot, but this idea that a particular kind of cognitive fitness, a facility with. I forgot if you put it this way, but disfluency, not fluent thinking thoughts that come easily, but disfluent thinking thoughts that come hard. A facility with that is something that, you're right, might have just been more common before it became so easy to just pull up a screen and dump our dopamine at it. Before we go, we've done a lot of criticizing. I want to make sure that we end on a recommendation note. We've been talking about the bad AI is also amazing, and it does some things spectacularly well. If I need a 10,000 word essay about Hoot Smalley tariffs of the early 1930s to understand great Depression tariff policy from several different angles, I can whip it up in five minutes. And that's pretty extraordinary compared to how long certain kinds of research would have taken for me to go to the library and track down the book and everything else, I'm wondering, not at any high philosophical level because I've really enjoyed the high philosophy, but let's get down into the straight practicality of it. How are you using AI? As a writer or worker, if at all.
- [52:31]
Cal Newport
Yeah, I would put. Yeah, it's a good question because I don't use it much professionally. I mean I use it in the sense of for articles messing around with all the different article, different models of this or that. I've tried to use it a lot of different ways and have come away very wary. So I use it in my personal life as a better Google. If there's specific information I need that is out there, but it's kind of annoying to synthesize. That's how I use it. Right. I do some electronics projects with my kids. It's really good. Here's like a perfect chatgpt question. Hey, I'm trying to wire up like this type of string of lights. What gauge? It'd be about this much voltage. Like what gauge wire should I use to be safe there? That that information is out there. But you're going to have to go to a couple discussion boards and be like, okay, I think it's this and it'll just answer that right away. I've been more wary about using it for research because I think on the other hand it's this interesting. We're in this interesting self reinforcing cycle where yes, there's certain research that it can get you faster but it's sort of more of a modern thing reality that we need this research faster. I was thinking the other day I was rereading Lewis Mumford's Techniques in Civilization, the sort of great early book and the technology theory and studies that came out in the 1930s and he has a great author's note where he talks about writing this. It took a long time. He was like, oh, there's not a lot of sources on the history of technology in Europe. So I went to Europe and it took two years. There's these small museums that they have in these small towns where they have these old millstones and stuff they're proud about. And I pieced together the whole history of technology and probably doing that over two years helped him form his theories. Right. Or this is like David Grann loves nothing better than just being in the archive and just turning those pages and just getting in. I want to learn all the nautical terminology and I'm going to spend six months doing that. But that also is probably the time there is not that bad. So it does help you find things faster. It's burned me before. There was an occasion and I will own this, but it's burned me before. There's an occasion where I was working on an Article, article. And I don't use, I really don't use it for research. But we were crashing a deadline, this thing, we're trying to get it out. And my editor was like, hey, can we add in a quote from this story you were talking about? And I was like, yeah, but I left the book at my house and I was at my office down the street and I was like, oh, well, this story is all over the Internet, so I'll just grab it off there. In fact, ChatGPT can grab it for me, says, hey, can you go find the story and give me the quote about this? And I was like, yeah, here you go. Then the fact checker calls like two hours later, I can't find this quote, right? What version of the story you're going on? And I go and I get the book and I'm like, oh my God. It just paraphrased. So it kind of burned me on that. I was like, ah, lesson learned. Even when you think like this information is out there. So I don't know. I've tried deep research, but I like going a little bit slower on the research. I think turning the pages sort of helps me take process it and imbibe it. So I use it for informational questions. I use it for grammatical stuff all the time. I think it's very useful for that. It's very good at sanity checking descriptions of how AI works. So it's very good for that. I feel like is this an accurate description of a transformer architecture? Because there's a lot of text on that and it's technical and it's good at technical text. So no, I don't use it a ton professionally. But I agree with you that this is a very cool technology. I think Google search is $175 billion a year business. Right? That was the 2023 numbers. And this does things. What's better than Google for certain types of questions? So that's nothing to sneeze at. I think that is a huge use case. Computer programmers, I don't do a lot of programming anymore, but they really love it. It really is great to be able to integrate this into the places where you type code and not have to go search for stuff in other places. I know a lot of people who love vibe code, whatever for small businesses, they for their own hobbies. It's just fun to be able to like make a program from scratch. It's just like a Steve Jobs, sort of late seventies early personal computer type of energy in it. I think that is really great. I find some of the, like, interactive brainstorming stuff to get a little parasocial to me, that creeps me out a little bit. It's like, you're so great and what a great idea. And that, that kind of, that kind of creeps me out a little bit. But as like a great Google, as like a programming aid, I mean, those things right there are nothing to sneeze at. Those are huge markets. And it's, it's hugely useful. And I use it for that. And I'm excited for, I'm excited for breakthroughs as well.
- [57:11]
Derek Thompson
Cal Newport, thank you very much.
- [57:13]
Cal Newport
Thanks, Derek. Sam.