← Deep Questions with Cal Newport
Deep Questions with Cal Newport
Ep. 370: Deep Work in the Age of AI
01:10:46
A recent study called into question a core assumption about the generative AI revolution: that these tools, at the very least, will make us more productive. In this episode, Cal dives deep into the study and argues that when it comes to efforts that require deep work, AI can sometimes make things worse. He then answers listener questions and then takes a closer look at an article claiming that the lack of Wi-Fi in a West Virginia school is making their students dumber.
Loading summary
Transcript39 lines
- [00:02]
A
In recent weeks, there has been a lot of turmoil surrounding AI technology. It increasingly seems now like those grand promises of superintelligence or AI systems automating most of the economy. These grand promises are probably not going to come true. But what about the more practical promise? The one that AI tools are going to make knowledge workers more productive, Especially if you do something like computer programming, for which AI is well suited? That's still true, right? Well, the answer turns out to be complicated. A recent study about AI productivity yield a completely unexpected result. It created a glitch in the matrix, so to speak, that leads to some deeper truths about this technology and its potential role in our work today. I want to explore all of this. If you're at all interested in how AI might impact your job in the near future, this is an episode you won't want to miss. As always, I'm Cal Newport and this is Deep Questions. Today's episode, a Glitch in the AI Matrix. All right, so I previewed. There was a study that caught a lot of people off guard. I want to load this on the screen here for those who are watching instead of just listening. This study came out in July of 2025. It was produced by a nonprofit called METR. That's often pronounced meter. It's a nonprofit that does evaluation of AI and its capabilities. They do high quality studies. They don't have any particular bias. AI companies often cite their results when they find them to be useful or impressive. So this is sort of a neutral body that does these reports. Here's the title of their July report that caused a bit of a surprise. Measuring the impact of early 2025 AI on experienced open source developer productivity. So what did they do? I'm actually going to read from the methodology section because I think it's important to understand like exactly what they were testing here. So let's read here together. This is from the paper. To directly measure the real world impact of AI tools on software development, we recruited 16 experienced developers from large open source repositories that they've contributed to for multiple years. Developers provide lists of real issues that would be valuable to the repository, bug fixes, features and refactors that would normally be part of their regular work. Then we randomly assign each issue to either allow or disallow use of AI while working on that particular issue. When AI is allowed, developers can use any tool they choose, primarily Cursor Pro with Claude 3.5 or 3.7 Sonnet, which were the frontier models at the time of the study. When Disallowed they work without generative AI assistance. Developers complete these tasks, which average about two hours each, while recording their screens, then self report total implementation time. They need it. All right, so that's the setup. It's a very elegant experiment. Here's developers working on the normal stuff they develop and issue by issue, hey, for this issue, I want you to use AI. For this issue, you don't. For this issue, you don't use AI. You do, right? So you get this nice randomized control of. All the developers are both using AI and not using AI in a randomly selected manner. The simple thing they did wanted to figure out is how long did it take to do these tasks when you were using AI versus not now. Here's where the conventional wisdom says computer programming is something that language models do well. So of course it makes productivity go up. I'm going to load up on the screen here the core chart of this whole paper. So if you're looking at this, what you're seeing right now is a bunch of green dots. So what these are showing, the things you can see, the data points you can see on my screen right now is predictions from various people about the increases of productivity that they thought AI would give them. So if you'll notice, there's a line in the middle here that's neutral. So a data point on that line means AI makes no difference. If a data point is below that line, it means AI slows you down. If it's above it, it means AI speeds you up. So the first point is from economic experts. They ask economic experts, how much more productive will this make these programmers in this experiment? And on average they said around a 40% speed up. They asked machine learning experts. So people who knew the technology, they were similar, yeah, should be about a 40% speed up. They asked the developers themselves during the study, like, hey, how much more productive do you think this is making you? And they were like, yeah, between 20 and 30% more productive. And. And when they asked them after the study was over, they were, you know, around the same place, like a little bit over 20. What was actually measured. I'll scroll up to show you that result. The observed result is, on average, they were about 20% slower than the people not using AI. So the AI tasks were slower than the tasks in which the programmers did not use AI, AI. This was an unexpected result. They thought they would just be measuring how much more productive AI made these programmers. They were unexpected to see that it makes them slower. So this is a bit of a mystery, and it's one we're going to look into because it gets to something core about AI and doing knowledge work. Why did programmers, despite their predictions, become less productive when they used AI? All right, so if we want to solve this particular productivity paradox, I think what we need to do next is understand what type of work are these computer programmers doing. We got to get specific here. Now I'm going to use a term that most of my audience is familiar with, computer programming, handling these tasks, which are a combination I looked into more deeply. But it's a combination of creating original code or fixing code. They require what I call deep work. Deep work, as you know, is tasks that require you to focus without distraction on something that's cognitively demanding. So deep work is when you give your full attention to something that is demanding and you try to keep your attention on it as intensely as possible. I sort of wrote the book and it's sort of. I wrote the book on it. I coined the term. It's been 10 years, Jesse. Isn't that hard to believe? Yeah. In January it's going to be the 10 year anniversary of deep work. And I guess I'm going to follow through with the promise I made my readers that if this book is still selling after 10 years, face tattoo, Boom.
- [06:59]
B
Like Mike Tyson.
- [07:00]
A
Like Mike Tyson. Deep work right across the forehead. Ryan Holiday put his book title on his forearm, going to one up him right across the face. So what a real committed author does. So anyways, here's the type of work we're talking about. We talk about computer programs. Deep work, right? It's something that requires focused. You're creating something complicated from scratch using your brain. It's cognitively demanding and it benefits from being able to give it undistracted focus in the modern knowledge economy. The core argument I make in my book on deep work is that this is the effort that creates most of the stuff that's actually valuable to the marketplace. All the other things we do, the emails, the meetings, the filling out the forms, the submitting the things, the tickets to it, the making the PowerPoint presentations, that's basically all efforts that support deep work. The stuff we have to do to keep the lights on and tell people about what we did. But you can't run a profitable company just off of email, meetings and PowerPoints. Ultimately, someone has to actually create something valuable using their brain, and that requires deep work. So in this study, we had programmers doing deep work who used AI to help them and they thought that would make their deep work more productive. But in reality, it made things worse. Great. So now we're narrowing in on the answer to this paradox. So we know what these programmers are doing, deep work, and we know that AI wasn't helping them. So let's narrow in this question even farther. How specifically were these programmers integrating the AI tools they were using into their deep work efforts? According to the study authors, the programmers were working interactively with these tools. So what they were doing was a lot of, can you produce me this code? I'll look at this code. Hey, can you fix this about it? Or this is not working. So kind of back and forth like that. Or here's some code. You see any mistakes, great. The programmer would run it and be like, it's still not quite working, right? Can you try to fix it here? So it was a very interactive loop. They were going back and forth, having code produced, checking the code, having the AI check the code, looking at what it sent back. So think of it as like a back and forth interaction. Here is how. Let me see here. Here is how the paper itself describes this. Let's see here. Okay, here we go. Here's text from the actual paper about this collaboration. When allowed to use AI, developers spend a smaller proportion of their time actively coding and reading, searching for information. Instead, they spend time reviewing AI outputs, prompting AI systems and waiting for AI generations. Interestingly, they also spend a somewhat higher proportion of their time idle, where their screen recording doesn't show any activity. So the non AI group, when they're working on a task without AIs more time, makes sense, like actually writing code. And the AI group is spending more time prompting, asking for code, asking for it to check code. So it's more of this like, interactive loop. I'm going to give a name to this. Let's call this cybernetic collaboration because these programmers are collaborating on their deep work with a computer. So it's cybernetic and they're collaborating. Let's call it cybernetic collaboration. They're basically trying to split the cognitive effort of producing this code or fixing this code between them and this digital mind. Now, intuitively, cybernetic collaboration should make you more productive. Why not, right? Like you don't have to do as much deep work anymore because you're offloading some of it to a machine and machines are fast and machines are precise. And hey, this seems like you've just made things more productive. But of course, that didn't happen. So now we've really narrowed in this paradox. They're doing deep work, they're collaborating interactively. Doing cybernetic collaboration with the AI while they're doing deep work. And that is not returning more productivity. It's still taking them longer to produce stuff than when they weren't using AI at all. So let's narrow in even farther and let's step back now and ask the question, what role does collaboration play in deep work? Is that something that you can do with multiple minds? Now, this is actually an important question because a lot of people who had a glancing encounter with my book on deep work assumed that it had to be solitary, but it doesn't. Actually, I know a lot about collaborative deep work because this was one of the core skills I had mastered as a, as a theoretical computer scientist. Like this is, if anyone knows how to do collaborative deep work, it's scientists that do math theory like me. I learned how to do collaborative deep work first at MIT when I was doing my doctoral work, and then refined these skills as faculty, computer science science faculty at Georgetown. I write in my book Deep Work about how to do it successfully collaboratively. So, yes, deep work can be done collaboratively, but how is it done successfully? If you want to do deep work with someone else, you want to collaborate in practice. If you're hanging out with mathematicians, how do we do collaborative deep work successfully? Well, here's the thing. Talk to any professional thinker who does this. They'll say the same. The reason why collaboration, I guess I should say the way you make collaboration help with deep work is that you use the presence of other people to increase the intensity and duration of your focus. So the underlying formula, focus produces results, deeper focus produces better results. That formula reigns supreme. So when I would sit down with my frequent collaborators to work on some sort of theorem or some sort of paper we're working on, the whole game was trying to get even more intense focus. Here's how this works. If I'm sitting at a whiteboard with two other mathematicians, a couple things happen. One, I'm going to maintain my focus on this problem much longer than if I'm just alone looking at a notebook. Why? Because if I'm alone, there's very little penalty for me allowing my attention to wander. Ah, this is hard. Let me just, like, let my attention wander. Let me let some of the steam out of the metaphorical steam engine here. Let me go check something else. Nothing bad happens. If I do, it's just me trying to contain myself. If I'm at a whiteboard with two other mathematicians, however, and I let my attention wander, I lose a thread of thought and then what do I have to do a couple of minutes later? Something that's kind of embarrassing. I feel like, hold on, hold on, go back. I missed what you guys were just saying here because I let my mind wander. Everyone has to stop. Everyone has to go back. So the social pressure of keeping up means that you keep your intensity longer. The other thing that happens when you're working with other people is it pushes your intensity of concentration deeper, right? Because you're sitting there trying to understand this. Someone has a breakthrough to, like, okay, hear me out, and they go up to the whiteboard and they start working on some sort of, like, complicated simplification of an equation or some sort of graph construction. And you're like, man, I gotta really lock in to follow this. Right? Someone is trying to download something complicated. They just had an insight about to my brain. And the only way to get there is to, like, really lock in and focus. So it's a focus accelerator. That is what you get when you're doing deep work. I think it's off there now. But, Jesse, that whiteboard we had here in the HQ for. I used to bring, like, during the pandemic, starting the pandemic, I used to bring my collaborators from Hopkins and Georgetown. They'd come here and we'd work on the whiteboard. Yeah, I don't know if it's still up there or not, but, like, the last thing that there was. So we would sit at that literal whiteboard and the last thing that was up there for a couple years, we actually won an award for that paper. So maybe I should have kept that. It was a good paper. But, yeah, that's what we do. So there's other advantages of working with other people. But this is like one of the big things you get out of deep work if you collaborate. Right. Is that it makes you focus harder and focus longer. So, again, there's this underlying equation. Intensity of focus times time is how much you produce when you're doing cognitively demanding work that still reigns. So collaboration, all you're doing, if you're doing it right on hard things, is getting more focus. You're squeezing more focus out of it. I call that the whiteboard effect. In my book, Go back to cybernetic collaboration. That's not what these programmers are doing. They are using collaboration with AI to reduce the amount of intensity of focus they have to experience to get those breaks. You produce it. I'll look at it. It's easier for me to try to debug what you did that was broken than it is for me to have to produce it from scratch. It's easier for me to have these nice breaks while I'm waiting for the AI model to generate the code I'm going to look at, which can take a little while. We know this, Jesse, because we get this question all the time now. What should I do while waiting for the AI model to produce my code? We only put it on the show once, but we get this question a lot. So that gives you a break. It makes the experience more pleasant. There's actually the authors of this say this somewhere in the study, that it was a more pleasant experience for the programmers because you get all these breaks. You don't have to be locked in. When you're just looking at the bank blank coding page, no word, no piece of code is going to get there till you come up with it and write it. It's really hard. There's no real break until you tell yourself to take a break because you're trying to get all this code together. Compiling for small programs is fast, too. You don't get much of a break from it, and so it's much more pleasant. Cybernet collaboration means much less intensity of focus, much less duration of focus. It takes less energy, it feels nicer, but that's why they're slower, because intensity of focus is what tells you how fast you're going to go. Intensity of focus tells you how good the stuff you're going to produce is when it comes to deep work. So the whiteboard effect says, yeah, come work with me so I can focus harder. Good stuff gets produced. Cybernetic collaboration, by contrast, says, I want a computer to offload some of the work, so I get a lot more breaks. But that means my brain is producing a lot less slower, less quality stuff. And that is why the time required to get the work done begins to go down. Now, like, look, if the machine was actually able to take over all of the deep work, that would be different. If you could literally just say, vibe, code this program and then commit it, that'd be great. Because now you're like, I don't have to do any deep work at all. But of course, the machine can't do that in cybernet collaboration, they can't do that yet. So now you're doing this back and forth dance where you ultimately still have to come up with the questions and edit the code and get everything to work, but you're cutting down your intensity of focus, so the whole operation inside your brain is going slower. I want to bring up another quote here. This is from an article from the Atlantic that was written by Roger Karma, and it talked about this study, among other things. The title of this article is Just How Bad Would an AI Bubble Be? And it really is about AI and productivity. I'm quoted towards the end of this article. It's a good article. But here's what he heard from the meter developers about what was actually happening as they were doing the sort of cybernetic collaboration. So let me quote here. Even the most advanced systems make small mistakes or slightly misunderstand directions, requiring a human to carefully review their work and make changes where needed. This appears to be what happened during the meter study. Developers ended up spending a lot of time checking and redoing the code that AI systems had produced, often more time than it would have taken simply to write it themselves. Right. So they were just in this loop of, you write, I'll look at it, you debug, let me look at what you're doing. But because they were avoiding the full intensity of focus that they're capable of, because they avoided maintaining their focus without having context shifts or their distraction, the work they were doing was just like, not at their peak. So now you. You're doing all this work of cleaning stuff up and you're not operating at your peak. This is the potential danger of cybernetic collaboration that when you downshift your mind. Let me downshift my focus intensity. It just doesn't work as well. It might feel nice, but deep work doesn't really have a lot to do with nice. All right, so that's what seems to be going on there. So some combination of this rhythm of going back and forth creates, like, a lot of work you might not have had to do before. And more importantly, it reduces the gear at which your brain is operating. So these programmers were just. It was pleasant, but they were producing, you know, stuff slower. It was taking them longer to figure things out. That's my. One of my understandings from what is going on there. But let's summarize this all. The end of our deep dives, we like to do some takeaways. We've been experimenting with music. Our last takeaway theme music, which I thought sounded like the beginning of a cable show about bass fishing. A little aggressive. So we're going to go a different way today for our takeaway theme music. We're going to do something a little more. What do you say, Jesse? Intellectual, calming. All right, so let's do our takeaways with a little bit of calming background music. All right, so what are our takeaways? Deep work rewards, intensity of focus. And if you add anything into your workflow that's going to reduce this intensity, you'll probably get less productive. This seems to be the trap that a lot of knowledge workers experimenting with AI right now are falling into. Focus is hard. It doesn't feel pleasant. It's tempting to try to make it go away, but that doesn't make your work better. Now I'm absolutely convinced that there will be upcoming applications of AI that will help our productivity. I think they will focus more on automating shallow tasks or speeding up things that don't require you to focus. But cybernetic collaboration, that is not the key to the AI future. Focus remains absolutely a essential to doing deep work. And for now it remains something that has to be hard to do. Like we should slowly high five or something. We did it. There we go. There's a lot of other things going on in that study, but I mean, I think that's a lot of what's going on to it. It's like, look, deep work is hard. It's tempting to make deep work easier. I can help you do it, but doesn't mean you're going to produce more work.
- [20:52]
B
I love the takeaway portion of the deep dive segments now.
- [20:55]
A
I'm working on it. I think we're still working out the kinks, but I like, because while you're.
- [20:59]
B
Going through it, I just. So the audience knows. I don't necessarily know what you're going to say. And I was like, well, what do I do?
- [21:04]
A
Yeah, so I mean, I think it's good. Like sometimes it's advice and sometimes like, here is the core message. Like, here's the, here's the thing. I'm taking away from it. So that's a cool study, man. There's so much controversy around it. Like, a lot of programmers are like, well, they don't know how to use the AI tools. Not true. Actually. These are tools. They got to choose which tools they use. And these were developers that largely were already using these tools. So there's always that argument of like, well, they're just using it. Wrong. Another argument. It's a small study. Yeah, it is small, but it's a good signal. It was a pretty clear signal. It's pretty well designed. Another argument which I think is more important is some people will say, yes, that use of AI or you have it generating code from scratch is slower. Other uses where you're just automating, like looking up information, making that faster. I believe that would make a programmer more productive. So it does kind of. That's not trying to get in the way of the deep work. It's trying to speed up the things that's not deep work so you can spend more time doing deep work. But basically, I'm sort of like a humanist, intellectual chauvinist. Here. The brain focusing hard is an incredibly powerful tool. Be wary of things that gets in the way of that. I mean, unless you really can just outsource to work completely, it might be some fool's gold. All right, we got good questions coming up. Got a good final segment. This one will be interesting. Jesse, I did a little bit of original data journalism. There's a claim out there that I'm looking into. And the scene like, is this claim true? I actually looked up some data from something. Something. It'll be kind of fun. Just investigate journalism. But first, before we get to the questions, we have to do the thing that everyone is most interested in, which is hearing from a sponsor. Jesse, we have to talk about Cozy Earth. As listeners know, I'm a huge fan of their sheets. They're the most comfortable sheets we own. I think we have something like three different sets to rotate through. We travel with them when we're away in our house rentals in the summer. I will say this, Jesse, if I was invited to stay in the Lincoln Bedroom at the White House, I think my first question would be, what type of sheets do you have? Second question would be, why are you inviting me? But first question would be, what type of sheets do you have? But Cozy Earth, here's what I'm excited about. They've got a new product, one that I have been waiting for, right? So I'm a professor, I'm a writer. This means I'm not wearing a suit and going to an office every day. And so I often wear pretty casual clothes. And as the weather cools down, I tend to wear sort of like athletic type pants. You know, I'm often on camera from the waist up. It doesn't matter. It doesn't matter what I'm wearing. I mean, I'm wearing bikini briefs right now because, you know, it's hot. You got to get people to want. Cozy Earth has a new product called Bamboo Joggers. So that sort of like casual pant in their super comfortable style fabric that we already know and love, you can wear their irresistibly soft jogger pants just about anywhere. They're made from that viscose bamboo fabric. These joggers are the softest staple wardrobe you're going to one of the softest staples you're going to have in your wardrobe, whether you're staying in or stepping out. And what about if you're going to like a slightly more formal affair? So like you know me going to the Lincoln bedroom to investigate their sheets? Well, they also have you covered with their everyday pant, which I have a pair of and I also love. They they look snazzy but they're also like super comfortable with that fabric. So Cozy Earth nailed comfort with their bamboo joggers and their everywhere pant. That's next level. Everyone I know who is trying these things out says this is really comfortable. So I'm a big Cozy Earth booster. You should be too. Go to cozyearth.com deep and use my code DEEP for 40% off the best pants, joggers, shirts and everything. And if you get a post purchase survey, tell them you heard about Cozy Earth right here. Built for real life, made to keep up with yours. Cozy earth, that's cozyearth.com deep and use the code Deep. This show is also sponsored by Better Help. We talk a lot about the different types of advice that might help you be more organized or make more progress on the things that might matter. But you know what advice is probably the most important of all. You need to cultivate a healthy relationship with your own mind. Everything else rests on that foundation. If you want to improve this relationship, therapy might be exactly what you need. A professional who can help you understand your own mind and improve that relationship. BetterHelp makes therapy easier with over 30,000 therapists, BetterHelp is the world's largest online therapy platform. They serve more than 5 million people globally. It works. They have an average client rating of 4.9 out of 5 stars for a live session based on over 1.7 million client reviews. It's convenient, too. You can join a session with a therapist at the click of a button, helping you fit therapy into your busy life. Plus, you can switch therapists at any time. As the largest online therapy provider in the world, BetterHelp can provide access to mental health professionals with a diverse variety of expertise. Talk it out with BetterHelp. Our listeners can get 10% off their first month at betterhelp.com if they go to betterhelp.com deep questions. That's BetterHelp H-E-L-P.com deepquestions All right, speaking of questions, let's move on to our own.
- [26:16]
B
All right, first question is from Sam. I found that using ChatGPT questions can act as project diaries. I now keep multiple chats for projects I'm working on and enables a shutdown and a place marker for where I need to restart work. Can ChatGPT chats act as single purpose notebooks for projects?
- [26:34]
A
I mean, sure. Couple questions here. What is a project note? Like this idea of like, I need a notebook to keep track of the progress on my project. Was that something you were doing before? I guess it's useful if you don't know where you left off and you want to have external notes on it. But here's my concern. My concern is that you're doing something like cybernetic collaboration with ChatGPT. We talked about this in the deep dive. When you have this running list of notes and conversations with ChatGPT, I'm concerned what you're doing is transforming your work into something where you go back and forth with the chatbot, you hand off more, you know, have them do some stuff and you look at what they did and you ask it questions, you go back and forth. This is more pleasant, this feels nicer, it makes work seem better. But just as the meter study found with programmers, if it's reducing your peak intensity of focus or the sustained duration of your focus and you're doing anything that's non trivial, it also could be slowing you down and hurting the quality of your results. The key with any type of effort that requires deep work is not reducing the difficulty of the deep work, it's not reducing the intensity of focus. It's setting things up so that you can reach high intensity of focus. The focus mind is what produces value. So that's my only issue. I would be wary if you're trying to make work in this cybernet collaboration that might not be making you as productive as you think, even if it feels nicer.
- [27:56]
B
Last week you talked about leaving a narrative for personal projects, and originally I think this question was in relation to that. But now that you talk about the cybernet collaboration, it kind of makes sense. But if you could touch on the narrative part too.
- [28:08]
A
Yeah, so like understanding. So what I meant by that is like understanding how to hit the ground running. So you know, if I'm writing something, I will end a writing session. It's like, okay, I've gotten up to here and then I have a few notes. I'm like, okay, here's what comes next. So if I could, like I'm missing this example, but if I could get this key example. You can, we could finish this section and try to get to the end or something. Right. So It's a little note I left for myself so that when I load up that writing project again, I know what I'm doing. I would do this a lot with mathematics papers as well. Right. I would put notes right into the document where I was typing in my results. Or I would email these to my collaborators and be like, okay, well here's where I'm stuck. But I think here are three ways I was thinking might be useful for getting unstuck. It's like a little narrative, a little note so that when I get back to that work, I'm like, okay, here's where I was or here's where I was going to think. So I do think that's useful. But I usually would just leave these notes right where I'm doing my work. It's in the document where I'm writing, it's in the document where I'm putting out my math equations. And I think that's useful so you know where to start again and it's easier to get into it. The thing that makes me more nervous is this idea of this long ongoing interaction. I don't actually think that type of cybernet collaboration in most cases is a useful new form of knowledge work. I think it's just escaping strain. It's a fancier version of like, I'm looking at my phone a lot. It sort of makes the work feel less hard. But hard is sometimes what you need. Who do we got?
- [29:35]
B
Next up is Mark. When using active recall to study a large volume of material, how soon and how often should I revisit material I've already mastered? Right now I find that when I return to something a few weeks after learning it, I've forgotten many of the details.
- [29:48]
A
Well, I mean, it depends what you're trying to do here. Like what are you learning and why are you trying to hold on to it? If you learn something that you then use, it will stay sort of active in your memory and eventually will really dig in. If you're memorizing some sort of information that you just basically don't come back to again, you're going to need to retouch on that within a two week window. Like if you let about two weeks or more go, your mileage will vary depending on the information in your brain. But if you let more than two weeks go without using information at all, it's going to start to lose its location. That is just like informal, based on my experience as a student professor. So if it's like this is arbitrary information, I want to remember every week or so you want to keep going back and doing that. Active recall. Eventually, if you do active recall enough, it will lock in. But again, be sure that you need to. If you're not using this stuff, be sure you need to remember it. The best way to submit something in your memory is to use it to do it. I, you know, you teach the concept a couple times, you really remember it, you explain it to someone else because you're using it for a project, you know, you really remember it. Active recall, of course, simulates that. By recalling the information from scratch, you're firing up all the circuits you would use if you were actually applying the technique. So it's using your own brain's own memory apparatus. But yeah, I would find if I memorize something using active recall and just never touched it again, within about two weeks, I might start to forget it. So I don't. Two weeks, Ish. But what are you trying to remember and why? And why aren't you using it? So maybe it doesn't have to be. All right, who do we got?
- [31:17]
B
Next up is Beth. I've seen articles ranging from AI is terrible for the environment to study, suggesting its carbon emissions can actually be lower than traditional methods. Can you help put this in perspective, for example, by comparing the emissions from one person's ChatGPT use to something more familiar, like a plane trip, and by clarifying whether AI's footprint is unique to platforms like ChatGPT or similar to energy used in everyday tools like Google Search.
- [31:42]
A
All right, I mean, I do have a hot take here if you want like the actual answers. I don't know, I don't have the numbers in front of me, but something like a Google search is sufficiently significantly more energy efficient than like a ChatGPT query. A ChatGPT query. You have to have this frontier model loaded up in memory. Now this thing could have hundreds of billions if not a trillion parameters defining it, right? So you're not going to fit into the memory of a single GPU chip. You're going to have to shard this thing over like 3, 4, 5, maybe 6 GPUs that are then going to have to be running all out just to generate tokens for your answers. I compare this to a Google search where they use commodity intel chips. Most of this stuff is cached. They can be dynamic of like, oh, there's little downtime on this chip. Hey, can you like go look something up in a hyper efficient sort of cached, you know, search index and the, the amount of computation they've got it's down, you know, minuscule. Probably the answer you're looking for is in some CDN server that's like 10 miles away. So no, it's a lot of computational power. But here's my hot take. I might, I might get yelled at for this one. Jesse, you ready for it?
- [32:47]
B
Yeah.
- [32:48]
A
I think right now the focus on the environmental impact of AI is in part a way for people who are critical of technology or big tech to get in on. I don't like AI in a territory that they're much more comfortable with in the technology. So if I'm just like, I don't know if I want a typical left leaning critique of the tech industry and AI. I'm super comfortable for talking environmental stuff like you don't care about the environment. I do. I saw the Al Gore documentary. You don't know science. The environment's important. I feel like I'm smart there. When it's me talking to other people, that's where I'm smart. I know more about this than you. You don't know science. Global warming is a real thing. So that's a very secure place of critique. A less secure place of critique is the type of stuff I've been doing in my writing recently. Coming in and be like, here. I don't think that reinforcement based learning post training refinements is giving you the sort of loss reductions that you would have expected on a more elongated power law curve. Getting into the technical details is very like, it's hard. And you're like, I don't really know this details well. But I don't like this and I don't like the people involved. And Sam Altman's kind of creepy. And Mark Zuckerberg still talks like a robot whose emotion circuit board is shorted out. We've covered this on the show and this is a territory that I'm much more comfortable in. Just also really leaning into the specific AI safety concerns over the particular words that a particular chatbot will or will not say. I'm comfortable there. This is inappropriate speech. But I think if we're really thinking about AI, these are kind of transient concerns because I don't think a future of these massive frontier models being queried for everything is a future that makes any economic sense. Anyways, here's the right way to think about it. If querying one of these models is bad for the environment, that means the amount of computation it's using can't possibly be profitable. So they can't be bad for the environment too long because it correlates also with expense and people aren't willing to pay for the massive computational expense. I mean, it takes money to create heat because that requires electricity and electricity costs money. So like my thought about this is yes, I would be concerned if we had ChatGPT used all the time, but I just don't think it's profitable. I think the future of AI is going to be. It has to be systems that have much smaller models, machine native models if possible. Meaning like this thing is running on my iPad, not in the cloud somewhere on a bunch of GPUs, small specialized language models combined with other specialized components like policy networks for evaluating options, future simulators for trying to understand which actions to take next. Control logic that's specific to the task at hand, put together into a program that you can get your arms around, that can run on existing hardware and does really good at being intelligent about a very specific thing that I'm convinced is the future of AI. And it doesn't have an environmental footprint that's different than other things that we're doing right now with our computer. So there probably is. It does use too much electricity, but I just don't think these massive frontier models are going to. It just doesn't make sense. I cannot be querying a trillion parameter model that requires six H100 GPUs to even produce a token for me for everything I'm doing in my life. That would probably be an environmental catastrophe. But that's not sustainable. I think frontier models like F1 cars, it's a way of showing off your technology, improving that your company has the best technology, but it's not the car we're buying on the Ford lot. You know, a year from now that's going to be a much simpler car, but we'll buy it from the company who had the best F1 car because that convinced us. So I don't know. That's right. So that's my hot take is of all the issues we have with AI, the focus on environment right now, we're not in some steady state yet. I just think it makes people more comfortable because that's territory where they're familiar, being the sort of higher level of the hierarchy in the argument and they're uncomfortable getting the tech because let's be honest, my people are nerds and are kind of uncomfortable to be around and we don't talk very normal and we know too much about algorithmic nonsense or whatever. So I don't know. That's what I think is going on. So I'm not as worried about the environment because I don't think this industry can survive in a mode that is super bad for the environment. It just costs too much money. People aren't going to pay $1,000 a month to pay for all their queries for things. I think smaller models can work. Did you know for example the model. No. And Brown's model, Pluribus, which was the first AI system to beat actual tournament players in Texas hold' Em Poker. The original way they built it was with very large neural networks and they used a supercomputer system center at U Pittsburgh to train it. It was like, okay. And then they figured out like, oh, we could have a couple different networks and what matters is the logic of how we connect them together and the logic of our AI and the system. They built Pluribus that can beat the professional players you can run on a laptop. Because it wasn't just like we have 100 billion parameter network that we're just like learn poker. No, we have a future simulator that simulates the possible mind states and cards of the other players. Then we have a smaller neural network trained just to understand different poker positions and say what would be good or bad. And then we have some logic that we, symbolic logic, we hand programmed in about like, okay, let's calculate the value, the expected value. If we do this and this was the case, here's what would happen. Let's give that a number. What about this? It's just like straightforward mix of neural network, unsupervised learning with symbolic old fashioned hard coded AI and the whole thing fits on a laptop and it can beat professional poker players. So hopefully the future of AI, I think is most likely the environmental concerns are not going to be, I think, substantially different than sort of the environmental footprint of the computing we have right now. But we'll see. All right, who we got next?
- [38:40]
B
Next up is David. If you left academia, would you spend your days as a writer and podcaster? Would you enjoy this or become antsy?
- [38:50]
A
Yeah, I would continue to write and I would continue to podcast. I don't think I would become antsy. So I've kind of two reactions to this. No, I want to become antsy. Writing and podcasting actually takes up a lot of time and it's not like I'm hurting for other things to do you know how many teams I'm coaching at my kids like schools and sports leagues right now?
- [39:11]
B
Let me guess, 3K6.
- [39:14]
A
It's three. I'm coaching three teams right now. I spent a lot of time Doing that. I have a lot of things to do. That's fine. But here's the bigger point, though, is writing and podcasting is like a lot of what I do as an academic right now. So I don't know how different it would be. So for those who know my trajectory, I've been a writer my entire adult life. I started writing in college. I trained as a theoretical computer scientist focusing on the theory of distributed systems. So I studied at Nancy Lynch's Theory of Distributed Systems group at MIT and did my postdoc under Hari Balakrishnan and his Network and Mobile Systems group. So I had a computer science specialty that was on the math behind distributed systems. Went to Georgetown. This is where my NSF funding was all about this. My papers were all about this. My grad students, we did a bunch of. I was pretty good distributed theoretician, published a bunch of papers, ran a bunch of steering committees. Somewhere around the time I got tenure, then I got full professor. Around the time I came full professor. Georgetown, where I work, they began making a real move for like, we want to be one of the places grappling with technology and its impacts. They call the field digital ethics. Like, we want to be at the core of this because we're in Washington, D.C. we have a big ethics background here. We were the university that figured out bioethics in the 20th century, the Kennedy center for Ethics. We want to do the same thing for tech ethics. We're here in D.C. we have all these policy centers. We have one of the biggest tech law faculties at our law school. It makes a lot of sense. And I was like, this is kind of what I'm doing already with my writing. I write about technology and how it impacts us. I want it. And so I've really been focusing on that. So I was one of the founding faculty members of Georgetown center for Digital Ethics under inaugural director of our Computer Science Ethics and Society academic program. It's the first major in the country to integrate computer science and ethics in a combined major. There's majors where you're a computer science major and you throw on a little bit of ethics, first integrated one in the country. This is actually what I'm doing largely right now as an academic is technology, how it impacts us, what to do about it. And I do it in a lot of different forms. And I think public outreach is important. I think the podcast is very pragmatic, but I reach a lot of people this way. My writing for the New Yorker, now we're talking to a little bit more of A rarefied crowd. This gets me in front of policymakers. This gets me in front of the Senate and on NPR or whatever. But it's a way, again, to work on tech and its impact. I do some academic papers on tech and its impact, which is more for an academic crowd. And then my books fall somewhere in between, so I don't really know. My life wouldn't be that much different if I left academia to write in podcasts. Writing podcasting is what I'm doing in academia. The main difference would be there'd be less time around really smart people and less time around students, both of which I like. Also teaching, but teaching increasingly, I can teach things that, like, helps me think about these thoughts anyway, so, you know, no, I want to be antsy. And also, I don't know how different it would be.
- [42:11]
B
Do you still do any math?
- [42:13]
A
I haven't very recently. I haven't. Yeah, been a year or so.
- [42:17]
B
So do you still teach undergrads, like, traditional computer science classes?
- [42:20]
A
I do a mix of traditional computer science and stuff that's relevant for the computer science and ethics program. Yeah. And I teach less than I did before, too, because I'm doing. Running these things or whatever, so. That's a good question, though. I think we have a case study.
- [42:36]
B
Yeah. Before we do the case study, if folks have calls, then just go to the deeplife.com listen and submit some updated calls.
- [42:44]
A
Is that Deep Life or the deep life.
- [42:46]
B
TheDeeplife.
- [42:47]
A
TheDeeplife.com Listen, there's a link and you can record the call right there from the browser. Yeah, do that. We have a lot of calls, but they're getting kind of old, right?
- [42:55]
B
Yeah.
- [42:55]
A
You got a good shot. A good pithy call on a topic I've been talking a lot recently. You got a good shot now, so I would go record the calls. We do have a case study, however. I love case studies. This is kind of a long one, but it's great. Case study on my theories of lifestyle centric planning. I'm thinking so much about this because of my new book, so I'm happy to have this. We have what theme music for this, right?
- [43:18]
B
Yep.
- [43:18]
A
Do we. I don't remember. Is it just to introduce a segment or do we play it the whole time?
- [43:22]
B
I don't remember. Oh, the two episodes. We did one, but then last week you wanted to hear both. We'll play it now and see your mood at the end.
- [43:29]
A
All right, let's see what we. Let's hear it. Just want to do a deep thoughts with Jack Handy. All right, so today's case study comes from Sven. I hold a bachelor's degree in interface design and in my 20s first worked as a web developer and later as a UX designer for several digital agencies. I enjoyed my work, but I felt stuck in big cities and kept dreaming about a life closer to nature. So in my mid-20s, I did something that you would surely advise against. Oh, I threw my career into the trash and moved to Norway to live and work on an organic farm. I had a great time there and enjoyed every aspect of it. However, I didn't have a sustainable long term plan. I realized I needed and wanted to move back to my home country where my girlfriend, now wife, lived. Back home, I decided that my next big move would be to find a job that allowed me to work in nature. So I signed up for a nature guide program and actually managed to land a job as a guide at the very place where I had completed that course. For the last six years I worked there running workshops and similar programs. Sounds like a happy ending, right? Not quite. Life changed. I became a father of two children and we moved closer to my parents so we could get support from the grandparents with family life. This meant I was now living almost two hours away from my workplace, constantly torn between family and professional life. On top of that, since most workshops take place on weekends, I was away from my wife and our two little ones at the very time they needed me most. It turned out that the dream of working in nature definitely had its flaws. I'm going to stop there for a second. Let's take stock of where we are in the story so far. We're seeing a common issue when people think about the deep life. So a common issue that people have is that you fixate on a single change that you begin to believe will make everything better. It's simpler to think about a single change. We can sort of inhabit that change. The idea of doing something radical itself makes us feel excited in anticipation and we can sort of take the imagined feeling of the best parts of that change and sort of expand that in our mind, like, my life is going to be better. So Sevin did this with nature. He's like, I like nature. So I'm just going to focus like a laser on, if I could just be in nature all the time, I would be really happy. The problem, as we often say when we talk about lifestyle centric planning, is that your daily subjective mood is not the result of a single decision or change, but on all of the relevant aspects of your life. Call this, your lifestyle, all the different things that are relevant during the day add up to give you your subjective experience of the day. So you might like the part where I'm giving a workshop and it's in nature and that's nice. But you have other parts of your life too. Like the fact that you had to drive two hours to get there and that you're kind of stressed out about what's going on with your parents and like the kids are, you're not around enough to help and it's creating tension with your wife. All of your houses may be nowhere near nature and you're actually spending most of your time in a car. All of these other aspects of your lifestyle matter too. And this is how you can make one radical change that you're excited about and you yet end up less happy than you were before because by making this thing better, you accidentally maybe made other things worse. So lifestyle centric planning says you got to construct your ideal vision for an entire lifestyle. All the parts of your life. How would I reorient my life so that all of the parts were something that resonated and then how do I make progress towards that? You're going to have. You're going to enjoy your life more if you explicitly construct it so that all of the things that contribute to your enjoyment are being pushed in ways that are better. Let's return the Sven because he recognized that and let's see how he made some changes. Around this time. I came across your books. Deep Work so Good they Can't Ignore youe and Slow Productivity. They gave me a framework to better understand my professional situation. I had to take a hard look at myself and realize that for the last five to six years, I hadn't built transferable career capital that could help me in a different role. After some reflection, I decided to revive my previously built career capital as a web developer. However, I hadn't been coding seriously for years and looking at job descriptions, I realized how much had changed in the web development space and I had a lot of catching up to do. All right, so he's doing something here called evidence based planning. So he makes this decision, he's looking at his life and is like, actually I need a job that is going to whatever the criteria here, but probably like more money, more autonomy, I don't have to commute so much. Whatever web development is like, I have career capital there. Let me start there. But he does evidence based planning. He actually looked at real job listings like oh, skill A, B and C. I don't Have. So I'm going to, I'm going to summarize a little bit here. But basically he uses like techniques from deep work and slow productivity, as well as from Scott Young's excellent book Ultralearning to begin a education process of learning the specific skills that his evidence based planning said were important, not what he wants to be true, but like I need to know A, B and C, these employers want exactly those skills and how do I get there as effectively as possible? I'll jump back to regular speed here thanks to your techniques and tools. It took me about a year to get out of my difficult job situation and back into my old career. The side effect, I now earn more than twice as much as before. My next move won't be as bold as the ones I made in the past. I want to become a valuable asset for my company, get so good they can't ignore me, and then use my career capital to reduce my working hours so I can have Fridays off to spend in nature without having to work there. So see what happened there is when he did lifestyle centric planning, he's like, I like nature. But by reorienting my life around my career takes place in nature. A lot of other things got worse. And actually being in nature is not as great as you think when you're also like working. He realized like, oh, if I could take my programming expertise, make it up to date, make myself really valuable, I can go back to programming, get good enough that I can cut my hours, still make more than my old salary, only work three or four days a week, and then spend time in nature very intentionally. But now without the commute, now being home with my family, now things being more stable, man, that's a better life. So lifestyle centric planning worked out better than just putting laying back, like, what's a radical thing I can do? I'll move to Norway. I'll get a job as a nature guide. Lifestyle centric planning was not as exciting, but it led them to a better place. All right, so we got coming up my investigative journalism beware. I do a little bit of data reporting. But first, before we get there, what you've really been waiting for to hear about another one of our sponsors. Here's the thing. If you run a small business, you know, there's nothing small about it. Every day there is a new decision to make and even the smallest decisions can feel massive. Jesse knows this. I get upset whenever anyone refers to our business here as small. I know how hard it is to run this. Just the other day for example, the nice woman who runs the gift shop down the on the next block here in Takoma park asked me how life as a small business owner was going. So naturally Jesse and I rolled our car into a swamp. We're not gonna put up with that nonsense. So here's what I'm trying to say. It's hard to run a small business, so when you can find decisions that are no brainers, you should take them. And when it comes to selling thing using Shopify is exactly one of those no brainers. Shopify's point of sale system is a unified command center for your retail business. It brings together in store and online operations across up to 1,000 locations. It has very impressive features like endless aisle ship to customer and buy online pickup in store. And with Shopify POS you can get personalized experiences to help shoppers keep coming back. And they will come back. Based on report from EY businesses on Shopify POS see real results like 22% better total cost of ownership and benefits equivalent to an 8.9% uplift in sales on average relevant to the market set survey. So get all the big stuff for your small business right with Shopify. Sign up for your $1 per month trial and start selling today at shopify.com deep go to shopify.com deep shopify.com deep I also want to talk about our friends at MyBody Tutor. I've known Adam Gilbert, MyBody Tutor's founder for many years and he is my go to guy for fitness advice. His company, MyBody Tutor is 100% online coaching program that solves the biggest problem in health and fitness, which is lack of consistency. They do this by simplifying the process into practical, sustainable behaviors and then giving you daily accountability and support what you need to actually keep taking action on your plan. You have an app, you check in every day like, hey, here's what I ate, here's how I did my exercise and your coach checks it. Hey, how'd it go? If you have a question like, you know, I'm really struggling with X, like great, let's fix this right away. Knowing there's someone checking on what you do every day and is there to help you if anything's not working, that's how you get consistency in getting healthier. It's not the information, it's the consistency. And having this online coach, it is a really, really good way to get healthy. All right, so if you're like, I want to get in better shape, this is the way to do it. Go to mybodytutor.com mention you heard about it on the Deep Questions podcast when you sign up and they will give you $50 off your first month. That's MyBodyTutor. T u t o r.com and mention deep Questions. All right, Jesse, let's move on to our final segment. So I wanna talk about an interesting article about technology and students and I want to look a little bit deeper at it and have a bigger point to make. This article actually appeared recently in the Washington Post. It came to my attention, however, when Tyler Cowen wrote about it on his excellent blog, the Marginal Revolution. I'll pull it up here on the screen, the blog post, because the Washington Post is behind a paywall, but Marginal Revolution is not. All right, so here is Tyler Setup just summarizing this article. He says until recently a nearby radio telescope meant a local school could not use WI fi. And how did that go? All right, so this article that he's citing here is about Green. Green Bank, West Virginia. Have you heard about this place, Jesse?
- [53:54]
B
No.
- [53:54]
A
It comes up a lot in technology circles because it's rural West Virginia and the world's largest steerable radio telescope is there.
- [54:04]
B
Oh, really?
- [54:05]
A
Yeah, and there's a town next to it and radio interference could mess with the telescope. So there's no cell phones and there's no WI fi. So it's like a town that's kind of like off the grid because of this telescope that's in the middle of the town. So what this Washington Post article that Tyler Cowan is citing, what it was about, it was actually an op ed and it was written by someone who has a book he's working on about Green Bank. What it was arguing is the fact that there is no WI fi in the town has been hurting the students in the school there. Green bank has an elementary, middle school, combined school, about 200 something students. He's arguing without WI Fi, they can't use like the modern Chromebook, online curriculums do online testing and this is hurting the students. So let me read here. This is a Tyler Cowan citing the Washington Post. So what I'm about to read here is from the Washington Post article. While the rest of the country rushed to bring tech into classrooms, Green bank remained stuck. In 1999, without Wi Fi, the school's 200 students couldn't use Chromebooks or digital textbooks or do research online. Teachers couldn't access individualized education programs online or use Google Docs for staff meetings. Even routine tasks such as state mandated standardized testing became challenging with students rotating through a Small hardwired computer lab where they took the exams. Some of the teachers say this has been a problem. Here's a quote here. The ability to individualize learning with an iPad or a laptop, that's basically impossible. That was teacher Darla Huddle. Here's another teacher being quoted in the op ed. Without the online component of our curriculum fully working, it's really detrimental to our instruction. That was Sarah Brown. So this is the issue here. They can't do this sort of modern ed tech stuff. And the argument of this article is like, that's held this school back. And how do we know this? Well, there isn't data on this in the article itself, like specific data. But here's what the author says while these discussions dragged on. So discussions about like, is there some way we can get like low powered WI fi or some other way to get WI fi in the school? While these discussions dragged on, students fell further behind in math and reading with Greenbrink consistently posting the lowest test scores in the county. All right, this is an interesting argument here that without modern ed tech, students are falling behind. And that's like a natural experiment. And look, this school has the lowest scores in the county. Now this is like a relevant time to be arguing this because there's so much discussion about phones in schools and technology in schools. And it's mainly negative. Like technology in schools is often negative. But here's this op ed in the WaPo that's being contrarian. And Tyler, who also has a real affinity for technology, is like, yeah, come on, let's be careful about it. Maybe some of this technology is really helping. So I want to look into this. I'm not an expert data journalist, but I was like, this is interesting and I want to look a little bit closer. So what did I find? Well, first, is it true that Green bank consistently posts the lowest test scores in the county? This is turns out to be called Pocahontas County. It's kind of interesting in West Virginia. Okay, so it's Pocahontas County, West Virginia. Yes, the Green Bank Elementary Middle school has the lowest scores in the county. But there's a problem here. It's a very small county. I mean it's a lot of geographical area. But you know, this is not Montgomery county here in like Washington D.C. the county, you know, there's the Green Bank Middle School and elementary school. The county has one other middle school and a shared high school. Then there's two smaller elementary schools that are in like another town over that are much smaller. One of which, which has the county's like, gifted and talented program. There's only 70 kids there. Half of them are gifted and talented kids. Like, we're, we're not talking a big county. It's a handful of schools of which like two of them, of the six are in Green bank, one is shared and then there's like one other middle school and two other elementary schools. They do. So yes, like the, the elementary school in Green bank is, has lower scores than those two other elementary school. The middle school has lower scores than the other middle school. All right, but. So it's a small county. But that doesn't necessarily tell us much like maybe Green bank. You know, for whatever reasons, that part of this, the county just is worse off and they just get worse scores. I don't know what's going on there. Do we know this is from not having WI fi or not? Well, what we really would need to test this is we need time series data. So we need to see test performance over time. Right. Because until somewhat recently, it wouldn't matter if you had WI fi or not. Right. So iPads were introduced in 2010. We see the rise of Chromebooks and classrooms picks up in the 2010s. That's really where this thing, this happens. So before the 2000 and tens, there would be a lot more technological parity between this particular school and the other school in the county. Right, because no one was using technology like that in school. So what we really need is like a time series that shows somewhere in the 2010s, the Wi Fi free school begins to separate. The impact of not having WI fi makes them worse because they might have always been worse. We want to see them get worse. All right, So I went looking for this data. It's hard to find time series data for the particular schools in Pocahontas County. But because Pocahontas county is so small, there's two middle schools, one of them is with WI fi, one of them is not. There's three elementary schools, one with, one without, and the other two add up basically to the size of the one that didn't have WI Fi. It's such a small county. What we can do is compare Pocahontas county to other counties in West Virginia, because presumably if like half the students in your county didn't have WI Fi and that affected them, that should bring down the whole county around the time that why Chromebooks, etc. Became big in school. So there is data, I can get data on anyap. You know, these are state mandated Test scores. I can get data on this county by county. So let's see what's going on here. So let me load up some charts. This is from Education Recovery Scorecard, which aggregates a lot of this data. It's really interested in, like, what happened after Covid, but they have data that goes back to as far as 2009. So it's perfect for us. What I'm loading on the screen here, for people who are watching instead of just listening, is math performance, grades three to eight, because again, going past middle school, it's a shared high school. So, like, we don't learn much from that from 2009 to 2024. All right, so as we see here, going from 2009, we see an increase. There's a bit of a gap in the data, but we see roughly, actually, math scores were getting better. Go. Pocahontas county math scores were getting better from 2009 until about 2017. In 2017. Oh, this is interesting. The score started going down. They went down, went down. Then we get the pandemic, where we don't have data, but they were falling multiple years before the pandemic. And then we see after the pandemic, we begin to get a recovery, which we've seen nationwide is things fell so low during the pandemic that there's a recovery right after. So that's the chart we're seeing. If we look at reading scores, it's something kind of similar here. The peak is around 2015. Then it begins to go down after that as well. The timing here is compatible with the WI fi hypothesis. Right? That kind of makes sense. Chromebooks and all these other things that require Internet begin spreading in the 2010s? It might make sense that where would you first start seeing the impact of not having this technology, like, midway through the 2010s, as these things were gaining traction, that might be where you start seeing performance go down. So this data from within Pocahontas county itself is compatible, roughly speaking, is compatible with the Washington Post hypothesis of without WI Fi, things started to get worse. But if we want to do a controlled experiment, we have to compare this. The similar counties. So what we need to do, because Pocahontas is half the non WI Fi, and we can't break that out. So let's find other counties in West Virginia that are similar in terms of size and demographics and socioeconomic status, like very similar other counties nearby in West Virginia where there were no radio telescopes and everyone is allowed to use WI Fi. What we would expect to Find if the WI fi hypothesis was correct, is that Pocahontas county should have a much more notable drop in performance starting in the mid 2010s than these other counties that didn't have any WI fi restrictions. We have that data. That's what's interesting. So let me scroll here a little bit. All right, so this is. We're going back to math performance here. We have three stack charts. So if you're listening, I'll explain to you what we see. Each of these charts has a downwards arrow and an upwards arrow. The downwards arrow, which is purple. I'll put this up here on the screen for Pocahontas county schools. First. That is the decline in math scores during that period up to 2019, where. Let me get the exact dates here, 2019 to 2022, where we had the steepest sort of losses we saw in those curves before. And then the green arrow is the improvements that they've seen since the pandemic. So this chart is showing here for Pocahontas county schools. It's just quantifying what we saw on that chart in that downhill period from 2019 through 2022, there was a 0.6, negative 0.6 drop in their performance compared to. I think this is like state average. So, yeah, they fell. And then we see the green. There was like this 0.36 increase. So don't worry about the numbers. But there was an increase. This is the recovery after the pandemic. All right, so I'm just quantifying what we saw on that chart. Here's what's interesting, though. They did the same thing for all of West Virginia counties. That's the next one. And for counties that had similar socioeconomic, demographic and size, so similar population. So if you're wondering, this is Nicholas County, Hampshire County, Barber County, Tucker county, and Pendleton County. Here's what's interesting. The similar West Virginia districts that were the same in terms of demographics, but differed mainly, and they didn't have the WI fi restrictions, they saw a bigger drop from 2019 to 2022, and they saw a smaller recovery after the pandemic. So the similar counties that had WI fi and never had that taken away did worse than the county in which, like, half the students are in this WI fi free zone. And in fact, if you look at West Virginia as a whole, that also was worse than what Pocahontas county did. So, look, I don't have school by school data. I hear this a lot. I found a lot of informal reporting online that did blame poor performance in Green bank by them not being able to use Internet in the school. So that might be true and these teachers seem to think it's a problem. But if we look at the data without having time series data from specific the handful of schools within Pocahontas county, we do not see here a data story that looks at that looks like the lack of Internet, once Chromebooks became a thing and these online curriculums became a thing, began to make Green bank much worse. It seems like that's just a bad school. Those schools are bad and they've been bad and I don't know why you'd have to know about like, you know, it's just like what this town is. I don't know what particularly is bad about that town. It could be little things, by the way, because these numbers are so small. Not to get too much into the data, but I was looking at proficiency on math and reading test scores broken down by the individual school. So I can get those numbers for the most recent times they've measured them and they're like 50% better, for example, in Hillsborough elementary versus the Green Bank Elementary. But Hillsborough elementary in Pocahontas county is the gifted and talented program. There's only 70 kids in that school, 70 kids total. So all it takes is like, oh, we have 30 of our gifted and talented kids are there. You're going to get 50% better, you know, number of people who are math proficient. Right. So these are small numbers. So we have to be careful. So yeah, there's not a lot of schools there and the Green bank one doesn't do well. But we don't have evidence because that it is because they couldn't use Internet connected Chromebooks. Again, the only way for this to be possible would be somehow the lack of Chromebooks and Internet connected tech really caused the Green Brink schools to fall really hard, but for unrelated reasons. Coincidentally, the other two or three schools in Pocahontas county did unusually well during this period and they sort of offset the fall that the Green bank was having. And that's why that county, even though other counties with the same demographics as Pocahontas county fell farther, that there was something special happening with the non Green bank schools in Pocahontas county where they offset the Green bank losses. Maybe that's true, but we would have to hear a plausible reason why that's true and actually see school by school data. So I don't know. The author of that op ed is writing a book on green Bake and he might have really good Data. But knowing what we know now, I think this is a nice cautionary tale and it's a good reminder for myself or anyone else who talks about technology trends. When there's an answer that we like, it's often easy to jump to it, find any point of data that seems to imply that, and expand it beyond what the data says. And I think that seems to be what was happening here. There's a nice subtle leap from the schools without wi fi are worse to the schools without WI fi are worse because they don't have WI fi. It's an easy leap to make, but the picture gets much more murkier once you pull even a little bit on the data storage. So anyways, I guess this is just a back to school note of we all have to be careful and take with grains of salt claims that are made that sound intuitive and there's a little bit of data to support doesn't necessarily mean it's true. When it comes to cell phones in schools, I have read exhaustively not just the research, but the debates that researchers are having about the research and the complaints and then how the complaints are answered. There's an hour long talk I give about the evolution of the research literature on harms from phones for kids. I feel like I know that data very well and it makes me confident to say these are often a problem. The benefits aren't worth it. You shouldn't have phones before high school if you're a kid, right? That's really different than like, hey, that school struggles and they don't have WI fi. Internet's good. So anyways, there we go. I'm not investigative journalist data. Who knows, maybe this, this professor has this school by school data. But I think the picture in West Virginia, Pocahontas county is more complicated than if only I could synchronize my Eureka math curriculum with like online resources, our kids would suddenly be much better. Harder reality. All right, speaking of hard realities, that's all the time we have for today, so thank you for listening. We'll be back next week with another episode. And until then, as always, stay deep. Hi, it's Cal here. One more thing before you go. If you like the Deep Questions podcast, you will love my email newsletter, which you can sign up for@calnewport.com each week I send out a new essay about the theory or practice of living deeply. I've been writing this newsletter since 2007 and over 70,000 subscribers get it sent to their inboxes each week. So if you are serious about resisting the forces of distraction and shallowness that afflict our world. You gotta sign up for my newsletter@calnewport.com and get some deep wisdom delivered to your inbox each week. Sa.