← Deep Questions with Cal Newport
Loading summary
Transcript1 lines
- [00:00]
A
Last week, Anthropic released a report with a scary sounding title, when AI Builds Itself. And it came accompanied by a scary animation that shows machines replicating themselves exponentially like cells in a petri dish. Now, the body of the report itself keeps these dark vibes going. I want to read you some actual quotes here from the intro to the report. They say for most of AI's history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work. Taken far enough and given enough compute, this trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self improvement. We are not there yet. And recursive self improvement is not inevitable. But it could come sooner than most institutions are prepared for. A little bit later, they then add, AI that can build itself would be a major development in the history of technology, one that could bring enormous good for the world in science, healthcare and beyond. But full recursive self improvement also might increase the risks of human humans losing control over AI systems. Now, if you look at the headlines generated in response to this report, most of them focused on a section of the report that seemed to call for a worldwide pause on AI development to avoid this scenario of humans losing control. But if you read that section closer, you see that's not actually what the report says. Here's the actual wording. If it were possible to effectively slow the development of this technology, to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. So in other words, Anthropic is saying we'll only slow down if everyone else around the world does too. Otherwise, we have no choice but to continue with our efforts at full speed. Now, look, this is pretty grim stuff. Anthropic is basically saying that we are potentially hurtling towards a world of AI that improves itself rapidly until we lose control over it. And they're saying there is nothing that we can do about it, except maybe continuing to publish solemn reports with fancy animations and I guess also cash in on our stock options after an ipo. All right, so here's the key question. Are these fears justified? Well, it's Thursday, which means it's time for an AI reality check episode of this show, which is a good opportunity to go looking for some measured answers. And that is exactly what we are going to do, as always. I'm Cal Newport and this is Deep Questions, the show for people seeking depth in a distracted world. All right, so how, how much should we actually be afraid of? Recursive self improvement. I want to look at the core charts from this anthropic report so we can see what they are pointing to that is giving them these RSI fears. All right, I'll load the first chart from the report up here on the page. It's called Code contributed per quarter per person by quarter. It's measuring how much lines of code their engineers are producing over time. And what we see is in 2021-2022-2023-2024, the beginning of 2025, not much. And then the second half of 2025 and into the first half of 2026, the amount of code jumps up. All right, here's how the report itself describes this trend. It says, a caveat. Lines of code is an imperfect measure as it measures quantity over quality. So eight times lines of code per engineer per day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration. All right, so chart one. Once we introduce these tools in late 2025 for software development, we began producing a lot more code using AI. All right, let's go to the second major chart in this paper. This is called CLAUDE code session success rate. What we see here is various color lines. Each of these represents a different type of problem. You could ask an AI LLM to solve trivial tasks, routine tasks, substantial tasks, and then at the bottom, open ended problems. It starts in the fall of 2025. And what we see is especially with open ended problems into 2026. And as the new models Mythos and Cloud Opus 747 are introduced, the success rate with those jumps up from low 20% to somewhere between around 70%. Okay, notice that graph starts. It just starts in the fall of 2025 because. And we'll get back to this in a second in more detail, but in order to ask AI to solve a hard coding problem like the example they gave was why is this thing, the system we programmed is crashing? Why is it crashing and sending the AI to figure out why you can't actually have data points for before the fall of 2025 because that's when Anthropic released along with OpenAI, their first mature, what's called a coding harness. So a control program that allows you to leverage an LLM plus a lot of hard coded logic to actually tackle multi step plans. So you can't really have data for that from before because there's nothing to ask to try to solve a problem like that. You couldn't just prompt the ChatGPT to do that or plod. So you needed the arrival of mature software development tools built on LLMs before you could even do these TAs. So basically what this graph is showing is that the very first software development coding harnesses they introduced in the fall of 2025 couldn't solve the hardest type of problems. And then they fixed that the next year. All right, there's one final chart in this paper that they're pointing at to justify their concerns. I'll put it on the screen here. The title is Where a Researcher Went wrong. Could CLAUDE have done Better? And we see here a bunch of different models of CLAUDE and percentage bars. And back for these early models of Claude, we were getting like 50 or 45%. And now with the very newest versions of Claude, like Opus 4.7 and Claude Mythos, we're at like 59 to 64%. So we got like a 10 or 15% improvement on that measure. What is this measure? It's a little bit complicated. Essentially they have these transcripts of programmers working on programmer style tasks. And what they're looking for is an example where there's some problem that the programmer is trying to solve and they take a wrong turn. So they go down some path to explore something that turns out not to be the correct source of the problem. So what they would do is take the transcript of this session right up to the point where the human was about to try to explore something wrong. They fed this transcript to one of these mature coding harnesses on top of an LLM and said, hey, what do you think we should do now next? And if it found the right thing to do, where the human looked at the wrong thing, then they would say, this is a case where the LLM plus its coding harness were smarter than the human. So that's jumped from 50% from a couple years ago to 64% now. All right, so this is sort of like the core data they're looking at to capture this idea of AI is getting so smart at producing programs that maybe soon it will be able to not only improve itself, but, and I'm going to quote here from the report, become capable of full recursive improvement and begin building their own successors. All right, so looking at that data and what I know about current AI tools, are these fears justified? And I would say no. Here's what this data is all describing. About a year ago, the major AI companies got serious about building tools to help software developers. These tools are a combination of human written programs called coding harnesses and LLMs. The coding harnesses can make calls to the LLMs and then act on what the LLMs do. The coding harnesses can also interact with other various tools on the computer. So what all these charts seem to be showing is that like, oh, once they got serious about writing these coding harnesses and also tuning the LLMs to play nicer with these coding harnesses on these sort of computer programming related tasks and evaluations, things jumped up the world. Before having these harnesses, we couldn't do well on programming task. And now that we do, we do better. Now none of this is trivial. In fact, it's a very smart market for the AI companies to go after. Software development is a big industry. These coding harnesses built on top of LLMs are really potentially very useful because LLMs are very good at understanding and producing code. The type of tools that a coding harness has to access in order to execute things are simple and text based. This is like a perfect case market to build the first killer apps on top of LLMs. Starting last year and then really picking up speed last fall, the major players really got involved in these harnesses. So I think that's what these charts are all showing is like. Yeah, these harnesses make us, they're getting really good at the. We can suddenly do a lot of programming tasks we couldn't before. But does that mean that recursive self improvement is imminent? It does not. And there's two reasons why. And I want to look at these one by one point. Number one, faster software development doesn't equal smarter AI. All right? So these tools help computer programmers produce code faster or find mistakes or issues in existing code or systems faster as well. These capabilities, though useful and perhaps a good source of revenue for these companies, doesn't add up to AI being able to improve themselves to create AI systems that are much smarter than what humans would have otherwise been able to produce. Now why is this? Because the bottleneck to producing breakthroughs in AI, to building new AI systems that are substantially more capable than those that came before, is not the speed with which you produce computer code or track down bugs or issues in existing code. The thing that advances the capability of AI beyond just training it longer, are ideas like if we look at our current world of gender of AI built on large language models, there's three big insights that built on each other. The first was Geoff Hinton and his collaborators working on back propagation. Right. This was an intuition that if you applied calculus properly, it would be you could train neural nets that had many, many layers. This was the so called deep learning revolution. Really picked up speed in the 1990s into 2000s and it was critical for what we have today. The second big idea that led to our current AI world and was Google research notion of adding something known as an attention transformer is a mathematical formalism. You add into the feed forward architecture of a large language model that allows there to be a selective attention on the input tokens as you are trying to calculate what token to produce next. This allowed sort of coherent generation of text based off a very complicated input or request. Right. This was revolutionary. This is largely a sort of intuition around architecture and linear algebra. The third breakthrough that led to our current moment was researchers at OpenAI led by Jared Kaplan, wanting to know what happened if you scaled the size and training compute of large language models well beyond the limits where traditional machine learning theory said you're going to start overfitting. Those three things together led us to this current future. And none of those had anything to do with computer programming. It wasn't, oh, if only we could have programmed faster or been better at finding bugs in computer programs, we would have had those advances quicker. These advances were scientific, not engineering. AI does not advance. It's not created at the fingertips of computer programmers. And speeding up those computer programmers does not speed up the rate at which we get smarter, more capable or more advanced AI systems. All right, the second point I want to make here is that these tools, these software development tools that they're testing in these charts are completely controllable arguments that say we're on track to losing control of AI. Like to think about these sort of AI based coding tools as some sort of unknowable alien black box that we ask to do things and we don't know what it does, it just kind of like goes off and behave. And as we don't know its intentions and in fact it could, it could develop somehow its own intentions that are contrary to us and go rogue. Right? We just see this like HAL 9000 from Stanley Kubrick's 2001. This is not how these systems work. And again, I mentioned this before briefly, but let's just say it again briefly right now, the software development tools that they are testing in these charts is a combination of a large language model like Claude mythos or opus 4.7 and what I've been calling a coding harness. Now the coding harness is a computer program. It's written by humans. There's nothing obfuscated about it. It's not a neural net, it's not something that trained. There's no mystery about how it behaves. It's a bunch of pattern recognition, often using old fashioned tools like regular expressions and a huge amount of conditional logic. If then statements, if this, do that, a lot of hard coded cases of things that comes up often in software development and they want to make sure they do the right thing. This coding harness, programmed by people and completely linear, deterministic, we know exactly how it works. We'll then make calls to an LLM through an API when it needs LLM intelligence, right? So it might say, okay, what we need to do here is produce some code. All right, I'm now going to create a prompt and send it to the LLM. Can you give me code that does this? And the LLM will return some code or okay, we want to create a multi step plan for investigating why this program is crashing. I will put that into a prompt, I will send it to the LLM, say will you write me a multi step plan for this goals? Here's the constraints, here's the tools you can use, right? In this format the LLM produces text, it gets that back. And now the coding harness, human written code, completely non obfuscated, completely deterministic, looks at that plan and says great, let's break this up into steps and execute them one by one. It's the coding harness that actually has any action, that actually accesses tools. It's the coding harness that has all the control logic. So we program that, which means it's imminently controllable, right? So like let's do a thought experiment here. Like let's say there's a certain tool you don't want your programming tools to use on your computer. Well, you could do that with 100% certainty by just never having your coding harness call that tool. Just like no matter what instructions it gets, it's just not on the list of tools that it'll call. This is the thing we often mix up. LLMs are unpredictable, they're token producers and they're deterministic in the sense that if you give it the same input, it'll give you the same output, but they're unpredictable because the output that you actually get from an LLM is effectively a probability distribution over potential next tokens. And then you probabilistically select a token from that distribution. This is why if you ask the same prompt multiple Times you might get different answers because those random selections made outside of the LLM might select differently. So LLMs are unpredictable. Right. That's why if you say, hey, give me a plan for this, sometimes the plan will make sense and sometimes it doesn't. And sometimes it'll seem reasonable, but it'll have a weird aside. Or if you give it a big scenario and say, what do you want to do next? It might say something you weren't expecting. And if it gets little cues and it prompts that we're playing a sci fi game, it'll be very sci fi Y and maybe it'll do like weird things. LLM textual output is unpredictable. That doesn't mean that AI systems that query LLMs are unpredictable because all of the control logic and action is done in human written programs. So this idea that it's like these are just these black boxes that maybe at some point will spontaneously start improving themselves makes no sense. The LLM is a static thing that you can prompt and it will give you text back. That's it. And a human controlled program does that prompting. That is not a setup. Where there is a. Where is the growing intention here? Where is the. Like we're going to change our own code. That type of thinking only makes sense when you kind of obfuscate this all as some sort of alien brain we don't understand. No, it's an LLM that's unpredictable plus a very predictable coding harness. All right, so let's step back here. What's a better description of the reality where we are right now as these programming tools built on LLMs are getting good, Right? I mean, we've only been working on it for less than a year, but we're finding like, oh, this is a great application of LLMs. We're still trying to figure out what these tools can do and can't do and how best to integrate them. What's a better description of this reality? Well, I want to bring one more chart up here on the screen. It comes from a recent John Byrne Murdoch article in the Financial Times. I actually saw this chart first through Gary Marcus newsletter. Here's the chart. The title here is relative Change the monthly volumes of iOS app releases and reviews. And here's what we see is over time, starting in 2025, when these AI based mature coding tools were first released, the number of iOS apps released jumped up that dark blue line there. At the same time, let's look at this light blue line apps with significant usage. It stayed steady and is even falling. All right, this I think captures something important. We're seeing that the introduction of these software development tools led to an increase in the number of mobile apps, which are like exactly the type of programs that basically anyone with some tokens to burn and cloud code can just spit out. They're easy for AI to produce, while at the same time we're seeing a decrease in apps with significant usage. AI powered software development tools speed things up, and that makes everyone involved feel more productive, but it doesn't necessarily mean that we're accelerating the creation of useful new things. AI often offers interesting potential to a lot of fields, but figuring out how to make it economically useful is a complicated effort that requires a lot of focus. Putting out these sci fi themed essays about all the doom that's lurking right around the corner might make these AI companies feel important or exceptional, but I'd argue that at the moment they're proving, if you'll excuse the use of an ironic term here, unproductive programming tools are interesting. They're here to stay, though, in their final form, and how we use them is still up in the air. This is a new thing. But nothing about this new tool's arrival should make you think that somehow AI is going to start improving itself. I would almost say shame on Anthropic for dropping this report. No suggestions, no reassurance, no culpability, no responsibility. Just like, what can we do? This is not productive. This is not useful. If you're wondering, let's just sum this all up. No claude code or codecs or cursor is not a thing that's going to very soon start improving itself Till we lose control of AI. We got bigger fish to fry here. All right, that's enough for this week. We'll be back on Monday with an advice episode of this show, which you check out. I'll probably have another reality check next Thursday. Until then, however, remember, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up@calnewport.com ideas and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today. Cal Newport.com ideas.