Why AI Is Not a Normal Technology (with Peter Wildeford)

Summary

Podcast Summary: Why AI Is Not a Normal Technology (with Peter Wildeford)

Future of Life Institute Podcast | Host: Gus Docker | Guest: Peter Wildeford | April 29, 2026
[Listen to the episode for individual nuances; this summary distills the full conversation, focusing on major arguments, illustrative quotes, and critical timestamps.]

---

Episode Overview

This episode features a conversation between Gus Docker (Future of Life Institute) and Peter Wildeford, head of policy at the AI Policy Network and a distinguished forecaster on AI trends. The discussion critically examines whether AI is "a normal technology," explores AI’s economic and societal impacts, discusses forecasting methodologies, and considers the national security and policy challenges posed by advanced AI systems. Wildeford offers insights informed by forecasting experience, trend analysis, and engagement with AI policy and governance.

---

Key Topics & Insights

1. Is AI a Bubble?

• Wildeford argues that "AI is very clearly not a bubble" ([02:15]):
  • Unlike historical tech bubbles (e.g. the dot-com era), AI firms like Anthropic and OpenAI face overwhelming real demand and have credible revenue growth.
  • “Anthropic has more demand than they have the ability to serve. They're like batting away users right now because there’s just too much demand…” ([02:28])
  • Revenue growth described as "astronomical" ([03:40]), with some firms "outpacing some of the greatest companies of all time, such as Google or Uber" when using annualized run rates ([05:13]).
  • Spending outpacing revenue reflects a strategy to build larger, more valuable models, not unsustainable economics.

2. Is AI a ‘Normal’ Technology?

• Wildeford's thesis: AI is fundamentally not a normal technology due to its adaptive agency and emergent capabilities ([07:31]).
  • "Normally when you have a technology like a hammer or a toaster, it just does what you tell it to do… But with AI systems… they're not being programmed line by line, [but] learning… not tremendously different from how a human would learn." ([07:40])
  • AI can, almost unpredictably, develop new abilities with scale—e.g., becoming adept at hacking without being explicitly taught.
  • “If you built a bigger toaster and all of a sudden your toaster could like hack the Internet... that's a very, very abnormal thing.” ([08:31])

3. Current Economic Impact & Diffusion Lag

• Despite impressive model capabilities, macroeconomic indicators have yet to move:
  • “If everyone in the entire planet were to immediately adopt the latest models… you would see much more impact.” ([09:43])
  • The lag arises because most users have not adopted, or are underusing, the most advanced AI (e.g., “Claude code” or GPT-5.4 Pro) ([13:06]).
  • Adoption is still at an early stage: only ~50% of US workers use AI tools at all ([13:10]).
  • Anticipated noticeable impacts on unemployment and productivity are expected within 1-2 years, with “a extra percentage point of unemployment attributable to AI” plausible by next year ([11:25]).

4. AI’s Exponential Growth & Surprise Capabilities

• AI capability trends are extreme and accelerating:
  • “Models are doubling in their time horizon every approximately 144 days... That’s just a very, very high number.” ([20:41])
  • Extrapolation suggests AI will soon surpass humans’ solo work endurance (“time horizons”) and remote freelance breadth, with 20% of remote tasks automatable by AI by year-end 2026 ([22:58]).
  • Wildeford concedes that trend skepticism has historically been misplaced: “All these people who have said AI has been hitting a wall have been just, like, very clearly wrong.” ([24:08])
  • Claude Mythos’s “superhuman vulnerability discovery happening this early on” surprised even cautious forecasters ([16:46]).

5. National Security & Governance Implications

• Frontier AI presents growing national security risks:
  • Vulnerability discovery by Mythos validated by external partners and governments ([17:00]).
  • AI could soon surpass humans not just at cyber offense but “military strategy, weapon design and deployment, cyber attacks, bioweapons” ([18:44]).
  • Policy urgency: “How are you supposed to run a government if you have private AI systems that can then just outmaneuver the government?” ([00:28], [64:29])
  • Wildeford supports integrating AI into military strategy, but only if well-controlled and robustly secure, warning about adversarial tampering ([28:54]).

6. Forecasting Methodology & AI Progress

• The "straight line on a graph" (benchmark trend extrapolation) has proven best for AI capabilities forecasting: “One of the dumbest ways of predicting this has actually been very successful…” ([20:41])
• Forecasters must balance 'base rates' and the history of “nothing ever happens” with openness to truly novel, discontinuous changes ([50:30]).
• Notably, AI itself is becoming a competitive forecaster—placing near top human forecasters (e.g., 11th out of nearly 3,000 in the Astral Codex 10 / Metaculus tournament; [72:07]).

7. The Debate on Technological Unemployment

• Wildeford: “It is different this time.”
  • AI could substitute for all human work, not just some, exceeding past pattern of job transitions ([42:26], [46:52]).
  • Classic economic optimism about future job creation may be misguided; even “weird jobs” like “professional partygoer” would struggle to absorb labor supply ([46:52]).
  • Social and democratic responses (welfare, universal basic income, bans or job protections) will shape transition outcomes ([38:53], [40:28], [42:26]).

8. Robotics and Physical World Transformation

• Wildeford challenges the necessity of robotics for transformative AI—much can be done digitally and via human proxies, but concedes that “really high skill robotics” could enable “runaway economic growth” ([32:34], [36:06]).
• Robotics remains a bottleneck, especially in dexterous manipulation, but breakthroughs seem likely, possibly accelerated by feedback from AI itself ([34:12], [35:14]).

9. Policy, Governance, and Democratization

• Significant policy challenges ahead:
  • Policymakers are not yet “attending to [AI] with the seriousness and urgency” warranted by current trends ([64:29]).
  • Historical precedent is for government “underreaction, until a crisis,” risking both sluggish and overzealous responses ([68:30]).
  • Key question: “Is this AI superintelligence thing a real thing, or… does the curve just flatline?” ([64:29])

10. Forecasting—Meta and Method

• Professional aggregators and elite panel forecasts outperform simple prediction market averages, but even broad community predictions are often useful ([78:43]).
• Liquidity and payoff delays are barriers for long-term bets and market development ([80:29]).

---

Notable Quotes & Memorable Moments

• "AI is very clearly not a bubble... AI companies have very clear demand... and just have this path to immense value.” (Peter Wildeford, [02:15])
• “If you built a bigger toaster and all of a sudden your toaster could hack the Internet... that’s a very, very abnormal thing.” (Peter Wildeford, [08:31])
• “A hallmark of a really good forecaster is knowing when things will be like the past and when there’s serious risk of things not being like the past.” (Peter Wildeford, [49:46])
• “At this point, I’ve kind of given up on trying to call the top of AI... just be very prepared for like human, if not superhuman, capabilities coming soon.” (Peter Wildeford, [24:08])
• “If you can build an AI system that exceeds human capability in military strategy, weapon design, and all sorts of national security domains... the implications are just kind of unprecedented and tremendous.” (Peter Wildeford, [64:29])
• “[Forecasting tournaments]...I ended up winning that, but AI ended up getting 11th place... and I’m eyeing my AI competitors not that far away on the leaderboard, which was definitely not something possible even like 1-2 years ago.” (Peter Wildeford, [72:07])

---

Timestamps for Important Segments

• Not a bubble: [02:15]–[06:26]
• The abnormality of AI: [07:31]–[09:21]
• Economic impacts and AI adoption lag: [09:43]–[14:33]
• Forecasted unemployment impact: [11:25]
• Trends in AI capability/extrapolation: [20:41]–[25:38]
• National security discussion: [16:46], [28:54], [64:29]
• Forecasting methodology: [50:30], [53:48], [78:43]
• Debate on future job creation and economic model: [42:26]–[48:57]
• Role and pace of robotics: [31:21]–[36:06]
• Policy challenge for governments: [64:29]–[69:38]
• AI as a forecaster and forecasting tournaments: [71:13]–[76:04]
• Community prediction markets and their limitations: [78:43]–[82:54]

---

Further Reading & Links

• Peter Wildeford on X (Twitter): x.com/PeterWildeford
• Peter Wildeford’s Substack: petervilleford.substack.com

---

Conclusion:
This episode provides a clear, rigorous, and candid account of why advanced AI is unlike previous technologies, the imminent economic and policy disruptions, and how forecasting—and even forecasting by AI—is rising to meet the challenge. Wildeford cautions against complacency, advocates for strategic policy engagement, and tempers optimism about adaptation, emphasizing the uniqueness of the AI transition.

Transcript

[A]

If you're someone who's using Claude code, you're kind of already in the top, like 0.1% of AI adopters. And then if you're able to use cloud code really well, then you're just like so much further ahead. I think at this point I've kind of given up on trying to call the top of AI and you just kind of have to ride out the line and see where it goes and just be very prepared for like human, if not superhuman capabilities coming soon. You see that the models are doubling in their time Horizon every approximately 104, four days. I think that, like Mythos will just be the beginning. I think this will only get more and more of a national security concern as AI companies build towards so called AI superintelligence. How are you supposed to run a government if you have like private AI systems that can then just outmaneuver the government if the government's not really getting its head around, like the national security implications of these AI systems and know how this, they could be used against us, know how we could be deploying them against our adversaries, and also know how AI might be like an independent threat actor. Like, I think it's just very important that the government is taking kind of a leading role here.

[B]

Welcome to the Future of Life Institute podcast. My name is Gus Docker and I'm here with Peter Wildeford. Peter is head of policy at the AI Policy Network, and he's also one of the best forecasters in the world, especially when it comes to AI. Peter, welcome to the show.

[A]

Thank you. Thanks. It's good to be here.

[B]

Great. So the setup I want to do here is for us to sort of walk through some popular narratives in the AI space and you tell me how you approach actually rigorously forecasting what's going on there. Now, I'm especially interested in sort of your methodology or how you think about the question, where do you begin thinking about these things? Does that sound good?

[A]

Yeah, that sounds good. I'll do my best.

[B]

Amazing. Great. Okay. I'm thinking we start with whether AI is a bubble, and we can define a bubble however we want. That's maybe part of the methodology to define what you mean. If you say whether, if you ask whether AI is a bubble, but how would you begin answering that question?

[A]

Yeah, I think AI is very clearly not a bubble. I guess this was maybe more of a popular talking point earlier last year when AI capabilities seemed to be phasing out. I guess like when you, when you think about a bubble, I Guess you kind of think about like the dot com era where you had like companies like spending a dollar to buy 80 cents and it was just like clearly unsustainable. But like AI companies, like they have very clear demand. Like Anthropic has more demand than they have the ability to serve. They're like batting away users right now because there's just like too much demand and they have like very clear revenue. And they also just have like this, this path to just like very immense value. Like Anthropic and OpenAI are basically just kind of creating machines that can just replace entire companies potentially in entire portions of the economy. Like that's just like such a huge opportunity and I think it's clearly panning out. So it's just like very clearly not a bubble.

[B]

There's a very clear story there of how these companies could generate enormous value and there is, as you say, the revenue to back it up. Would you look at the difference between revenue and valuation and say, okay, there's actually a lot of revenue here and not just high valuations, sort of in hopes of selling these shares to other people in the future?

[A]

Yeah, I mean, I think the biggest thing is just like the revenue and the revenue growth has just been astronomical. Companies have been deck toppling their revenue over the portion of just a year. I think the main reason why people point to it being a bubble is just that there's massively more spending right now than there is revenue. But I think like this does make sense. Like basically every time they build a new AI system, that new AI system manages to capture even more revenue. There's no requirement that companies have to be profitable at all times or in order to not be a bubble. Like you could certainly be a very legitimate company and still spend more than you earn, like Amazon has famously done that for decades. I think the main point is just like whether you project that revenue could exceed spending at some point, if you like eased up on investment and whether you have good unit economics. And I think it's clear that like hypothetically, if AI companies stopped trying to build even bigger models and just focused on serving their current models, like they would be able to do that in a profitable way, though they would of course also just lose out to competitors. But I think both OpenAI, Anthropic and other major AI companies have credible paths. Just like tremendous amounts of revenue that could then just like pay back all their spending and then some revenue growth

[B]

is pretty insane right now and that probably can't continue. But how do you, how do you Think about projecting revenue in the future.

[A]

Yeah, I mean I think you can look to companies own projections. I think in some cases the companies are actually projecting less revenue than I expect them to have. And that's because I think that the way that traditional finance analysis works is you're not allowed to price in transformative AI capabilities because that's unprecedented or weird or bad for financial reporting. So I think that I'm generally even more bullish than these companies official projections because. And I think that's like historically been the case where companies have like Anthropic and OpenAI have like constantly beat their, their projections. And then of course you can also just try to like look at kind of some of the fundamentals, like what do we think they're on pace to automate? And like how, how much value do you think that has been? But with like anthropic already posting $30 billion in ARR. The kind of like where you take the most recent month's revenue and multiply it by 12 to kind of create this like projected annual revenue. Like they're already kind of outpacing some of the greatest companies of all time, such as Google or Uber in terms of like revenue growth.

[B]

Yeah, that's pretty wild. All right, so that's the bubble. We can set that aside I think, because maybe that narrative is not as popular as it once was, given the increasing capabilities we've seen recently. What about AI as a normal technology? That's also sort of a common framework for thinking about AI. Maybe, maybe we should explain what we mean when we ask whether AI is a normal technology. It means basically that it's going to. Saying that AI is a normal technology is compatible with saying that AI is going to have large effects in the world. We might see AI as sort of the next Internet. And so it's, it might be a technology that keeps growth rates in the economy at large at maybe 2 or 3%, but not beyond that. It's not going to replace human workers or push growth rates above 10% or anything like that. I know this is a very broad question and forecasters don't love these broad questions, but how do you think about whether AI is a normal technology?

[A]

Yeah, I think that there's definitely some incredibly abnormal things about AI. I think the biggest thing for me is just the agency, like normally when you have a technology like say like a hammer or a toaster, it just does what you tell it to do. And there's like no chance that like if you put in the toast in the toaster. That it will like, end up creating some like, completely different thing or end up like disobeying you and going off and doing something else. Like, we understand how toasters work and they have like a very fundamental way of like bread in, toast out. But when it comes to these AI systems, they're not being built like typical software. They're not being programmed line by line. But instead what you have is just these machine learning algorithms learning new ways of doing things, new patterns just based on ingesting tons and tons of data and solving tons and tons of problems and learning not tremendously different from how a human would learn. So like, this means you can get new AI systems like Claude's latest mythos model, that just because it's bigger and trained on more data, it like, just becomes better at software engineering in a way that like, no one really fully understands. And then it just becomes a lot better, not just at software, but also at finding software vulnerabilities and engaging in hacking. It's kind of like if you built a bigger toaster and then all of a sudden your toaster could like hack the Internet in addition to making toast. Like, I just feel like that's just like a very, very abnormal thing. And like, if you're talking about AI systems that could eventually just like substitute for human labor or exceed human skill at like a wide variety of tasks, like, that's just going to be transformational in a way that like no other technology ever has been.

[B]

But it seems right now we have pretty amazing capabilities and not much economic impact in terms of unemployment numbers, in terms of gdp, in terms of productivity. Tell me if I'm wrong here, but it seems that sort of the classic economic indicators are lagging. We're not seeing massive impacts even though we have these models that are incredibly capable.

[A]

Yeah, I mean, I think it's still just very early. I think if you look at kind of just like the adoption of AI technology, I think if everyone in the entire planet were to immediately adopt like the latest models like GPT 5.4 and Cloud 4.6, sounds like, know how to use them really well, that you would see more, much more impact than you currently have. But like, kind of when I, when I walk around offices and just like, I'm. Sometimes I'm a nosy guy, I look at other people's computers and I've been seeing more people using ChatGPT now, but it's not been like, for building web apps and like the full range of capability. It's been more for like proofreading. Essays or writing first drafts. And so people just aren't using the full range of capabilities. But I think if you come back to me in two years or so when model capabilities have improved and existing models have diffused more into the broader economy, I think I would expect to see more economic effects on the employment side. I would expect to see more economic effects on productivity. I think that right now it's just kind of like too hard to see and all the, the noise because I guess like labor statistics are just kind of very noisy. It's like hard to get like a really fine grained signal. But I think the signal will become pretty clear over the next next two years. So maybe we'll do a, a revisit podcast and we can see like, I guess two years from now if there's still, still no effect. I think then I'm, I'm pretty clearly mistaken about some of this.

[B]

How large would you expect these effects to be from our standpoint today? Of course.

[A]

Yeah. So I mean, right now I think that they're so tiny as to not be noticeable in the statistics. I think that a year or two they would start to be noticeable. I had tried to kind of look at how AI systems are substituting for being able to do remote tasks and then like kind of accounting for a lag in diffusion and then trying to estimate the employment effects and kind of like when I was doing this modeling I was seeing that like right now the expected employment effects like just kind of are indistinguishable from noise. But that when you wind to this time next year, I would think that you could have like a extra percentage point of unemployment attributable to AI. And so I think that that type of thing should be noticeable about a year from now.

[B]

Yeah, and that's, that's actually a pretty big effect, if I'm not mistaken. That's, that's not sort of 1% doesn't sound like a lot, but it is quite a big effect, right? Yeah.

[A]

I mean if you think about how many workers there are in the US economy times 1%, like that, that's like a large number of workers.

[B]

Yeah. So I mean a lot of people use these tools. ChatGPT has something like a billion weekly or monthly active users. I think there are other companies out there with, with a lot of users. Also Google's models are used a lot. It seems like the models have diffused. But your theory here or your hypothesis is that people just aren't very good at using the models or they haven't sort of figured out what the models can do yet?

[A]

Yeah, I mean, I think when you dig into the numbers, there was like some, some recent analysis on this from Epoch AI and they found that kind of like when they surveyed people, only half of employed Americans had been using AI tools at all. So there's still a full half of people that are just like, not even using AI. And then when you also dig into like, what AI are they using? They're not using CLAUDE code, they're not using GPT 5.4 Pro. They're like, more like using Microsoft Copilot or like ChatGPT instant free tier. And so they're not getting the latest and greatest capabilities. They're kind of getting a pretty sizable lag from just kind of either not using the latest models or using the models at kind of like a much smaller scale than is currently available. I would say, like, if you're someone who's using CLAUDE code, you're kind of in like already in the top, like 0.1% of AI adopters. And then if you're able to use cloud code really well, then you're just like so much further ahead. But like, eventually kind of when these AI more authentic capabilities can diffuse into the workplace, I think there would just be more clearer substituting of human labor. But it is going to take some integration work, definitely. And that's kind of what creates some of this lag.

[B]

What about economic growth as one of these sort of metrics that are difficult to push up? When would you expect to see AI beginning to affect that number?

[A]

Yeah, I mean, I would still look in a year or two. I think that you would see kind of like higher productivity per worker as a result of AI augmentation and that that would lead to economic effects. I don't think that you would get some of these, like, bonkers. Economic growth of like 10% plus, I think is still. I think it's very feasible if you can like automate large swaths of the economy with like this like artificial general intelligence or something that can just do a lot of expert tasks. Like, we're still a little bit away from that. So I don't think you would see very large economic effects for another decade or so. But you should start to see some of the early signs like within, within the year.

[B]

You mentioned clogged Mythos. Had you sort of, had you already factored that into your predictions? Did it surprise you in, in terms of its capabilities?

[A]

I mean, I think that if you look at like some of these capabilities, like they do seem to Be kind of on the trends that have already been established, which I mean, I guess when some people say like, oh, it's on trend, it sounds like, oh, there's like nothing interesting or nothing to be to feel crazy about. But like these trends are already so like ginormously fast. Like you have like them deviling in capability like every four months or less. And so like the existing trends are just already so crazy that just continuing on these existing trends should suggest like very crazy capabilities coming in the next year or so. I think like yeah, Claude, what I was kind of expecting as a result that we would get some sort of AI system that had very strong cyber capabilities that might be too dangerous for public release. But I guess I was still a little surprised that it happened so soon. Like I think maybe I wasn't expecting that the current trend would result in like superhuman vulnerability discovery happening this early on.

[B]

And it seems like Anthropic, at least in the way they're talking about it, was sort of. They were surprised themselves around how early this arrived and they had to. That sort of explains their response to it.

[A]

Yeah, I mean basically you have this AI system that's like just found tens of thousands of previously undiscovered vulnerabilities, many in like well tested and hardened AI systems, many of which like professional human vulnerability discoverers like tried to find vulnerabilities and failed. And like many of these vulnerabilities are like high, high severity. And like this has also just been validated by the UK government. It's been validated by some of the partners of Anthropic, like including their competitors in Google and Microsoft. Like I guess some people have thought that, oh, maybe Anthropic is holding back this model because of marketing buzz or it's cool to have a model that might seem dangerous, but I think Anthropic is just very serious about this and I think there's just very clear and compelling evidence that this AI system is indeed exceeding human professionals in vulnerability discovery. And I think this is just also points that AI capabilities are coming very quickly, that AIs are now just very clearly a matter of national security. And I also think that Mythos will just be the beginning. I think this will only get more and more of a national security concern as AI companies build towards so called AI superintelligence.

[B]

Yeah, and it seems like cyber is a natural place for these very advanced capabilities to show up first just because the models are best at coding and sort of coding work is similar to discovering these vulnerabilities. And so, in some sense, you could have predicted this perhaps, but it's still sort of wild to see it actually happening.

[A]

Yeah, it definitely hits different when it actually occurs in real life, and it's no longer a matter of theory. And I think that some people did indeed predict that this would be the case. And these people that successfully predicted vulnerability, discovery, super AI are also predicting that AI won't stop there, that it will just keep going until it becomes AI superintelligence, which is this, like, AI system that just, like, greatly exceeds human experts in all domains, including national security, domains such as military strategy, such as weapon design and deployment, such as cyber attacks, such as bioweapons. Like, I think that kind of, like, bigger AI is really coming soon, and that Mythos is, like, while it's very capable, it's also like a signpost of what's to come and a kind of a credible down payment on some of these predictions.

[B]

So one thing about Mythos and one thing about many of the at most advanced models is that they have longer time horizons than we've seen before. There is this organization called Meter. I've discussed this before on the show, and they measure time horizons for the best models we have. Basically, you have a bunch of humans complete tasks, and then you measure the time it takes for a human to complete a task. And then you see whether the AI systems can complete these tasks or with what reliability they can complete these tasks. And then we see that, okay, the most advanced systems we have right now can do tasks that would take a human on the order of 12 hours. But this is moving very fast. And so, honestly, this will be different in two months. How do we think about predicting this? Specifically, how do we think about where time horizons will be at the end of the year, say?

[A]

Yeah. So I think one of the dumbest ways of predicting this has actually been very successful, which is just that you plot all these dots on a graph of, like, this model had this time horizon. This model had that time horizon. And then you just draw a straight line of best fit through those dots. And when you look at that, you see that the models are doubling in their time horizon every approximately 144 days. So that's like a little over three months, like, three to four months. And many people have thought, like, oh, this doubling rate, like, that's, like, crazy high. Like, surely it'll stop at some point, right? But, like, we were able to draw these lines, like, last year, and this. This line kind of held up and predicted all the model releases over the past Year. It even kind of actually clocks like Cloud 4.6 opus quite well. And it probably clocks Mythos well as well. And the crazy thing though is that if you continue that line throughout this year, you end up with like time horizons that are approximately 90 hours, maybe even 100 hours of human labor being able to be done by an AI in one sitting without human assistance. And that's just like a very, very high number. I don't even think meter will be able to measure a number that high just because they will eventually run out of tasks that they can assign these AIs that have these like really long horizons. You can also look at another measure I like, called the Remote Labor Index that is run by the center for AI Safety and Scale AI. They're measuring kind of like what percentage of freelance remote tasks AIs can do. And currently Opus 4.6 can do like 4% of these tasks. But if you plot the same kind of rate, you end up being able to do maybe 20% of these remote tasks by the end of this year and over half of the tasks by the end of 2027. So like kind of all these measures are fitting pretty well and predicting kind of like, yeah, much more powerful AI coming within the next year or two.

[B]

Yeah, that is pretty crazy. I mean, so the trend has held in the past, and this then gives us some sort of trust in that trend. And now we're extrapolating into the future. At some point it must bend, right? At some point it must sort of stop working. Otherwise we would land in predictions where these models can do hundreds of years of work or have a time horizon of 100 years or something. I mean, so when you extrapolate out and you see that sort of 100 year time horizon, do you think that the measure has, has. We're not measuring the, the correct thing here or how do you think about that?

[A]

Yeah, I mean, I don't even know really what it means to have a 100 year time horizon. Or like.

[B]

Yeah, that's what I mean.

[A]

Yeah. What is my personal time horizon as Peter. Like, I, I don't really have a, a good sense. Like, do I even, like, I won't even probably. I don't even know if I'll live a hundred years, let alone doing a hundred year long task. So it's like kind of like really unclear to me sort of what this even means when you get to that point. But I would say, like, I guess I've always sort of been a skeptical guy. I've kind of like tried to call the top on AI a few times over the past few years, lots of other people have been like, oh, AI is hitting a wall. Like, oh, GPT5 means that scaling's over. And like, all these people who have said AI has been hitting a wall have been just, like, very clearly wrong. Like, AI progress just kind of continues to march more and more forward. And I think Cloud Mythos is like, the clearest demonstration of that. And so, like, I think at this point, I've kind of given up on trying to call the top of AI and I think just, you just kind of have to ride out the line and see where it goes and just be very prepared for, like, human, if not superhuman capabilities coming soon. I feel like another. Another good example might be be AI chess models. Like, if you plot AI chess performance versus human performance, like chess AI never really, like, had a bent curve at any point. Like, it just kind of approached human level, eventually beat Gary Kasparov, eventually just became better than humans, and then became increasingly more and more superhuman. And it hasn't really stopped advancing. So, like, I think that does suggest that, like, AI systems can just cross human level and then just keep going on and on without really even a good sense of, like, where they could end up or, like, what it even means to, like, outscore humans by that much.

[B]

I think they even have championships now between different chess AIs, whether they have Elo scores of 4000 or something, which is like far, far beyond the human range. And so, yeah, this is evidence of AI becoming superhuman in a narrow domain, though. But I totally take that point. So you think, is it fair to say that perhaps this sort of the human level is something that we care a lot about because we're human. It's not really sort of a deep fact of nature that the models can't just run through and become better than us.

[A]

Yeah, I mean, I think that the AI models can just exceed human skill and potentially by like, a large margin at some of these tasks that we care about. I think Claude Mythos is already showing AI systems that can exceed human skill at vulnerability Discovery. I guess whether that still doesn't necessarily translate into the ability to do, like, cyber attacks, but, like, I think it does show AI becoming more capable at that as well. And I guess, obviously cyber attacks is a type of thing that we care about very deeply that affects the balance of geopolitics. And an AI being superhuman at that definitely just affects geopolitics and is a matter of national security. I think there's going to be Other domains too, where AI will just exceed human experts in domains that matter a lot for, for geopolitics especially also as AI builds out agency and can do more of its own thing on its own time frame, I think that we could potentially just have AI systems that are geopolitical actors in their own right if we do have these super intelligent AI systems. So I think it's definitely just kind of a lot to take in and a lot that we need to prepare

[B]

for, say more about these domains. So there's cyber, where AI could become geopolitically relevant. Which other domains would you be worried about?

[A]

Yeah, I mean, I think that there's a lot of different domains and it may be kind of hard to predict in advance. I guess there's like a sense that maybe bi superintelligence would mean some kind of nerdy intelligence. Like you're like a high IQ person, but that doesn't necessarily mean that you're going to be dangerous or able to take over the world. But I think in the case of super intelligence, it's not just like nerd intelligence, it's also just like broad and deep capabilities. So it could be with regard to military strategy or with regard to creating novel weapons, or with regard to kind of like doing things that we don't even really expect and know to prepare for. So like it might not even be domains that we're really expecting or can articulate in advance. Like I think that's kind of the, the potential of AI superintelligence.

[B]

Yeah. Is that an argument for more tightly integrating AI into our militaries, learning how to use it, or is it an argument for keeping AI out of our militaries because it's not controllable or it's difficult to know where things will end up if we do that?

[A]

Yeah, I mean, I think ultimately we need both. I'm definitely strongly in favor of integrating AI into our military. I think that that's kind of what's necessary to keep ahead of our adversaries and make sure that we have a strong and powerful military to kind of lead the free world. But I think when it comes down to it, there's a few important things to keep in mind. I think, like if we also need to be able to deploy the AI technology in a way that is under control and responsible. Especially like today's AI systems are built for commercial use, they're not built to be military grade. So we have to understand that like frontier AI systems may not be reliable enough for these like mission critical use cases and Also, they may not be sufficiently hardened against adversarial sabotage. Like, if China or Russia could tamper with our AI systems or steal the model weights for our AI systems, like, that could really affect our advantage there. And then I think thirdly, just like the prospect of superintelligence does really loom large here, like, if we're. If the government's not really getting its head around, like, the national security implications of these AI systems and know how they could be used against us, know how we could be deploying them against our adversaries, and also know how AI might be like an independent threat, threat actor. Like, I think it's just very important that the US government is taking kind of a leading role here and making sure that we have a way to control these AI systems and do it safely, do it responsibly, and do it in a way that's like, secure from. From our adversaries as well.

[B]

So right now we have a lot of progress in, we could say, the cognitive domain with AI. We. We have these models that can do well on tasks that you do on a computer. It seems that robotics is lagging behind and that robotics is necessary for AI to have truly transformative impacts on the world. So where are we with robotics? What's your sense of that? And where will we be? Will there be perhaps sort of a similar moment to the transformer or LLMs in robotics? Yeah. What do you think?

[A]

Yeah, I'd say two things. First, I think I would fight the premise. I don't think that robotics is strictly speaking necessary for transformative AI technology. I think you can do an awful lot over the Internet. You can also do an awful lot if you can direct humans to kind of be your hands and legs in the real world. And you can definitely just do a lot if you can successfully cyber attack a lot of critical infrastructure or military infrastructure. And so I think kind of.

[B]

Yeah, yeah, sorry. I totally take that point. If you have a system that's unaligned, for example, or misaligned, it can definitely do damage to the world without also having an army of robots. What I'm thinking about here is if we are to get to a world where we have insanely high growth rates or where we sort of transform the world in the way that the Industrial Revolution transformed the world, that will require robotics to do all of the physical labor to build out all of the infrastructure and so on. So there's definitely danger without robotics. But my point was that robotics is probably necessary for truly transformative changes.

[A]

Yeah, I guess, again, to Sort of argue with the premise. I think that there's a lot you can do with the remote economy too. Especially if you could just develop better and more efficient industrial plans or more efficient economic designs. You could then maybe like leverage physical industry more efficiently even without having direct robotics. But yeah, I agree that like having really high skill robotics would be pretty important for like turbocharging manufacturing and like building out a lot of like what could be tremendous economic growth. I think that robotics is definitely lagging. It's still kind of in like an early phase trying to acquire training data and trying to build the AI systems. I think miniaturization is also a big deal if you have like, normally these AI systems take an awful lot of computers, but if you have to fill, fit it on a smaller robot machine for latency purposes, I think that's more difficult also just like having all the sensor data like human hands can just detect so many more things, can be manipulated in so many different ways that we're still approximating with robotics. But I do kind of expect there to be sort of a, a big breakthrough moment in robotics and I, I don't think that the robotic build out will really meaningfully slow down like what may, may be possible. Especially also if as I mentioned, AI systems themselves could be meaningfully helpful in like designing robotics. You could potentially get some sort of like feedback loop that way.

[B]

Do you think there will be, do you think we can use the models we have now to further our progress in robotics? Do you think there'll be transfer learning from the digital to the physical domain?

[A]

Yeah, I think so. I mean I think that AI systems already are being deployed in these robotics context and I think that they do have just like pretty good world understanding and know kind of how objects relate to each other and like what each object is and what it means. I think that the difficulty we're still encountering is I think just like how do you move through the physical world? Like how do you actually position your arms in order to effectuate things? Kind of like the so called muscle memory I think is something that is still taking a while in bottlenecking AI systems. And that might just be a matter of sufficient data collection and then sufficient miniaturized processing to kind of be able to do that without needing to take several seconds each action to process like what's going on.

[B]

Yeah, it seems like the models right now are quite capable of understanding a scene. You can just, you can take a picture of your kitchen with your phone and then ask, ask an image model or any Model, basically, how would you. How would you navigate this kitchen to make a cup of coffee and that it can sort of list out how it would do that. But I think you're right that the actual challenge is then in sort of actually navigating as the robot through the kitchen where we are not. We are not in the end zone yet. Yeah, yeah. Do you, but, so you don't. You don't believe that robotics will be necessary for,

[A]

For.

[B]

No. Tell me about the importance of robotics for, for transforming the economy. Where's the role there? What can be. Which economic effects could we see if. If we don't have advanced robotics yet?

[A]

Yeah, I mean, so, like, I guess on a very basic level, the economy is just like a series of interconnected goods and services. And many of these services can be done remotely, and many of these services cannot. And then so obviously, for all the services that can't be done remotely that you want AI to be able to do, you would need robots to do them. And for all the goods and manufacturing that you require, like, physical processes to create and distribute, you would also need robots for that. If you did want, like, say, a fully automated economy. I think that, like, some of these manufacturing robots, I guess, like, a lot of manufacturing already is automated in some ways, and you can have more customers built robots for that. But, like, insofar as, like, a lot of the economy is not just done from your computer, you would definitely need robots to fill in all those aspects. I think part of the reason why this would actually be turbocharging as opposed to just like, recreating the current speed, but with robots is that you could have, like, many orders of magnitude more robot labor than you could have human labor, that these robots would be many orders of magnitude cheaper to run than human labor. So that you could just do things a lot more cheaply and pass the savings on to consumers. And that also you could just have AI systems that could physically move a lot faster and maybe a lot more intelligently than humans if you did build such systems. And I think that's kind of what could really lead to runaway economic growth. Of course, it's, like, hard to escape the fact that in this situation, you also would not really have any meaningful human employment that would be kind of like humans, that, assuming everything goes well, humans would be very wealthy, but kind of wouldn't have anything in particular to contribute to the economy themselves. Kind of similar to how chess grandmasters don't contribute much in these kind of like AI tournaments, because the AIs just kind of do everything they can do and more.

[B]

Yeah. How should we think about human wages in this transition then? It would seem to me that human wages would go up as AIs substitute different forms or as AIs begin to replace different tasks in different jobs and where humans become more valuable. Because we are the bottleneck. Say you need human hands to do something in the world. Maybe companies would be willing to pay people a lot to sort of use their hands to work in the world. And so it could seem to me that the wages could increase and then fall off a cliff as we get full automation. Is that a plausible picture?

[A]

I think it's a plausible picture. I think ultimately it's hard to really forecast because the economy is just very complex. And so like a lot of these kind of like fine grained effects are just going to be very inherently speculative. I do think that if you're in the position to be an important bottleneck for like a multi million dollar process, that you could then kind of really command a very large, a very large compensation as a result, at least until you're no longer the bottleneck because you end up getting automated yourself. So I feel like it would be inherently temporary. I think a lot of it also will come down to government response, kind of like, yeah, how, how do we as a society want to organize around very capable AI systems? Like are there parts of the economy that we don't want to outsource to robots? Like would you kind of prefer that like, say like childcare or elder care or school teaching is done by humans? Even if robots could do it just fine? Like would you want some sort of jobs program? I guess like you also could theoretically have a government that just like has the capability to automate a lot of AI, but chooses not to because it's just like too fundamental to the human condition. I think there's just, or maybe we just have a lot of redistribution of AI wealth. I think it's kind of like really unclear I suppose, like how the human population will like respond to essentially being obsoleted by AI technology. And I expect that's not something that people are going to take, take lightly.

[B]

Yeah, I think you're right. I think this, the social response will play a big role in how AI is ultimately adopted here. I think there will be lots of sort of calls for protecting various industries and from those industries themselves also. I think there will be sort of political debates around which jobs we can't allow AIs to do and so on. But there will be, on the other side of that, there will also be this enormous economic incentives, incentive for governments and companies to implement AI in order to save time, to save money, in order to compete with other companies and other governments. Do you think this is now about the broadest question I could ask, but how do you think those two different effects play out against each other?

[A]

Oh boy. Yeah, I mean, I think I'm one of the world's best forecasters and I really just don't have an answer for you. And I think anyone who thinks they can forecast that is probably not really thinking very clearly. I think it's just like, yeah, one of the most complex interconnected dynamics you could ever possibly try to forecast. So sorry to not have a good answer for you.

[B]

Yeah. Even with that being said, I'll take a tiny stab at it and say that historically it seems to that governments have sort of allowed industries to evolve and we have seen a lot of creative destruction and job displacement. But the trade off there for the citizens have been that we've also seen in sort of, you could call them advanced democracies that spending on social welfare replace a is a larger and larger fraction of gdp. So you see like replacement of tasks and jobs, but then also more spending on social welfare. And maybe that trend just continues. But as you say, it's very difficult to speculate about.

[A]

Yeah, I mean, I think generally kind of like every time we've had technological developments it has resulted in shifting jobs. But there's definitely been some structural unemployment where people are disadvantaged by that shift and end up unemployed potentially for large periods of time. But like others can advantage from that. And like generally speaking, the total level of unemployment kind of doesn't really exceed say like 10% or so, except during very severe recessions. So a lot of people who are looking for work are still able to find work and things end up kind of sorting themselves out over time. I think that what makes this very different though is like we're talking about potential AI systems that don't just like substitute for some forms of work, but actually substitute for all forms of work such that a human couldn't necessarily find a different job because the AI would be able to do that job too. And so every time you're kind of looking for the next thing to do, you're kind of outfoxed and out competed again. And so this could potentially yield just incredibly high unemployment rates, like unsustainably high. And like in that kind of situation either we would have to just like essentially pivot to a completely different economic model where there's just like Tons and tons of essentially welfare, or we just kind of have like a system where people are no longer generally expected to work. Maybe like everyone is kind of like a retired person or like a rich person who's just living off their investments and not really working particularly hard. And like, that could be a society that we end up accepting and it could be a society that people end up quite enjoying. I don't think there's anything like inherently terrible about it. But like, I think also a lot of people do find their personal meaning through work and I think would not be excited about this kind of society. And I think that's like a very important democratic question that we're going to have to answer collectively have, based on where we might be heading. And I think it will come at us like much faster than the Industrial Revolution did. I think in the Industrial Revolution there was definitely a lot of change, but it kind of happened through multiple lifetimes. And so there was way more time to adapt and pivot and there were other things you could adapt and pivot into. But I think with the AI revolution, it will happen all kind of really fast within one bewildering lifetime and there won't really be anything you can pivot into because by the time you find something to pivot into, the AI will be there as well.

[B]

Yeah, this all sounds pretty plausible to me. So we say that when my office job or your office job or whatever office job is automated, we then have to retrain as a plumber. But it takes four years to become a plumber. And in those four years, robotics have advanced such that robots are now better plumbers. So when I interview economists, they seem to believe that unemployment will never sort of go out of bounds except in, in these economic downturns. And, and then it'll return to this, say under 5% or something. And, and this is, this seems quite plausible when you look at economic history. This is how, how it has been as you describe also. But it, it seems to me that maybe economists have sort of overlearned that lesson and now believe that nothing could ever push unemployment, be sort of permanent unemployment caused by technological change. And that, that I think is, is, isn't true, actually. I think if you look at, so the move from agriculture to factories to offices, there's always been an alternative job for, for people to take. And now it seems that maybe when AIs can, can do all the jobs there, there won't be any jobs. And, and the, the respond that the response I, I tend to new jobs will arise. These jobs will be. We can't predict them in advance. And these jobs might seem super weird to you right now. And so maybe people earn time, earn, earn wages in the future as professional party goers or sort of streamers or life coaches or all this sort of. Yeah, jobs we can't even imagine. How do you, how do you, how do. Yeah, you've just described where you sort of land on that. But what is your response to the traditional economic perspective here?

[A]

Yeah, I mean, I think the traditional economic perspective, like you said, is just trying too hard to shoehorn the past into the future and kind of not really accepting like what it means to have general purpose AI systems that can just do every single human task, including going to parties or streaming and potentially quite compellingly. I mean, I do think that there might just be fundamental human preference jobs where like I said, kind of like even if a robot could do the job just as well, if not better, you still kind of just prefer a human to be in that role. And maybe that is like elder care or child care. Or maybe you still do want a human doctor to deliver you the diagnosis, even if the diagnosis was treated entirely by an AI system. And maybe you do just prefer a human streamer or a human party goer to an AI system, even if the AI system is also very charming and creative. But like, I am just like, just very skeptical that this would be enough to employ the entire human population. Like how many parties do you, could you possibly be having such that like you can have everyone employed as a human partygoer? Like I think I just don't see how this like gets you more than like 10% total employment or labor participation or something. And it does just sort of seem like a failure of imagination to kind of like, like economic models have never had to contend with some alternative population that could just do all the jobs that humans could do. So it does just suggest that really strange employment effects may be possible. And I think this just all gets back to AI not really being a normal technology specifically because of the agency. Like these AI systems are just kind of able to do their own thing, do kind of lots of things, and especially in the future, just like do them very creatively and very competently. And it doesn't just leave a lot of room for humans to do things as much if they're not going to get out competed.

[B]

Yeah, I think the prediction that we want to see high unemployment, sort of permanent technological unemployment, that instinct comes from, from, from a good place actually. It comes from like you don't want to Deceive yourself into thinking that something is a bigger deal than it, than it actually is. We've seen there's been plenty of sort of predictions and hyped technologies in the past and that hadn't really moved these standard economic metrics out of their normal ranges. And so I think you would be sympathetic to thinking that sort of things will be as they have been in the past as like a good methodology for prediction. Right. But of course, sometimes things really are different and it's important to, to be aware of that.

[A]

Yeah, I mean, I think the hallmark of a really good forecaster is kind of knowing when things will be like the past and when there's serious risk of things not being like the past. There's certainly been lots of trend breaking things that have happened all the time and lots of things that have happened that were not like the past. And so like kind of breaking traditional models is certainly a thing that can happen. And I think the true wisdom of a forecaster is kind of knowing when that's plausible and when that's not.

[B]

Yeah. You have a post where you describe your forecasting methodology as sort of half jokingly, I think, as like nothing ever happens. That's like your main principle. Maybe you can explain that.

[A]

Yeah, I mean, I think that comes down to exactly what you said, which is that like, you know, in many, many cases the future does look like the past. And in many, many cases, like people speculate about future events that end up just being like tremendously out of the norm and tremendously unlikely. And I think it's just very easy as a forecaster to like apply these base rates and apply these historical trends. And I think in many cases you will end up just being right just because there's like so many speculative things that people might speculate on and like a lot of these things end up not happening. But like at the same time, like if you always thought that nothing ever happened and you applied that in 100% of cases, like, you would just miss out on some of these really important trends. Like, I guess, like, I feel like Covid is maybe a really strong example. I guess Covid obviously did have historical analogs such as the Spanish influenza in the early 1900s. But like, I guess like Covid definitely looked like something very out of the ordinary and I think it was very easy to be skeptical of it by applying kind of like a nothing ever happens principle, but you would have been very wrong in that case. And I think kind of like economic modeling of AI that like heavily relies on the past like also just kind of can't really capture this trend of where things are going.

[B]

Yeah, it seems like maybe it's a bit of a risky approach if you are worried about downside risks. Right. If you think the world is, nothing ever happens, the world is going to stay relatively normal, then you are right 19 out of 20 times. But the one time you're wrong is the thing that really matters. Like Covid, or I would say at this point, it's fair to say that AI has also surprised a lot of people and AI is sort of out of the ordinary thing. The economist Brian Kaplan has this, has been betting on his beliefs for a couple of decades now, and he has an amazing betting record. And the one bet that it seems like he's now quite likely to lose is on AI is on whether AI can get a high grade in his economics exam. And so maybe that tells you something like that. By, by betting on base rates, you are very often right, but when you are wrong, you're sort of wrong on, on an important, on an important question.

[A]

Yeah, I think that's, I think that's definitely true. And I think that AI has really kind of surprised a lot of people, including myself, for kind of exactly that reason.

[B]

Yeah. What. So again with your methodology, which sources of information do you, do you tend to rely on, where do you start, when, when making your forecasts? I know you also have sort of a roster of experts that you consult and you don't have to tell us who those experts are. They are like a source of your sort of edge in these forecasting events. But in general, where do you start? Do you start with like data sets or news or rumors in San Francisco or what do you implement?

[A]

Yeah, I mean, I think it kind of depends a lot on what you're forecasting. And I think some of the part of being a good forecaster is kind of knowing what sources are good to trust on what sorts of issues and what sources would lead you astray. I think when it comes to predicting future AI advances, though, it's pretty hard to just beat the like, straight lines on the benchmark approach. I think that has kind of like steered correctly way more than kind of like any other source of information out there. And if the benchmarks start skewing in another direction, I think you'll, you'll see that kind of earlier than anyone else just by kind of looking at those dots and trying to draw lines through them. The rest kind of is just sort of commentary. That being said, when it comes to commentary, I'm Still a pretty big fan of Twitter. I kind of maintain these Twitter lists of people I like to follow and kind of read what they have to say every day. And I think that's, like, also been pretty helpful at forming my views.

[B]

And are you then sort of rigorous about measuring who is actually trustworthy, who is actually making good predictions, and who you should actually listen to?

[A]

Yeah, definitely. I think in a lot of cases, when I follow somebody and start to kind of consume, first of all, I try to follow people who kind of make testable predictions as opposed to just kind of spouting off opinions. And then I try to kind of measure, like, whether I disagree with them, whether I agree with them, and, like, whether they end up being right. And a lot of times I've, like, unfollowed people because they end up just kind of being wrong or unhelpful. In some cases, like, someone's still helpful, but I know to discount them because they end up kind of tending to overstate things. But, like, I can mentally adjust for that. In other cases, I think someone's just, like, very valuable because they end up, like, having insights earlier than I do, and they end up being kind of a reliable source of that.

[B]

Yeah. What about the statements from the CEOs and employees of these companies? How do you weigh those? There's a lot of talk around whether this is sort of marketing hype, which I don't think is the right frame for understanding this, but again, it might not be smart to take them at face value. On the other hand, it seems like you could have done pretty well predicting AI if you had listened to Dario and Altman and Hasabis over the past five years, say.

[A]

Yeah, I mean, they've definitely all been bullish about AI, and they've been very right about that. I think that their past track record means that we should take some of their future statements about super intelligence and about the risks and, like, take that very seriously. But that being said, they do definitely have an incentive to over hype. And I think I've definitely seeing kind of around some of these model releases. They end up getting hyped way more than they actually pan out just because the hype is off the charts. And I still think that at the end of the day, just drawing lines on benchmarks ends up, I think, doing as well, if not better at kind of more reliably steering you as to what to expect than listening to what CEOs have to say.

[B]

When we think about timelines, there's been a massive drop over the past five years, say, and it definitely over the past 10 years. If you asked people in 2016, when we would see AGI or superintelligence or human level AI, maybe they would say 40, 50 years or something. Some would say 100 years, some would say never. And now it seems more people have timelines in the order of five or 10 or 20 years somewhere in that range. Taken from a sort of. Taken a very broad view on this, it seems like that we could pay attention to the direction of change in timelines and say that, okay, maybe we don't know whether it's two years or seven years or 15 years, but what we can say is that almost everyone agrees that the time to AGI is now much shorter than it was just five years ago. Is that a smart way to think of things? If you want to have a sort of a broad overview?

[A]

Yeah, I think that is a smart way of thinking about things. I mean, I think it's also just, like, very pragmatic. Like, basically, like, what would you do differently if AGI were 3 years out as opposed to 10 years out? I think it kind of has very similar implications, and I think that that kind of means that maybe the minute differences between those views are not really worth obsessing over. So I think the fundamentals are kind of still. Still the same in terms of, like, what it means for society. And, like, it is kind of happening quickly. Whether it's happening super fast or merely fast is, like, I think, kind of not really important. I also think that, like, AI, like, there. There aren't really these, like, concrete milestones where everything is, like, totally normal, and then you have, like, AGI or you have super intelligence, and all of a sudden, like some random Tuesday, everything just, like, completely falls apart. Like, I'm expecting this to be far more, I mean, I would say gradual and continuous, though. Gradual and continuous also sounds sustainable. And slow, I think, like, incredibly fast, but also incredibly continuous, insofar as that makes sense. But basically, that being said, that I think, like, AI with Claude Mythos, I think, is already pretty crazy. And that's like an AI that currently exists. And I think in a year from now, I don't think we'll have, like, AI that can do absolutely everything a human can do, but I think we'll have AI that can definitely do very important things that are very relevant for national security and very relevant for society. And that made Cloud Mythos look kind of really modest and weak by comparison. And I think that will be really important to attend to regardless of whether it meets this definitional milestone.

[B]

Yeah, I've been having some of the same thoughts around this that it's definitely not the case that it sort of happens in an instant or you're going to get AGI on 27th March 2028 or something like that. But it's also, as you, as you mentioned, is also somewhat misleading to think of it as this gradual process. If we zoom out and we are in 2100, we will probably be able to say that within these two or three years, that is when we really got highly advanced models and it will seem concentrated from that perspective would be my best bet on the question of whether it changes anything how far AI or AGI is actually away from us, whether it's. It's three or seven years. I think it's mostly right that we should be working on the same things. We should probably, we should do alignment research. We should figure out what to, how to handle the economic impacts and so on. There's, there's one example, one counter example that I, that Toby Odd mentioned at one point, which is that export restrictions on chips to China will have, will sort of have the opposite effect that it's that the proponents of those restrictions are after. If, if it turns out that AI AGI is further away so China will build up their own industry for chips in the meantime, say if, if AI is 15 years away as opposed to 5 years away, does that make sense?

[A]

I mean, I, I don't agree with that. Take. I, I don't know what your audience's tolerance is for very long nerdy digressions on export controls, but it is something that, that's great. That is something that I, I care a lot about. I've definitely been like kind of a, a strong proponent of strong export controls on China. I think like when it comes down to it, I think again it's like not really about some milestone and when that milestone occurs, but it comes down to I think two things. I think first that AI is already super relevant to national security. I think if China was able to get all the compute they could ever want, like I think they might also have Claude mythos capabilities and then we would kind of be in a way worse cyber picture than having anthropic be able to kind of keep it under some semblance of lock and key, at least for now. And I think that will only matter more and more as AI kind of again continuously increases in how much it matters for national security. And then I would say secondly, I think that there is some sort of Sense that like, if we just gave China all the compute they could ever want, that this would like somehow slow them down in the long run. And I don't really understand how that would be the case. Like, I think China very clearly wants to build their own domestic compute capability. I think they would be building that out regardless of how much export controls there are, just because they really value self sufficiency. And I think that they're already kind of driving hard at this at the maximum speed they can sustain. And I think that in some sense they want to have their cake and eat it too. Where if they can bring in American chips, but also at the same time drive hard to build Chinese chips, then they're kind of in the best possible position. I mean, basically like China, they're not like a capitalist country. You can't just like bring in American chips and displace Chinese markets because the Chinese government would just like use regulations to force companies to buy domestic chips. Like they can do that and they in fact do in fact do that where there's basically like, okay, you can use American chips, but you also have to use all the Chinese chips as well. And then they can just kind of force the creation of a market. And so in that sense, I think given that strategic picture, it just doesn't really make sense to give China the chips, which is like the one thing they don't have and the one thing that's kind of stopping them from also having very powerful AI systems.

[B]

Okay, I will let the, if listeners scroll back in the feed. I've had guests on with the opposite perspective, but I'll let that stand. Interesting.

[A]

Yeah, please. Yeah, I mean, happy to debate with any of these other guests. Like, yeah, let them, let them know.

[B]

Yeah, maybe we could set that up. Actually, I want to know which sort of, which questions that we can do forecasts on are most policy relevant. So what are the most interesting unanswered questions from the policy perspective?

[A]

Yeah, I think that the policy perspective is not really reacting to very discreet forecasts. I think it's more just like a general vibe shift around. Is this AI thing a big deal and how big of a deal is it going to be? And I guess you have people out there, including myself, who are saying crazy sounding things like that all human employment may kind of become obsolete. That you could have like some AI system that greatly exceeds human capability and like military strategy, weapon design, and all sorts of national security domains. That this would be a real challenge to the primacy of the United States and the government. Like how Are you supposed to run a government if, like, you have, like, private AI systems that can then just outmaneuver the government? Like, the implications are just kind of unprecedented and tremendous. And I think this kind of makes for a defining governance challenge. And I think that people in Congress, I think they're starting to become aware of this, but I think they're not really attending to it with the seriousness and urgency that if they took these views seriously, like, that they would have. And so I think kind of really, the fundamental forecasting question is just like, is this, like, super intelligence thing? Is it like, a real thing, like. Or is it kind of, like, all a bunch of kind of hooey that, like, doesn't end up panning out? Because actually, when you get, like, GPT7, the curve just flatlines there and, like, never improves? Like, I think that's kind of the most important thing.

[B]

And probably also, when is this thing coming? It must be a very difficult perspective to inhabit if you're a politician with many other concerns. Like, if you think AGI is coming by 2030, for example, that's probably not easy to balance or to sort of integrate into all of your existing commitments.

[A]

Yeah, I mean, I do have tremendous sympathy for the burdens that policymakers are under. For me, I can just focus on superintelligence all the time. But, like, for senators or members of Congress, they have these, like, very broad portfolios that cover every conceivable issue. And they have to cover, like, Veterans affairs, and they have to cover the war in Iran, and they have to cover Social Security and, like, every sort of other conceivable issue. They have to cover all of technology, not just AI. And so they don't get that much time to think about it. And I think Congress is just generally an inherently reactive institution. It tends not to be very proactive again, because there are just, like, all these demands on their. Their time and everything they have to do. And so, like, I think it really. Yeah, just do have a lot of sympathy for that. But, like, I think we also have seen, especially with COVID that you can just get tremendous policy response tremendously quickly in tremendously bipartisan ways when, like, a genuine crisis does kind of emerge. Like, you kind of had a Republican Congress and a Republican president kind of engaged in this, like, really huge kind of social welfare project. To make up for Covid, you had Republican state governments, Democratic state governments doing lockdowns, which was, I think, some, like, unprecedented thing. That definitely has like, just, like, very large implications for freedom, like a very big civil Liberties trade off. But it was just kind of done, done very quickly out of kind of the urgency. But so I think that means we can react very quickly. But like at the same time the level of preparation for Covid was very minimal. And then like we had Covid and then the level of preparation for the next bio situation I think is also just back to being incredibly minimal. It's almost like we learned nothing. And so I think that just does really show you what to expect from Congress. And I think that also applies to the AI issue as well.

[B]

Yeah, to be a bit pessimistic here, it seems that the response from governments in general, perhaps we're talking about the US government now, is that not much happens until the thing is actually staring you in the face, until sort of the effects of COVID or AGI is already being felt in an undeniable way. So people will wait until they can actually see it, until they believe it. I think right now it's still possible to say, for example, AI is. I think this is a wrong perspective, but I think it's possible to say, you know, AI doesn't really, it won't matter that much because it's not sort of, it's not being forced on everyone yet. But then perhaps you have this because it's difficult for governments to prepare because again there are so many other priorities. Everything is on fire all of the time. It has to be handled. We might not prepare and then maybe we get perhaps even an overreaction when, when AI or Covid is finally staring at us in the face. Is that something you're worried about?

[A]

Yeah, I think it's certainly possible. Yeah. I guess right now my primary concern has been that we would kind of underprepare and underreact. But I think it's certainly possible we could overreact as well. I think that there's definitely like you can kind of see like Bernie Sanders and Alexandria Ocasio Cortez that proposed banning all data centers. I think that would be a potential overreaction. I think sometimes like a lot of this like concerned populist energy can be like channeled in unproductive places. And it's like I think like you've seen this kind of historically with like nuclear energy. I think that's like just been very over regulated. I think I've also seen kind of like waymos something that I think is very safe and very good for humans to be able to have these AI driven cars kind of end up being stalled and opposed due to concerns that don't really make sense. And I think this could happen to AI as well. If we did have some way to deploy these tremendous capabilities in safe and reliable and controlled manners, there still might be unnecessary opposition to that. And I think that's something that we'll also need to take a close look at. But I think that's ultimately just like how the democratic process works and like how things work by design. So like I'll certainly be doing my best to kind of like really care and carefully target the correct reaction. But like, I mean ultimately it's like for society and for Congress to decide sort of like what, what the actual reaction is supposed to be.

[B]

Yeah, yeah. One thing I want to ask about also is how good is AI at forecasting at the moment?

[A]

Yeah, I mean I think AI has been getting very good at forecasting. I think it's kind of right now, if you like type of forecasting question into a chat bot, I think you would get an answer that is exceeds the skill of most humans. Basically any human that's not a professional forecaster. I guess I am flattered to say that it does not yet exceed my personal skill. But like I think there was like a head to head tournament in forecasting where there were like hundreds of participants and AI still got like. I ended up winning that, that tournament, got first place, but like AI I think ended up getting 11th place or something like that. So it ended up doing very well.

[B]

How many questions do you remember? Do you remember how many questions?

[A]

Yeah, There were like 35 questions in this tournament. It was run by the Astral Codex 10 Scott Alexander in partnership with Metaculus. Yeah, and there were, I think, yeah, there were ended up being 2,975 competitors. So it was like a very competitive tournament and I'm honored to have placed first, but also kind of eyeing my AI competitors not that far away on the leaderboard, which was definitely not something that was possible in even like one or two years ago. And I think also like when you think of the unit economics, like it's so much cheaper to get an AI forecast than to get like a, a human forecast. Like I'm very busy and very expensive, but like an AI system, you can get like a pretty good forecast for just like a penny or two. From a unit economics perspective, I think that's definitely something to, for professional forecasters to pay attention to. I think we're going to be out of work pretty soon.

[B]

So what would your forecast be there for AIs exceeding human capabilities at forecasting?

[A]

Yeah, I mean I Think with a lot of forecasts, it sort of depends on how you define the question. So I had kind of broken this down into two concepts. One being economic parity, which is just basically like, at what point would it just make more economic sense to outsource all your forecasting to AI instead of outsourcing it to professional humans just because the AI is so much cheaper and so much more available. And then this sense of true parity, which is like, okay, if you did spend a million dollars to get a top super forecasting team and they had access to all the best data and they were like working together full time, like, I feel like those are like two very different setups. I think that you could get this like so called economic parody within like a year or two. And then like true parody feels kind of more like five plus years away. Especially kind of like insofar as humans can like systematically access kind of private data sets or other expertise that's like harder for AI to do the same.

[B]

But isn't that a sort of mundane limit then or sort of limitation then? I'm guessing that isn't the real competition between AIs and humans given the same data. What do you mean by accessing private data sets? Is that like a substantial limitation to AIs?

[A]

I mean, yeah, I think data acquisition is actually like a pretty difficult thing for current AI systems to do. It's kind of like, I mean, I don't know if you've heard the joke that like, like there's this joke that like a lawyer is like, oh, I'll never be replaced by AI because AI can't go golfing with the judge. Like the sense that like what, what is truly their replace irreplaceable skill is like their human relationships less so their like professional skills. And I think that does kind of matter here as well, which is just like humans will still kind of maintain an edge for a while in their ability to kind of like know how to acquire data, including kind of from like really human to human interactions that might be difficult for AIs to replicate. I've kind of been seeing this. I've been doing some data science projects using Claude code. And Claude basically has like no idea whatsoever where to get the data from and then even still make some like strong questionable assumptions about like what the data is like and how to go about doing the data. So I, I do feel a bit more confident in some of my skills not being automated, at least for a little while. And by a little while I mean like probably still I'LL be automated in a decade or so.

[B]

Are we talking about data that's available online or are we talking about sort of truly difficult to get data? Would you have to go somewhere physically like an old library or something? Which types of data aren't accessible to AIs?

[A]

In this case, the data I'm using is primarily behind a paywall, so you have to buy it. And so that's currently not something that AIs are able to do as easily. But like you said, that is kind of just like a mundane barrier. Right. And then still like once you have bought the data, like it is kind of scattered across the website and like I've tried to have Claude automate the like scraping of the data off the website and it's still kind of proven to like, so basically like it already has my login information and so it's gotten past the paywall step, but it still like struggles to scrape it because of how the, the like insane website is formatted. But again, I do also think that's kind of just like, also like a triviality that will be fixed soon.

[B]

It seems like a limit of computer use, but I totally take your point that data acquisition could actually or is actually a difficult task.

[A]

Yeah. Or maybe another more tangible example is I'm involved a lot in congressional policy and there's a lot of forecasting questions about, oh, will this bill pass and when and by what margin. And there is just kind of a limit to how well you can forecast those questions if you're not talking to congressional staffers and getting some of that like behind the scenes knowledge. And I assume that staffers don't just answer any AI system that reaches out to them over email, but they may be like more inclined to answer me because I'm fun and charming and like they have a good relationship with me.

[B]

Yeah, yeah. AIs, at least based on the sort of emails I'm getting that are clearly written by AIs, they're not there yet. In terms of convincing me to do something or to have some guests on the show compared to getting an email from an actual person.

[A]

Yeah, yeah, I appreciate that.

[B]

So maybe you can resolve a confusion for me here around forecasting in general. If I'm looking to, I'm seeking out the answer to some question. Should I is it the case that the markets, the prediction markets are, are efficient? Should I be looking at sort of the aggregated prediction of the community as a whole, or should I seek out the very best forecasters and rely on their specific forecasts?

[A]

Yeah, I mean I think generally like an aggregation of the very best forecasters has outperformed like the aggregation of the entire community. And so insofar as you can get like a panel of the very best forecasters, like I think that's what you should use but I think that's like very difficult to do. And I would say that like the kind of like community prediction tends to just be like a pretty good estimate in general such that like I question whether it's like worth the time to really get like an even better estimate. I guess it really depends on how much the question means to you and like how important it is to be like oh, it's like not 60%, it's 65%. Like does that actually make a difference? Yeah but like if those fine grained things make a difference and if the question is very economically valuable, then it could make sense to shell out for yeah like top predictors and aggregate them. But like, I mean I would say like if you wanted to know, like oh, will Democrats win the Senate or will Republicans win the Senate? I think it's like pretty hard to just beat looking up at Kalshi and seeing what odds they offer.

[B]

It seems that many of these odds on various sites are around politics, around sports and so on. And some of the markets that are more interesting to me like around AI and technology aren't as popular. Is there a way for us to get more attention to these? Some of the questions that we've been discussing in this conversation are extremely important I think, but still don't get as much attention as sort of near term politics or, or sports. How do we, Is there something that can be done there?

[A]

Yeah, I think so. I mean I think that there's like two main reasons why they don't get that much attention. I think one is just like what do people actually care about? And they care about sports, they don't really care as much about this AI thing even if maybe they should. And then secondly, I think that on these platforms like you generally you want to get your feedback, get your returns, get your points whatsoever. You want to get that sooner rather than later. So there's like just a very strong preference to questions that close by the end of the year, that close in a few months or like close in a few weeks so that you can kind of get that fast loop and so these questions that are like 10 year long questions, like it's going to be 10 years before you find out if you're right or wrong and before you get your juicy points or Whatever. Like, I think that's just like a huge barrier to people. But that being said, I think if there was like an enterprising market maker liquidity provider that did want to put a lot of money down on this and like, try to attract the, the best people and like, did just like provide a lot of economic value, if you're right, or maybe even compensate people for their time, in addition to having this like bonus system for accuracy, like, I think you still could get some pretty meaningful engagement.

[B]

Yeah. Would it be, how does that work? How do you provide liquidity to markets? So you just. Yeah, no, explain how that works.

[A]

Yeah, I mean, so generally the way that it works is that you kind of need volume. Basically the more volume you have in the market, the like, more you can bet on the market and the more you can stand to make. And so basically you just need somebody who's kind of willing to take bets on both sides at like kind of the market line. And like, that's kind of like, I guess they in some sense are market neutral. Like it doesn't matter whether it like wins or loses. They're kind of like taking a bunch of bets on one side and then offsetting it with a bunch of bets on the other side. But by taking these bets, they're able to build this market and engage in price discovery by kind of seeing what prices people are willing to take bets on.

[B]

Yeah, makes sense.

[A]

Yeah. So there's a free project. There's a free project to any billionaire listening to this podcast that doesn't know what to do with their money.

[B]

Yeah, I think there'll be actually a good project to support and sort of useful for the world to have higher quality information on these questions. Peter, it's been really great talking. Do you want to provide our listeners with some links or where should they go if they want to follow you? Because I think you're a voice worth following. It's clear that you are sort of able to predict many important things in advance. And so. Yeah, do you want to say something about that?

[A]

Thanks. Yeah, I would say I think there's two places that I would love for people to go. I think the first is my Twitter account. I guess we call that X now. So that's like X.com PeterWildeford which is W, I, L, D E, F, O, R D. And then I also have a substack which [e-mail]. yeah, I would love to see people there.

[B]

That's great. We'll put both links in the description of this podcast. Peter, thanks for chatting with me.

[A]

No problem. Thanks for having me on. It was great.