← 80,000 Hours Podcast
Loading summary
Transcript189 lines
- [00:00]
Helen Toner
One thing that was reported was the UAE getting hundreds of thousands of next generation Nvidia chips. And that is just an unbelievable amount of computing power. A big thing that the companies will say is if the US doesn't sell it, then China will sell it. So that China is waiting outside the door. If the deal doesn't go through with the us, they'll just offer the exact same deal.
- [00:19]
Rob Wiblin
But China can't even make the chips.
- [00:20]
Helen Toner
Exactly, exactly. So it's totally disconnected from the reality of what they can do. The UAE is an autocratic country. Political parties are banned, they do mass trials of dissidents, they persecute the families of dissidents. Economy runs on immigrant labour. In the worst cases, they are essentially forced labour. There's pretty strict rules around, quite strict rules around what the media can say about the royal family. It's a hereditary autocracy with a royal family that is going to stay in power.
- [00:44]
Rob Wiblin
So when I think one of these deals was announced for the data center in the UAE, it was a collaboration, I guess, with OpenAI was involved. They put out a press release saying this is a huge win for democracy. Is this just kind of boundless cynicism or is there something else going on?
- [00:57]
Helen Toner
It certainly looks cynical to me.
- [01:02]
Rob Wiblin
Today I'm speaking with Helen Tonor, the interim executive director of the center for Security and emerging technology, or CSET, based here in Washington, D.C. cSET is one of the most, possibly the most, influential think tanks, doing analysis of the security implications of advances in artificial intelligence and suggesting potential policy responses. And Helen is also well known for having been one of the board members of the OpenAI nonprofit back in 2023, and having been one of the four board members who voted to remove Sam Altman from his role as CEO. Thanks so much for coming back on the show, Helen.
- [01:30]
Helen Toner
It's great to be back.
- [01:32]
Rob Wiblin
So you tried to remove Sam Altman from his role as CEO of OpenAI back in 2023. I think people are less confused now than they were back then about why you were interested in doing that or why you and three other board members were interested in doing that now than they were back at that time, when I think people didn't necessarily understand what the motivation was. But I hear from people who, and I imagine you hear from people as well, who say even if the goal was a noble one, it was handled perhaps somewhat naively or amateurishly, and maybe there wasn't enough forethought given to how staff will react, how investors will react, how the public is going to perceive and understand what's going on and as a result you kind of were perhaps caught flat footed and didn't actually manage to remove Altman in the end. I know there's like a lot of confidentiality issues around here which mean that you maybe can't say everything that's on your mind, everything you'd like to say, but. Yeah. What's your reaction to that?
- [02:25]
Helen Toner
Yeah, I mean, I don't think I can give a fully satisfactory account of all the decisions we made and why we made them. But two things I will say. One, I totally understand why people come to that conclusion. I really understand why it looks that way from the outside. And I think at this point I've basically made my peace with the fact that it's going to probably for people who see it that way, I don't think I'm going to be able to convey something of why it looks differently from the inside. But what I will say is that there have been a lot of, at the time and to some extent since, but especially at the time, a lot of reactions of oh my God, why didn't the board do X? Why didn't the board think about X? And for pretty much every single X, it was something that we were thinking about, we considered carefully, we had good reasons not to do. And in most cases I still stand behind those reasons. So was a very complicated situation, it was a fast moving situation. I think it's just probably not something that I will ever be able to kind of fully convey my perspective to outsiders. Not just because of confidentiality, but also just because there's so many little details of specific things that happen, specific people involved, kind of best guesses and judgments about those and lots of different trade offs. I mean, I will say for anyone who is sort of interested in this and who hasn't seen since it happened, hasn't seen some of the more information that came out about kind of how it actually went down. I think that can also help explain some of the decisions that we made. So one place to look There is an interview I did in May of last year on the TED AI show where I talked, wasn't able to share everything, but kind of really tried my best to give a kind of clear description of the basics of what happened. And then there's also been these two books that have come out actually this year, one called the Optimist by Keach Hagee and one called Empire of AI by Karen Howe, and they both include reporting about what happened on the board. I don't want to vouch for every single detail of their reporting. But I think both of them, kind of in broad strokes, have it just about right. And in particular, Karen's book gets into quite a lot of detail. So for people who are sort of interested and confused and haven't seen that material, that might also kind of help make sense of it.
- [04:29]
Rob Wiblin
So the bottom line is maybe you made some mistakes, but if you think it was so obvious what to do, if you think it would have been so easy if only you're in that position, then perhaps you don't fully understand how difficult the position was.
- [04:41]
Helen Toner
Yeah, I think the big decision, I mean, we made hundreds of small decisions over the course of that weekend. I think the big decisions, I basically stand behind. Yeah. And there's lots of ways that it could have gone worse as well, I guess.
- [04:52]
Rob Wiblin
Yeah. Back in 2023, you and the other board members came in for some pretty harsh criticism, I think, particularly from the tech industry, which was really, I guess, blindsided and confused and a bit dismayed by the decision, by and large. But my impression was that you got a much more sympathetic hearing, I think, in D.C. among policy folks and also in the press and the media, because they were inclined to, I guess, understand the situation pretty differently as maybe a power struggle within a company. Although it isn't entirely a company, it's actually a nonprofit. How has that kind of played out for your career over the last couple of years? Do people kind of respect what you tried to do?
- [05:27]
Helen Toner
Yeah, I definitely see really wide ranging reactions about what we did. Another big difference I see is between people who have experience in kind of high profile boardroom decisions and people who don't. But yeah, what you said about kind of tech industry versus more policy folks is definitely true. I think. I mean, to speculate a little bit, I think on this coast, in this city, there is maybe. Well, for one thing, there's much less of the cult of the founder, which is really incredibly pronounced in Silicon Valley, of the founder. The CEO is a genius and always correct. And the board's job is to sort of stand behind and cheer. And that's just not like a cultural expectation here or organizational expectation. I think maybe a little bit more inclination to not assume that companies are always doing the right thing. That probably contributes. And yeah, I also think something I really noticed is just different expectations around how much information is going to be shared out of a sensitive and complicated situation. I think as well, maybe the most important piece about this from my perspective was I was the only person on that board who really had strong professional roots outside of the West Coast. And I think that it's not a coincidence that I've been the person who's been able to sort of talk about this a little more than the others. And I think it was a big. Contributed a lot to how we were able to make the decision in the first place is basically not being vulnerable to having my reputation destroyed in Silicon Valley. I think at this point, I don't know that I think my reputation has recovered somewhat, as you said, because of the changes in perceptions of our decision. But I think for many others in and around the company, they are just very vulnerable to perceptions in that one specific community. And so the fact that I had networks and respect and people who know and understand and value my work over here that weren't going to be subject to that was really important. I mean, I still meet people who know CSET and don't know my name. So they know our organization, they know that we do really high quality work on AI and national security and that's what they know, which is very different from on the West Coast. I'm much more likely to meet people who only know about me from the board and have no idea kind of what I spend most of my time doing.
- [07:36]
Rob Wiblin
Yeah. So while more people probably have heard of you because of OpenAI, so you were on the board for two years and it's an unpaid role and I guess a part time role. Yes, definitely. And by far the thing that you've been working the Most on since 2019. Right. Is center for Security and Emerging Technology. What should people know for the purposes of this conversation about what CSET is and I guess what actually you do.
- [07:58]
Helen Toner
Yeah. So we were set up like you said, in 2019 and that was actually last time I came on. The show was right as we had gotten set up and I went back and looked at that interview and it was interesting because at the time to sort of describe CSET was basically saying, look, AI, national security, these are big topics. Emerging tech and national security. We decided there should be a center like really important. Which at the time was kind of a novel idea. Yeah, right, exactly. These days I think the way to understand CSAT is that is very much still what we do, but to explain a little bit more what makes us special. So we are really focused. We're based within Georgetown University, but we're not academics. We don't do the academic journal thing or don't have tenure track academics. We're very policy focused, very focused on what is going to be relevant and useful for policymakers and other decision makers. And our work, we really strive to be independent, evidence driven and technically informed. So what do you mean by that? Independent, nonpartisan, not coming in with an agenda, not sort of primarily there to do advocacy, but really to just sort of tell it like it is. Evidence driven, data driven. Where we can be we have a data science team who do incredible work. We have large data holdings of different kinds, data on kind of papers and publications, financial flows, job postings. We have an in house translation team. And so we're really focused on using data and evidence to show what is actually going on out there in the world. And then technically informed, meaning our work is we really specialize on a small number of technologies, primarily AI, semiconductors, biotechnology. And that means that we can have people on staff who really understand those technologies really well. So an example, just to give us a sense of our work, a piece that came out fairly recently was looking at several thousand, so kind of combining our data and China translation capabilities, looking at several thousand contracts from the pla, the Chinese military on AI and analyzing. In this case we have more coming out from that data set, but in this case it was looking at what are the organizations that are working with the PLA and what does that tell us about China's military Civil fusion strategy, which is a strategy they have to try and sort of bring together their public sector defence ecosystem and their private sector and really interesting findings. When you look at this sort of big data set of I think almost 3,000 tenders around the kinds of new and non traditional vendors, so universities that haven't traditionally been linked with the defence ecosystem or smaller newer companies coming up that aren't just the gigantic state owned enterprises that have traditionally done Chinese defence. So that is the kind of work that we like to do to kind of bring more clarity, shed more light on these issues to help policymakers make more informed decisions.
- [10:43]
Rob Wiblin
I think one of the big influences that CSAT has had is I think you were one of the groups that provided a lot of information and analysis data and suggestions that I think culminated in the imposition of export controls on semiconductors and semiconductor manufacturing equipment to China. I think that started around 2019 at the very beginning and then I guess it's gotten more intense and now it's I guess been very topical this year. How do you feel about that influence? With the benefit of hindsight?
- [11:10]
Helen Toner
Yeah, so I guess there's some kind of technical nuances to unpack here. I guess we have time, so I'll unpack them. So CSET's work. One of the first things that we looked at in 2019 was looking at kind of inputs into AI advancements. And so we did a lot of work on kind of talent. We did some work on data. And then we also looked at the compute, the semiconductors as an important component and wanting to understand how do semiconductors, access to semiconductors affect AI progress. And it's really important to distinguish between the two kinds of export controls you mentioned, namely sort of semiconductors, the chips themselves versus semiconductor manufacturing equipment, which is sort of the gigantic pieces of machinery that get put in the fabs, the factories that make the chips. And so our research really focused on the semiconductor manufacturing equipment, that supply chain. And the reason that matters is I think there's actually just a really straightforward, really solid case for why we should be controlling certain kinds of semiconductor manufacturing equipment. So the kind of most famous one for people who follow this space is EUV lithography. So that's extreme ultraviolet lithography. These are these machines that are made by this one company in the Netherlands and flown over to Taiwan in multiple jumbo jets, because they're these very, very complicated, large machines and no one else in the world can make them. And there's other examples. So another sort of choke point is EDA software. Electronic Design Automation, I think is the acronym, which is primarily made by US providers. And the key things about these are if you're going to control something, if you're trying to prevent someone from accessing something and you could supply it to them and you want to put a control on it, the question is, what will they do instead? And so if you want to prevent a country from having light bulbs, you're going to have a really hard time because if you prevent them from buying your light bulbs, they're going to be able to buy them anywhere else. So it's kind of pointless. You're just sort of shooting yourself in the foot. And so the key thing about these pieces of semiconductor manufacturing equipment, like lithography machines and EDA software and so on, is if they really are a choke point, meaning there really is only one or a very small number of providers, and it's not going to be easy to sort of start a new company that can replace that, then you can potentially really slow down China's efforts to build up its own domestic supply chain, which means that they stay dependent on US Chips, which I think is just strategically very solid. It sort of doesn't actually really antagonize them very much at all or cause problems for their economy, but it does give the US a significant and the US and allies a significant strategic advantage. So in 2019, that was what we identified at CSET based on our research, and that has been controlled. It's kind of a shame that that semiconductor manufacturing equipment or SME people in the field will say those SME controls have kind of fallen out of the limelight or not been a major focus as there's been so much debate over chips. And so we can talk about chips in a second, if you like. But ideally the sort of stuff around chips is a little more complicated, a little more trade offs, a little bit different theories of change. And ideally, if you were going to dabble in that, you would sort of keep the SME controls as this very obvious baseline that you're really focusing on. But instead, because of sort of limited resources, limited focus, I think those controls have actually been not implemented particularly well or particularly rigorously, while there's been a lot of discussion about the chips instead, which I think is kind of a mistake.
- [14:39]
Rob Wiblin
You think machines are being kind of smuggled into China?
- [14:42]
Helen Toner
I would have to go look at the details. I think there's licensing and there have been lots of licenses granted that colleagues of mine who are deeper in the space are sort of saying, why are we granting these licenses? Things like that. So it's not necessarily just smuggling, but also a question of focus on which actors get access to which things.
- [14:59]
Rob Wiblin
I'm surprised you said that you thought the export controls weren't antagonizing China. I would have thought they would feel like they're being put in a pretty vulnerable position, security wise, that maybe their economic ambitions are also being somewhat constrained by this. It's just kind of a reasonably hostile move, you would think, to deny them access to these machines, to the chips as well. Am I understanding wrong?
- [15:20]
Helen Toner
So again, we should separate out the SME from the chips. I think I was saying the SME is not necessarily very antagonizing. I think the chips question is really interesting. I mean, it's important to look at the backdrop here. So I do think that the export controls on semiconductors themselves, China did not like that. Obviously the backdrop, though they already were saying in at least 2015, if not before, that they were assuming that the US was going to do this kind of thing, that they were assuming that they needed to indigenize, that the US was a hostile actor that you couldn't trust and that was kind of against them. And then in the first Trump administration, there were a bunch of actions around semiconductors and sort of the tech Industry more broadly targeting Huawei as sort of the well known case. But there was also zte, another company which was targeted beforehand. And so kind of already then in I think it was 2018, I want to say that was sort of confirmation for China of what they had already believed, which was that the US was going to behave in this way and that they had to indigenize and they had to kind of be looking out for themselves. So I think it's a real open question, like what was the marginal effect of the different controls? And so I would say that the marginal effect of the SME controls was probably not very large. The marginal effect of the chip controls may be a little bit more debatable, certainly some marginal effect. But I think sometimes I hear people in this space who are sort of very focused on AI and not focused on the US China relationship acting as though that was like a bolt from the blue, gigantic hostile action. Often nothing was going wrong in the relationship. And I think that's not right. I think it's sort of a little bit trickier to say what was the effect and notably China's reaction. I mean, there's now been multiple years of sort of a little bit of back and forth. But certainly at the time the reaction from China was actually not very retaliatory, was not very sort of escalatory, which I know was another thing people were concerned about was, you know, will this create a huge backlash? So, yeah, I mean, I don't want to say it was all Kumbaya and no problem whatsoever. But I also think it's possible to.
- [17:23]
Rob Wiblin
Overstate a strange thing that's happened this year. There was this big debate in the US about whether the US should be selling or allowing more Nvidia chips to be sold to China. And people would argue it's bad to do that just for the obvious national security reasons. And other people would argue back, no, actually it's good because what we want to do is maintain China's dependence on these chips rather than basically force them and indeed encourage them to develop their own local semiconductor industry and give that industry a big boost. And then recently China, I think, made the decision to not allow the Nvidia chips that the US was debating whether or not to allow into China itself. So in a sense, China has imposed exports controls on itself, at least on these particular Nvidia chips. Can it be the case that it's both like a smart move for the US strategically to deny China the chips and a good move for China to deny itself the chips? It seems like it's a reasonably zero sum situations. How can it be in both of their interests to do it?
- [18:18]
Helen Toner
Yeah, lots going on here, I think. So one immediate question is, does China really mean it or is it an ambit claim? Is it a bargaining position? I think at this point most countries around the world know that President Trump loves a deal and loves making gigantic opening bids on deals and then retreating to something more reasonable seeming. So I think it's very possible that this is just sort of mostly signaling from the Chinese government saying, you're going to have to beg us to buy your chips and we don't care anyway. And so far, I think the effect of that rhetoric on the US Debate does seem to have been, oh, we should just let them buy. If it is a move by China to be allowed to buy the chips, it seems like so far it's working. So that's one possibility. I think though, this broader thing and also what you said about how the US Debate has shifted really shows how there's been very little clarity and very little agreement on what the goal of these export controls is and what the theory of change. And I think partly that comes from the fact that this is a complicated technology and relates to AI, and AI is general purpose technology with lots of different components. I think partly it comes from this sort of situation in US Policy over the last few years where one of the few things people could agree on was essentially China bad. And so if your policy was like, oh, this is going to China bad and therefore we'll do something that will hurt China, then people could kind of get behind that without necessarily needing to have a very clear sort of position on the nuances of the policy goals and how you would tell if they were succeeding. But I do think the fact that we've gone from the initial messaging around this was really about China's use of chips for military applications and for human rights abuses. And astute observers pretty quickly were like, hang on, military applications, a lot of what you need for that is going to be much smaller chips onboard. Like if you're having something on a drone or on any kind of equipment, you don't use these gigantic data center style chips. You use something much smaller. So if we're primarily targeting the military, this is kind of a weird way to do it. Likewise, human rights abuses, that's often going to be like image recognition, speech recognition, other things that you don't need these gigantic clusters for. And so then the rhetoric has sort of shifted around to, well, it's actually about sort of China's ability to build very advanced AI systems. I think that's the most sort of sensible or like that won't actually make sense if that's what you're trying to prevent, then preventing large accumulations of the most advanced chips is a good way to do that or a reasonable thing to try at least. But then now, yeah, the rhetoric has shifted again to, oh no, what winning means. And David Sachs, the White House AI and crypto czar, has said this very explicitly. What winning against China means is having more market share in chips. And so as long as we're selling them chips, we're winning. And that's just sort of a very different theory of change. From winning means that we are using the chips domestically to build something that gives us a strategic advantage that they're not able to build or that helps our economy because we have this broad base of computing power that they don't have. And so we have some kind of strategic advantage. So there's really, I think, a lot of confusion and a lot of, of almost rug pulling over sort of what exactly are we doing here and how will we know if it's working?
- [21:25]
Rob Wiblin
Yeah, I've heard this a lot this year, that it's really important for the US strategically to get the rest of the world to start using its AI models and to be part of its AI stack. To me, I guess it's not intuitively obvious that that is a national security priority or that it really does make a big difference whether someone in Australia or South Africa or Brazil is using ChatGPT or whatever the Chinese model might be. And I mean, a cynic might say, well, this is in the AI industry's interest to convince the government that it's really important that they provide support so they can sell their products more and compete for market share. Why wouldn't they try to persuade people of that if they can get away with it? Does this argument make sense to some extent or how strong is the argument and how should I think about it?
- [22:08]
Helen Toner
Yeah, I think the version of it that makes sense to me and something that colleagues of mine at CSET wrote about is basically as a soft power play, meaning that there's just going to be lots of probably like if you look back at the history of technologies and sort of who's providing what. There's just a lot of cases where you get some kind of broad, diffuse soft power benefits from being the provider of a key technology. A sort of different example would be like Hollywood and American music. And why exactly is it good for America that Everyone around the world enjoys Hollywood movies and listens to American music. Sort of like kind of hard to put your finger on it, but I think it is clearly good. That's one of the canonical soft power examples. Likewise, I think it does seem right to me that it's going to be better for just a wide range of reasons in terms of sort of influence and standard settings and expectations and relationships if American AI is used around the world. But sort of big caveat there is, I think if that is your focus, then it doesn't necessarily make sense to focus on these sort of frontier systems, meaning computing clusters that have hundreds of thousands of the most advanced chips and are being used to train or run the very, very most advanced models. I think if you actually talk to people who are going around and trying to sell an AI stack or get government partnerships with AI stacks, often what people want is sort of more of a turnkey solution of like, okay, well what is this going to do for me? And there you often don't necessarily need huge amounts of computing power. You don't necessarily need the very most advanced models. Instead, maybe you need some kind of support to figure out how to use a model for some particular use case that's interesting to you or to use it. If we're talking government usage or sort of large enterprise use, the support you maybe need is how does this fit in with our existing kind of IT procurement policies? And what is the ux? How should we build it into our existing offerings for our staff? Or if it's a government, how should we fit this into our sort of online digital government initiatives? And that's much more the kind of thing where you need, need sort of embedded software engineers to go in and understand the problem and sort of help make progress as opposed to the kind of thing where you really need hundreds of thousands of chips. So I think the argument is basically reasonable and I understand why people coming out of the tech world sort of want to say let's do this full stack American AI stack approach. But I think it's sometimes used for this like, and therefore we should let them build their own world class supercomputers, which I don't think follows.
- [24:34]
Rob Wiblin
Yeah, something that's been even more confusing about this argument to me is I think people will make that argument and then say, and so it's very important that we have leading open source AI developed in the us But I think if you have a brilliant, if meta, open source is a really good model and so anyone anywhere in the world can use it and run it on their own equipment, then they can just fine tune it to have whatever kind of properties, whatever personality, whatever sort of values they like, and then just run it locally on some equipment that's not in the United States. How does this really help the US's national strategic, national security goals?
- [25:06]
Helen Toner
Yeah, I don't know. Again, I think from a soft power perspective it maybe makes more sense. And I also think that often people in an open source ecosystem, people will kind of engage and go back and forth. They don't just sort of grab the open model and then run away and hide in a cave and do their own thing with it instead. There's often more of a rich kind of back and forth and engagement. So I tend to think I am pretty in favor of there being more of an effort to produce really high quality open source models. I signed onto this project called the Atom Project, the American Truly Open Models Project by Nathan Lambert, who's a researcher at AI2 in Seattle. And the key thing in my mind is actually separating out two different questions, which is should we be trying to have leading open models versus should we be trying to open source or open weight the most frontier models? And I think those should be separate questions. So I think that, that it does make sense for US companies to be trying to offer models that are at least sort of roughly as good as the best open models openly. But I don't think that necessarily means they should be racing to open source their very best models just because it creates security risks. Yeah, I think that my sort of overall stance on open source is to think that starting point open source is great, sharing everything as widely as possible is great and has lots of diffuse benefits that are kind of hard to count up, but really meaningful in terms of accountability and access and broad benefit. But the main exception in my mind is these frontier models, the most advanced models, because we just understand them least well and we're least well positioned to sort of mitigate any new risks that they pose. And so just having some amount of the phrase that's sort of gotten a little bit established is precautionary friction, meaning having a little bit of lag. Having is it 6 months, 12 months, 18 months to test and understand those systems, to figure out what can they do if you really push them to their limits, what kind of unexpected effects emerge and having that happen in a time where you can still sort of pull them back from the API, do some fine tuning, put some new safeguards on, rather than having it be fully open.
- [27:11]
Rob Wiblin
The rhetoric in D.C. is really around the U.S. being in a race to develop AI, an AGI against China. Is China actually racing to build AGI?
- [27:21]
Helen Toner
It's a great question. We actually recently put out a short post on this because there have been these two recent op EDS by one by Eric Schmidt, one by Jack Shanahan, both very smart and well informed people saying the US is doing it wrong because we're focusing on AGI and China is doing it right because they're focusing on kind of applications and diffusion. And we put out a post, two of our researchers, Bill Hannis and Toimei Chang, who are two of our sort of of most senior China researchers at ccert, put out a post saying like, well, that's actually not right. Actually China is doing all of the above. So it's definitely true that China's big push right now, or one of their big government pushes is called the AI plan and it's AI plus because it's AI plus some other sector. So AI plus manufacturing, AI plus healthcare, looking for those intersections and those useful applications. That is a big focus. But they can walk and chew gum at the same time. And they, they are continuing to also emphasize AGI, general purpose AI. So some wrinkles here. One is that in Chinese the word for AGI is the same as the word for general purpose AI. So it's just. So it means like general purpose. It's very unhelpful. So in the us, you know, or in the west or in sort of Anglophone discussions, we have for a long time talked about AGI and then after ChatGPT came out, this sort of idea of general purpose AI kind of got established, which is this more sort of mundane thing that can be used for many things, but it's not the full on human level or building towards superintelligence. In Chinese it's the same word. So it's a little bit unclear. That being said, they do sometimes just use the English letters AGI and they do sometimes talk about superintelligence. So I think it's very clear that they are right now pushing for both. Something that's less clear is the question of sort of how, I don't know of a better term than AGI pilled Chinese leadership is meaning they're really all in on AGI is going to be an enormous deal and it's going to happen soon. And the way to do it is to scale up. And the best thing that I've seen on this is actually from Jordan Schneider at chinatalk, just has a debate sort of posted on his substack. I Think if you say Chinatalk, AGI debate or something will probably come up on Google and it's basically formulated as a debate between a believer and a skeptic. The believer saying obviously China is very AGI pilled and here's all the evidence and the skeptic saying obviously they're not. I tend to come down with the skeptic on that debate, meaning I don't see the evidence that they are really gunning for sort of AGI or superintelligence in the same way that sort of the leading US companies are or the leading Chinese companies. I think Deepseek is certainly very AGI pilled as a company, so there's sort of degrees of what you could mean for is China racing towards AGI. I also don't think the US government right now seems very AGI pilled and I think that could be for good reasons. I think there's a lot of assumptions bundled into what that means that may actually not be correct. So I don't want to say they're not AGI pilled and that's a mistake. I think there's reasons to have lots of space for different views on that.
- [30:26]
Rob Wiblin
So D.C. has a pretty hawkish attitude towards China, I would say. Do you think this is kind of appropriate given how China has behaved? I think my worry is it's become such an easy thing to say, oh, we've got to do X or Y things that I liked anyway because of China. And it's kind of uncomfortable, I think, for people to push back and say is it really necessary to be this antagonistic towards China? When something becomes that degree of conventional wisdom and so easy to say and so awkward to oppose, it can just get basically never gets challenged and just gets accepted by everyone, despite the arguments perhaps being weaker than they seem. What do you make of that?
- [31:01]
Helen Toner
Yeah, I mean, I think groupthink is bad. I think people feeling that they need to self censor is bad. I have good news for you. I think over the past, even just the past three to six months, this has been softening a bit. I think President Trump's stance on China is pretty unclear, shall we say, and, or maybe somewhat fluid. And that has created some space. I think also certainly in the AI side, the top folks advising him on AI seem much more interested in commercial engagement with China, selling US chips to China in a way that is not as compatible with sort of the traditionally China hawk stances. So yeah, I mean, I think it's really valuable if there can actually be many voices in the debate and many, many perspectives. And I think there's been a little bit of a lack over the last few years maybe of creative thinking about what are all the different ways that we could relate to China. It all gets collapsed into Hawk vs Dove, which is maybe sort of a callback to 20, 30 years ago of do we assume that China is on this path to being a responsible stakeholder and a more politically liberal, nice, friendly country, or do we assume that they are on track for war with the US and then we have to be maximally hawkish. And of course, there's a very wide range of possibilities in between there. So I think it's valuable to be able to consider a wider range of options.
- [32:29]
Rob Wiblin
Something I've been a bit confused by is it doesn't seem like there's any direct diplomacy going on trying to move us any closer towards having some sort of treaty or agreement with China on governance of AGI, preventing AI being integrated into military prematurely before either country feels comfortable that they actually have a grip on this technology and understand its pros and cons, or let alone a treaty to govern superintelligence and say, well, maybe we should be kind of cautious about that one, even if we're barreling ahead on all the applications that we want in the economy right now. Am I right that there kind of is very little intercountry discussion of that? And if so, do you think that's a mistake?
- [33:06]
Helen Toner
I think you're right. Especially at what gets called the Track 1 level, meaning official government to government talks. I think there's a few reasons for it. One basic reason is one sort of AI specific reason is I don't think that the groundwork is there in terms of concern about this as a problem or concern about this as a major problem to prioritize. So what is AGI? What is superintelligence? Are there things we could build ever? Are there things we might build soon? Would that be good or bad? And for whom? I think there's just very little, especially within, at the government level, very little agreement on those questions. In contrast to, I think, perspectives in the AI safety community, for example, which is.
- [33:46]
Rob Wiblin
Or the companies or many of the companies?
- [33:48]
Helen Toner
That's right. Well, in the companies it's, we can build this, we will build it soon. And then in the AI safety community, and it's going to be really bad. And so we should be talking about it. But I think that is a relatively insular set of beliefs still. That's kind of the AI specific reason that the sort of groundwork is not there. I think it's also really important to look at the broader relationship as well and understand that from the perspective of, of a diplomat or certainly of a president or a chairman, this has to slot in with everything else that is going on in the relationship. And so what else is going on, the relationship? Well, one thing is the US And Chinese governments are barely talking at all. So over the last couple of years, there have been some direct country to country talks on some small number of issues, but they've also often been completely suspended. So for a period, I think it was after Nancy Pelosi went to Taiwan, I think just basically all talks were suspended.
- [34:35]
Rob Wiblin
I mean, isn't that just crazy? It, it's mind blowing to me. This is the two most powerful countries in the world. They've got so many things to talk about.
- [34:42]
Helen Toner
Yeah, I mean, I think generally my understanding of this is that usually the US Is the one that has more appetite to talk. And so China is using, and China knows that. And so that becomes a bargaining chip for China to say, we don't want to talk to you. We're not going to do these military to military talks about extremely sensitive important issues because we're mad. And if you want us to do them, then you have to give us something in return to come back and join these talks. Right.
- [35:04]
Rob Wiblin
And I guess you don't want to give into that.
- [35:05]
Helen Toner
Well, yeah. I mean, if the whole point is that it's mutually beneficial, then we don't want, want to be acting as though we're making a concession by, or that they're making a concession by letting us talk. There's a lot of context here and a lot of baggage. A couple of other pieces of context and baggage I would give one is I think a fair amount of skepticism among US Diplomat. Diplomats in general are usually pro engagement, pro negotiation, pro conversation. That's why they become diplomats. I hear among US Diplomats a lot of scepticism about the value of that with China based on their track record of what happens when we do that. Sort of important example is in 2015, President Obama and Chairman Xi had a long discussion about this problem of China spying on U.S. companies and sort of stealing trade secrets from U.S. companies, which is different from. So there's a long established countries spy on each other for sort of strategic reasons. But China was doing something different, namely corporate espionage, where they kind of take trade secrets, benefit economically, hand things to their companies very much not okay, very much out of the norm internationally. In 2015, the end of a long series of discussions There was a big deal signed between Obama and Xi saying that China was going to stop doing that. And consensus is that they basically started a few months later again, so they stopped very, very briefly and then restarted. That's sort of one emblematic example that's in cyberspace. So a lot of the people who do AI stuff have previously worked in cyberspace. So that's a very salient example for them. So there's also skepticism about are they a good negotiating partner. And another example I would add there is the big example of Track 1 government to government talks on AI was in Geneva last year. This sort of initial foray into this conversation, my sense is that that didn't go terribly, but it also didn't go great. And part of the reason, which again is sort of emblematic of US China negotiations more generally, part of the reason was the US sent over some really technical, well informed, some of their best AI policy people to have quite in depth conversations. And the Chinese sent their America specialists. So they're like US Diplomats who knew almost nothing about a or just specialists in trying to handle the Americans. The term that sometimes gets used is they sent their barbarian handlers, meaning they specialize in going out there and playing nice with the foreigners. And so that again, just again, I don't think that dialogue went terribly, but it wasn't a good start. And again, it sort of suggests that the groundwork is not there in terms of this actually being a priority that the Chinese government actually wants. In contrast to the default explanation for why would China agree to talks is because it makes them look like a great power and they're up there with the US having bilateral talks on AI. Doesn't that show that they are doing so well at AI and so that is also a motivation for them to talk. And if that's why they're there, then it's really not going to be very productive.
- [37:50]
Rob Wiblin
I mean, they are a great power and they are doing pretty well on AI.
- [37:53]
Helen Toner
I mean, sure, yeah, but if the reason they want to be there is to be able to brag about it, then you're not going to make much progress on reducing risks.
- [37:59]
Rob Wiblin
So you would say it's mostly on China that there aren't more negotiations along these lines, some shared responsibility, but more on China than the U.S. i think.
- [38:07]
Helen Toner
There'S plenty of plenty of fault on China.
- [38:08]
Rob Wiblin
Yes, I guess let the record show that I'm disappointed in China over that. You can tell them I'm frustrated. Is the US on track to stop its best AI models being snatched by China at the last minute to have the weights exfiltrated and then just used by the Chinese military.
- [38:24]
Helen Toner
No.
- [38:27]
Rob Wiblin
What should it do that it's not doing?
- [38:30]
Helen Toner
I think the best hope right now is that to the extent that China perceives the gap between the best open source models and the best closed models to be small, they might just not invest in trying to steal steel weights or steel models. And I think that is a pretty widespread perspective right now. So maybe they just won't bother. Basically, I think that's the best hope. But I don't know, man, cybersecurity is really hard.
- [38:51]
Rob Wiblin
But I think isn't there a bit of a contradiction? If you're saying we want to, we're kind of like racing with China, we're trying to get to AGI first, but also it's all going to be so close together they're not even going to bother to steal the thing that we're making that we're putting so much effort into to. How can these things coexist?
- [39:05]
Helen Toner
Yeah, I mean, I don't think it's that clear that we're racing with China towards AGI. I think there's sometimes a set of background assumptions here around even the language of a race. I'm perfectly happy to say that we're competing with China on AI. The thing with a race is there's a finish line and whoever crosses the finish line first wins for real. And if you cross the finish line second, then that's useless and doesn't matter. And I think sometimes in some AI circles there's an assumption that AI or AGI really is a race with a clear finish line and the finish line is whoever builds sort of self improving AI first. Because then if it's true that once you get to a certain level of AI, then your AI can improve itself or can improve the next generation and you have this kind of compounding improvement, then that could genuinely be a situation where whoever gets to a certain point first then sort of ultimately wins. I don't think it's at all clear that that is actually what it's going to look like versus the systems get more and more advanced, they're used in more and more ways, they're sort of diffused through multiple different applications. And in that case, I think we're sort of in this state of ongoing competition with China, but not necessarily a heated race where whoever is a hair ahead at the very end at the finish line ultimately wins the future or something. Yeah. So I think the shape of the competition is actually pretty unclear. And when people treat it as though it is very obviously just the this sort of winner take all race, I think that is a pretty risky proposition because it sort of implies that certain kinds of trade offs and certain kinds of decisions are obviously a good idea when in fact I think it's not at all clear.
- [40:34]
Rob Wiblin
So two years ago my memory is that the attitude was we've got to tighten up security, we've got to make sure that they can't steal the weights. And it sounds like you're saying there's been a bit of a shift in attitude that people are more fatalistic now.
- [40:46]
Helen Toner
Well, I mean, I think even at the time, the trouble is that stopping a sophisticated state actor from stealing anything online is really, really hard, or not even online, any kind of digital infiltrating, any kind of digital system, really really hard. And so I think the sort of best known work on this is by Rand. They have their sort of security levels that they came up with. I do think that there's plenty of. It's sort of very clear that we should be trying to make it harder for a wider range of actors to steal advanced AI models or otherwise algorithmic secrets or other proprietary information. Because if you are just nihilistic about it, then maybe it's not just China, but it's also North Korea and it's also a terrorist group or it's also a disaffected young person who uses not this generation of ChatGPT, but two generations from now ChatGPT to carry off a somewhat sophisticated cyber attack. So I do think we should be very much focusing on, on improving security. But I also think it is going to be a really, really big lift to try and actually make these systems resistant to sophisticated state based hacking attempts.
- [41:56]
Rob Wiblin
If that's the case, it does seem like it really undermines the idea that we have to train the best model in the US because if it matters, if it actually does provide a big strategic advantage, we can fully expect that China will have it pretty soon after it's trained or we can think that that's pretty likely. I mean, the thing that would remain is you want to ensure that there's more compute in the US that China is kind of denied the ability to do massive amounts of inference. Even if they can steal the weights, that could still provide a strategic advantage, but they need to train the thing urgently to get ahead. That argument feels a lot weaker.
- [42:27]
Helen Toner
Yeah, again, I think it's this question of what is the shape of the competition or what are the parameters here? And maybe another way to say it is. I think if it is purely about who has the most advanced model, then certainly the fact that China will probably be able to steal most advanced model makes it less important sort of where it is developed, or makes it less beneficial to be racing to develop it first. But as you say, there's also a question of sort of deployment. There's also maybe a question of, you know, if part of why you want the most advanced models to be able to build more advanced models, then maybe you need the computing base for that. And so maybe they could. I think this is reflected in some of the compute modeling in the AI 2027 scenario, for example, is like China in that scenario, China steals a model, but then they can't actually use it to make as much progress because they don't have as much compute to make the progress progress. Again, that's baking in a bunch of assumptions about what will progress look like and what kind of advantage will accrue to which kinds of actors with what kind of resources. So I don't think it's like a knockdown argument to say that because China can steal the models, therefore we shouldn't be trying to have the best models. But I do think it should at least temper that claim or should be always in the foreground when we're thinking about to what extent does it make sense to move fast, Especially if we're saying, well, we have to move fast, even if it's risky because we have to get there first. Then it's sort of like, well, which risks? And how are you weighing them against the possibility that your model will just get exfiltrated?
- [43:39]
Rob Wiblin
A few years ago, you wrote that you thought China was two to three years behind the frontier on AI. I guess that feels like a lot now. I imagine that the number is lower. What would you say it is now?
- [43:48]
Helen Toner
Yeah. So this was two years ago at this point in a piece in Foreign Affairs. And the point of that piece was actually to say that we shouldn't use the concern that China will race ahead as a reason not to regulate our AI sector. I still very much stand behind that. And there's sort of a bunch of other things in the piece that kind of back that up. But yes, the estimate we used in there, which was actually coming from some Chinese investors and folks over there, was two to three years. I think that's clearly not right anymore. At the same time, Deepseek had its big moment in January, obviously made a huge splash. And in the wake of that, I saw some people pointing back at this piece and being like, oh my God, two to three years, what a joke. They're right exactly on our heels. There's no gap whatsoever. And I think that is also overstating the case. So I think the smallest gap that we have observed is three months, which is the time between when OpenAI put out 01, its first reasoning model, and when DeepSeq released R1, its first reasoning model. That was very impressive. I was surprised. That was really well done by Deepseek. But Miles Brundage, for example, who's a former OpenAI researcher at the time, said, look, the way that this works is this is a new paradigm, this reasoning paradigm and right now they're at the very low compute, sort of earliest stages of it. This is going to be the easiest instance to recreate this sort of O1 level. Right. And since then we've seen OpenAI scale, that approach to create O3 and O4 mini and I think also 04, but I'm not sure what the is 04 now GPT5 thinking, maybe unclear. And China has, or deepseek, which is I think essentially the leader in the reasoning space. Maybe Alibaba's clean series has something that could compete. To the best of my knowledge, they have not previewed anything as good as O3, which confusingly is the second generation of the OpenAI models. Because O2 is a telecommunications company, they didn't want a trademark dispute. So OpenAI put out their second iteration. They previewed it in December and they released it in January. We're now in September when we're recording this. And as far as I know, no Chinese company has has previewed a similarly sophisticated model, which means we're now at something like nine month gap and that might be longer depending on how long it takes them. And also it's not clear how to account for the O4, which is the third generation OpenAI model, which certainly they don't seem to be competing with. So it's all very murky, it's hard to put an exact timeline on it. I've been saying something more like 6 to 12 months now. I could imagine pushing that out longer depending on how long it takes to have that sort of competitive with O3 performance. And then it's also hard to compare because OpenAI is releasing this as a product, as a sort of full system. So O3, for example, is well known for being really good at tool use. And deepseek or Alibaba would put out an open source model which is a little bit different. So then how do you Think about the tool use. So it's very hard to compare. But the short answer would be right now, I would say something like 6 to 12 months.
- [46:42]
Rob Wiblin
Probably CSAT was years ahead of the curve, I would say, in recognizing the national security implications of AI advances, and I guess probably years ahead of the curve in realising that export controls were going to be a huge deal. Is there anything you're doing now that could be the next thing where that CSET is ahead of the curve, where other people haven't appreciated just how important some issue is.
- [47:00]
Helen Toner
We haven't actually spun up work on this yet. But the one that comes to mind for me is robotics and advanced manufacturing, which I think is sort of sometimes treated as two separate topics, but I think is very closely related, which is basically producing really sophisticated industrial machinery at scale, which could include robots or could include other things. Yeah, this is something I want to learn more about. I want to look into more. I think sometimes the conversation about manufacturing in the US can become too much about jobs, about manufacturing jobs, and that sort of neglects the strategic importance of having really sophisticated high volume manufacturing capabilities here from a strategic perspective of what we can actually produce domestically or sort of among our allies and partners. So I think, I don't know. A thing that I've been thinking about is this. I forget where I got this term. It's not mine. This idea of an industrial explosion, meaning if AI is actually succeeding, will we see a huge buildout of energy, a huge build out of data centers, a huge build out of increasingly advanced sort of robotics and the manufacturing you need to produce them? And I think right now China is sort of really well poised to make the most of that if things do go in that direction. And so I think some interesting questions around what does the trajectory of that technology that build out look like and what would it look like for the US to position itself better?
- [48:18]
Rob Wiblin
Despite being pretty bullish about AI, The Trump administration has done a bunch of stuff that I think is kind of inconsistent with wanting to win any sort of AI race. I guess it's made it more difficult for people to immigrate to the us, including skilled workers, and generally just seems kind of hostile to scientists and students coming over. It's been negative on rollout of energy generation, at least renewable energy generation. I guess the tariffs have made it more difficult for the manufacturing sector in some ways because it increases the cost of industrial inputs, basically of the machinery that you might import from overseas. I guess they had that raid on the Hyundai battery factory. I think Trump then said that he actually didn't support that. So maybe they kind of rolled that one back. But it feels like there's a bit of a contradiction here. There's a bunch of stuff that you probably would be be doing like encouraging high skill immigration if you wanted to be, I guess, as far ahead of China as you possibly could be, that things that aren't happening, indeed it's kind of going the other way. How can you make sense of that?
- [49:15]
Helen Toner
I think I make sense of it by there being sort of different factions inside the Trump administration, different specific officials with different agendas and different priorities and them not necessarily coming together into a coherent policy vision. So for example, I mean on immigration, that was one of the topics CSET looked at at earliest. And when we were in the space in 2019, there was sort of this common wisdom that high school immigration was good economically because it benefited US Companies and helped the economy grow. But it was only a downside from a national security perspective because people could leak information, steal information, or at a minimum just get educated here and then go back to their home countries and benefit their home countries. And so some of our earliest work was on understanding actually the national security benefits of having high skill immigrants here, including just the fact that the US is high tech ecosystem is so driven by immigrants. Depending on which numbers you're looking at, they're at a minimum something like 30 or 40%, sometimes up to well over half of any given pool. I think it's something like half of the founders of top startups are foreign born, things like that. And that makes sense if you look at if the US is going to compete internationally. We just have a much smaller pool of domestic workers than somewhere like China or somewhere like India. So the fact that we can import them and that we can can draw the best talent to this country is a huge asymmetric advantage. But that is just obviously in contrast with the Trump administration and sort of the Trump the MAGA movement's perspective on immigration more generally. And so I think that sort of policy wonky set of considerations around pros and cons just gets lost in the sort of broader push to be anti immigrants. So I do think that among the the policy actions that the Trump administration has taken, the many things that are going to deter high skill immigrants from coming here and are up there with the cuts to science funding as some of the most damaging to US competitiveness just in terms of being technologically as sophisticated as this country has been in the past. And I don't think that my understanding of it is not that there is a clear, well thought through strategy that explains why that is. It's just these sort of different components of the coalition sort of doing their own thing.
- [51:32]
Rob Wiblin
The center for New American Security and Leonard Heim, the compute governance expert who was actually on the show two years ago, they've recently been pushing pretty hard this line that the US should be renting compute to other countries rather than selling the chips itself. Basically just because there's I guess, a clear security win that if you're just renting access to them, if people are just basically buying access to the compute on the cloud, then you could always cut them off if they're using it for some nefarious purpose or just against US interests for any reason. Do you think that's a good argument?
- [52:00]
Helen Toner
I think it's pretty good. I think it comes back to this question of what are your objectives and how do you know if you've achieved them? So I think this is quite good. If you are trying to be able to continue to profit from Chinese companies using AI and you think you're going to be able to sort of monitor and shut off concerning uses, then I think that's great. If instead you're trying to maximally profit from the Chinese market, then probably you do want to sell instead. And if instead you're trying to maximally limit China's access, then you don't want to rent. So I tend to think that it's a good way of meeting the trade offs involved. But again, I think this debate is so lacking in agreement on objectives and how you know if your policy is working that it's a little bit hard to know if this will hit the spot for kind of different, different people involved.
- [52:54]
Rob Wiblin
What's the best argument against the rent don't sell approach? I mean, I think one thing is if the UK just couldn't buy any Nvidia chips, even allied countries of the US couldn't buy any chips, then they would feel in a very vulnerable position basically because they could always just get cut off. It's not really a security situation that any country would gladly accept. So I suppose that would create a lot of pressure or a lot of enthusiasm perhaps for just sourcing AI chips from anywhere else, including in particular, I guess China would be the main alternative source. I guess that could be a negative side effect that it might have.
- [53:25]
Helen Toner
Yeah, I mean, I think with a lot of the chip controls, a huge question is how much does it help the Chinese ecosystem? And that's just, I think, an empirical question that I haven't Seen very good answers to. So this is true for the original October 2022 controls is the big downside. The big thing you want to avoid is by preventing them from buying US chips, you supercharge Huawei, you supercharge smic, which is their best chip manufacturing company. And I think people who are against the chip controls often assume that preventing them from buying US chips will be a boon to their domestic industry. And people who are for the chip controls often assume that it won't help them that much because, for instance, and this is true, it was already a huge priority for the Chinese government to be supporting their domestic semiconductor ecosystem and pouring just hundreds of billions of dollars into trying to subsidize it. And they were already struggling. So I think similarly with the rent don't sell, it's just an unanswered question of how much does this actually help them indigenize their supply chains? How happy are different countries to just rent instead and say, that's totally fine, no problem versus do they then go looking elsewhere? And if they do go looking elsewhere, does that help? Or is it just so technologically difficult to reproduce this supply chain that it doesn't make much of a difference?
- [54:44]
Rob Wiblin
The US has approved the construction, I think, of big data centers with Nvidia chips in several Gulf countries. I think Saudi and uae, possibly Qatar as well. What are the pros and cons of that, in your mind?
- [54:55]
Helen Toner
Yeah, I think the deals that you're talking about are, as far as I understand, provisional or sort of early stage announced without the details being hammered out. So I think it will really depend on what the specifics of those deals turns out to be. The big pro is that these are countries, if you look at the different inputs to AI, if you think computing power is an important input, then what do you need for that? You need chips, you need land, like permitting, the permission to build, and you need energy. And so there's sort of a natural trade that you could do and say, look, in the US we have lots of chips. Permitting is a nightmare. Grid is struggling. But in the Gulf, they have plenty of land, they have plenty of sunlight, they have plenty of oil. And so we bring the chips over there, they let us build and they give us lots of energy. Great deal. I think that's sort of the main pro is being able to build out more computing capacity than you otherwise could. I think the cons depend a lot on the specifics of the deals. So there might not be that many cons. If it is, I've heard a comparison between U.S. data centers in the Gulf and U.S. military bases in the Gulf. It's like this is a US asset, it is US soil, it is fully under US control. Maybe there's not that many downsides if that's the case. But the more that the countries themselves have ownership rights over the chips or usage rights or ability to access the facilities, then there's sort of two big potential downsides. One is the connections with China that these countries have. So this includes doing joint military exercises or having very tight personal and commercial relationships with Chinese leaders and Chinese more political leaders and business leaders. Meaning does this help China Reverse engineered chips have more access to kind of advanced chips in a way we wouldn't want. The other big set of downsides is just specifically around the fact that these countries are autocracies. They are not like nice governments.
- [56:50]
Rob Wiblin
It might be worth elaborating a little bit on that. I read a blog post you wrote about this and I was shocked by the degree of repression that there is in the uae.
- [56:58]
Helen Toner
Yeah, yeah. So I think people tend to know that Saudi Arabia is not a democracy. Famous for not letting women drive, famous for hacking apart a journalist in Jamal Khashoggi in the Saudi Embassy, assassinating him in cold blood. Which is sort of emblematic of how they think about free speech and free press. The UAE though, I think people just have a bit of a sense of like, oh, Dubai and Abu Dhabi are kind of nice places to visit. It's a bit too hot, but they have big skyscrapers and fun indoor skiing or whatever. But the UAE is an autocratic country. I think the score on the Freedom House Democracy Index is something like 18 out of 100, which is really, really low. Political parties are banned. There isn't. There's sort of one body that's half elected and half appointed by the royal family, but it doesn't actually have formal power anyway. So there's kind of elections, but they're sort of fake elections. They do mass trials of dissidents that are sort of clearly not, not due process and actual real rule of law. They persecute the families of dissidents. The economy runs on immigrant labor, which those people have very, very few rights. So at a minimum they're sort of non citizen workers who have very little ability to participate politically. In the worst cases they are essentially forced labor. So this is really not a country that respects rule of law or that is interested in empowerment of its population. There's pretty strict rules around, quite strict rules around what the media can say about the royal family. It's a hereditary autocracy with a royal family that is going to stay in power. The shape of these deals is still up in the air. It's not clear exactly who gets what, but if you believe, as some of the leading companies making these deals have said, if you believe that access to compute is going to be a huge determinant of national power in the future, and the deals are structured in such a way that, that the royal families, the autocratic governments here, get access to essentially world class supercomputers, then that's pretty concerning because you're handing over a large amount of power to actors that are really not. Whose interests are not in line with the public generally or even with the US in terms of a long term strategic outlook where their priority is staying in power. Essentially, yeah.
- [59:25]
Rob Wiblin
So when I think one of these deals was announced, I think for the data center in the UAE, it was a collaboration, I guess with OpenAI was involved. And I think they put out a press release saying this is a huge win for democracy and democratic AI or something along those lines. How should we interpret that? Is this just kind of boundless cynicism or is there something else going on?
- [59:45]
Helen Toner
It certainly looks cynical to me. I think it's really playing fast and loose with what sort of democracy means, what democratic AI means. If you read sort of line by line, they never say that, they never imply that the UAE is a democracy, but they sort of talk about building on democratic rails or promoting democratic AI internationally. I think there's kind of two ways you could interpret this. One is, I think both could be reasonable in principle, but aren't really present in this deal. So one is that access to AI can be a democratic force for good in general. For example, if OpenAI were to make ChatGPT available freely in a country that previously had restrictions on speech or restrictions on access to information. I do think that having an AI system that you can ask questions of, that you can get information from, that you can learn things from, that you can use to further your own goals, I think that is obviously empowering of the individual in a way that is compatible with democracy, but it doesn't seem to be part of the deal with the uae. Shortly after that deal was announced, one of their executives was on stage and was asked by a journalist, a US journalist who had lived in the uae. She said, I've lived in the uae. There's these red lines of what media can and can't say, are you going to build those red lines into ChatGPT and the UAE and he was sort of like, oh, we'll see, which is not very encouraging. So this sort of first theory of change, of giving more people access to AI, and that being the sort of democratic, grassroots, empowering force, it doesn't seem like the case. The other one, which I think they are trying to sell more, is this idea that American AI is inherently more democratic than Chinese AI. And so anything that furthers American AI or helps America win against China is good for democracy. And I think again, that one in principle could make sense. And sort of coming back to this story of the big pro here is being able to build more data centers in a way that you couldn't do domestically. The problem is, on the one hand, again, if we think that the UAE government is going to be empowered by this deal, that they're going to have access to really sophisticated AI, then why exactly is that better than China having access, given that they are so autocratic? But secondly, also what are the parameters of the deal and what did the US have to give up in order to get this strategic benefit? So again, if it's a data center that is essentially like a US military base and fully US owned and controlled, and, you know, everything accruing to the us, great. But if, you know, a big thing that the companies will say is if the US doesn't sell it, then China will sell it. So the China is waiting outside the door. If the deal doesn't go through with the us, they'll just offer the exact same deal.
- [62:26]
Rob Wiblin
But China can't even make the chips.
- [62:27]
Helen Toner
Exactly, exactly. So it's totally disconnected from the reality of what they can do. And actually, I think this has been a recurring theme in the debates about chip controls and sort of US versus China capacity is really neglecting the difference between what are the specs that are announced versus what is their actual production capacity and what are their real specs. So just to close the thought on the UAE deal, one thing that was reported was the UAE getting hundreds of thousands of next generation Nvidia chips. And that is just an unbelievable amount of computing power that China absolutely cannot match. They are really struggling to meet their own domestic demand. There is no way that if that deal didn't go through, they could step through the door and offer the UAE the same thing. So I think that's just a really bad motive. Innovation. I think in the discussions about chip controls more generally, yeah, it's been really neglected. Like what number of chips are we talking about? What is China's capacity to manufacture them? And when they demonstrate manufacturing, how have they done that? So, for example, it's now pretty well demonstrated, pretty clear that one or two, I forget, generations of Huawei's chips, their most advanced chips, they've demonstrated this pretty impressive performance. And do you know the way they've done it? Have you heard this story?
- [63:33]
Rob Wiblin
No, I haven't heard this one.
- [63:34]
Helen Toner
Basically, they fooled TSMC into making chips for them. So this is this Taiwanese semiconductor manufacturing company that makes the most advanced chips. So this was after the controls were in place. TSMC was not allowed to manufacture chips for Huawei. Huawei set up a front company, sent in some chip designs that were clearly for Huawei chips. But TSMC just either turned a blind eye or didn't have the processes in place or something else and went ahead and manufactured a large number of chips or gave them dies, I forget the exact specifics, but basically handed them this giant stockpile of very advanced inputs for their chip making process. And so they are now kind of marketing and selling those chips and acting as though it's sort of their own sophistication.
- [64:15]
Rob Wiblin
But it's like, well, no, if anything it shows the opposite.
- [64:17]
Helen Toner
Exactly. So yeah, there's often these sort of headlines of oh my goodness, Huawei announced this chip. Huawei demonstrated this performance. Huawei manufactured this small run of phones with advanced chips in them that really isn't looking at like, wait, how do they make that? And how many are they going to be able to make in the future, which is what you need to do to try and understand and the actual sort of effects and trade offs with these controls.
- [64:36]
Rob Wiblin
So we almost always assume that frontier models are going to be developed and first deployed in the US or China, that those are the two leading countries? Is it crazy to think that however you want to define AGI superintelligence, it could first be trained in Saudi or the uae. I mean, I think they have a lot of disadvantages, but if they have access to an enormous amount of compute and these countries, I mean the benefit they have is they have just enormous stockpiles of cash basically that they could throw at this problem. Problem. They have a lot more kind of disposable income than a country like the US has. Maybe not as much as the US but being autocratic, they can direct money much more with far fewer constraints. Is it possible that superintelligence could be developed in Saudi first?
- [65:18]
Helen Toner
I think it's possible. I would say the UAE more than Saudi Arabia is my impression. So certainly by all accounts the UAE government is just very, so called AGI pilled, very into this idea that AGI is going to be a huge deal and, and that the way there is scaling up compute and things. So if that's true, then they're at an advantage because they already believe that. I think, as with all questions, AI depends on the timelines, depends on how long we're talking. I think in the next couple of years they would struggle to stand up a really competitive effort. But if we're talking a little bit longer scale, then potentially I think also they're very well positioned to benefit from the US rejecting high skilled engineers and scientists. They can really offer great compensation packages, great quality of life, and I'm sure that they are trying to do that, if not already doing that. So it's not what I would bet will happen, that the most advanced models will be developed in the Gulf, but I definitely think it's possible. And certainly if we continue to make these giant deals and really build up a huge amount of computing power there, then it becomes even more possible.
- [66:20]
Rob Wiblin
Do you have any proposals for how we could go through the transition to AGI, potentially superintelligence, without ending up with an absurd amount of power being concentrated in the hands of some companies, possibly some governments, possibly some individuals. Maybe you could explain for people why that feels like it could be a natural outcome.
- [66:38]
Helen Toner
Yeah, I mean, I think there's a couple of reasons why AI might be very power concentrating. The most obvious one is if it continues to be as capital intensive as it has been recently. So I think the clearest trend over the last 10, 15 years of AI development is this sort of compute scaling idea of the way to make progress is to have the biggest computing clusters and to spend increasingly gobsmacking amounts of money training individual AI systems. So if that continues to be true, then you're naturally going to have actors that have access to large amounts of money, whether that's very well capitalized companies or whether it's rich governments, they're going to be the ones developing the AI. Another way that it might come to be would be if you have a regulatory ecosystem that protects the top players so that there's some kind of very stringent requirements on how to develop AI. And that's very hard to break into. You know, have like a regulatory capture and there's a small number of players where maybe those regulations were introduced with good intentions. But if they end up concentrating power, then that's not great. And I think there's a natural tension here as well between, I think, among some people who are very concerned about existential risk from AI really bad outcomes, AI safety. There's this sense of well, it's actually helpful if there's only a smaller number of players because one, they can coordinate better. So maybe if racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or ten. And also a smaller number of players is going to be easier for an outside body to regulate. So if you just have a small number of companies that's going to be easier to regulate. So I think there's often a sense of, well actually that concentration is valuable. I see the logic there. But the problem is then sort of the. Then what? Question of if you do manage to avoid some of those worst case outcomes and then you have this incredibly powerful technology in the hands of a very small number of people. I think just historically that's been really bad. It's really bad when you have small groups that are very, very powerful and typically it doesn't result in good outcomes for, for the rest of the world and the rest of humanity. I guess that's all preface to your actual question of how do we avoid this? I don't really know. I would love for there to be more work on this. I think a lot of the thinking that has happened about this has been between people who say well the risks are really large and so we have to try for concentration because otherwise we all die and other people saying that's stupid, we're obviously not going to all die and so therefore we should just diffuse power maximally. And I think trying to get those conversations, those people to actually engage with each other and say like maybe there.
- [69:12]
Rob Wiblin
Is what if risks are medium then.
- [69:15]
Helen Toner
We'Re in a real or both risks are high. Right? Yeah. I think sometimes when I talk about this people think that I'm optimistic and I'm like it's all good, it'll be fine, just let there be less power concentration. But I actually think my take is a more pessimistic take of like well, I don't think concentrating is the solution and I don't think maximum diffusion is probably the solution. And so how do we navigate that middle ground? I think it's really hard. I think one answer might be we might get lucky in terms of how the technology develops. It might be the case that actually things develop relatively gradually. There's time for sort of fast followers to catch up and there can be relatively broad access to capabilities and there aren't these really decisive, huge civilizational downside risks that you'd need to Manage in a concentrated way. So I think we might just get lucky. That's sort of my best hope. I think there's also tools that I would love to see explored more that would target this sort of. So at a very basic level there's things like AI literacy, meaning how do we get a larger range of people to understand what is going on with this technology, to be able to engage with it, to be able to think about trade offs, pros and cons, risks and benefits. How do we think about worker empowerment or the role of workers, including workers at frontier companies, in shaping the development of the technology? There's also very basic things like taxing companies. So. So if the only problem is just, well, it's a naturally capital intensive technology and so large actors are going to build it and we don't have to worry about the downside risks, then you have sort of very traditional tools like tax antitrust that can come in and help try to diffuse that concentration of power as well. But that doesn't solve the safety challenges.
- [70:54]
Rob Wiblin
The concentration of power worries also I think point to some extent against the export controls or against being very intense on the rent don't sell of compute, because I would guess if almost all of the COMPUTE is concentrated in the United States and even Europe or Australia or the UK or basically no countries can get a foot in the door to have significant data centers. That I guess leaves them very vulnerable to exploitation by the United States or it puts the United States in a troublingly powerful position. Do you agree with that?
- [71:24]
Helen Toner
Yeah, I think that's probably directionally right. I haven't thought about it much, but yeah, interesting question point.
- [71:28]
Rob Wiblin
You wrote recently about how maybe one of the underlying disagreements in AI that people don't necessarily address directly is between people who favour kind of decentralization and dynamism and those who feel apprehensive about a dynamic open AI ecosystem. Yeah. Can you explain that idea?
- [71:45]
Helen Toner
Yeah. This was coming from a book that I read late last year by Virginia Postrel called the Future and Its Enemies, which is a great title. It's actually from the 90s. So it's sort of in the era of environmentalism and sort of cyberspace just becoming a thing. And I found it really helpful because it sort of for me, helped me put my finger on something that I think weirds people out about AI safety discourse that people in the AI safety community are often not as aware of, which is basically this instinct of if there are these big risks, then the solution is concentration. So the solution is there's this risk that that AI will kill all humans or this risk that AI will take over or something like that. And the solution is, well, we have to make sure that the right people build it and that it's a very small group. And I get the logic there, but there's sort of a lack of it.
- [72:34]
Rob Wiblin
Gets stuck with the creeps.
- [72:35]
Helen Toner
There's a lack of like, oh, there's sort of different ways you could do this. Right. Sometimes when I talk about this and also when I wrote about it, people come back and they're like, wait, so you think we can just not worry about that at all? Just let everyone build AI? It'll be totally fine. Why do you think that? And that's actually not my take. My take is more like we should be more worried, we should be more pessimistic about these factors. We have to balance. To put that in another way, I think an issue with the. I see the logic when people very concerned about existential risk from AI say we're going to have to concentrate in order to manage these risks. But then it's lacking the oh no, oh, we think we're going to have to concentrate access to this technology in order to manage this very severe risk. That's terrible. How do we handle that? How do we think about all the downsides of that? How do we make sure that it doesn't just end up in massive disempowerment of most of humanity? And I think a lot of people, especially in the tech world, but I think also more generally subscribe to what this book, pastoral's book would call a more dynamist view, meaning that a huge source of success, a huge source of, of human prosperity well being is having a world that allows for more trial and error, that allows for sort of decentralized experimentation, that allows for things to fail, that is more focused on sort of resilience and recovery from problems than on kind of preventing every problem in the first place. Because if you try to go with preventing every problem, then you end up with a much more authoritarian, centralized, what she calls stasist kind of stability, focused, control focused regime. And again, I don't mean to say, therefore we should just not worry about it and just let people do whatever they want. What I more mean to say is if we really think that these sort of more quote unquote stasist solutions of control and stability are really necessary, then we need to really be apprehensive about that and we need to be thinking really carefully about, okay, if we really, really think that it's necessary to concentrate power in this way? How do we think about how to deconcentrate it again at the end? Or what are the guardrails that are put in and how short are we? Like, what evidence can we gather about what the nature of different AI risks is and whether it is worth this trade off as opposed to I think sort of the vibe of the AI safety community, which can often be like, it's okay, the solution is just concentration, then we'll be fine.
- [74:51]
Rob Wiblin
I think that's shifted quite a bit in the last few years.
- [74:53]
Helen Toner
I think it has, yeah, I think it has. But I do think it is still a strong undercurrent in a lot of the thinking. I think there's been more discussion of risks of concentration of power and so on, but I think there's still a strong default of like the answer is just have one project and then you can manage the project.
- [75:08]
Rob Wiblin
Well, I think this is one of many other kind of assumptions that get baked in that go all the way back, I think, to the very. To the 2000s, when the picture of how this would play out was very rapid recursive self improvement loop based potentially on not that much compute thinking that is overwhelmingly the most likely way for things to go. So. So power will end up concentrated whether you like it or not. It's still possible you could end up with a very strong process of improvement loop. But I think people believe that much less than they did 15 years ago. But it's so hard to go back and remove all of the assumptions that were kind of baked into the language and just how you were thinking about the problem from the beginning. Yeah, yeah.
- [75:53]
Helen Toner
I mean this is like a sort of mini hobby horse of mine for the last couple of years has been. And personally, I think something that we really need in this space is people who are coming in who are willing to take these ideas seriously. So willing to think, like to engage with the idea of AGI, of superintelligence, of existential risk, of human extinction, of disempowerment, AI takeover, these sort of sci fi wacky risks. So people who are willing to engage with that, but who are not coming out of that sort of social and intellectual milieu that I think both you and I come out of. And that I think does contribute to sort of groupthink and sort of bubble dynamics. Not bubble as in a financial bubble, but being in a social bubble. And that's actually something that I really value at CSET is being around people who are mostly not from that world and who Mostly don't bring in all of those sort of, as you say, those sort of baked in assumptions that are hard to kind of go back and unbake. So yeah, personally I think that in this space it's just really, really helpful. And I sometimes meet people who are sort of getting interested in these issues and they're like, well, I haven't been thinking about them for as long as some other people. And I think that can actually be a real advantage because you're bringing fresh eyes, you're bringing sort of fresh perspectives. And the key part for me is, yeah, not shying away from being interested in these questions as opposed to sort of having a particular background in some particular set of answers.
- [77:07]
Rob Wiblin
I feel like we're in a very difficult spot because so many of the obvious solutions that you might have or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. And so, so what policies you favor and disfavor depends quite sensitively on the relative risk of these two things. The relative likelihood of things going negatively in one way versus the other way. And at least on the loss of control thing. People disagree so much on the likelihood. People who are similarly informed know about everything there is to know about this. They go all the way from thinking it's a 1 in 1,000 chance to it's a 1 in 2 chance 0.1% likelihood to 50% chance that we have some sort of catastrophic loss of control. And discussing it, it leads sometimes to some convergence. But people just have not converged on a common sense of how likely this outcome is. And so the people who think it's 50% likely that we have some catastrophic loss of control event, it's understandable that they think, well, we just have to make the best of it. Unfortunately, we have to concentrate it, it's the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with. But we have to bear that cost. And the people who think it's one in a thousand are going to say this is a terrible move that you're making thinking, because we're like accepting much more risk, we're creating much more risk than we're actually eliminating. I guess the idea would be if we could find policy responses that help with both simultaneously or help at least one of them without harming the other. I don't know how. I think people probably have underdone attempts to find win wins or at least win neutrals, but it's easier said than Done. It's not a simple matter.
- [78:32]
Helen Toner
No, definitely. I mean, I think there's sort of a set of policy recommendations that I do think can help help us navigate this a little bit. So these are things like everyone's favourite consensus recommendation is sort of more transparency, more disclosure from the AI companies. I think that is really helpful. I don't know. I guess my stance here is that it's very unlikely that there's these sort of two futures that we imagine and the future goes down one of those tracks. I think it's very likely there will be unexpected twists and turns, new things that developed that we didn't can anticipate. Technologies never develop exactly the way that we think they will. So I think our stance here should not just be kind of which of these two camps is right, but being open to and ready for many, many different possibilities. So yeah, I think transparency is helpful for that in that it lets information propagate about kind of what is actually going on, what is sort of the ground truth of what the systems are being developed, what systems are being developed, what risks they pose and so on. Yeah, this sort of AI literacy point or technical capacity in government is sort of a similar thing of, of making sure that you're hiring people or training people. So there's not just a very small group of experts inside the companies that are kind of making all these decisions, but having kind of a broader set of eyes on it and then sort of resilience focused approaches. So this is sort of cyber defense, biodefence, AI control type approaches, meaning how do you invest in building AI systems that can kind of monitor other AI systems and things like that. And one other piece that I think can be helpful as well is really investing in the science of AI. So people sometimes talk about, oh, the government should be funding AI safety because the companies are funding AI capabilities. There's actually a different thing that I'm trying to say with this is, which is I think we would be in a much better position if we had a more grounded understanding of how AI systems work and how do they generalize from their training data? Why are they so jagged? Why do they fail in these weird ways, is how could you understand whether a given system is likely to succeed or fail in a given setting without just trying it? Because right now we basically just have to try it. Interpretability is another kind of example of a sort of science of AI approach. And I think right now we're really under investing in the science of AI as opposed to sort of the engineering of making AI that is better, that can do more stuff. So I think those are some ideas that I think could be helpful. Sometimes I think the AI safety community can reject anything that is not a full solution. I feel like there's something about the community's background in mathematics or computer science or philosophy where they want like a full. The solution that you can prove is adequate. And if it's not that they want to reject it, I think that's really mistaken. I think we're going to have to kind of do our best, figure things out as we go. And I think investing resources now to try and position ourselves better so that we can respond better one year from now, five years from now, now is really valuable and is one of the few things that we can do. I think it's also worth thinking through what are sort of more elaborate plans for actual sort of full solutions. But even there, I mean, the idea of a full solution suggests that there is like a clear end state, which, coming back to this idea of dynamism, I think is the wrong way to think about it. I don't think there is a one best way, an optimal endpoint that we should be aiming for. Instead, I think we need to be trying to get to a world that is dynamic and open and free and empowering and that that's not going to look like sort of we fixed it. Instead, it's just going to look like maybe we got through an acute risk period or something like that.
- [81:59]
Rob Wiblin
Yeah. The desire for a full comprehensive solution is another one of these assumptions. I think that goes back to the Yakowski Miri vision. I guess we're talking, I think in the week of the release of if Anyone Builds It, Everyone Dies. It's in the title saying we really do need to solve this completely, comprehensively. Muddling through is that is definitely not going to work in their view, and that may yet be vindicated. But I think it's very far from obvious and on many plausible views, muddling through is an option. A bunch of half measures might well work. I think more and more people have appreciated that in recent years. I think I hear, at least, I think the people who want the full solution, they're doing it because they think think loss of control is overwhelmingly likely. The technical problems are overwhelmingly hard, rather than just because they've made that as an unquestioned assumption.
- [82:47]
Helen Toner
Yeah. And I think there's real like a big spectrum of how competently you muddle through. Right. So I think sometimes muddling through, I don't really love that phrase because it sort of Sounds like, oh, we'll just take it as it comes and sort of see and it'll be fine. I think a different version of muddling through, a more competent version is this idea of plans are useless. Planning is everything of like, like how do you set yourself up with lots of different possible levers to pull lots of different things on the shelf that you could take off the shelf if needed and also set yourself up with the kind of visibility and understanding to be able to tell what paths you seem to be on. Yeah, I think in a sense that's also muddling through because you're not kind of pulling the trigger ON here's our 12 step plan and we're going to go through it comprehensively. Yeah, but I think there's a big range of what it looks like to muddle through. I mean, arguably we muddled through with nuclear weapons, but that involved a massive international treaty and a huge monitoring apparatus and that's really valuable. And I think it clearly reduced the number of nukes in the world and the amount of nuclear risk in the world.
- [83:44]
Rob Wiblin
I think most of the technical research agendas are either win win or win neutral on this. I guess one that is a win win is the ability to inspect the models and detect hidden goals or hidden agendas that might have emerged unintentionally or have been deliberately inserted in there.
- [84:02]
Helen Toner
There.
- [84:02]
Rob Wiblin
That is really helpful, I think, for avoiding concentration of power, I guess. Tom Davidson in an episode earlier this year explained how this problem of secret loyalties does create a vulnerability for a power grab, basically. But it's also, I mean the same techniques might well just work for outing any unintentional goals that have ended up getting baked into the model through training that we didn't intend. I guess like model organisms, understanding AI misbehavior when it generates, when it doesn't, what things you can do to reduce it almost across the board. I think, I think funding for all of those sorts of research agendas does just seem like a real no brainer in my mind.
- [84:36]
Helen Toner
Yeah, I think that's right. I do think that there's maybe a different axis where they are sort of like win neutral as opposed to win win, which I've been thinking of this more and more as really central to disagreements about risk, which is this question of like, will AI just be a tool or will it be something more than a tool? This is sort of part of, but not the full disagreement with the AI as normal technology. Authors Arvind Ryan and Saesh Kapoor, who are both really excellent and I think are great examples of people who are taking these issues seriously and engaging with them seriously, but coming with a very different set of assumptions and I think really adding to the debate. I mean, I think that if AI stays as a tool, continues to just be something that people sort of use to their own ends, and maybe sometimes it breaks or fails, but it doesn't have its own agenda, it doesn't have its own goal goals, then you could still be very worried. Then it makes sense to be worried about concentration of power. All of the takeover stuff doesn't really come into play. And so I think that is another place where there's different assumptions, different expectations and different trade offs look better or worse to different views. But I think it's been really valuable to have that AI as normal technology view articulated because I think it is sort of a funny thing where I think they wrote it because they sort of see the super intelligence view as sort of dominant. But in fact what they articulated was a much more widespread unarticulated view. I think most people, when thinking about AI.
- [86:00]
Rob Wiblin
Surely overwhelmingly that's the most common approach.
- [86:02]
Helen Toner
Most people assume that it's going to be more like a regular technology, it's going to be a tool, we're going to decide how to use it. And this was explicitly.
- [86:09]
Rob Wiblin
Someone made the case that that's really what it is.
- [86:11]
Helen Toner
Yeah, yeah. And this was explicitly part of why they wrote that paper was to say we think this is a common view, we want to put words to it, we want to sort of describe the components of it. It.
- [86:19]
Rob Wiblin
Where do you come down? Is AI tool or is it a new form of life?
- [86:23]
Helen Toner
I don't know yet. I think I agree with some of the things they articulated. I think it was really useful to articulate the distinction between capability and power. So an AI system that is able to do something does not mean that it's actually set up in a way that it can do that thing. And we as humans have a lot of agency in deciding what AI has the power to do. I think sometimes they say at the beginning of the paper that it is, is both a description of the present, a prescription about how they think AI should be handled, and a forecast about how it will be handled. And I think those three things sometimes I agree with them that AI is a certain way, for example, that we are deploying it relatively cautiously at the moment, relatively slowly. And I agree that we probably should continue, especially in high stakes settings, to deploy it relatively slowly and cautiously. But I maybe disagree that we obviously will and in particular I think, and I know that they're working on this actively right now, they excluded military AI from that paper entirely. And they also included a section about, I guess another debate which I think is really underexplored, which is how superhuman can AI get in different areas. I would love to see more work on trying to pull apart what are the reasons to believe that there's a very high ceiling above human level at different capabilities. And one thing that Arvind and Sayash say in their paper is that, for example, being faster than humans is not that useful in many settings. It's useful in high speed trading, but not that many other things. For me, battlefield management is going to be. Their speed is absolutely a huge advantage. People in the military talk about the OODA loop, the observe, orient, decide and act loop. And you want to quote, get inside your opponent's OODA loop, meaning be faster than them, be able to drate around that loop faster anyway. So that's one example of an area where I think the AI as normal technology view is not necessarily accounting for the incentives to hand over power, hand over autonomy in order to get benefits like for example, speed.
- [88:17]
Rob Wiblin
You've written that you think the AI security ecosystem, it's often appropriated the language of non proliferation, I guess from nuclear weapons, from biological weapons. And you think that might have been a wrong turn or at least it's narrowed people's scope a bit and instead we should think about things in terms of adaptation buffers, which is a thing, I think the term that you introduced. Yeah. Can you explain why you think they might be a mistake and what adaptation buffers are instead?
- [88:38]
Helen Toner
Yeah. So this is specifically in response to concerns about misuse. So the idea being we're building these more powerful AI systems, they're going to be misusable in particular ways. Two of the most common people bring up are being used to create bioweapons or being used for cyber offense, so hacking and that the response is the AI systems are going to be so powerful they would enable such bad attacks that we have to do non proliferation, meaning prevent them from spreading. And I think the challenge with this is it won't work and it will be very invasive and very sort of authoritarian. And the reason for that is basically because of this very clear trend, this sort of maybe somewhat contradictory seeming trend to people who are not following AI closely of on the one hand, as we're building more advanced systems, the amount of computing power that you need the first time to build an especially advanced system is going up over time. So you need More and more and more compute to build something at the frontier. But as soon as you build something at the frontier, so say an AI system that is as good as a software engineering intern or an AI system that can get some score on some test or whatever, as soon as we build that for the first time time, the amount of computing power and also just broadly technical sophistication that you need to build, that is going to fall pretty rapidly over time. So for example, at the time when the Alexnet paper was published in 2012 by Alex Khrushchevsky and Ilya Sutskever, Jeff Hinton, that was very, very hard, very complicated, but now it's incredibly easy, incredibly cheap. Likewise for a GPT3 type model at the time, really at the cutting edge, quite expensive, very, very difficult now, very easy to recreate. What that means is for misuse. So for this idea that we'll have AI systems that are very powerful at hacking or very good at helping with bioweapons, maybe initially it'll be a small number of companies or research groups that can build a great model, but a couple of years later, five years later, it's going to be very easy for many groups to do that. And so in order to do non proliferation, you're going to have to get very, very invasive about, about restricting the number of chips that people have access to or monitoring what they're doing with AI systems. And I think it's just really not likely to actually work. If you have a determined actor, a bad actor who is trying to misuse these models, they will find a way to do it. And in the meantime you will be really restricting the ability of many, many normal people to use and build AI systems in ways that would be perfectly fine. And to me the sort of best alternative that we have is, yeah, I call it a sort of adaptation buffer, basically meaning this period, this buffer period between when we know that some capability of concern is going to be very accessible at some point in the future and when it's actually accessible. So I think we are in this period right now for AI systems that can help novices to create known bio threats, if that makes sense. So it's important in the bioweapons conversation, I think sometimes people, two threats actually get threat models get conflated, one being novices creating known bio threats. So this is like a teenager in their basement creating smallpox kind of thing. That's different from helping experts create much more sophisticated bio threats, which often is going to involve the use of biological design tools or it's sort of a different process. I think we're clearly in the adaptation buffer for that first one. And the thing to do then is to try and put as much in place, sort of societal resilience style efforts in place to prevent that capability from having massive negative impacts when it is widely available, as opposed to trying to sort of clamp down and prevent it from ever being widely available. So that's this idea of the buffer is that the time period between when we can forecast it pretty specifically, not just like bioweapons, something something, but something pretty specific and the time when we can get it. Now, I don't think this is a perfect approach. In particular, if buffers are short, then maybe we don't have enough time to get to them. And it's very much focused on misuse, not on kind of more misalignment focused threat models. But that's kind of the basic idea.
- [92:44]
Rob Wiblin
Yeah, I think people have not always been drawn to the adaptation buffer framework and I think it's not for no reason. It's often because they think there aren't any good adaptations, I guess, especially on bio. I think this has been the view that, that if it's easy for amateurs to create new pandemics, it's just so costly to combat them that we just have to stop them from being created in the first place, I guess. I recently did this interview with Andrew Snyder Beatty where he presented a plan. Maybe there are adaptations that we could make that would make society very resilient to any new pandemics that someone tried to release. It's a heavy lift though, it's challenging and it's not completely obvious that it might work. So again, I mean, it's possible that we could go down the adaptation buffer other track and then look back and say actually that was a bad idea because in fact the adaptations were just too difficult and non proliferation was unfortunately the only option.
- [93:31]
Helen Toner
I mean again, I think it's like my point here is not non proliferation doesn't sound fun. My point is I think non proliferation won't work. They will proliferate anyway. And I do think also I think people. Yeah, I mean, time will tell in this case, I guess. I hope that I'm right sometimes I'm talking about risks and hoping that I'm wrong. But time will tell. I think people sometimes overstate the inevitability of really severe misuse. And I mean, if you look at when I wrote this post, for example, about adaptation buffers, one of the commenters was saying like, well, isn't this all Hopeless. I remember in college when a professor of mine was telling us about how easy it is to just drive up to a water treatment facility with a dump truck and dump a bunch of toxic chemicals right after the treatment commitment ends. And someone else was saying to me that they'd seen some video in the 2000s of how you can, in an airport, how you can make a bomb using materials that you obtain after security. So you go through security and then make a bomb. But these examples, these examples are both great. They're very encouraging. No one to my knowledge has done, I don't know about the water treatment maybe that's happened or if they did it, then they got caught and it got prevented. So I think sometimes there's a little bit of a neglect of, of what actually are all the options? What is the toolkit? Including just like the FBI monitoring terrorist groups. That's a real thing. Or the CIA as well. That's a real thing that happens. Or looking at materials, looking at synthesis of nucleic acids, which is obviously part of the discussion, looking at can you just have layered approaches with the chatgpts of the world. They're going to refuse to help you. They're pretty good control on that right now. So that reduces the number of people. And then if you do eventually have kind of attackers actually really producing bioweapons. I don't know. I feel like sometimes in the existential risk community there's this assumption that it's going to be this absolutely horrific event as opposed to something that potentially can be contained earlier. So I don't know, I'm not unworried about this. I think it is really worth trying to prevent, trying to mitigate. But I think at a minimum, any concern that we do feel any risk management sort of energy and resources that we have for preventing bio attacks should be doing a lot of this resilience focused stuff as opposed to focusing primarily on clamping down on models. Again, just because I think clamping down on models is not going to actually work.
- [95:58]
Rob Wiblin
It's not a long term solution.
- [95:59]
Helen Toner
Right. And so saying the adaptation buffers might fail. I agree, but I think saying and therefore we should just focus on preventing model proliferation is the wrong call.
- [96:10]
Rob Wiblin
You were recently involved in a CSET report titled AI for Military Decision Making Harnessing the Advantages and Avoiding the Risks. How are you worried that the military is going to end up deploying AI Poorly.
- [96:21]
Helen Toner
So this report was about a specific kind of AI called Decision Support systems. And the reason, or one reason that we wrote about it was because it's something, I think there's sometimes a bit of a gap between kind of external perceptions of military AI and what the military is actually looking at at in practice. And one, for example, I think externally people really focus on lethal autonomous weapons, especially drones with facial recognition, I think is like a really compelling use case that the public worries about. But if you look internally, there's many, many other potential use cases that the military is thinking hard about and where how they adopt them, how quickly they adopt them will matter a lot for what actually ends up happening. And so this paper was actually focused on a specific kind of system called decision support systems. And what we wanted to do there was really not get into this kind of binary of oh, AI in the military is bad and should be banned and we shouldn't do it, or AI in the military is essential and we should just be racing to adopt it as quickly as possible. But instead to say, take a clear eyed look at what are these systems, why do militaries want to use them and how can they use them effectively and not get caught up in potential pitfalls of AI systems? So that involves looking at what are the actual benefits of using decision support systems. It depends a little bit on the specifics of the system and what are the potential pitfalls. So do you have a clear understanding of the scope of a particular decision support system, what it was trained for or where might it sort of be out of distribution, so in a situation it wasn't trained for and fail, or how do you train your operators so that they can use them effectively? So that was kind of the set of issues that we were exploring there.
- [98:06]
Rob Wiblin
How much appetite is there for rapidly integrating AI into the military? I interviewed Dean Ball yesterday and I think I was putting to him, might we see premature applications of AI and the military dangerous ones. And he was like, it's going to be so hard to do it at all. It's actually probably going to happen very slowly and in a very hodgepodge fashion. Yeah. What do you think?
- [98:25]
Helen Toner
Yeah, I tend to agree. I think there is a lot of appetite, but institutionally the ability to ProCure or build AI systems and then roll them out at scale is tough. We have another paper at CSET that I wasn't involved in called Building the Tech Coalition, which is a case study of a sort of successful adoption of AI. And really what it's the exception, not the rule, that they got this system to an operational state where it's actually being used in practice. And the case study is looking at what are the factors that actually made them able to succeed in that case. And one of the key parts there was having the military sometimes talks about having bilingual leaders who are competent both in sort of technology and also military operations. And one thing we really identified in that case study was you actually need trilingual leaders who are are competent in technology and military operations. And also these kind of acquisition, procurement, kind of questions, contracting, getting through all the legal language. So it's tough. There's a lot of barriers. The military is not set up the way that it does research and development, is not designed for software the way that it does testing, is not designed for non deterministic AI systems. So I think the appetite is very much there. But I tend to agree with Dean that in practice it's going to be slow and piecemeal and a slog.
- [99:44]
Rob Wiblin
Do you have any read on how worried the military is about AI being backdoored or having secret loyalties or agendas?
- [99:51]
Helen Toner
I think they're most worried about that the military is naturally set up to think about adversaries. Yeah, sabotage, Right, right, sabotage, exactly. So I think they're certainly worried about that. In the context of what an adversary, whether it's China or a different potential adversary could do, that's certainly a reason not to use Chinese models or to be very cautious. I think about even using US models that are trained on the broad Internet. There's already evidence of Russian groups, for example, doing essentially a version of data poisoning, trying to seed online data sets with pro Russian views. So I think that's primarily the lens that they're thinking about it through.
- [100:31]
Rob Wiblin
Do you hear any discussion of this question of if you're having AI operated military equipment, should it decide whether to accept orders based on what it thinks the law of war is, or should it just follow orders of whoever its operator is? I guess each of them has its issues. If your tank is having to make independent judgments about the law of war, about military law, maybe it's not equipped to do that. And that also creates a lot of vulnerabilities, I guess, that adversaries could try to use that against you. On the other hand, if your equipment just absolutely follows any instructions that it's given whatsoever, that creates a lot of opportunities, I guess, for coups where previously you just wouldn't have been able to get human collaborators to go along with it. Do you have any thoughts on this?
- [101:14]
Helen Toner
I think mostly this is sort of only relevant in as much as you're thinking of AI as very much not being a tool. So you're Thinking of it as being kind of having its own agency making its own decisions. And I think the discussions in military circles are very focused on AI tools.
- [101:29]
Rob Wiblin
They're just not thinking about independent agents yet.
- [101:31]
Helen Toner
I think that's just not really a part of the discussion yet.
- [101:34]
Rob Wiblin
Is it because it's unacceptable or because it's technologically not feasible yet?
- [101:37]
Helen Toner
Yeah, I think it's sort of too sci fi for the military at this point and also based on sort of where they're at with adoption, the level of sophistication of the tools that they are looking at. I will say that when these kind of topics come up, there is kind of a natural, to the extent that AI is operating in a more sort of agentic, independent, autonomous way that is more equivalent to a human operator. There is a whole set of institutional expectations, standards, rules, laws for military personnel that you could in theory port over to an AI. So, for example, lower level service members are expected to follow the commands of their commanding officers, but they are supposed to, not if the command is illegal, but also things that they do that go poorly, that does then reflect back up on the commander. So there's ongoing questions about, about how does accountability and responsibility for the use of AI systems flow back through the command chain so that if something goes wrong, who is held accountable? Which can actually work if the AI is primarily a tool and can potentially also work if it's sort of operating in a less tool like way. But yeah, I think the conversations about AI that is not a tool are pretty nascent. Yeah.
- [102:50]
Rob Wiblin
Yet to come. Yeah, you recently wrote that we should maybe stop using the term AI alignment almost at all and instead talk about AI steerability. I tweeted this out recently and it got quite a bit of play. It seems like a lot of people are on board with this in principle, although I suspect it won't happen because terms are just very sticky. If you're used to saying something, it's so hard to stop saying it. Can you explain why AI steerability is better?
- [103:09]
Helen Toner
Yeah, I mean, I think it's better in some ways. Since publishing the post, I've maybe come around to also thinking it's also not quite right. And yeah, I would never expect that. I don't think we can get the word alignment out of our vocabulary at this point. So I was more making the post for people who are, are trying to explain it or want an alternate sort of lens on it. The basic idea is, I think the term alignment sort of embeds in the word this idea that there is sort of a clear thing to be aligned to. So, like, if we think of alignment, we think of like aligning the tyres of a car where they're supposed to be straight. We think of like, you know, aligning a picture on a wall where it's supposed to be like, you know, perpendicular to the whatever. And so I think if you start talking about alignment, people's minds just very naturally go to aligning to what? Like, what is. What is the standard here? Which I think is an important question, but is getting ahead of itself where the underlying problem is more one of. Now, AI control means something else, but it's more one of controllability or steerability in the sense of can you actually direct the AI, can you build it in such a way that it behaves how you want at all in the first place? So I think steerability is the AI steerable conveys this better. The challenge with steerability that I've kind of come around to thinking is more of a problem than I did when I wrote the post is it doesn't distinguish between steerability at the development phase of, like, when you're training the AI, when you're setting it up, can you steer it to be the kind of AI system that you want versus when it's in operation, can you go in and intervene and steer it in sort of ways that you like? So I think some people who've read that post thought that by steerability, I was talking about, for example, when you're with a chatbot and you can can chatting with a chatbot and you can sort of steer it to roleplay in one way or another, it will go with you and it will sort of follow your lead in lots of different ways, which the current chatbots are very good at, and which is kind of a different thing than, for example, ensuring that it can't be jailbroken, which is something that we're less good at. So, yeah, I don't know. I think overall, I think that the idea of, well, we actually don't know how to steer AI systems in the first place, or we're not very good at it is, is a more intuitive way to put the problem then we don't know how to align them. Yeah. But I think no term is perfect.
- [105:20]
Rob Wiblin
My impression is that there's been a big shift in AI discourse in Congress over the last year or two, I guess. Again, every week or every week or two, I feel like I see new congressional hearings where it feels like either members of the House, Representatives or Senators are asking questions that feel like they're almost out of the pages of some Miri report, or they have a very, very Yudkowskian energy to them. Were they very worried about superintelligence or saying, you're planning to build superintelligence, how is that going to go? Might we not lose control? Am I just getting a kind of biased selection of things? Or has there been a bit of a. At least some people starting to get more open to that worry?
- [105:58]
Helen Toner
Yeah, I think the really huge shift that I saw was after the release of ChatGPT. So ChatGPT came out late 2022 and in 2023 and 2024 there was definitely a huge uptick of, you could say AI being the main character on Capitol Hill or being a huge topic of discussion in a way that really hadn't been before. It is interesting, I think you're right that over the Last sort of six or 12 months there has been more discussion of more of the superintelligence type concerns. I don't know exactly why that is. I wonder if part of it is just a sort of. Of a feature of the discourse getting a little bit more sophisticated. I do think in 2023 there was just this basic getting up to speed of what is AI? What is going on with AI? How should we think about AI at all in the first place that a lot of people, not just members of Congress, a lot of members of the public needed to do. And now, I mean, even things like, I think our ability to think and talk about AI has just gotten more sophisticated over the past few years. Even things like AI 2027 and AI as normal technology being published, these two sort of different visions is very helpful for kind of crystallizing and making concrete different kinds of ideas and different kinds of disagreements. So I guess that would be my guess. At the same time, I guess we've also seen some of the CEOs getting more explicit about the level of disruption they expect. So certainly Dario Amade at Anthropic has been been outspoken about job loss and things like that. Sam Altman at OpenAI has written that they think they're on a path to superintelligence. And so I think there's probably also a sort of normalization and an ability to ask about these topics in a way that wasn't there when these sort of high profile figures hadn't kind of talked about it in their own words.
- [107:44]
Rob Wiblin
Some people have had the impression that AI progress is slowing down. At least some people have had the impression that GPT5 was a bit off track. It was kind of disappointing. Do you think it is slowing down?
- [107:53]
Helen Toner
I don't really buy it. It's really hard to say because our metrics are so bad, we're really bad at measuring progress in AI or knowing what it is that we're even trying to measure. That being said, I sometimes think that it often seems to me like people who started paying attention to AI after ChatGPT, their sort of subjective impression of kind of what's going on in AI is like sort of nothing was really happening. This my little chart with an X axis of time and the Y axis of how good is AI, nothing's really happening. And then suddenly chatgpt like big leap. And so for those people that was pretty dramatic, pretty alarming. And the question was, okay, are we going to see another big leap in the next couple of years? And we haven't. And so for people whose expectations were set up that way, it looks like, oh, okay, it was just this one off big thing and now back to normal, nothing to see here. I think for people who've been following the space for longer, it's been clearly this sort of pretty steady upward climb of increasing sophistication and increasing ways. And I think if you've been following that trend that seems to have been continuing basically since 2012, when deep learning really started to work, there haven't really been huge breakthroughs. The biggest ones are things like the transformer in 2017. But that was a new architecture of deep learning. Yes, it was a really big deal, but also was kind of building on LSTMs and RNNs and this like longer tradition or making Internet scale pre training work was like that's a big deal. But it's not like a huge breakthrough, it's not a big paradigm shift. And so I think we've kept seeing this level. Reasoning models coming out is another example of in one way very big change. In another way it's sort of just building on the paradigm we had. So my kind of zoomed out view is that it looks to me like GPT5 is sort of continuing on that trend. And I think that trend, to be clear, is like, like one that should make our eyes go wide and we should really pay attention to is like, wow, this is a huge amount of technological progress happening in a very short time. It's just not the insanely dramatic level that people might expect if they sort of just noticed the ChatGPT one. And we also don't know for any given step on that trajectory, we don't know sort of what Real world effects will result. So I tend to think that it doesn't really seem to be slowing down. It also doesn't seem to be obviously speeding up. So I think also you saw some commentary from people at the start of this year, maybe when the reasoning models were sort of being demonstrated and you had that first kind of 01 to 03 jump. I think some people said, oh my goodness, we're going to have really, really advanced systems by the end of 2025, end of 2026, even more so. And that hasn't been borne out. So I do think it makes sense to kind of shave some of the sort of fastest progress scenarios off the top of your confidence interval. But that doesn't mean that things are just sort of plateauing or hitting a wall.
- [110:35]
Rob Wiblin
Yeah, I think people are getting a little bit tricked because the releases come so much more frequently now that often a release can feel a bit disappointing or only incremental. But you got to remember the last model came out maybe only three months ago. It used to be like more periodic releases. The jumps were much bigger, much more stark.
- [110:53]
Helen Toner
Yeah, I think that's right. People have sort of seen there was GPT4 and then sort of 4.0 and 4.1 and 4.5 and different sort of updates to 4.0 and, and that's just in the OpenAI space, let alone the different kind of Google Gemini and the Claude versions. So yeah, it's sort of more of a like drip, drip, drip epoch had a really good chart, you've probably seen this of taking some sort of illustrative benchmarks for the GPT3 to 4 and GPT4 to 5 and they're different benchmarks because they basically saturate but showing like in both cases there were really significant jumps on sort of key benchmarks of the time. And it's only because if you're comparing GPT5 to whatever came out six months ago or three months ago, that's what makes it look less impressive.
- [111:33]
Rob Wiblin
Like you said, you wrote in your notes that there's this funny phenomenon that there are multiple different camps that have self consistent, very different worldviews about AI and they both feel like they can explain satisfactorily kind of all of the empirical observations that we're making about how things are progressing. Explain how that can be.
- [111:51]
Helen Toner
Yeah, so this came out of a workshop we ran at cset, which we're going to have a paper about hopefully in the next couple of months on automating AI R&D. So this is sort of Ideas related to intelligence explosion or recursive self improvement or at what point are the AIs the ones doing the AI research and things maybe take off or maybe don't. And something that was really interesting at that workshop, we had people with pretty different views there and it was surprisingly difficult to get them to come up with different predictions for what would happen in the lead up to potentially very automated research. And part of why this is, is that these sort of different worldviews I think have good ways of explaining contrary evidence and why it's going to converge back to their expectations. So to go one by one, a worldview that is expecting that you're not going to get very superhuman systems. You're not going to be able to fully automate things, you're not going to be able to have this sort of self reinforcing dynamics. That worldview expects multiple things, it expects bottlenecks to arise. So if one thing gets accelerated, then it'll get sort of held back by some other factor that can't accelerate the same way expecting plateaus on ability. So maybe the AI system gets much better, but it can only get so much better. It can only hit a certain level of biology skill or coding skill or something like that. And so for that worldview, if you see evidence that things are going quickly or that things are becoming more automated, your expectation is just okay, so they're going to hit the limits quicker. So sure it's going faster now, but we know that the limits are there and so it's just going to hit them faster. On the flip side, people who expect there to be these really dramatic rates of increase in AI automation or AI getting much far superhuman at different things, when they see sort of bumps along the road or hiccups or difficulties, they are often able to explain them in terms of, oh, this will just lead to more speed later. So like an example that I've heard recently is if you look at the different kind of agents, have you seen the Agent Village? I love this thing. So if you look at. So the Agent Village is this nice demo of different agents being put in a computer use environment and asked to do some tasks. It's very fun. I highly recommend for anyone who hasn't seen it. But if you look at them, they have a lot of the agents in question, which is some of the best models. Gemini 2.5 and they must be doing with GPT5 at this point and Claude Claude 4 they struggle with some really basic computer use tasks like clicking an interface or figuring out that something isn't working. And someone who has a view of well, everything's going to go very fast looks at something like that and instead of saying like, oh wow, actually the models are less good than I thought. Sometimes they'll say like, well, that just shows that the UI that the models are being presented with isn't very good or they haven't been trained enough on how to get the pixels right for their clicking or whatever. But once they get that, then we'll have even more speed, speed up. Or similarly, you can see this with compute of like, well, if they're going to be limited by compute, then that just means that once we do build more compute, they're going to really take off. So it's kind of disheartening because that sort of means that you have less. It's actually surprisingly difficult to get some kind of agreement on if there's some potential point where things might get totally insane or might just not.
- [115:03]
Rob Wiblin
What would we see in the lead up?
- [115:04]
Helen Toner
What would we see in the lead up? Are we going to see anything that's actually going to distinguish? So yeah, I think that there are still some ideas, but it's almost like there's sort of basins that both views are drawn to and even if they see a little bit of a roll towards the other basin, they just have such a heavy gravitational pull towards their own view in ways that I think are fully self consistent that it makes it really hard to distinguish.
- [115:29]
Rob Wiblin
So I think around October last year, OpenAI said that it was going to maybe kind of spin off its non profit and basically disempower the nonprofit, not allow it to have much control over the business anymore. I think in March this year they did a seeming about phase where they said no, the nonprofit is going to retain control. But I guess I did an episode around that time with Tyler Whitmer where we explained that, or at least he explained that he was pretty skeptical and he thought in fact the nonprofit was going to lose substantial control, maybe like almost all significant control. Do you have any thoughts now on OpenAI's restructure efforts, what we should think of them?
- [116:02]
Helen Toner
Yeah, I mean I haven't followed this as closely as Tyler has a couple of thoughts. So I think it was really good that they made that sort of seeming about face. I think it was clearly the result of pressure from sort of external groups. There's various sort of now like a mini ecosystem of OpenAI watchdog groups and also pressure from the attorneys general, which is really appropriate because the attorneys general are sort of of legally the Parties that are responsible for ensuring that a nonprofit carries out its nonprofit mission. I think there's still a lot of open questions about what that restructure is going to look like. And I'm not particularly encouraged by the most recent update they've put out about the nonprofit getting a 20% stake in the PBC, the for profit. So I think there's very open questions. Yeah, there's very open questions about they say the nonprofit will retain control, but there's a wide range of possibilities for what that might look like. And a key feature of the current setup is that obligations to the nonprofit mission are sort of the primary legal obligation on board members who are ultimately responsible for the operations of the whole company. And I think trying to retain that setup, where the nonprofit mission, which is to ensure that AGI benefits all of humanity, that that remains as sort of the primary legal obligation, as opposed to being something that a small fraction of shareholders are pursuing or something that the typical way a pbc, a public benefit corporation works is that the PBC is allowed to pursue both missions. It can maximize profit or it can choose to pursue a socially beneficial mission, which is just much, much, much weaker. So this question of what does it mean for the nonprofit to retain control, I just really hope that the attorneys general, when they're looking at this, are thinking really hard about how it can be that so many of the same people are on both sides of this deal. So in particular the board members, perhaps especially Sam Altman, they're negotiating with themselves essentially for what is the nonprofit retaining, who ends up with board seats, who ends up with equity. I think it's a really tough transaction to do at arm's length, which is legally required.
- [118:14]
Rob Wiblin
Right. It's meant to be an arm's length.
- [118:15]
Helen Toner
Transaction, as I understand it. Yeah, yeah. And so I just hope that the attorneys general are really looking very closely at what the company is telling them about who is recused and who is involved in decision making and how they're ensuring that the nonprofit's interests are actually truly being represented.
- [118:28]
Rob Wiblin
Yeah, I mean, I guess being more familiar with UK charity law, it's kind of mind blowing to me that kind of the effective trustees of this charitable organization are selling the assets of that organization to themselves. I almost don't know what more to say than that, but it's extraordinary to me that, that you could in any sense be on both sides of a transaction like that in the uk. I think it would never be permitted.
- [118:52]
Helen Toner
I mean, we don't know what is going to be permitted yet. They haven't got final approval. So yeah, we'll wait and see.
- [118:58]
Rob Wiblin
An audience overridden with this question. I've heard Tona described as extremely normal in a complementary way. Is this true? And if so, how does she manage being normal around so many non normal people? I'm not sure who the non normal people are being referenced here. I guess maybe me. I don't know.
- [119:12]
Helen Toner
It's very challenging being normal around you, Rob. I don't know if I'm normal. I don't think of myself as very normal. I do think of myself as enjoying translating between groups or something both in a literal way of. I've always loved languages. I speak German at home with my husband, who's German, and our kids. I loved learning Chinese when I lived in Beijing. But also in a more metaphorical way, translating between the weirdo existential risk community in the Bay Area and the more sort of staid national security community in dc. I find it really fun to try and understand what are the different perspectives coming in here. What are the assumptions? What are the social norms? So I think that is more how I would describe it. Definitely there are ways that I feel. Yeah, I guess I don't know exactly what is meant by normal versus weird. I certainly identify as at least somewhat weird, depending on the social context I'm in.
- [120:14]
Rob Wiblin
I guess in the Bay Area there's degrees of weird. And I think you're not weird by Bay Area standards.
- [120:19]
Helen Toner
No, that's right. But I really loved living there and being in a social setting where weirdness was totally accepted and not judged. That's one of the things that I miss about living in the Bay. For sure, sure. Actually, one other thing that makes me think of is probably people listening who've heard me speak before are confused by my accent, which happens a lot. Maybe related to the translator thing. So I'm originally from Australia, as you know, as are you. And usually I sound American when I'm speaking with Americans. Basically, at some point, after living in the US for multiple years, it just felt both sort of more natural and more fun to speak with an American accent. But when I'm around Australians, it feels really artificial to keep speaking an American acc way.
- [121:04]
Rob Wiblin
So are you conscious of your accent changing? Because I think when I talk to Australians, my Australian accent comes out much more at this point. I've lived in Britain for a long time and I think my Australian accent has really weakened, but I think it probably has come out a little bit more in this conversation than usual. But I'm not aware of That, I.
- [121:17]
Helen Toner
Mean, if I listen to it, then I can hear it. It's like partially, I would say it's somewhat conscious and it depends on the setting that I'm in. But yeah, I mean, I think for some people. Some people seem to find it weird or I think some people really identify very closely with their accent as being a key part of who they are. And I've just never really felt that way. I mean, maybe it's like as a kid I enjoyed putting on different accents and learning about them. I think also these days, because I am speaking German every day at home, I'm code switching anyway, so it's not like there's my real me and then there's fake something. It really just feels like there's just different ways my voice can sound. But yeah, I definitely get a lot of people being like, wait, what's going on with your accent?
- [121:59]
Rob Wiblin
So do people immediately know that you're Australian? Just in normal life in the US.
- [122:06]
Helen Toner
If I sound American, then no.
- [122:07]
Rob Wiblin
No. Okay.
- [122:08]
Helen Toner
Yeah.
- [122:08]
Rob Wiblin
Interesting. Yeah. Do people usually. Are they usually able to give up their accent to that extent or change their accent to that extent?
- [122:14]
Helen Toner
I don't know. I mean, when I speak German, people also tell me that I sound German. So, yeah, I don't know. I feel like there's a wide range. Like another, you know, a more similar thing that I think quite a lot of people do is people Americans from the south will pretty often sound more southern around family and friends. And then when they're talking to northerners, they sort of go towards more of a standard American accent. So I think there's lots of versions that different, I don't know, communities or cultures handle it differently.
- [122:42]
Rob Wiblin
Yeah. Another audience question was, this field is changing so rapidly, both via technological improvements and via shifting global geopolitical relations that affect the semiconductor supply chain. How do you keep up with all the stuff that's going on?
- [122:53]
Helen Toner
I mean, it's a challenge. This is my full time job and it's hard to keep up with things. I do think that Twitter X is still quite useful as a source of news.
- [123:05]
Rob Wiblin
Better or worse.
- [123:05]
Helen Toner
I have a set of substacks that I get in my inbox that I try to read some of them, I try really hard to read every post. Other ones I just sort of skim and carry on. And you can go to my substack, so helantona.substack.com, there I have a kind of list of the recommendations of different subs that I find really useful. If you want a Lighter touch, way to catch up. Then CSET publishes a monthly newsletter about kind of the biggest AI news, which I actually find helpful as sort of a roundup because it's like a fire hose day to day and then once a month it's like, okay, here are actually the big stories that really matter.
- [123:41]
Rob Wiblin
I guess. I think Transformer has a weekly update on Friday. I think I could probably recommend that one. You want like a once weekly sense of what's going on.
- [123:49]
Helen Toner
But I think it's hard and I think I mean something that I've kind of accepted working in the space that I work in is that I'm in the intersection of many different fields. So there's AI, there's national security, there's China, there's US politics, US policy more broadly, big tech regulation, platform regulation, social media, economic. And so I've just accepted that I'm never going to have the, I'm never going to be an expert on every single thing that touches my work. And so it's just always a matter, there's never perfect. I'm never like, oh, I have to stay up to speed. Am I staying up to speed, yes or no? It's always just a matter of prioritisation and trying to choose where to put emphasis and where to make sure that I'm sort of spending time, work in progress.
- [124:31]
Rob Wiblin
How do you keep up to date or try to have some sense of what's going on in China? It seems particularly hard to gauge.
- [124:36]
Helen Toner
Yeah, I mean, so a few different answers. I do think the best substack that I know of on this is chinatalk by Jordan Schneider, which is really good. CSAT actually has a tool called Scout. So if you go to our public data platform which is called the emerging tech observatory ETO Tech, there's a tool called Scout which does auto translated and summarized but then sort of human curated news from China which is very helpful. And then yeah, lots of other sort of miscellaneous sources. I do think it's helpful. One thing I will say, if you're interested in China and AI, I think it's really worth sort of laying some foundations of China and non AI topics. So whether that's like reading the Beautiful country in the Middle Kingdom, which is a sort of comprehensive history of the US China relationship, or learning about the history of the Communist Party, or looking at some of the history of sort of tech development and the economic development in China generally since the 80s, I think it's really valuable. I think people sometimes come in to this topic sort of purely from the AI angle. And only know about China inasmuch as it relates to AI. And I think that leaves you with a bit of a shallow foundation. So I think it's also. I also try to not always just be keeping up with the news, but sort of going back and reading the fundamentals as well.
- [125:52]
Rob Wiblin
You were part of the founding team for CSET and I guess you're leading it now. What's your vision for what CSET can uniquely do or uniquely add to, I guess, policy understanding and dc?
- [126:02]
Helen Toner
Yeah, I think it's a much more crowded space now than it was when we started and I want to sort of partially claim a little bit of credit for that, that we were early to saying having this in depth, technically informed analysis of emerging tech and national security matters. I think a lot of other organizations have kind of got on that bandwagon, which is great and really good, really positive for the world. I think what CSET can still offer uniquely is a combination of being really sort of independent, neutral, vigorous, so not coming in with a particular point of view that we're trying to push, but really just trying to, to shed light on what is going on in reality. Kind of calling balls and strikes. I think our data team is a huge asset. So this is a set of data scientists and engineers and sort of data focused researchers who help our research analysts, research fellows, senior fellows, to make use of different kinds of data sets for different kinds of research questions. But so things like looking at the semiconductor supply chain, for example, we have on that same public data observatory at ETO Tech, we have an interactive tool that lets you look at all these different parts of the semiconductor supply chain, for example, and which companies and countries are involved in which parts of that very, very complicated, very, very large supply chain. Or looking at translation and having not just machine translation is really good. We use a lot of machine translation, but for important documents it's also very helpful to have a human editor go through and make sure that they, those key terms are translated appropriately. That all the weird CCP jargon is like using the correct terms so that you can connect it to past documents. So I think that is a big asset as well. And yeah, having a team that is focused on AI and emerging tech and not just sort of a small part of a larger think tank, but really having the whole organization dedicated to those questions.
- [127:53]
Rob Wiblin
Yeah, the conversation is more crowded now. I feel like unfortunately there's more and more actors who are being paid by vested interests to push a particular line, to push a particular agenda. I guess you can't blame tech companies for having an interest in what's going on in D.C. and hiring lobbyists to make the case for the things that they want. I mean, I guess one that stands out to me is Nvidia's lobbying around the export controls. They've paid people to say some sensible things. They've paid people to, to say some things I think are extremely not sensible and not always true either. Do you feel like the conversation is getting a bit degraded or a little bit corrupted by the fact that there's now people with clearer agendas, a clearer ideology as well, who are trying to have their way with things?
- [128:34]
Helen Toner
I mean, I definitely think you're seeing that the companies are staffing up and they are becoming increasingly active on AI policy topics. And I think that has pretty predictable effects in terms of, of pulling things towards the company's interests. So, yes, one counterpoint I would give is I don't want to as we record this, this is not yet finalized, but in California there's this bill called SB 53, which I think is kind of an astonishing triumph of sensible policymaking over distorted rhetoric. This is the Successor Bill to SB 1047, which was enormously controversial last year, subject of huge amounts of rage and fear mongering and hate. And what happened in the aftermath of 1047 was that Governor Gavin Newsom vetoed it, but set up a commission to sort of study what would be the appropriate way to regulate and how concerned to be about the kinds of issues that SB 1047 was designed to help with and created an expert commission. The expert commission wrote a report that was, was really high quality And Scott Weiner, the California senator who put up SP1047 last year, put up a new bill that was drawing heavily on this report and has gotten a lot of support and a lot more consensus and it's passed through the legislature. And as far as I know, Governor Newsom is sort of indicating that he will sign it. Don't want to get ahead of the.
- [130:03]
Rob Wiblin
Game there, but it's been massively less controversial anyway.
- [130:06]
Helen Toner
It's been massively less controversial and I think we'll be certainly a smaller step than SB 1047 would have been in both good and bad ways. I think that the potential ways that it could go wrong are much lower as well as potential benefits if 1047 had really worked out the way that its supporters hoped it would also lower. But I don't know. I put that out there as an example of I think there is much more lobbying in the space. There is much More sort of corporate influence. And that is probably going to stick around. But there are still ways that kind of good policy can be made. And I mean, I do also think of that as a big value add that CSAC can provide. Is trying to. Often policymakers talk to companies because the companies can tell them what is real and what is realistic and how the technology actually works. And you don't want to do a Europe GDPR of creating all these rules that have all these unintended side effects because they're disconnected from how the technology actually works works. So it makes sense that they want to talk to industry lobbyists if they're going to have sort of that real talk for them. And that is a place that CSEC can step in as well and organizations like us and have that technical depth to be able to go in and say here's what's realistic, here's potential unintended side effects, but not be sort of ultimately promoting the company's interests instead having a broader public interest in mind.
- [131:24]
Rob Wiblin
Do you have to worry about CSAT's research being influenced by the opinions of its funders and maybe wanting to. To pander to them too much?
- [131:33]
Helen Toner
I mean, I think any organization that every organization has to pay attention to that. Yeah. Is dependent on something. We work really hard to bring in funding that is sort of compatible with our model of following the evidence, where it goes, being independent, not pursuing a certain agenda. So when we take industry funding, which is a very small amount of our budget, we make sure that that is really sort of cordoned off in a way that makes it very difficult for them to certainly, as with any funder, totally out of limits, off limits to have that influence any of our findings. We're especially cautious about that with industry, but also with philanthropic funders. We have a large amount of funding from open philanthropy. And we also work really hard to make sure that our research and findings are not just trying to show whatever we think that they would want us to show, but that instead is what we think is true. And that's really what has built our reputation as well. So at this point, something we hear regularly from funders is that is why they want to support us, because they know that we are going to do that. We are going to make sure that we are not going where our sort of financial backers might be inclined to go, but we are going to really be independent and tell it like we see it. And so that is a value add in itself.
- [132:50]
Rob Wiblin
Yeah, I think people can often tell when you're being paid to push a particular line because it's like the conclusion will always be the same, no matter what the incoming evidence is or what the updates are and people tune out?
- [132:59]
Helen Toner
Yeah, I think that's right. And I think. I mean, I also think even if it's not a financial thing, I think there can be ways in which, in AI debates, for example, if people have the feeling that your conclusion is going to be risks are really high, we need really severe interventions. If they feel like that's going to be your bottom line in every case, on every paper, and you're not curious.
- [133:19]
Rob Wiblin
To change your mind or learn something new.
- [133:21]
Helen Toner
Yeah, Then they also tune you out. And I think that is also a strength that we bring, is not doing our work that way.
- [133:27]
Rob Wiblin
Are there any particular talent bottlenecks in D.C. or what other skills or knowledge that you could bring into the conversation? Have people work at CSET or wherever else that would improve people's understanding, that would actually push things in a smarter direction?
- [133:42]
Helen Toner
Yeah, I think there's all kinds of things. I think having a basic understanding of AI as a technology, which doesn't necessarily mean a computer science PhD, I think, but being comfortable enough with calculus and linear algebra and a bit of programming to have played around with models yourself and have a sort of basic intuition for how they work, and then bringing some other kind of expertise, really any kind of expertise. I think something that is a bit overrated in AI is purely computer scientists coming in and combining that with and learning the policy side, because AI touches so many different fields. I think if you're an economist, if you're a historian, if you're an anthropologist, if you're an accountant, if you're just so many ways, if you're a teacher, I don't think there's a lot of space for a very wide range of different kinds of expertise. I do think it makes sense, inasmuch as you can, to try and get that sort of basic understanding of AI, largely. So it doesn't seem like magic trying to get deep enough on the technology so that you have a rough sense of how it works. But beyond that, I think there's a huge range of needs.
- [134:43]
Rob Wiblin
What's something that you think a lot of listeners might get wrong about the AI policy discourse in D.C. i think.
- [134:50]
Helen Toner
One thing could be thinking that policymakers are really uninformed or sort of stupid on AI issues. I mean, certainly different policymakers interact with these topics differently, but I think.
- [135:09]
Rob Wiblin
Is this a widespread perception, do you think?
- [135:11]
Helen Toner
I think a lot of people think that policymakers have no idea and are really ignorant about tech topics. I think it's really important to remember two things, both just how busy policymakers are and how depending again on the policymaker, how many topics they might need to cover and how many issues, including often just like truly urgent crises they need to handle. And so if they don't know something or misunderstand something, it doesn't mean they're stupid, it means that they're doing something else. And I think another thing that people underestimate is how insular some of the conversations and some of the sets of ideas are where from the perspective of many people in dc, they've just never really encountered an argument about, for example, why export controls would be good or why we should expect AI to develop in a certain way because so many of these conversations happen in these pretty self contained social groups or sort of online networks. So I tend to think there's a huge amount of value in trying to create more accessible, no prerequisites, required introductions to concepts or explainers of key concepts. We do quite a lot of explainers at CSET as well that bring people into these sort of key ideas and let them sort of understand them and play with them in their own head as opposed to just having them be discussed sort of verbally in conversations among friends.
- [136:29]
Rob Wiblin
I guess you engage a lot and I think very productively with people across the full spectrum from being, I guess, very optimistic about AI to being very doomy, very untroubled about the future. Is there any particular evidence you think that could help to settle in your mind or give you a big update as to how worried to be?
- [136:47]
Helen Toner
I think probably no one single piece of evidence, but different things that we could see over the next few years. I do think this, I guess two things we've touched on, but to say them again, one is this question of how much is AI going to be a tool versus how much is it going to be really sort of agentic, autonomous out there, kind of doing its own thing. That's something I expect we'll get evidence on. I think this year the updates have been mostly that it's turning out to be more challenging than people expected to have. AI agents that can even do pretty basic things like booking flights reliably. I'm sure they'll get there. But it's turned out to be a little bit more challenging than some of the most bullish people are expecting. But I think the more that we see AI systems that can really go out and do in practice on a regular basis, sort of longer time horizon tasks or also messier tasks. So things that aren't as sort of cleanly boxed and specified as like here is a nicely chunked out thing for you to do, but instead here's a squishier thing. Can you make some progress on it? That's one big piece and then another piece is this question of how super intelligent things can get and I think there's starting to be a little bit more discourse on this, a little bit more looking for sort of evidence or breaking down different types of capabilities of how far above human level could you get on a given thing. I would love to see more of that because I think assuming that things will plateau at human level seems clearly wrong. But also the sort of boss German is in superintelligence, he talks about these superpowers of the AI will have superpowers and all of these different things. And that also seems pretty unlikely to me. And so trying to figure out kind of where in the middle we will land seems really important.
- [138:23]
Rob Wiblin
So is CSAT hiring for any roles that people in the audience, if they want to dive into this world or maybe already like someone in this world but are considering moving to CSAT or DC should think about.
- [138:31]
Helen Toner
So depending exactly when this goes live, we're hoping to have a post up pretty soon for a research fellow or senior fellow. So this would be focusing on frontier AI. So this would be a. A role to lead research at CSET would need someone with a graduate degree, a few years of research experience at a minimum, maybe multiple years, maybe some government experience for a senior fellow position. I think this will be a really exciting role. We'd love to see lots of really great applicants. If that role is closed or also if it's not the right role for you, then we'll likely also be hiring for additional research roles, data roles, communications and external affairs roles, potentially operations roles sort of through 2026. So definitely keep an eye on our careers page. Sign up for our newsletter Stay in.
- [139:14]
Rob Wiblin
The loop Is CSET growing in general?
- [139:16]
Helen Toner
I think we're roughly happy with our size. So we were I think in 2019. The interview was about the 30 person research group in DC. We're now more like 50. We may fluctuate a little bit around that, but I think pretty happy with the size. It's a nice, I think a big part of CSET's model is being able to sort of have a cohesive team culture where people are collaborating across teams and, and there's sort of a sense of as an organization, what are our sort of key priorities, which gets difficult if you're scaling up much beyond that level.
- [139:44]
Rob Wiblin
The last six years since conversation in 2019 have been pretty eventful. I can only imagine the next six years are going to be as eventful or more so.
- [139:52]
Helen Toner
And talk to you in 2031.
- [139:54]
Rob Wiblin
Hopefully we get to talk before 2031. But yeah, my guest today has been Helen Turner. Thanks so much for coming on the 80,000 Hours podcast, Helen.
- [140:01]
Helen Toner
Thanks, Rob.