← Future of Life Institute Podcast
Future of Life Institute Podcast
How to Govern AI When You Can't Predict the Future (with Charlie Bullock)
01:07:11
Loading summary
Transcript89 lines
- [00:00]
Unidentified Expert
Evaluation for the sake of evaluation doesn't make a lot of sense unless you really have an action that would follow the evaluation. We already see some evidence that prolonged use of AI systems might contribute to deskilling effects. For example, if you give doctors AI systems to assist them with the identification of tumors, then after three months of using those systems, once you take the systems away, their performance drops by about 6 percentage points. Now we have hundreds of millions of people who are using AI companion apps. And the first studies do show that heavy use of AI companions seems to be associated with decreased social contact with other people. Take a step back and think about the big picture. Is this actually how I want to live my life? Systems are now able to distinguish regularly between test and deployment settings. If we can't ensure that a system behaves in a test environment same way it behaves in the deployment environment, then that really undermines our oversight measures.
- [01:04]
Gus Docker
Welcome to the Future of Life Institute podcast. My name is Gus Docker and I'm here with Charlie Bullock. Charlie, welcome to the podcast.
- [01:11]
Charlie Bullock
It's great to be here, Gus.
- [01:13]
Gus Docker
Great. All right. Do you want to introduce yourself?
- [01:16]
Charlie Bullock
Sure, yeah. I'm a senior research fellow at the Institute for Law and AI. What that means essentially, is that I work on US policy, AI policy specifically. I do things like I advise state and federal lawmakers on what good AI laws look like. I do stuff like draft legislation sometimes. And then I also do research. I write research papers about legal. Legal issues, usually with practical relevance to AI governance. This, this, this paper I'm. I'm talking about today is, is a little bit less legal, but still policy relevant.
- [01:46]
Gus Docker
Yeah, we're talking about a. A draft paper called Radical Optionality. And, and maybe let's start with why we need. Before we get to Radical Optionality, let's start with the pacing problem. What is the problem with regulating emerging technology like AI?
- [02:04]
Charlie Bullock
Yeah, so this has been talked about by sts, science and technology scholars for a long time. The issue is that the pace at which technology develops is quite fast relative to the pace at which our institutions and laws develop. What this means, essentially, is that often laws are outdated when they're intended to regulate emerging technologies by the time that they come out. Right. Or very quickly afterwards. There are a lot of striking examples of this throughout history that we could go through, but essentially this is a pretty well recognized problem and most people agree that this is the issue. Right. So a lot of AI policy prescriptions are intended to address this problem, essentially.
- [02:40]
Gus Docker
Maybe some historical examples would be good here just to get a Grasp on what we're talking about. What's the best examples of sort of the mismatch?
- [02:50]
Charlie Bullock
Yeah, one of my favorite examples is the Audio Home Recording act, which is a law that came out, I think, in 1989 or close to it. And it was meant to resolve this long standing debate between sort of like advocates for the home recording of audio and record companies and stuff like that who were afraid this was going to lead to piracy. And so Congress hashed out this issue for years and years. They came up with this grand bargain that fucked up, finally resolved all these issues and then tucked somewhere in this exhaustive legislation that was going to create this whole mechanism, this whole framework for governing these digital audio transcribers, ds, which are supposed to be like the wave of the future. There's this little line that says, oh, and anything that goes through the hard drive of a computer is not covered by this law. And then of course, the MP3 is invented shortly after. And so by the time the law goes into effect, it's essentially obsolete. Right. And this is a common problem. Right. When you try to look forward and predict where the technology is going, inevitably you're wrong because not because governments are necessarily stupid, but because this is impossible or at least very hard to do with any consistency. And so then, you know, the, the legislative process, which takes years, and the process of updating the common law, which can take even longer. And of course, you know, not to speak of like amending things like constitutions, which happens on the scale of centuries. Right. Often these are often considered to be inadequate to the task of governing emerging technologies that are changing the world extremely rapidly.
- [04:12]
Gus Docker
And what's the lesson for AI here then? What should we learn from this problem?
- [04:18]
Charlie Bullock
Right. Well, I mean, first of all, that there is this problem. Right. And then secondly, that you're going to need things like flexible rules and definitions in order to counter it. Right. And that you're going to need flexible governance mechanisms in general. Right. This is sort of the insight that's motivating proposals like the private governance proposals that you've seen from people like Dean Ball or Gillian Hadfield. Right. Regulatory markets, that sort of thing. And it's the, it's the insight behind a lot of, you know, other, other, like anticipatory governance is one on the left that gets brought up a lot. Right. The idea that we need to like, look forward and, you know, be better at anticipating things before we, before we make rules. So, yeah, that's, that's sort of a lesson for AI.
- [04:58]
Gus Docker
I think it could it be the case that that AI is different here? Or would you say that that AI is perhaps going to be the most striking examp. It's moving so fast.
- [05:11]
Charlie Bullock
Yeah. I think it's kind of like the perfect example of this, of this problem. Right. Or like the, the final culmination of it or something. Right. Like the technology is moving so fast. It's so potentially transformative and it's so poorly understood even by the people who create it. Right. I mean, this, this black box idea is maybe a little bit outdated with advances in mechanistic interpretability and things like that, but there's still a large element of black box in how AI is understood by the people who make it. So it's like, you know, you're trying to make rules to govern something that you don't understand and for which the, like, the risks and the benefits that it will create are particularly poorly understood and like, sort of hard to foresee, despite the cottage industry of people whose job it is to try and foresee it.
- [05:48]
Gus Docker
Yeah. We have sort of lack of understanding and uncertainty about how the technology works, what it can do. But then you couple that with how it's going to interact with society and it becomes even more difficult to predict what's going to happen. Like making predictions about how AI is, is going to change the economy, for example, is pretty difficult.
- [06:10]
Charlie Bullock
Yeah. I mean, it's very fun to speculate about, but it's also incredibly hard. No one's confident in their predictions. Right. I don't think. Unless they're crazy or something.
- [06:18]
Gus Docker
Yeah. And so you have this proposal, radical optionality. What is that about?
- [06:25]
Charlie Bullock
Yeah. Okay. So the core insight of radical optionality is that what you want to do when faced with this kind of decision making under extreme uncertainty is, is you want to avoid both overregulation early. Right. Because regulation in ways that I think people who don't study this full time sometimes don't fully understand is sticky and path dependent. Right. Once you make a rule, you tend to keep on making rules like that. And it's very hard to change your institutions to accommodate totally different rulemaking approaches. But at the same time, we need to prepare for potentially really difficult regulatory challenges that might emerge that probably will emerge if the technology ends up being transformative. So what you need to do is you need to prepare a lot beforehand by building up government capacity in order to be ready when the regulatory challenges do come.
- [07:11]
Gus Docker
Yeah. So there's the issue of overregulation. There's also this issue of sort of the wrong regulation and Locking in the. Focusing on the wrong thing in law and then locking that in. So even if you are interested in sort of heavily regulating AI, I believe there's virtually thinking about optionality just because we could end up in a situation where we are regulating the wrong thing or focusing on the wrong thing in our regulatory approach. Yeah. Maybe talk about the idea of building up governance or government capacity.
- [07:45]
Charlie Bullock
Yeah. So basically, complex systems like LLMs or corporations or government agencies thrive on a diet of information. Right. They really perform better when they have better information. And there's a lot of scholarship backing this idea up. And so if you have a government agency or something, Right. You want them to have good information gathering authorities. Right. You want them to know a lot about the subject of the regulation. Right. Ideally we'd have people in government who understood AI very well. Currently, we kind of don't have any of that. Right. There's, there's no government agency who's remitted to govern AI. Right. We have plenty of government offices and agencies dedicated to regulating things like railroads, like electricity, like nuclear power generation, et cetera, but so far nothing for AI. So that's a glaring lack of capacity in the United States that needs to be addressed. I mean, other countries do better. The U.K. i think, does the best. They have the U.K. safety Institute, which is funded to the tune of I think something like 80, 80 million dollars a year or something. And they have a lot of great people there who work very hard. But even that, I would say is insufficient to the scale of the, of the challenge presented. Right. If you believe that this is going to be the most important technology ever invented or something close to it. Right. And that's going to fundamentally transform our society. We should be willing to invest almost an unlimited amount, I would say, in the, in anything that increases your chances of making the transition go better, even a little bit. Right.
- [09:03]
Gus Docker
Yeah. Explain that one. Because it seems like if you say we should be able to, you know, if you have some problem in society and you say, okay, we should be, we should be willing to invest almost anything. That seems like the wrong path to go down. If we, we might want to improve our schools, but we don't want to invest infinite money or all of the state's budget in that, why is it that the optionality here has to be radical?
- [09:25]
Charlie Bullock
Yeah. I mean, so this is like, as you know, where the radical comes in, Right. Is this prescription that we should be able to spend almost an unlimited amount on increasing government capacity. Right. Like it's actually kind of a milquetoast proposal at its core that like, oh, like, don't, don't regulate. Just like, sit around and like, prepare or something. But like, the radical element is like, that you should be able to. Should be willing to spend almost an unlimited amount of resources and political capital on this particular issue. Right. As opposed to others. To be clear, I don't expect this to happen. It's not realistic. I'll take what I can get, even like a little bit, a few million dollars. We're taking crumbs at this point. But I do think, if you think that the transition is going to be this important and if you think that it's possible, plausible that governance will have an effect on it, which I think more or less everybody thinks that, you know, whether you're very pro regulation or very anti regulation. I think most people believe that, like, there is some chance that regulation affects the trajectory of the technology in some way. Right. Just like the fact that this is some sort of hinge of history or something means that, like, this is like, the most important thing happening. Right? And it's in one way or another going to determine how our future looks. So it's kind of unlike investing in schools or something where, like, it's obviously important. I'm in favor of funding schools. I'm not suggesting that we defund schools in order to pay for AI governance or something like that, but on the margins, investing in schools results in some amount of extra learning per student or something like that. And if you believe that transformative AI is going to really radically change our society in the very near future, any chance of impacting that transition outweighs the marginal dollar spent on basically anything else by a factor of a great deal.
- [11:04]
Gus Docker
Let's walk through the assumptions of the paper. You sort of cleanly laid these out in the beginning, where the first is that transformative AI might be developed in the next 15 years. And that's something we probably don't need to go into. Listeners of this show will have heard the story of how that might go many times. The second one is more sort of contentious and interesting, which is the benefits of transformative AI will probably outweigh the risk, especially if we have sensible governance measures. And yeah, I think you can land on either side of that. Tell me why you think that's a plausible assumption.
- [11:43]
Charlie Bullock
Right? So, I mean, reasonable people can and do disagree about this. Right? The reason, like, I have this assumption personally is like, just based on rough heuristics about, like, okay, historically, technological change has usually been good. It's usually produced Net good results for humans. There are all sorts of reasons to believe that that doesn't hold true here. Right. But I think that the, the arguments that it'll be worse than it is good are kind of equally based on rough heuristics or something. Right. I mean, not all of them. Some of them purport to be based on like, you know, cold rationalist logic applied in a series of 36 steps that proves, you know, for a fact that we're all going to die in next years or something. But I, I think that those takes are like overly confident or. Yeah. Kind of based on logic that doesn't always hold up or something like that. But yeah, I mean, it's just a rough heuristic. And I think that radical optionality is a good option even if you don't believe that the technology is going to be net good. Right.
- [12:38]
Gus Docker
Yeah, actually make that case because that's pretty interesting if radical optionality is a good path to go down, even if you perhaps land on the side of this assumption where you think it's probably going to be net bad.
- [12:52]
Charlie Bullock
Yeah, sure. So, I mean, this is less of an argument for the correct theoretical answer or something, because if you believe the technology is net bad and the correct theoretical answer is not to build it. Right. Obviously. But I would contend that most people agree that it's not possible to not build the technology right now. Right. Like somebody's going to build it either in the US or in China or in some other adversary state. And because of that. Right. If we agree that a pause is not politically realistic right now, if you're a pause advocate or something like that, you could continue militating for that and then also push for, in the meantime, okay, sensible governance measures. Right. Information gathering, like optionality increasing things are good. Because if you believe that it's not true, that it's totally hopeless, that any regulatory response to this problem has any effect, then it's probably good to have some government as muscle in place for when things hit the fan and you need to respond to them.
- [13:41]
Gus Docker
Yeah. The third assumption you make is that there is significant uncertainty about how and when transformative AI will be developed and what the best way to govern it will be. And that one, I think is also quite plausible on its face that we are faced with this, a very high level of uncertainty about what we do in this situation. And so the sort of case for optionality almost makes itself if you accept that last assumption. Maybe talk about, talk about the uncertainty we're under here.
- [14:16]
Charlie Bullock
Yeah, I mean, so there's a number of unknown unknowns here. Right. And there's a number of known unknowns as well. Right. I mean, if you talk to someone from anthropic or something, often they're pretty confident that the technology is coming out very soon. Right? The next 12 to 18 months. Some people have longer timelines. I personally think it'll probably take a bit longer than that, but I'm not a professional timelines guy or something. But then, okay, I think there's also a great deal of uncertainty about what the technology will look like. I mean, okay, if you look at the early Yudkowski predictions about what the technology would look like, right. The prediction was that oracular AI was more or less impossible. Right. It would necessarily have some sort of agency, it would necessarily have goals that it was pursuing. Right. What we see right now with top of line LLMs is more or less just an oracular AI as described. Right. So this is like sort of illustrates the problem of how difficult it is to predict the future, like what the technology would even look like. Right. But even if you assume that, okay, AGI is coming quickly, it's going to be based on like an LLM architecture, Right. And it'll kind of look like an extremely good version of Opus 4.6 or something like that. Even if you assume that. Right. Like we, I don't think anyone claims to fully understand the effects that that kind of change would have on society. Right. Some of them are obvious, Right. People might start losing their jobs. Okay. What does that do to society? What does that do to our politics? Right. I think there's, these are very complicated questions and people are working on them, which is good. But like when I look at the state of the art for like predicting what's going to happen, it's like, you know, people are doing good work, but it's kind of primitive and so forth. Right. So it seems like investing marginal dollars into figuring that out is good. And then simultaneously.
- [15:50]
Gus Docker
Right.
- [15:50]
Charlie Bullock
Preparing for a wide variety of different futures is also good. Right. Because we don't know what the future is going to look like. So it's, you know, it's good to have tools in place that are going to be applicable in a broad range of situations.
- [16:01]
Gus Docker
Yeah, I, I see, I see the case for this. I do worry that this is awfully convenient for AI companies in the sense that perhaps it's, it's useful for their purposes to avoid regulation, even if that regulation would be good for society at large. And this is an argument that, that we need to sort of wait and see and gather information and hold off on regulation. And is this. Is this perhaps like when tobacco companies were interested in avoiding regulation, and so they. They sort of played up the uncertainty around whether cigarettes cause cancer?
- [16:40]
Charlie Bullock
Yeah, that's a good question. You know, I don't think you're wrong. This is in some ways a convenient argument for AI companies. It's kind of convenient by design. Not because I'm a shill for anthropic or something like that, but rather because, like, I think it's, like, very hard to pass AI legislation in the US Right now at the federal level, but also even at the state level. And so, like, if you have laws that are designed not to tick companies off, you have a much easier chance of passing them. Right? You'll notice that laws like California's SB 53, which is kind of like. I mean, it's almost like. It's funny to me. I think a lot of people don't understand what the law actually does, so it's not funny to them. It's, like, hilariously not burdensome to AI companies, right? Like, what it essentially says, like, the way that OpenAI, for example, complied with SB 53 was the law says, okay, you have a safety and security policy. OpenAI, which they do, right? That policy is currently on your website. Leave the document that's currently on your website on your website. And so what OpenAI had to do is not take a document off of their website, right? It's sort of like, hilariously easy to comply with in light touch, right? And even this kind of has companies like, you know, screaming and wailing and gnashing their teeth and running their clothes about how oppressed they are by overregulation, right? So it is very hard to pass anything that does a lot more than that. And so it's in some sense convenient for me to make an argument about, like, oh, this is per. This is a great approach. Like, let's do this thing that's like, hey, look, we're not hurting you. We're not touching you. Just like, you know, please let us invest some money in, like, government agencies, right? Or, like, evals running an eval system or something. It seems like something that you could feasibly do or, you know, at least do a version of it and the sort of, like, it's an argument that's designed to be kind of hard to say no to if you fully accept the premises and stuff like that. So I think that's kind of why it ends up being quite convenient for AI companies. Which is not to say that I don't believe what I'm writing. I mean, I do believe it. Right. I'm down to talk all day about why I'm a techno optimist or something like that. I understand that not everyone is a techno optimist, but at the end of the day it's also an attempt to point to a set of win win situations that seem like very low hanging fruit that hasn't been picked. And I'm like wondering why it hasn't been picked and hoping that I can, somebody will pick it at some point.
- [18:47]
Gus Docker
Yeah. So how do you imagine this goes? Is it the case that we spend time building up government capacity to handle AI before we act? And if that's the case, are we going to be building up government capacity while Anthropic is sort of having their AI do recursive self improvement? So like, what I'm asking is, is there a slowdown in our approach? Does it make sense given the pace of AI development?
- [19:20]
Charlie Bullock
Yeah, I guess I would want to know the alternative realistic approach that would happen fast enough to deal with recursive self improvement. Recursively self improving powerful AI seem like a very difficult regulatory challenge to handle in general because of this. Like, I think that having knowledge about them. Right. Can only be good. Having a government agency that like is aware of this and is like keeping it tabs on anthropic is good. Right. Like if you're in favor of a, of a pause or something like that, or like a ban on super intelligence. Right. Being able to say, hey, super intelligence is coming and we know this and here's how we know and here's the information that proves it is like very good. And I think the current trajectory we're on is that you might get superintelligence and no one would have any idea except the lab. That was crazy creating it, maybe not even them.
- [20:07]
Gus Docker
Right, yeah. So what you're saying basically from any sort of regulatory point of view, recursive self improvement is just a massive challenge to deal with because it's something that's happening perhaps or internally and perhaps sort of unknown to the outside world, maybe within even one lab or one AI company?
- [20:26]
Charlie Bullock
Yeah, that's right. And I think that things like transparency requirements, things like reporting requirements, things like whistleblower protections are like robustly good for letting us know when it's coming so that we can, you know, potentially stir up some, stir up some political will to do something about it.
- [20:41]
Gus Docker
Yeah. Do you think, do you think this approach takes into account the Political sort of the real world political incentives here. If we imagine the current US Administration building up capacity in the government, sort of considering different approaches, then having a, then they've spent their time building up a strong government to then hand over to the next administration. Is this something that. This doesn't really sound realistic or there will be some incentives counter to that? I think.
- [21:13]
Charlie Bullock
Yeah. I mean, I think this is like a very realistic objection. Right. Is that like, okay, we live in a heavily polarized political moment. Right. And one government is. There's a sort of like, you know, a long standing understanding on the conservative right that like all else being equal, like Democrats tend to want to do more with government than Republicans do. So building up government is like good for Democrats and bad for Republicans or something like that. I mean, there's a lot of counterarguments to that. One of them that I would like to point out is that like, okay, currently the government, all three branches of it, are controlled by Republicans. Right. So you're building up capacity for you, the Republican government currently to handle transformative AI. Right. The alternative is like, you know, anthropic, builds it in a lab somewhere and then you're, you know, like the, the Aschenbrenner situation. Right. You, you know, situational awareness. He predicts that like by 2027, necessarily. Right. The U.S. national Security Enterprise will like put all the labs together, nationalize them and develop AGI and sort of a Manhattan Project secretly within the government. Right. If that's going to happen, which, you know, I'm not saying it will, I don't, you know, I think his assumptions are kind of like overly confident or something. Probably it can go better or worse. Right. And if you're more prepared for it, it's going to go better. If you're not as prepared for it, it's going to go worse. Right. And there's lots of different things you can do to prepare for it. So you don't want to sort of scramble to accomplish it at the last minute or something like that. And if this is going to happen under Republican government, it seems like they have lots of incentives to want to do it. Well, in an ideal world, is this.
- [22:43]
Gus Docker
So the kinds of things that we're talking about right now, is this something that's known to the people actually implementing this regulation? So would Republican lawmakers at this point, would they think about AI timelines and when they might expect this technology? Are they even thinking in these terms or is this sort of like over intellectualizing the process perhaps?
- [23:04]
Charlie Bullock
Yeah, I mean, a Lot of them are not, right? This is a question, as I understand it, about, like, how AGI pilled different people in different areas of government are. And the answer is it varies, right? I mean, there are people within, like the military, there are people within the executive branch. There are people in Congress, right, who are like, quite AGI, right? Bernie Sanders recently was saying, like, oh, my God, like, you know, AGI is coming. I endorse the views esposed by the. Espoused by the Machine Intelligence Research Institute or something like that. And there are, you know, Republican lawmakers as well, who have said, like, very AGI pilled things. I think it kind of depends. I mean, like, when we talk to congressional offices, right, Often we find that we're knocking on an open door, right? It's like you go and you. You talk to some staffer and they're like, dang, this stuff's crazy. Have you read, you know, if anyone builds it, everyone dies. Like, somebody should do something about this, right? But there is not, like, currently the level of, like, I guess, AGI pillness or whatever that would be required to implement, like, the extreme version of our recommendations, right? I think there is potentially enough, like, enough realization that, okay, like, this could be pretty weird. This could stuff could be very impactful in various ways. I think there's enough understanding of that that it's not totally unrealistic to say that we might implement some of these. I don't think it's crazy to a lot of people in Congress the idea of having a government agency that does safety evaluations or something like that. I do think that the trend has very noticeably been that people are getting more AGI pilled as time goes on. I think that's been very apparent in early 2026, and I think that's only going to increase. And I think that once you start seeing things like people losing a lot of jobs, unemployment gets up to 4, 5% or something like that, and that's really going to be hard to ignore. And I think there's going to be a ton of political will at that moment to do something, anything, right? And so the idea is to kind of have a slate of potential options in hand, right, that you can then, you know, a playbook that you can parlay that sort of maybe upsurge in political will or something into some kind of, you know, productive and useful government structure.
- [25:01]
Gus Docker
Do you think regulation is inevitable? So is it the case that we can either prepare and have some proposals ready, or we will just have a scramble and then sort of implement Some regulation that's grabbed out of thin air.
- [25:16]
Charlie Bullock
Yeah, I mean, I think that at some point regulation is inevitable. Right. The question is how much. Right. And so the sort of pitch to AI companies here is, look, if you build AGI as you claim that you're going to, it inevitably will have to be regulated. Right. Even if you're like the massive libertarian or something, no one believes that like nuclear weapons should be totally unregulated, that you should be allowed to build a nuclear weapon in your backyard. Right. Like that would be crazy. So if we assume that AI is going to have military applications that are like on the level of like the military applications of nuclear fission or something like that, which I don't think is a very unreasonable assumption given like how important the technology seems and how critical, I mean, our military establishment is saying it's going to be on the battlefield, then obviously there are going to be some basic regulations at least. Right? No, you cannot have your own army of hunter killer drones in your backyard. Right, of course, Right. So my pitch to AI companies is sort of like, look like you're going to get regulated sooner or later. If you build what you're saying you're going to build, that's inevitable. If you don't fight us on the preparation stuff, then the regulation can be more even handed, it can be better. That's a win win. Right. There's a sort of deadweight loss option which is like everyone freaks out, it's too late, they overreact, they do a bad job and they shut off a lot of the benefits of the technology without addressing the risks adequately or something like that. And so that's bad for everyone, it's bad for the labs, it's bad for us, it's bad for everyone. So the goal here is to sort of avoid that lose lose situation and try to capture as much of the win win as possible where like, okay, we protect people's safety, we protect people's interests, we avoid risks and also we do it in a way that's kind of smart and designed not to unnecessarily limit innovation and stuff like that.
- [26:48]
Gus Docker
You have a section in the paper about private governance and why we can't rely on that exclusively. But I want to perhaps you can provide the best possible version of that argument because as you mentioned, I don't think many people believe that we should allow companies to develop that's as powerful, as potentially destructive as nuclear weapons. And so what is it that's actually proposed when you propose that the AI companies govern themselves?
- [27:19]
Charlie Bullock
Yeah, okay, so this is the suggestion that Dean Ball has argued for and that Gillian Hadfield at Johns Hopkins has argued for and others fathom is the organization that pushes these sort of IVO bills, independent verification organization bills. They've got one in Minnesota, now they've got another one in California. So this argument is sort of that like, not, you know, the strawman would be like, companies are governing themselves, right? I don't think that's what's being proposed. If you take the argument seriously, it's that AI governance should depend on private governance mechanisms, like, just as much as, if not more than, government laws and regulations. So you still have government laws and regulations, but additionally you have private governance mechanisms. Right. Not that the labs are governing themselves, but that private entities are governing the labs. These would be the ibos, as I just said, they've been called multistakeholder regulatory organizations, MROs before. Whatever. The idea is, and the idea is essentially, look, it's essentially a way of responding to the pacing problem that we talked about earlier. We need flexible rules. The rules we have are inflexible. The fastest way to change legal rules currently is usually an agency rulemaking. Right? And like real notice and comment, agency rulemaking takes many months to pass a rule. Right. And if the technology is changing every two weeks, if it's recursively self improving when you do something about it now, that's not good enough. Right. You need something much faster. And there are, you know, there are existing ways in government to make it faster, but it still ends up being extremely slow and kind of hard to adapt. So the idea is that we need to take advantage of AI to govern AI, right. And invent technologies, including like regulatory technologies that allow us to do a better and more flexible job and sort of keep up with the pace of technological development. So that's the, that's the steel man of it. Right. And the, the proposals are often, are often good. Right. But I think, like, fundamentally, like, I, I don't think that private governance is all you need. This is, this is an argument. This is something I argue against in the paper. Right. I'm not in favor of just relying on private governance. And so the reason I think it's not all, it's not quite good enough is that like, there's a, they never quite answer the question of, like, okay, how do you make sure that these private regulatory organizations are aligned to the good of the people? Right. We have an answer to that question. It's an imperfect answer. We have an answer to that question. In the case of Government, it's called democracy. Right. Somebody votes for them. Right. But the idea of taking advantage of like market incentives to govern AI relies on the idea that like these companies are going to be competing with each other and that competition will create better rules or something. But if the customers of these companies are the labs, right. They're not going to be competing to create better rules. They're going to be competing to like create the least possible burdensome rules. Right. Like it's going to be raised to the bottom essentially. Right. And so there are, you know, kind of ways to address this race to the bottom dynamic, but they end up being a little bit hand wavy in my opinion. Right. You just say things like, oh, there's a government agency and the government agency does a super good job of, of making sure that they only license the very best IBOs that are staffed by good and honest people. Right. And this is a solution. But I like, it's not like the best solution. I don't think it's not like an adequate solution because like, like you just. We're back to square one right now. You have to get a good government agency, which is the whole problem in the first place. Right. So building capacity is like critical for that, I would say.
- [30:28]
Gus Docker
Yeah. Now, now building up this capacity, does that put us at risk of sort of enabling authoritarianism? So I'm not talking about the current administration or any sort of particular administration, but just in general, is that a worry that if we are building up this massive government apparatus, then we somewhat might come grab it and use it against the people?
- [30:52]
Charlie Bullock
Yeah, I mean this is a very good objection. I think this is kind of the best objection of radical optionality is like misuse risk. Right. From the government. Right. Essentially. Okay. There's a couple different versions of this argument. Right. One is increasing government authority increases the risk of authoritarianism. I think my response to that is kind of that concentration of power risks cut both ways. If you don't enhance government authority to govern this technology and you leave it to the private sector companies, then there's a risk that the power will become very concentrated in the labs themselves, which seems worse because they're not in any way democratically responsive.
- [31:30]
Gus Docker
Right.
- [31:31]
Charlie Bullock
I think there are ways to address the risks of like authoritarian takeover, something. One very good idea is like a paper written by my colleague Colin o'. Keefe. It's called Law following AI. Right. They're, they're have a paper out and they're trying to come up with a benchmark to judge how law following AI systems are Right. And so you could pass a law, right. You could write draft legislation that says, okay, you, if you are procuring a, an AI system for government use, it can only be procured if it passes, you know, a certain set of benchmarks, if it's a certain amount of law falling. Right. In other words, if it refuses illegal orders, the alternative is, okay, you've got a bureaucracy that's mostly staffed by AI or controlled by AI and perhaps a military that's controlled by AI or something like that in some form. Right. And I don't necessarily mean this in like a sci fi way, I just mean like, you know, there's a lot of AI in government, right. And you need, you know, the, the essential problem of like planning a coup is that you need a lot of people to say yes to it, right? And some of those people will say no or like tell on you or something. And so if you have AIs that are not law following, then potentially you get a situation where one person or a few people can just say, okay, AIs take over the country for me. And it happens without the need for all that buy in and stuff like that. Right. This is the risk of an automated military and stuff like that. So I would say that, yeah, radical optionality has responses to that, but it is a serious concern and objection that by creating all this capacity within government, we're also increasing the government's capacity to do bad things.
- [32:53]
Gus Docker
Now, law following AI sounds pretty good and I'm sure the models are quite smart. The models are quite flexible in how they understand the law and so on. But just given the recent anthropic department of war situation, it seems like some of the laws in the US are not necessarily set up in the right way where perhaps it is. Yeah, you tell me if this is the case. Right. But my understanding is that it's technically, it's technically legal to do what you in normal language would call mass surveillance, given that data is procured in a specific way. And so even law following AI might not be enough if it's just following the current laws. And so it would have to be trained in a way where it's following the sort of, what we want. So this is, in the end, this is sort of the alignment problem all over, I think.
- [33:44]
Charlie Bullock
Yeah, it's like what who wants, right? What the government wants. Because the government may not want good things, right? Like, but what anthropic wants that a lot of people have a problem with it being, being aligned to what anthropic wants, right? Because anthropic are a bunch of, you know, lib defective altruists, whatever, blah, blah, blah. Right. So it's like the, the law, the advantage of the law is that it's like a neutral thing. Right. And so like kind of everyone agrees on it more or less to some extent. And that's like a big advantage because like having it just like Claude, obey your own constitution. A lot of people don't trust Anthravic to have a good constitution for Claude. Or on the other hand, a lot of people don't trust, you know, Grok to have a good constitution when it's saying that it's Hitler or whatever. So you need to align it to something and it's maybe harder than you might think to come up with something that the country in an organized political way can agree to. And so that's kind of the benefit of law might be that it's like the worst alignment system except for all the other ones or something like that. Right.
- [34:38]
Gus Docker
You're saying that in general people sort of can agree on the constitution or some very basic legal principles.
- [34:46]
Charlie Bullock
Yeah, I think most people in the U.S. by end of the U.S. constitution, there's like a majority. It's not a huge majority, but sure, it's enough. And then like most people agree that you should follow the law. We kind of all, like, not everyone agrees with every law, obviously, but in general. Right. And of course, I mean, the claim of law falling at AI is not that the AI should sort of like slavishly follow each individual law very literally, like a stupid robot or something like that. It's that it should like have like good, robust intuitions about which laws to follow and when. And like when it's okay to kind of break a trivial or outdated or unenforced law and et cetera.
- [35:17]
Gus Docker
Yeah. And I think it's actually like current models can actually be flexible and sort of subtle in that way. Like people can. It's not that they are sort of reading the law and then following it in a dumb way, as we might have expected 10 years ago.
- [35:31]
Charlie Bullock
But you need to be principled. Right. Because there are also lawyers often are very good at being sort of finding loopholes and stuff like that. And you don't want an AI that technically follows the law, but is like super good at scheming and coming up with like little, you know, Office of Legal Counsel style tips and tricks like, hey, actually you could totally destroy anthropic if you use this one weird overlooked law or whatever like that.
- [35:53]
Gus Docker
Yeah, this is making Me feel a bit down about the whole thing because just if the companies can't really be allowed to sort of not have any oversight. Right. But if you push for government oversight, you then create this risk of power concentration or authoritarianism or it's basically power concentration within the government. There's no middle path. There's no sort of like, yeah, how do we navigate through this given sort of the dangers on both sides?
- [36:29]
Charlie Bullock
No, yeah, it's a skill and Charybdis situation. Right. Where you have dangerous to each side of you. You've kind of got to. My claim would be you've got to try to find the middle path or something like that. Right. And you know, do a good job of weighing these different dangers. But like I think like, like I said earlier, there are low hanging fruit and like sort of obvious things to do here that I think like there are ways to like increase government capacity that like, I don't think very meaningfully increase the risk of authoritarian takeover or something like that. And they do meaningfully, like provide information and government capacity to respond to like really bad events. And there are institutions that most people kind of trust. Like, you know, courts are sort of trusted sometimes a lot, you know, things like that. You know, I, So yeah, that's the, I think the best we can, we can hope for is sort of to chart a middle course and try to do what we can to get ready.
- [37:21]
Gus Docker
Yeah. Now this is, this is something that's, you know, a middle course here might be something, a government institution that is semi independent of the federal government, like the Federal Reserve. Right. They have some independent power and they're not fully beholden to the, to the President, for example, is that. I know this will take time to set up. I know I haven't provided any details, but is that a plausible sort of way through here?
- [37:48]
Charlie Bullock
Yeah, I mean it's less plausible than it once was. There's been recent court decisions essentially indicating the Supreme Court is very likely to, if it hasn't already, sort of gut the system that allows for things like the Federal Reserve. I mean, kind of hilariously the opinion said like, oh, the Federal Reserve specifically for like historical reasons or something is exempt from this. But every other independent agency the President can basically gut at will. Right. So this is an unfortunate development, I think, and it would be nice to have a little bit of insulation from the executive. But it's not impossible. There are ways to insulate it. Right. I mean, like, unfortunately, historically one of the big ways that insulations happen is just like bureaucracies are Permanent. And they kind of have their own opinions about things and it's kind of makes it hard to steer the ship or something. This is like the deep state problem, right. It's often viewed as a bad thing. Some people view it as a good thing, but like that kind of gets solved by AI, right? Like if you get like the, the pro. The problem that creates the deep state is that like the president can't be everywhere at once, even if he's like a huge like technocrat or something. Because the federal government is huge. And so it's very hard to have like your guys controlling every single decision that gets made when all of these like important but kind of like maybe obscure decisions are getting made every day by agencies like the EPA and so forth. But like if you have like loyal AI subordinates in every agency, that kind of solves that problem. Right. It's just like, oh, align it to whatever the President wants. And every decision that get made has to accord with what the president wants. Right. That's like a kind of scary prospect, I think, in ways that are not fully appreciated. Now because the current president is Republican, like, I think it's asymmetrically scary for Republicans because like Democrats tend to want to do stuff with government more than Republicans do. So like the idea that a Barack Obama could enforce his will in that way throughout the executive branch is maybe frightening or something like that.
- [39:27]
Gus Docker
Yeah. So even if you have some semi independent institution like the federal governing AI, you have other problems there. You have the problem of sort of a bureaucracy governing the most important technology. And if you try to oversee that, you then have. Yeah, basically. This sounds very scary to me to have sort of the President being able to stay up to date on anything that's happening in the government all the time.
- [39:52]
Charlie Bullock
Yeah. I mean, even with the Federal Reserve, like who appoints the Federal Reserve Board? Well, it's the president, Right. So ultimately they're supposed to be answerable to him. So it's like it's quite hard. I mean, you have to balance like Democratic values basically against like sort of technocratic incentives or something like that. And it's like pretty hard to strike the balance because like, you're genuinely uncertain whether you're going to like who's in charge of the governor or not, or whether they're going to be doing good things or not. So like you, you want them to have power because they need to do things. But then what if they do things you don't like? It's like a very, very hard problem.
- [40:20]
Gus Docker
What we really Want here is a balance of power between different institutions. And it seems like the system set up by the founding fathers of the US is perhaps not up to date on. I mean, it survived a long time, but it's not perfectly up to date in an AI world. I guess my question is just again, how do we preserve the balance of power?
- [40:47]
Charlie Bullock
Yeah, I mean, no, it's sort of. This is like a hilariously overblown example of the pacing problem, right? Is that the law is literally hundreds of years old. But. Okay, I mean, this is a good example of how you address the pacing problem traditionally. Right. Very early cases in U.S. constitutional law say things like, we don't forget that it's a constitution we're expounding, so it can't partake of the prolixity of legal code. Right. And what that means is essentially we wrote it vague on purpose so that it can kind of stay updated on its own. The interpreters can like, you know, repurpose these rules to anything. If you had a legal code. Right. Which is sort of, I mean, how a lot of countries worked a couple hundred years ago or something before the innovation of like, constitutional government sort of like replaced those systems. Then, like, you have to spell out the exact precise rules for every little situation. And so that kind of code becomes obsolete very quickly, right? It's a sort of governance misspecification problem, basically. Right? You, you, you make a rule and then, you know, you have purposes that you intended the rule for, and the rule becomes obsolete very quickly because circumstances change or something, and then the rule is counterproductive or something like that. So, yeah, having a constitution that's nice and vague like ours is and says kind of general principles that are like, you know, kind of agreed on by a lot of people and foundational and maybe don't change as much is useful. So I think, like, having governance structures like that helps somewhat, but then of course, you have the problem of who's interpreting it. Right. And if you have bad interpreters, you're kind of just screwed. Right? So I guess the answer is you got to elect good interpreters of the thing. Right? And if you don't, you're kind of in a lot of trouble. So that's. I mean, there's a political problem that has to be addressed before anything else, I think.
- [42:26]
Gus Docker
Let's get into some of the concrete policy proposals you have in the paper because I think, you know, people can agree or disagree with your approach in general, but many of these are quite, These are quite easy to support. And I Guess that's, that's by design. Right. So maybe let's start with gathering information and, and transparency in general. Why is that a good thing for, for the seminary segment? I would like for us to be a bit concrete about what it might look like. And so what specifically are you proposing here? Sure.
- [42:57]
Charlie Bullock
So we talked a little bit about this earlier, but basically, right, information is good. There's a lot of evidence saying that having good information increases the effectiveness of governments. As we discussed before, this could be good for companies as well as for the public. And specifically what we mean by information gathering authorities. Right. You can have transparency requirements, things like SB53 and raise. The nice thing about these authorities is that they can be built on. Right. It's a start. So even if you are like a person who believes that we should have much more intense regulation or something, I would argue that you shouldn't oppose these kinds of measures and you also shouldn't dismiss them as not enough. It does sometimes feel like very light touch requirements like SP53 or you're spitting into a wildfire or something. It's like there's this huge problem of generative AI being the capability is increasing very rapidly. And then your solution is somebody's got to publish a safety and security policy or something. Okay, I mean, what does that do for us? But it's a start, right? And it makes it easier to build on. And this is sort of a self reinforcing, virtuous cycle. Right. So first you require companies to have and publish a safety and security policy. The follow up might be, okay, maybe there's now auditing requirements, right? Maybe now you have to have an outside organization verify that you followed your safety and security policy. Right. You have to verify the safety and security policy is actually good and provides adequate protection against risks. Right. The EU typically takes a different regulatory approach than the U.S. they're much more pro regulatory, which you could argue is why they are not building this technology in the eu. But also at the same time provides real safety and protects people from real harms. And the EU approach is a bit more intense than that right now. They're already at the stage of like, okay, it's got to be like cutting edge in terms of your safety and security policy and so forth. And the ways you have of implementing it have to be very good. And you know, it's. They still need to add sort of like external verification metrics and stuff like that. But it's, you know, a decent start. Maybe plausibly, I think, you know, a lot of conservatives Object to the EU AI Act. And to be clear, I'm not saying that like support every part of it, but like specifically the like transparency requirements in it seem fine and generally good to me and like quite light touch. And I think that's not always understood. There's other sections of it that kind of are less focused in Frontieri that are potentially burdensome or something. And then like there's also like reporting requirements, right? So we have the authorities at the state level that say, okay, here's the information you have to tell the public, right? But obviously like there are like intellectual property reasons and other reasons, like national security reasons even that like AI companies might not want to reveal everything about their models, right. Even if you're like a huge safety person, like you don't want them publishing model weights. Right. Because open weight models could be used by bad people to do bad things and et cetera. So we also think there should be reporting requirements. Requirements. You have to report things to the government, right. A specified government agency in like a classified way where the information is handled kind of like securely. Right. So the stuff that might get redacted from your ssp, you have to put in somewhere and give it to some of the governments that they know stuff. And that's something that we really don't have right now. I think either in the EU or the US there was a proposal under the Biden administration to do some reporting requirements, but the legal basis for it was in Defense Production act. And a lot of people thought that that was kind of like an emergency authority that shouldn't be used for that. So I think like a new independent statutory basis for robust reporting requirements would be really good. And you could have that at the state government level as well as the federal government level. Although I think it's better suited to the federal government, like a lot of these things.
- [46:15]
Gus Docker
And then there's sharing the information you're gathering. So which avenues are you imagining this information to be shared in?
- [46:24]
Charlie Bullock
Yeah, so there's sharing between governments, right. And this is sort of critical. I think the EU AI act addresses this. There's a section about sharing information between, you know, the EU office and other government agencies of other countries. I think this is really critical between allied nations in Europe and the U.S. i think even if you're like a huge like America first type person, it's, you should just generally understand that it's in your best interest to do this. Right? Like, because you do like need allies, you have enemies in the world, right? There's, there's China There's Russia, there's countries like this that we generally don't like and that, you know, don't like us. And so it makes sense to like to. To share information that you have. Right. And other countries are ahead of us in some ways, regulatorily. Right. The UK Safety Office currently shares a lot of information with the. With the US AI, with the uskc. Right. Center for AI Standards and Innovation. But that could happen on a greater scale, and we think it should. Right. And then there's also information sharing within government, which is very important. Right. Like, if you have an agency collecting this information, is it getting to where it needs to go? Currently, the way that AI is regulated in the US is very sectoral. Right. There's no, like, one AI regulation office. It's like the Department of Energy does some AI stuff, and then the Department of Commerce does some AI and Commerce stuff, and then the, the, you know, the FCC is charged with like, some, like, AI transparency stuff or is, you know, plausibly suggesting that they might be, and so on and so forth. So you need ways for all of these agencies to communicate with each other, which they often don't like to do. And it's kind of hard to arrange in a bit of a bureaucratic nightmare. But there are existing regulatory technologies to, like, make this easier, and there are, like, you know, attempts to solve this problem that have been made before. So, like, we can look at which ones went well and which ones didn't go as well and try to adopt the best ones or something like that.
- [48:00]
Gus Docker
Yeah, yeah. Is there information sharing that makes sense between the US and China? So I'm thinking, even if there's a new Cold War or however you want to put this, could it make sense to share some information about perhaps what's not going on, what the US Is not doing? Maybe. Is there a way to prove that you're not engaged in a new Manhattan Project to race to AGI, for example?
- [48:24]
Charlie Bullock
Yeah, I mean, I think there are win wins from sharing information with China. Right. Certain information, not all the information, obviously, but yeah, I think it can only be good for the US government and China to, like, exchange information that they both believe to be mutually beneficial. I think so far that hasn't really happened. I think, like, there was an effort during the Biden administration to reach out to China on AI and kind of have serious talks about it. And my understanding is that the US sent sort of their top people and their real, like, AI experts, and China sent their, you know, what's called colloquially like. Colloquially like Barbarian handlers. Right. They kind of sent their like, you know, just. They didn't seriously engage. Right. So at that time there was not an appetite. I think it would be good to continue reaching out to China and say, hey, like, you know, you want to have dialogues about this? There are efforts on going to do that. Right. There are track to dialogues, but track two dialogues are just basically like a group of random people from the US going and talking to a group of random people from China. Right. It's not that helpful in most situations. Right. It would be good for the governments actually to get together, but they're currently not doing that and I don't really know how to solve that problem. Right. I think there are efforts ongoing, both in China and in the US to kind of AGI public government. Hey, this is a huge deal. You should talk to China about it. There's probably some like, easy win wins lying on the floor, but it's like, yeah, kind of difficult to make progress on that, which doesn't mean we should stop trying.
- [49:35]
Gus Docker
Yeah, yeah. And then you have evaluations. This seems like an area where there's sort of room for, for more players. What you have right now is mostly third parties trying to develop benchmarks and then getting whatever information they can from the, from the companies before they release a new model. Is there a more structured way to do this? Is there a way where we can perhaps. Yeah. Doesn't this just seems to me like a, like an area where the government could spend money and get good results and good information and you could make that information public.
- [50:07]
Charlie Bullock
Yeah. I mean, so this is a lot of what Casey currently does and it's a lot of what the UK AC currently does is like do evaluations of models or subsidize pri. You know, independent third parties who are doing evaluations of models are incentivize companies to do their own evaluations of models. Right. I think like the science of evaluations is still like in its infancy. Right. There's a lot of progress that needs to be made and that can be made. It takes a lot of money, it takes a lot of effort, it takes a lot of time. This is sort of a prime example of a place where like, you know, everyone I think should agree that there's like, it makes sense to invest here, even if you're kind of skeptical or whatever, like invest on the off chance that it's useful because it's like very unlikely to be bad to know more about these models. Right. It's like, probably not going to hurt us unless we like discover some dangerous capability and then disseminate the information widely or something. So yeah, I think like there have been bills like what's called air. I forget what that stands for, but it's proposed by Senators Hawley and Blumenthal. I had objections to that bill. I thought there were things that it didn't do perfectly. But like overall it was I think a good idea. Right. It's in the Department of Energy. It's going to set up a, a system for, for mandatory evaluations of AI systems. Right. Mandatory is obviously a little bit burdensome on companies, but it's not that big of a burden. You just have to share your models with them at some point. Right. I mean talking to people in labs, they often think that like pre release evaluations are kind of very burdensome because like they want to release their models pretty soon after they invent them or something. It's often they're releasing a lot of models these days, so they don't want to have to go through the government every single time. But at the same time, I mean you could do post release evaluations, right? Just like, okay, you release the thing and then that doesn't address sort of the core issue of like, well, what if it's a super intelligence that destroys the world but it can like still provide information about future models. You can see where the trajectories are going. Right. You can like there are some risks that can be addressed even post release. Right. You can ask them to undeploy it or something. That won't help if somebody's like, you know, downloaded or whatever. But it, it can help if the, there's some like easily addressable danger. Right. Some current harm that, that people are worried about.
- [52:03]
Gus Docker
Then you also write about sort of beefing up security, securing the model weights and securing algorithmic secrets within the companies. Do you have a sense of how that project is going and how would the government help here? It's not a usual situation that the government is sort of helping private companies with their security. How could they directly help?
- [52:28]
Charlie Bullock
Yeah, this is a situation where perhaps it's not quite as unusual as we think. Right. AI safety people like myself have a tendency to sort of obnoxiously assume that they're the first people ever to encounter a certain problem and that they're like, let's just, let's from first principles address this thing with our huge brains from scratch. Like this is like, I think I've over time kind of come to see this as an issue that's like quite common and that there are already, like, ways of addressing. Right. Like, it's very common that the US Government pays a defense contractor to create a thing for them, and they want it to be, like, secure from cyber attacks. It's like, it's like every item of defense that has any sort of, like, cyber component. You don't want China sticking malware in as very important. Right? So, like, we already have existing, like, standards promulgated by the Department of Defense and like cisa, the cybersecurity agency, that, like, if, you know, if they're like, contract conditions where if you enter into a contract with the federal government to procure something for defense purposes, it has to be cybersecurity. It has to follow this long list of detailed standards about how to promote cybersecurity. Right. So I think what we need in the AA context is just basically more of that, right. Like a more intense version of that same process. Right. You have very detailed standards, right. That are very, you know, always being updated. And if you don't abide by those very detailed standards that are sort of, you know, perhaps administered by a third party or something like that, that's really good at making these kinds of standards, then you don't get your stuff acquired by the U.S. government. Right. And, you know, you could also apply that more broadly in the private sector. Right? Like, follow the standard or else, you know, you don't have a license to distribute your thing or something like that. Right. I'm not in favor of a licensing regime currently, but, you know, you can see how these have applications for sort of, like, future times when, like, more intense regulatory regimes are needed.
- [54:03]
Gus Docker
Yeah, I totally get your point here, that this is like a known thing, actually. But I'm wondering what it looks like in practice. Does it look like sort of banking regulation? Know, your customer laws where you have to fulfill certain obligations and you have to report something to the government. I'm just thinking, what does it look like for the government to concretely step in here and improve cyber security? It's not governments government agents in the building of anthropic.
- [54:30]
Charlie Bullock
No. Yeah, I don't think it is government agents in the building of anthropic. But I think what it is is, like, okay, there's only a few major labs right now, which makes it kind of a little bit easier to keep an eye on them. It's arguably bad for the concentration of power reasons, but it's good because it theoretically makes it easier to regulate or something like that. And so what you do is these companies all more or Less except for probably anthropic. Now for obvious reasons, want to be doing business with the United States military. Right. And the United States government generally. Right. So if you have a requirement that says if you do business with the United States government, here are the cybersecurity and physical security standards that your lab must abide by. Right. And if they're very like, good. If they're, if they really reach like, you know, Rand has a paper about different security levels, if they really reach a high security level and are very good at securing your model weights and stuff even from nation state level attacks, then you can do business with the government. It might be worth it. It might be in those companies interest to abide by those standards just so they can do business with the government. And that has the like tack on effect of okay, now they're safe from everything else because you know, these are general requirements for what your company has to do. They're not like specific to one contract or something like that.
- [55:34]
Gus Docker
Okay, got it, got it. Then there's the issue of allowing the government to hire more people and better people. What's the situation like now? Because I sense some frustration in your writing about this.
- [55:47]
Charlie Bullock
Yeah, no, I mean the situation right now is just, I think we're doing basically nothing. We have our head in the sand and we're not at all ready for AGI. Right. So the nice thing is there's nowhere to go but up. Right? I think, yeah. In a lot of. I mean this applies. I mean, we've been talking mostly about the U.S. because I'm from the U.S. this paper is one that I co authored with my boss, Christoph Winner, a brilliant guy who kind of came up with the idea. And he's European, he's involved in EU AI governance. So this is applicable both in the US and the EU and in the UK for that matter. Although the paper mostly focuses on US and Euro. But yeah, I mean most people agree that the government needs to get better at hiring AI piece. Right. The Trump administration's AI action plan says that it emphasizes it repeatedly. The Biden administration emphasized it repeatedly. So everyone kind of agrees that it's a key priority. Right. This is like a easy thing for politicians to say, ah, we need to hire more technical talent in government. Right. Well, there are existing ways to do that and they're often not very good. And this applies to the state level too. Right. Like California now has a P53 free. They need to hire very good people into the California government to, to. To boost their capacity. There are existing ways of doing that. They're like fellowships. Right. So private money can pay for an outside researcher, you know, to come in and, and work in the government for a while. Obviously there are alignment problems there. You want to make sure that the money's coming from the right place and the researcher has the right motivations. But it's a way of addressing the capacity issue. Right. The general problem is that, like, having technical talent with like, machine learning is like an extremely valuable skill these days. And so you're getting paid a ton of money often to do this at companies. So it's like a little bit harder to hire people to come. But I think like balancing this out is the fact that a lot of people realize this is a real issue. And there are a lot of people who are kind of motivated by impact and stuff like that and are willing to like, maybe take a pay cut to work in the government to help us all have a better future. So I don't know, I'm not totally hopeless about it, but I do think there's a lot of low hanging fruit here.
- [57:31]
Gus Docker
Is there a limit to how much you can be paid as a government employee in the US Is that, is that the main issue, that they just can't compete with the, with the packages available at the companies?
- [57:43]
Charlie Bullock
Yeah, and there are ways around it. There are loopholes that kind of get you to a higher level. But in general, I think in the US and in the eu, like, government pay doesn't usually go beyond a certain level. I mean, with certain key exceptions. So, yeah, it would be nice to have structural reforms sort of addressing that. It's not my top legislative priority, but I would definitely take structural reforms. I mean. But yeah, this is a problem that's been kind of thought about for a long time. There are provisions, the Defense Production act, for like an executive reserve of like super talented people that you can like, they can work in industry and then you can sort of surge hire them. Right. My organization, my colleague Alex Jumper recently wrote an article about surge hiring authorities in the US So there are ways to do it, but I wish we had better ways to do it, basically is the long and short of it.
- [58:26]
Gus Docker
Yeah. The final item on your list is about avoiding premature preemption. You got to explain what preemption means in this context.
- [58:34]
Charlie Bullock
Yeah. Okay. So in the US we have in our Constitution, Supremacy clause essentially says the federal government is the supreme authority. And state governments, while they're sovereign in some sense, state law is subordinate to federal law in some sense. So if you have a federal law and A state law that conflict the federal law rolls out, the state law is preempted. And the federal government can do this explicitly by passing a law that says this kind of state law is preempted. And there is currently an effort among. This is sort of the primary policy goal of the the tech right at the moment. Or you know, maybe it's unfair to label the entire tech right like this, but sort of like David Sacks and similar figures and basically the very deregulatory pro accelerationist people are want to preempt state AI laws, right? So the argument is that there is a patchwork of state laws regulating AI, right? Or that there soon will be one. Basically this is very burdensome. It's harmful to regulation to have 50 different rules. It's difficult to comply with because companies have to like, ah, we're getting regulated 50 different ways. It's like very hard to comply with all at once. It costs a lot of money, it slows down innovation. And it's especially hard the argument is for like startups and small companies, because while the big companies can afford to comply with 50 different sets of laws, small companies cannot. Right? So what's being proposed is that we basically stop states from regulating AI at all for like a period of 10 years or something. That was the original proposal that was wildly unpopular and it didn't succeed politically. Last summer that proposal was defeated. They tried to put some preemption into their reconciliation bill and it lost 99 to 1 in the Senate. It was closer than that sounds, but it was going to lose in the Senate by a significant margin. And so they're still trying to do that. That's sort of the ongoing actual debate. I'm kind of tired of thinking about federal AI policy right now because all that's getting talked about is preemption, which is like kind of an uninteresting issue to me at this point or something because it's like I want to be done with it. But I think from the optionality perspective, the optionality framing, preemption of state laws is really bad because you might need state laws to address real risks from AI at some point, right? Like there are lots of, there are lots of local harms that AI could create, right? And so the attempt to address this comes from like you distinguish between developers and deployers, right? And so maybe development is like developing AI systems frontier AI systems is maybe like a very sort of interstate commerce y task or something like that. So it makes sense for the federal government to regulate that and then states can address Laws affecting deployment. Right. So that's more local. Right. It's like if you are, if you're a business that uses an AI product. Right. That's more deployment level. The problem is that all of the burdensome state laws are deployment focused. Right. And all of the completely non burdensome ones like SB53 that require almost nothing are developer focused. Right. So like, yeah, I don't know. The White House just released a framework saying, okay, we're going to preempt all like developer based AI laws basically. And this completely fails to address the thing that they claim to be worried about, while also like not like you know, having it having quite bad consequences for safety and so forth. So I think it's a bad idea to do broad and premature preemption of state laws. This doesn't mean you never preempt state laws. I think like the best way to approach this is a, is a sort of iterative process where like the federal government, the state governments worked out over the course of time who should regulate what. This is how it's like always worked historically with other emerging technologies. And if I think like doing it any other way would be a bit foolish, honestly.
- [61:52]
Gus Docker
I mean, you can understand having preemption for state laws if you then have some federal framework or if you're introducing like one unified framework for governing AI. But that's not what's being proposed here. Right. What's being proposed here is mostly that you don't have state laws and then maybe at some point, but not anything concrete in terms of federal legislation.
- [62:14]
Charlie Bullock
Yeah. So they learned their lesson from the first time when it failed, that they had to have a quote, federal framework. This is the phrase that always gets thrown out. Federal framework. But I mean, it depends what the federal framework is. Right. The federal framework that they're currently proposing that they've just recently proposed is like we'll do absolutely nothing, everything will stay exactly the same and states can't regulate AI. Right. That's not a good, that's a federal framework. Right. It's just not a good federal framework. So I think a better approach is to do preemption on sort of a piecemeal basis. Right. You pass a federal law in an area and then you preempt state laws that are inconsistent with that law or contrary to policy or something like that. Right. And I'm not saying it has to be a strict one to one thing. Right. There are instances historically of like deregulatory preemption where you preempt state laws just because you don't like that kind of law. But it's never been as broad as like a whole technology. Right. Like people always bring up examples like the Airline Deregulation Act. Right. Which kind of stops states from passing a specific kind of well understood law that affected like airline costs or something like that. Right. The price of airline tickets you can't regulate. Right. And it wasn't the federal government that is that they wanted the market to handle it. Right. Just it's going to be the free market. We're not going to regulate airline tickets anymore, the prices of airline tickets. But what they didn't do was say no states can pass laws relating to airplanes. That would be insane. Because even though air travel is like this quintessentially interstate issue. Right. You rarely fly between one city and a state and another. Unless you're in like California, some very big state, you almost always. Right. You're going to have tons of laws affecting airplanes on the state level. Right. Is it legal to drive an airplane on a highway? The answer is no in most states. In some states the answer is yes sometimes. Right. How do you license pilots? That's the thing that gets regulated by the states. Right. So there's always going to be a ton of issues for any general purpose technology that like need to be addressed at the state level. And I think it's very foolish to do this extremely broad version of preemption which like any law related to AI. I think because AI is such a new technology, people don't fully understand like how bizarre that is. But like the idea that you would ever stop states from regulating electricity at all in any way is insane. Right. We have all sorts of state laws regulating electricity just like we have all sorts of federal laws regulating electricity. Because you just need a ton of regulations because it's a general purpose technology that has all sorts of risks along with its benefits. Right?
- [64:21]
Gus Docker
Yeah. If AI is as big of a deal as I expect it to be, and perhaps you expect it to be, it's going to show up everywhere. And so it's going to be. It would be a weird situation to have preemption for laws relating to AI, which would be like sort of a preemption on laws around technology or something like very general, very sort of the show up everywhere in general. How much do you think we can rely on historical examples when thinking about AI? So in this conversation you've brought up sort of examples from history about how we've regulated things. In your paper you bring up historical examples. How much can we rely on these historical examples given that AI might be quite different from other technologies, given that AI might be a bigger deal, perhaps even than electricity.
- [65:08]
Charlie Bullock
Yeah. I mean, well, the best thing to do would be to look into the future and see what's going to happen then. But that's impossible. Yeah, that's impossible. So I think historical examples are often the best thing we have. Right. I think it's unsatisfying to a certain kind of thinker because it's not like you're just kind of looking at the past and making a rough analogy and then guessing what's going to happen. Right. I like historical examples because I think that accurately reflects how hard it is to make accurate predictions about the future. Right. All you have to go on is what happened in the past. Most of us do not have a perfect model of the world that allows us to completely understand in intricate detail everything that's going to happen in the future. And I think people who claim to have that kind of model are generally mistaken, even if they believe it themselves. So, yeah, I mean, I do wish we had better, something better than history to rely on, but I don't think we do. Right. I mean, there is. Maybe I'm being a little disingenuous. There is a balance to be struck between just looking at history and actually looking at the thing and what it does and building an accurate model of it and then predicting how it's going to go going forward. But I think a lot of these decisions are complicated and difficult enough that as mere humans, the best we can do is look at history and see what it tells us.
- [66:12]
Gus Docker
Yeah. Maybe we should think about AI as sort of a technology that's going to be a combination of something, something as potentially impactful as nuclear power, then combined with something potentially as destructive as nuclear weapons. So you have the dual use in one analogy almost. I guess that's one of the best sort of historical examples I've heard about.
- [66:38]
Charlie Bullock
I love the nuclear analogy. People have written sort of Google Docs about the nuclear analogy that get passed around among AI governance researchers. I. Some people hate it. Like Rand has a whole paper about how overused the nuclear analogy is and how they hate it. Right. Like, I think Greg Smith worked on this for, and it's a really, really good paper, but it's basically saying like, stop using nuclear analogy already. And I use it in radical optionality. So I'm like, no, I don't think I will. I'm going to keep using it. I'm going to use it even harder.
- [67:02]
Gus Docker
Yeah, yeah.
- [67:03]
Charlie Bullock
Well, yeah, I think it's a great. I think it's a great analogy. Yeah.
- [67:05]
Gus Docker
Yeah. Great. Great chatting with you, Charlie. It's been. It's been great.
- [67:09]
Charlie Bullock
It's been really fun. Thanks, guys.