← The AI Policy Podcast
The AI Policy Podcast
Unpacking the EU AI Act Code of Practice with Marietje Schaake
00:50:53
Marietje Schaake joins to unpack the EU AI Act Code of Practice, its drafting, compliance, and systemic risks.
Loading summary
Transcript72 lines
- [00:00]
A
Foreign.
- [00:11]
B
Welcome back to the AI Policy Podcast. I'm Gregory Allen and today we've got a really important episode at the center of the AI Regulation and Governance conversation. If you are following this podcast, you know that the EU AI act is a landmark piece of legislation that has been taking an awful long time to implement. And now some of the big machinery behind that regulatory approach is now in effect. And one of the most important things in it is the code of practice for general purpose AI models. This is a very complex document, but a really important one, not just for Europe, but for the entire world in the future of AI regulation and governance. It's also a pretty complex document and that's why we're extremely lucky to have a guest here today, Maricha Shock, who is one of the most important drivers behind this code of Practice. She was one of the working group chairs that drafted the Safety and Security section of the Code of Practice. She also has a distinguished background as a former member of the European Parliament. She also writes a column that you may have read at the Financial Times, which is excellent. And her current day job, I guess you would say, is that she's a fellow at Stanford's Cyber Policy center and the Institute for Human Centered AI and is also the author of the 2024 book the Tech Coup. So, Marie Cishak, thank you so much for joining the AI Policy PODC podcast.
- [01:36]
A
Great to be here. Thanks for the introduction.
- [01:39]
B
So we're going to get into the meat of the code of practice here, but before we do that, I want to understand a little bit more about your background. How did you come to work on tech regulatory policy? How did you become interested in AI? What's sort of the starting point of your career, which has done a lot of different things?
- [02:01]
A
I would say two starting points are important. One is when I studied at the University of Amsterdam, I I was constantly looking for how to apply any knowledge that I gained while studying. And I didn't even find it so easy. I studied American Studies, by the way. So, you know, like many I, I will claim to be an expert on American politics even though it's mind boggling what's happening there now. But there was a new miner at the time that was a new media. And new media meant questions around what is the World Wide Web? What is this Internet that is emerging? We're talking 1999, 2000, roughly speaking. And so I decided to enroll. And this was because I always had a curiosity about where change was coming from. And so there are many things that induce change. But of course Technology is one of them, and you could argue policy and politics is also one of them. So I studied it a little bit, was always curious about technology and as an outsider in politics, in my late 20s, when I ran for European Parliament for the first time, this coincided with the Barack Obama campaign in the United States, the emergence of social media platforms, where maybe this was the case in the US Too, you would have to tell me. But in the Netherlands, it was also this early phase where social media platforms were really platforms for political discussion. So it was a smaller group of people, maybe journalists, policymakers, experts in some fields that would gather there. And so I saw it as a way to reach audiences where I just didn't have a profile for national media at all. And so that was a sort of organic entryway into using the technology. But also, you know, when I got elected at the age of 30, being seen as someone who represented this younger, more tech savvy gener, I had this curiosity, this background in studying it. And so I continued to also just look at this intersection of technology and society, technology and politics. And then what really accelerated my role as a policy lead, as a representative, focusing on these areas were the Arab uprisings. And so one of the first.
- [04:30]
B
You talking about the Arab spring of the 2012 timeframe?
- [04:33]
A
Yeah, but it also started in Iran, if you remember. So I was elected first in 2009, and this was when the Gree movement happened in Iran as well. And one of the first trips that I took abroad was in early 2010, because I went to eastern Turkey to meet with refugees from Iran. You know, these activists were fleeing because they were being harshly persecuted, imprisoned, tortured, the worst conditions. And when I met these activists of which we had heard they were organizing on social media, you know, there was this famous delay in the update of Twitter, if you remember, under the Obama administration, because there was concern that then the Iranians couldn't post the eyewitness accounts of what was happening. And so we were very familiar with the story of sort of Facebook revolutions, Twitter revolutions, and that whole narrative, which was, you know, very flawed. But when I met these activists that had fled to Turkey, I also learned of their deep concerns about surveillance, about hacking. And I decided to dive in deeper. And so one of the first, I.
- [05:42]
B
Guess I should just say, like, this was a pretty idealistic moment in terms of what technology meant for the future of politics. I remember, I think it was 2012, when then Facebook CEO, now Meta CEO Mark Zuckerberg, wrote a post in which he basically said that the growth of Facebook is Sincere, synonymous with the spread of democracy. And that, like, everything that we're doing is bolstering free speech, liberty. And as you just saw, like Arab regimes like Libya fall over and Tunisia, it looked like Facebook was this tool for democratic promotion that was just going to usher in this beautiful era of peace and political harmony. Didn't quite work out that way, but it was this, this really idealistic moment. And you had a really interesting viewpoint on it from your seat in the European parliam.
- [06:34]
A
Yes. And being able to travel to these places, Iran first, but later also Tunisia, Egypt, meet with all these activists and hear their eyewitness accounts, their firsthand accounts of being arrested, being tortured, but being confronted with their own messaging, with their own emails. And then I decided to dig in further and I learned that some of the technologies that these dictatorships were using against bloggers, protesters, students, you know, human rights defenders, opposition leaders, were sophisticated surveillance tools. I think that was the sort of less idealistic story about what role technology played in these uprisings. But they were also made in Europe. And, you know, it was a real wake up call for me because on the one hand, we sort of had universal outcry of the repression of people. There was also, particularly from the democratic world, us, EU countries, the eu, a real hope that indeed this youthful uprising cries for more justice, more equal treatment, more democracy. We're going to succeed, and that technology could play a role in this. But then how could it be that our ministers were condemning the repression of these activists? But then our quote, unquote, European made technology was the tool of choice to repress. And of course, it's different if you can already surveil and intimidate and limit people before they have to take to the streets or before violence has to be used. So it's actually a very sophisticated tool because it's less visible. And it was also, you know, potentially not as violent, but certainly very effective. And so it would maybe lead to less outcry, et cetera, which I think was the case because it took forever to get restrictions on spyware. And if we look globally now, the market in spyware is still way too powerful and unregulated. So those were a couple of years where I spent a lot of time at the intersection of human rights and technology, democracy and technology. But it was mostly a foreign policy issue, I should add. And then, of course, as the years progressed, technology started to be used more and more. The tech companies, including the popular ones that we know, became much more powerful. So then all kinds of angles like antitrust Cybersecurity, copyright protection, export controls were added to that portfolio of policy issues that I worked on. But I have always been and continue to be motivated by the question of how can we make sure that the core values that define our quality of life, the rule of law, democracy, human rights, are the core lens through which we treat technology, that we're not too frivolous or too careless about these core principles as we celebrate the marvel of innovation, the hopes that we have of what beautiful solutions technology might bring. I mean, we're now in the age of AI. This is an AI podcast. I mean, you probably have a comprehensive list of promises that have been made by CEOs about curing cancer, ending climate change disaster, breakthroughs in science.
- [09:41]
B
Don't forget about ending poverty. We're going to end that one, too.
- [09:44]
A
Oh, that's true. Yeah, I forgot that one. There's such a long list of what AI is promising to deliver, but I think it's become my role to also ask myself, well, first of all, is this a hype? Second of all, what could it mean in a different context? What could it mean for the most vulnerable? How do we make sure that we don't just hope, but that we have safeguards? You know, where do checks and balances come from? And so on and so forth.
- [10:12]
B
Excellent. And, you know, I think when most people think about European regulation of technology, they're not thinking of those initial spyware kind of debates that you talked about. I think the landmark legislation that most people sort of anchor on is gdpr, which of course relates to privacy and therefore has a tie to spyware. But were you part of that sort of transition?
- [10:40]
A
Well, we were all voting on it, but I wasn't as directly involved. But it was definitely a piece of legislation that also focused the minds in Europe. And I think, you know, part of why data protection has always been such a key issue for Europeans is because of our recent past with fascism. You know, I sat in the European Parliament with colleagues who grew up under communism as well.
- [11:03]
B
Yeah.
- [11:03]
A
Yes. But that's, you know, clearly an anchoring identity for Europeans and one that, you know, most of us would hope never to repeat.
- [11:13]
B
Yeah, I'm going to just interject with a. A fun fact that has always blown my mind, and I always anchor on this in discussions about surveillance and technology. So if you've ever been to Berlin and you haven't been to the Stasi Museum, you've got to go. So this is the original headquarters of the European East East Germany's secret police, and one of the amazing data points that they have is that it turns out that one out of every 67 East Germans was either working directly as a Stasi employee or was an officially, you know, known informant. So think about how labor intensive the surveillance apparatus of that repressive society was. One out of every 67 humans in the country, of working age or no, not even of working age, of the.
- [12:09]
A
Whole country, you could probably remove.
- [12:11]
B
So if it's a working age, it would probably be like one out of every 40 people. People or something like that was working for the surveillance apparatus. And that's because they didn't have, you know, modern technology where computers can handle so much of the surveillance burden and AI. And so, yes, you're, you're totally right to point out, right, that those memories loom incredibly large in the minds of Europeans as they're thinking about privacy and other things.
- [12:37]
A
What's interesting is that, you know, in those years, the focus of Europeans was much more on data protection, while the focus in the US was already on data mining and, you know, training AI models. So I think, you know, one of the lessons learned there, and there are many when it comes to general data protection regulation, including how important enforcement is and not just the letter of the law, because the enforcement has, has just not really been up to par. But it's also that sometimes there can be a tunnel vision, right. If you, if you're focusing on one thing, maybe another thing is happening in the meantime. So it's, it's been a really interesting lesson learned. But yes, I think the reason why the GDPR is such an important, an important starting point of sort of tech regulation for Americans is that it was the first time that US companies started to take EU regulations seriously. So until then there was kind of sort of, you know, laughing and dismissal of what European regulation was. It was sometimes framed as a bit emotional and, you know, typical for Europeans, but not something that the tech companies and others, by the way, would have to reckon with. Because as you probably know, the general Data protection regulation regulation is a horizontal law. It applies to, you know, the local swimming pool as much as it does to Google. And so while tech companies have kind of framed it as a weapon to go after the US tech companies, and maybe unfairly so this is really one harmonizing 27 different systems of each member state into one standard, which is something that a lot of EU regulation is about to make sure that we harmonize over time. But it's, it also applies to everyone. But because it, it hit home in Silicon Valley that you Know, this really mattered. They had to change their. There could be serious fines. It is maybe the starting point of history for many in Silicon Valley.
- [14:27]
B
Yeah. And so coming back to you, how did you make the jump to interest in AI specifically?
- [14:35]
A
Well, again, I think this was organic. First of all, some of the things that we used to call social media, we now call AI. So it's just if you're interested in tech, you're sort of interested in AI.
- [14:47]
B
As a sort of tech policy interested person. As AI rises in prominence, it naturally falls on your desk. I mean, there's a very similar phenomenon in the US Government where the sort of first tranche of AI policy people in the national security community were almost invariably cyber people or space people, because those are the, you know, people who like, understand technology things. And so it just sort of naturally falls on your desk, I'm sure.
- [15:13]
A
Well, but it was also accelerated by the fact that. So when I left the European Parliament in 2019 after 10 years there, I got a job offer at Stanford and it was a double appointment, one that I still have one with the more political science kind of research clusters at FSI, wonderful place run by Mike McFall, which also has the Cyber Policy center, which is indeed the starting point is policy. But then I also had an appointment at this very new institute at the time called the Institute for Human Centered AI. And there the focus was really on AI. And so all my colleagues there are focusing on AI exclusively. And so as a result, I sort of, you know, deepened my focus on AI, was appointed to the UN high level advisory body to the Secretary General on AI. So, you know, that allowed, allowed us all to take a much more global perspective where, you know, before that my, my remit was mostly EU transatlantic, because I served on the US delegation of the Parliament and because so many companies are American. So it's just grown by the various things I've done. And then I became one of the chairs of this code of practice that I know you'd like to focus on as well. So it's been organic. But I don't consider myself an AI expert, if that makes sense. I'm much more of a broad tech policy person. And I just think we all have to wrap our heads around what AI means in our society and sort of weed through hype and hysteria. And to a large extent the same policy questions apply. If you ask me, you know, how do we make sure they're.
- [16:47]
B
Yeah, go ahead, let's turn to the code of practice. So first I kind of want to understand like the machinery for how this came into existence. So the EU AI act, you know, has a bunch of thou shalts, companies shall do X. And then there's the question of, like, how does the government and how do even the companies know and show that they are complying with the law? And it seems like a lot of the answer is going to be found in this code of practice. So help us understand, like, what is this thing? How does it fit into the larger picture?
- [17:23]
A
Right, let me start with the larger picture because I think it will help listeners understand. So in your introduction, you said the AI act is this law that's taken, you know, ages to implement, but actually by legislative standards, the process has been.
- [17:38]
B
Quick, draft and implement. It's taken ages.
- [17:40]
A
Yeah, it took about two years to draft, I would say. And you know, the irony is of course, that when the idea is to fast, vastly implement, companies say, hold on just a minute, we need time. And other stakeholders may also say we need to adjust because, you know, institutions have to be set up or reorganized in order to do the enforcement. So, you know, there's often a reason for time spent, which is to prepare stakeholders. In this case, AI companies that I think, you know, should be capable with their, with their multiple billions, because that's the group that has to do most of the preparation. But, but, you know, we can get to that. So when the work on the AI act started, a lot of people said, oh my gosh, the EU is too quick. Right. Why would you regulate AI now? You're going to stifle innovation, which is sort of the most heard mantra about the EU and regulation anyway, when it comes from tech and AI people. But the decision was made that this was going to be a very deeply impactful technology and that there needed to be a sort of conceptual framework in the law of how to assess risk and mitigate risk. And so the vast majority of use cases is not considered to be risky. But then there's a risk spectrum, you know, that goes from low level risk to high level and even unacceptable risk, which is how the law is built up. And so you could say it's very much comparable to liability rules and, you know, safety rules in anything from, I don't know, machinery to pharmaceuticals. Right. Like, where do we see the most risk? What are the obligations of the providers, the makers of this product, to mitigate the risk? But then while the law was being drafted, you know, generative AI broke through and the question became, as AI was progressing so quickly, what do we do with more general purpose AI models? So where it's not clear that it's used for biometric identification or for scanning someone's CV for employment.
- [19:43]
B
Because those risk classifications that you mentioned, clear, those are use case breakdowns.
- [19:48]
A
That's right.
- [19:48]
B
Like, this is high risk, this use case is low risk. And so what do you do when you have AI capabilities that don't just have one use case, they have potentially, you know, infinite use cases.
- [20:01]
A
Exactly. Thank you. That's an important clarification for those categories. A group of independent experts were invited to help sort of create clarity for companies when they were subject to this part of the law and how they might comply. And this is a process that's still unfolding where the European AI officer, the European Commission, still is doing work with regards to standards and clarifying what these companies should do. But as independent experts with various backgrounds, a lot of actual AI experts, but also some others, like in the space of copyright, or in my case, in policy, were asked to basically clarify for companies what compliance with the AI act could look like. And so it's sort of, you could see it as a sort of roadmap or series of steps that when companies take these, they show good faith, of course, but they also show, you know, their, their willingness and ability to comply, and it sort of maps out how they might do that. But the law applies to anyone. I think that's really important. So I. Even if you, if you're a company that doesn't want to sign the AI Code of practice, you still have to comply to the AI act, but then it's sort of up to you how you want to do it, and up to the AI office to see whether, what you do, if it's not following the code of practice, whether that's sufficient. So these are not two different laws. The one is the law that's the only thing that's legally binding, the AI Act. And then the code of practice is sort of a series of steps that companies can take and adhere to. So sign up to. That's where you have this whole question of signatories, are companies interested in doing that, signing on, committing to doing this? And actually, quite a few companies have, yeah.
- [21:52]
B
So breaking that down, the law says something like. I don't think it uses these exact words, but something like companies must take adequate precautions to ensure safety and reliability. And the question then becomes, if somebody sues you and you're going to go in front of a judge, how do you prove that you took, quote, adequate precautions? And the EU AI office has basically said, hey, if you do everything in this code of practice, we will treat that as though you have taken adequate precautions. So some companies have signed up to that, like Google OpenAI have signed up. They're like, this is what we're going to do in order to take adequate precautions. Other companies like Meta have said, you know, we hate this document and we know better what adequate precautions are than the people who drafted this. And if it goes to court or if the regulators come after us, like bring it on. I think is sort of the nature of the conversation. Right?
- [22:54]
A
Yeah, presumably, I mean you'd have to invite Meta to explain their reasoning.
- [22:59]
B
I doubt they would actually go on the record saying bring it on. But you know, ish.
- [23:02]
A
Well, I mean I've heard, I've heard pretty, pretty scathing statements by Joel Kaplan, you know, as maybe he was hired to send into the ether. But you know, I think they've, they've got to more and more brazen as a company.
- [23:16]
B
So now what? Now that we understand like the context for why this document exists, let's actually get into the machinery of it. And I want to focus on the part where you were a co chair, which is the safety and security chapter. But just for the context of everyone, there's a section on transparency which I think is very blase. I mean anybody who is familiar with a model card in AI, this is just sort of like, like the biographical data of a model. It's pretty tame, I would say, in terms of what it's asking for in terms of transparency, then you've got the copyright section, which is like, how do you handle intellectual property and copyright? But I want to mostly put those to the side. And now talk about what is undoubtedly the longest part of the code of practice. This is like 40 plus pages, whereas those other ones are like 6 pages each. And it also includes the most commitments. There's 10 commitments that signatories are signing up to when they do this. So sort of I realized it's a, it's a big document, it's very complex. I'm going to defer to you as to like, what's the, what's the best way to walk us through this story?
- [24:22]
A
I think we should just think about it in pretty basic terms. So the idea is there are general purpose AI models and there are general purpose AI models with systemic risk. And for those general purpose AI models with systemic risk, the providers have to show the steps in sort of, you know, formalized documentation and information sharing, manners of how they assess the risk, what they've done to mitigate it, and for how long. For example, they have to keep the documents, who they have to report it to, when they have to report it. So it basically spells out a way, again, companies can choose to do other ways, but a way to be rigorous, but also identify the people within the companies, for example, who should have executive responsibility so that it's not some, you know, niche group of juniors who are assessing the most serious risks, but that it actually is also in a governance sense, you know, with those responsibles on the executive level.
- [25:27]
B
Yeah. If you're, if you're a part of the safety team and you're not allowed to say no to anybody else in the organization, are you really part of the safety team?
- [25:36]
A
Yeah, well, that's a big question. And so there's also provisions for whistleblowing, for example, because we want to make sure that considerations for society writ large, for national security, for public health, et cetera, are informing decisions that companies make instead of the business cases, the market competition, and everything else that's driving these companies.
- [25:58]
B
So I definitely want to get there, but could we rewind one second? Because you basically said there's two paths you can go down. There's general purpose AI models and there's general purpose AI models that, that pose some kind of systemic risk. How does the law, how does the code of practice determine whether you're in one group or the other?
- [26:20]
A
There's a lot of responsibility for the companies themselves, but there are also standards forthcoming that will clarify more and more, you know, the details there, which the AI office will do.
- [26:31]
B
Okay, so it's not obvious and set in stone right now which general purpose AI models pose systemic risk and which ones do not.
- [26:40]
A
And it's also because, you know, you and I, or even, you know, my colleagues at Stanford who are deeply, deeply in the weeds of how AI models work, don't have access to this information. I think that's really important to understand. So a lot of what these companies do and decide in the process of building their models, testing them, putting them out into the market, tweaking them and so on, is of course entirely proprietary.
- [27:02]
B
Right, Terry, if memory serves, you know, the, the Biden administration, one of their executive orders, they looked at total amount of processing power used in training as a proxy for power and power as a proxy for risk. And the Biden administration sort of knew that was a very unsatisfying definition. But they're like, look, this is what we've got right now. And I think, I think an earlier draft of the EU AI act, maybe it didn't make it into the Final version, I forget, was also looking at like this 10 to the 25th flops used.
- [27:34]
A
Yeah, it has that. But of course, time progresses and models may become more capable. And then there's also questions of downstream uptake. So some models may reach many more people than other models. Right. So there's the power of the model, but then there's also the reach of the model. So I think those criteria would naturally be fluid. Right. I mean, this law is written now, but who knows where we are in terms of power or are new risks.
- [28:05]
B
Surely there are some models right now where we know, like that model is subject to the systemic risk evaluation. Is that not the case?
- [28:15]
A
I wouldn't be in a position to just say this model or that model because it's the companies that have to really combine a whole bunch of information for that.
- [28:23]
B
Interesting. Okay, so let's say we're in the world where the model has been designated as one, I.e. general purpose, with the potential potentially posing some kind of systemic risk. Now what happens? Like what are the. What are the commitments and what are the companies going to do?
- [28:42]
A
Also the companies are going to walk through. I mean, I don't know the code by heart, but you will see all kinds of steps in there. It's online for everyone to review and it's fairly technical, so I don't think it's interesting for listeners, but basically the idea is they commit to a number of steps of how to assess their models, how to also document what they found in this assessment. It's not just scouts honor, we've assessed all clear. I mean, this has to be documented for a certain period of time, has to be reported. So, for example, there can be a model that is being developed. So it's not on the market yet, but the developers have the idea that this may reach a certain threshold or a certain impact, or they may see certain risks. Then there's a period within which it has to be notified. You know, the information has to be. Has to be preserved.
- [29:37]
B
And notification in this case is to the EU AI office.
- [29:41]
A
Yes. And the AI Board, I believe, at the same time, or there's just hopefully going to be one contact point. But there, there's the AI office, which is sort of the central office at the EU level, which, which falls under the European Commission, which is the executive branch of the European Union. But then there are also representatives from the member states because, as is often the case, EU law still gets enforced and implemented on the local level, meaning country by country. And so every member state, the Netherlands, Germany, Italy, Spain, you know, France can designate the authority that is best capable within its own structure. So this could be the market authority, or this could be the data protection authority. You know, how oversight bodies may have different tasks in different contexts, but in any case, they need to be equipped with the right resources and mandate on the local level to independently verify whether companies comply. And so when you're talking about this transition between the law being politically adopted and entering into force, these are the kinds of processes you should think about that need to be executed. Meaning, you know, the offices may have to hire new people, they have to have AI experts that they may not have needed before, et cetera, et cetera. So this is all now happening, and representatives from the member states are part of this AI board. And then there's the AI office, if that makes sense. Cool.
- [31:12]
B
So there's a few products that companies are sort of required to produce as they sign on to this code, and these are things like, like a safety and security framework, safety and security model reports, serious incident reports, and some public access stuff. So I think the safety and security framework is doing a lot of, a lot of the work in how the EU is trying to deal with serious risks. But before we go there, because I want to go there next, but before we go there, there, can you just talk a little bit about, like, what are the big risks that the EU is trying to solve with this code of practice for. And I'm thinking specifically here of these quote, unquote, systemic risks. So, like, another way of putting the question is, like, what is the problem that the EU is trying to solve in terms of systemic risks?
- [32:07]
A
Well, you should think about risks to national security. You know, we all know the famous examples of, of bioweapons, and might it be easier for people to develop those through models? But it could also have implications for public health, for example, if certain, I don't know, like toxics or whatever could be easy to understand and spread or whatever through AI. But there's also a responsibility to sort of keep thinking about systemic risks.
- [32:37]
B
So I think cyber security was also on the list, of course.
- [32:40]
A
Yeah, loss of control is definitely there. So if, if AI models kind of go off on their own and create all kinds of problems through cybersecurity or otherwise. Yes, absolutely. But there could be other things, right? I mean, we could learn more about new models and what problems they create. I think that's the tension there. You write the law at a moment where this technology is evolving so quickly. And by the way, important jurisdictions such as the US not taking Comparable measures, but you have to kind of both be specific because that's what people need in order to comply with the law. It's what companies ask. But you also need to be flexible in order to anticipate what might be coming. And so I think we've constantly been in this tension of, okay, well, what do we know we need and what do we not know that might happen? And how can we still write it down so that this law is sustainable and it's not outdated, you know, three years from now, but at the same time is specific enough. And so there's also the. The opportunity for the AI office to update with new risks that need to be considered. But also the companies are just expected to act in good faith vis a vis systemic risk based on, for example, the body of scientific work, you know, latest information that is out there in the. So that when basically everybody knows that there's a new problem, the company can't say, well, we had no idea, because we just, you know, have to assume that they will study and be aware of the latest in AI, including systemic risks.
- [34:24]
B
Yeah, so let me chime in a bit here, because you've caught me just after I published a report on AI and bioweapons risk with my former colleague Georgia Adamson. So I'm chomping at the bit to talk about this systemic risk thing. So pardon me while I butt in here. I think it is a extraordinary gift of human history and fate that it turns out nuclear weapons are hard to make, right? And the reason is that separating uranium 238 isotope from uranium 235 isotope is just really, really hard. Hard. It's expensive and complicated. Saddam Hussein spent billions of dollars on it and failed. And so thank goodness, right, that nuclear weapons are expensive and complicated to build. If nuclear weapons were as hard to build as Legos, we would all be dead. We would all have already died if nuclear weapons were as hard to build as legos. And so the fact that these weapons of mass destruction are expensive and complicated is good for humanity to be. Now, you look at other kinds of weapons of mass destruction, like biological weapons, and it. The. The question you have to ask is, well, is that expensive and complicated now? And could that change over time? And AI is one of these areas where if you look at the existing safeguards, both those that exist in nature and those that exist in human society and technology, it's just getting a lot easier to create new organisms, to design changes to existing organisms, and then to go create those organisms. And so the plausibility of a pandemic that is human created and not human created by, like, North Korea spending many years to develop bioweapons, but by, like, one or more, you know, groups of people. Like, I think we're all familiar with the James Bond villain trope of, like, an evil genius on an island somewhere with massive resources and massive brains developing WMDs. But what if WMDs were within the reach of an evil average person? And that is the AI and bioweapons risk. And what's really interesting is that, like, it's not just, you know, national security types like me who care about this. Even the folks who are typically associated with being most hostile to AI regulation, like Meta CEO Mark Zuckerberg, he's like, yes, bioweapons is in a different category. Like, we absolutely need to do something about this. Because, you know, know, even if nine out of ten companies are deeply responsible and put in place a lot of safeguards, it only takes one bad actor to, like, screw us all over and cause another global pandemic. Similarly, in, you know, cyber weapons error, the sort of question is like, well, what if AI models, because they're very, very good at writing computer code and assessing computer code, can make it easier to generate offensive or malicious computer code? Do we want, you know, a group of three or four cybercriminals, or maybe even one, to have the level of capability that today we normally associate with nation states in cyberspace? So those are kinds of systemic risks where AI, as a helpful technological enabler, can enable a lot of good, can enable a lot of bad. And so the question is, what are companies doing to, like, minimize the chance that it's going to be enabling an awful lot of bad. And I think that. That. Tell me if you disagree, but that those. That category of risks is what the code of practice was intended to go after.
- [38:09]
A
Yes, I mean, part of the code of practice, indeed.
- [38:11]
B
Right. The safety and security part of it.
- [38:13]
A
I think it's a. I think it's a good. A good summary. You know, I think I see a really important analogy in what you're saying about, you know, how one company can spoil it for the rest, not only for, you know, life. Life on earth. And maybe all of us get impacted by a new pandemic or what have you. But I also think AI companies should think about how bad actors, maybe some of them, can really harm the trust in AI as a technology for many. And so I've heard a lot of criticism, of course, of the AI act, especially from the US but no surprises with this administration, but also from the companies. But I also see clear advantages which is, you know, this is a way that you can facilitate compliance and show that, that this is a trustworthy technology that can actually be assessed for risk. While you know, I, I don't know many AI experts who think that this is a risk free technology. You know, not that long ago we had academic experts, CEOs pending all kinds of letters that this needed to be paused, that there was going to be the end humanity potentially and you know, all the catastrophes you're talking about. But there's also risks in the present day. So forget about systemic risks that are horrible and non controversial. There are also problems in the here and now that could seriously impact the trust that people have in AI writ large. Cybersecurity on a lower level being one of them. Discrimination of course, impact on democracy being another one that I think is really important. And so, so I would imagine, and I think that the test case will be fascinating to observe between the US and Europe and other jurisdictions. Will this legal landscape where there are mechanisms, one, will it be sufficient, will it work out in practice? But two, will it actually help build trust in the technology? That has a lot of questions about what is it ultimately going to do? Are all these promises of good stuff going to materialize or are we going to continue to suffer from harms in this day and age, but also in the future? And so the analogy that one company can ruin it for the rest, I think applies broadly and I think here laws and standards can really help build that trust and predictability, which is something that companies typically want. Now that the federal government in the US is not going to adopt much regulation at all around tech, it will be really interesting to see not only what individual states will do, but also what the responses by the companies will be. Because this kind of fragmentation, different laws on relatively small scale of amount of people makes the work of companies harder because they have to comply differently for different areas, different states, potentially different countries. And so it may well be that when, I don't know, California comes up with a strict law, either that there will be the California effect, the way that there is a Brussels effect, that the highest standard will apply to all, or that companies will begin to lobby and say, hey, just why can't we have this Standard Instead of 5, 6, 15, 20 different states doing different things and us having to figure it out for every different state. So I think the whole process of how to regulate emerging technologies, including AI and what it means for the impact that it has on trust, but also on safety and other aspects. It's really interesting to observe now that we have such a big difference between the US and the EU in real time.
- [41:51]
B
Yeah. So the going back to the products that we're talking about, I think the safety and security framework is one that's really important for folks to understand. So can you talk a little bit about, like, what is a safety and security framework? What do the companies have to do and how's that going to interact with the EU AI Operations Office?
- [42:13]
A
Well, I think it's. It's what I just mentioned, so it spells out in greater detail and I don't know it off the top of my head, but all kinds of criteria that the companies have to, you know, assess for before placing it onto the market. But also while it's placed onto the market, they have to keep monitoring lessons learned, especially as bigger models get tweaked and used downstream, if that makes sense. So. So one company may put a model out into the world, but then others may use it in different contexts. And what happens there?
- [42:44]
B
Yeah, Just to give an example, Anthropic Claude is used a lot as coding assistance, but it's not just like, you might be a customer of Anthropic and not know it, because you might not be buying Claude directly from Anthropic. You might be buying it through a company like Windsurf or somebody else who adds on the sort of additional capability to make it more useful as a coding assistant. And you can imagine that, you know, that kind of downstream implications. There's a question of, like, who's really responsible to make sure that this thing is safe, secure, and abiding by regulatory requirements. So that's sort of like, yeah, I guess that sort of overarching question about, like, who's responsible for what in these obligations. The original developer of the model, somebody who's the actual person who's selling the model to the end user versus, like the end user themselves. Themselves.
- [43:39]
A
Yeah. And I think just, you know, from a practical point of view, this law is, is more about the upstream provider. So the, the handful of companies that have these rare capabilities of developing these very advanced models or the most advanced models today, could be different a year from now. But that is also just practically, and this is not spelled out in the law, but I, you know, as a policy person, just think it's helpful to think this True. True. If you think about it from a practical perspective, where would you want to intervene? Would you want to monitor the handful of companies that have these unique capabilities or would you want to have compliance from thousands, if not tens of thousands, if not hundreds of thousands of smaller players downstream? It makes no sense from a practical enforcement point of view either, but also because these model providers have the biggest insights, biggest handle on how these models get developed, designed, and they can see things that we can't see.
- [44:33]
B
Yeah. So, I mean, I think, I think it looks different on a problem by problem kind of a basis. Right. If you want to think about the issue of deepfake imagery, and some countries have put in place a requirement that deepfake generated imagery have associated metadata tags or like watermarks inside the image, so that, you know, anybody who has, like the appropriate tools can look at this and say, say this image is real, this image is a deep fake. You know, it's, it's built into the image. Well, if you want that regulation to exist, you really need to go upstream to the people who are creating the image generation software. Right. Like a watermarking requirement where the implementation is at the very last step of, like, the consumer who is using Photoshop has to go in and like, manually add a watermark. That is a dumb way to address that regulatory hurdle. However, however, like in other areas, like you can't stab people with a knife, like, the regulatory burden doesn't fall on the knife manufacturer, it falls on the consumer who is using the knife. And so, like, whether you want to target upstream or downstream, I think kind of depends on the nature of the problem you're trying to solve. And if I understand you correctly, you're saying that most of the problems that this regulation is looking at tackling are the ones that the EU has identified as better tackled in the upstream stream. Right.
- [46:01]
A
It's because they would be systemic. So the moment that it spreads, that becomes the problem, if it makes sense, you know, so the risk would grow if more people had access to the models.
- [46:15]
B
I see. Okay.
- [46:17]
A
And so I think you have to think about, you know, closer to the source is the best place to prevent the risk. But also who is ultimately the sort of owner of the model and the knowledge that goes into it and who has the levers before it gets put into society or the market? Some of them get released, some of them get sold is of course, the model providers. They have the resources, they have the insight, they have the knowledge more than anyone else. And, you know, I think one of the problems, and this is not in the, in the law, but I think, you know, know this will be a problem that we all have to reckon with is how unpredictable AI ends Up being. And so the question is, how much risk can you mitigate for at all? And that is a risk in and of itself. Because, you know, anthropic CEO Daniel Amoda has said it's a public secret that nobody really knows how generative AI works the way it does. You know, lots of people have basically said how unpredictable these models and their workings are. And I think it's hard to work with existing law and enforcement models to anticipate so much uncertainty. But that is something that we will have to also learn as a society, reform our legislative processes, our enforcement processes, because I just don't think that the sort of traditional legislative and enforcement bodies and processes are equipped to deal with so much change, unpredictability. And so I'm hoping that everyone who is working with AI or is working in policy is also interested in reassessing how we deal with these risks at all. Because I honestly don't know. We'll have to see whether the laws that we have now, now will be sufficient.
- [48:18]
B
Yeah. So as you think about that, you know, will this be sufficient? What are you thinking about for measuring the success or the failure of the code of practice? Like if we got in a time machine five years from now and you could track, I don't know, you know, three metrics that would tell you whether or not the code of practice is a success. Like, how are you thinking about that? What. What will you look to. To. To feel like whether or not this is working as intentionally.
- [48:46]
A
Well, I hope it will make the whole enforcement process more agile for everyone. So both for the companies and for the AI office and for the AI board, because, you know, this takes time and effort, and we should make it as easy as possible for everyone. And I'm sure there will be lots of lessons learned there in how to simplify the processes. But that is ultimately the intention. How can we make this predictable and manageable for everybody? And I hope that companies will say, look, we weren't sure about how this AI act would work. The administration at the time was very much against regulation of American companies. But because we want our products to be trusted and because we also feel ownership of avoiding risk and particularly systemic risk to society, we have now learned, five years after beginning to implement the AI act, that it has actually helped. We have caught things that we might not have otherwise caught, we have been able to build trusted relationships. We've been able to share knowledge also publicly, because it's not a requirement, but an encouragement to also share some of these lessons learned with the rest of the world, right? Not just in the very confidential, I should say, sorry, confidential context of the enforcer and the model provider, but also more broadly, et cetera. So I hope it will turn out to be a positive thing for everybody. Everybody.
- [50:12]
B
Yeah. Well I think that's a lovely note for us to conclude on even though we could easily go talking for another hour. But I know you have other important things to get to today, so let me just say Marita Shock, thank you so much for coming on the AI Policy Podcast.
- [50:26]
A
Thanks for having me.
- [50:29]
B
Thanks for listening to this episode of the AI Policy Podcast. If you like what you heard, there's an easy way for you to help us. Please give us a five star review on your favorite podcast platform and subscribe and tell your friends. It really helps when you spread the word. This podcast was produced by Sarah Baker, Sadie McCullough and Matt Mand. See you next.