← The AI Policy Podcast
The AI Policy Podcast
AI Safety Institute Rebrand, Congressional Hearing on Export Controls, and Meta's New Superintelligence Lab
00:31:37
We cover the U.S. AI Safety Institute rebrand, BIS export control updates, and Meta’s $15B bet on AI superintelligence.
Loading summary
Transcript37 lines
- [00:00]
A
Foreign.
- [00:11]
B
Welcome back to another episode of the AI Policy podcast. This week we're talking about the US AI Safety Institute rebrand, a recent congressional hearing on AI and export controls, and Meta's new so called super Intelligence lab, which is very interesting. Greg, welcome as always. So good to have you here. Let's talk USAI Safety Institute. It's been rebranded as the center for AI Standards and Innovation. On June 3, the Department of Commerce released a statement announcing several changes to the USAI Safety Institute. One of the main changes was a new name. So before we get into the details of the statement, can you provide us with a refresher on what the original AI Safety Institute is and what it's meant to do?
- [00:56]
A
Yeah, it's kind of weird to say, but this organization is coming up on 2 years old at this point.
- [01:03]
B
Hard to believe.
- [01:04]
A
Yeah. So its origins are in the UK AI Safety Summit, which happened in November 2023 and then shortly after that, or sorry, at that, at that summit then Vice President Harris announced that they were going to create one. They created one in miniature pretty quickly. And now this organization has been going and you releasing guidance, doing other kinds of things. And one of the interesting points here is that it was set up in nist, the National Institute of Standards and Technology. So Standards has always favorite building out 270. Yeah, one of its mission has always been connected to standards. So in that first fact sheet, you know, they said that one of the things the AI Safety Institute was going to take over responsibility for was NIST's AI risk management framework, which was a framework, you know, for companies or other organizations developing or using AI to assess their risk and manage it properly in an organizational basis. And that has continued at the AI Safety Institute. And what's interesting is now with this new rebrand, they're taking the word safety out and they're putting the word standards back in. So it's always been had a standards kind of mission, but now they're putting that front and center and then there's a bunch of other changes as well.
- [02:24]
B
Okay, so with the new name and direction, what parts of the original institute are sticking around and what do you expect to change?
- [02:31]
A
Yeah, I think it's worth pointing out, you know, that you and I have talked about this a bunch. How in Republican politics, really since January 6, 2021, the word safety has not been a favored term for any kind of reason. Right. It's been very controversial term in Republican political circles. And so I think it's been clear to a lot of folks for a long Time. As soon as Donald won the November 2024 election, if the AI Safety Institute was going to survive, it was definitely, at a minimum, going to have to have a name change. Because in some pockets of conservative politics, safety really has a connotation and an association with social media censorship. And so while there's, like, ideas behind, you know, safety that Republicans were on board with, they certainly were not on board with that term. And you can see that thinking reflected in the quot from the Secretary of Commerce, Howard Lutnick, where he said, quote, for far too long, censorship and regulations have been used under the guise of national security. Innovators will no longer be limited by these standards. Casey no longer AC will evaluate and enhance U.S. innovation of these rapidly developing commercial AI systems while ensuring they remain secure to our national security standards. So that's one change. It's just the overall framing. The second thing I want to point out is that the definition of risk has narrowed. And this is actually echoing what already happened with the UK AI Safety Institute. They rebranded even before the US one did as the AI Security Institute for the exact same reason. Right. They just didn't want the connotations of safety. So in this case, per the statement, you know, that the Commerce Department put out, quote, casey will focus on demonstrable risks such as cybersecurity, biosecurity, and chemical weapons. And if you contrast that to what, for example, the AI Risk Management Framework said several years ago, quote, AI technologies pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment and the planet. So this is really a narrowing of focus to really explicit and concrete national security type focuses. Now, it's worth pointing out that a lot of that narrowing had sort of already taken place after the election. I don't really think there was significant work going on on, like, for example, mitigating bias and discrimination in AI models going on at the AI Safety Institute. But this just formalizes it and makes it crystal clear this institution has two mandates. One is on innovation and helping US Industry succeed. And the second is on national security. And then leadership on standards is sort of the means to both of those ends.
- [05:24]
B
Okay, so we expect great things out of this institute. How safe are they going to, you know, without getting into, you know, names and calling labels and all that, Is safety really still a priority?
- [05:37]
A
Yeah, and I think, you know, when you talk about safety, obviously there's multiple ways that you can use that word. Right? One is thinking about harms like safety from harms like bias and discrimination. They're just not interested in that. Category of safety anymore. And the kind of risks that they're focused on mitigating are ones that we have a paper coming out on this in not too long. Right. Which is what if, what if AI systems can lower the barriers to entry for certain categories of malicious activity. Right. We think of developing a bioweapon as something that would require a nation state or a pretty well resourced terrorist group, or at an absolute minimum, an evil genius. Right.
- [06:22]
B
And these are, these are the kinds of things that keep AI people like you and, and by, and me also up at night because, you know, AI is so smart, it's getting smarter by the second. And if it can develop a bioweapon or another kind of weapon of mass destruction, look out. Right?
- [06:42]
A
Yeah, exactly. So what you really don't want is you don't want AI to be so smart that it takes, you know, developing a bioweapon or developing chemical weapons from being, you know, only within reach of a well resourced evil genius. And put it within reach of an evil moron, not an evil genius. Like that's the kind of risk factor that we're thinking about. And I think one of the ways that standards can be useful to that is you've got a lot of organizations that do care about AI safety. I mean, meta OpenAI, anthropic, Google, all of them talk about the national security risks that Kasey has been talking about. And they've been talking about it even before the government moved on these types of issues. But those are extremely well resourced, well financed corporations, you know, who can independently decide to invest in these security risk type issues and mitigating them. The question is like, what about this universe of AI startups? What about the open source community? What can they draw upon? And so the idea behind Casey is instead of just lowering the barriers to entry for bad guys, something that we don't want, how do we lower the barriers to entry for good behavior? How do we sort of give companies a total totally viable playbook that is grounded in the best technical expertise worldwide, that is grounded in best practices where companies can share this information with the government on in a way that they feel like is not going to put them at a competitive disadvantage and how do they put all of that into practice to again lower the barriers to entry, this time to good behavior? And so that kind of standards thing is there now. One thing I do kind of have in the back of my mind is just what is going to be included in this pro innovation part of the story. So we talked about the Right.
- [08:31]
B
Because let's face it, nobody, nobody, whether it's a startup or a nation state, is trying to slow down artificial intelligence. Everybody.
- [08:40]
A
There was, there was a community that was advocating for that.
- [08:43]
B
Sure, there's communities.
- [08:44]
A
They're really outside of policy making at this point. They're not really a part of the conversation.
- [08:48]
B
Nobody in policymaking really, because of our competition with China, because of industry competition, because of, you know, research competition, nobody is trying to slow down AI. But guardrails like this around bad actors are super important. That's why we have a podcast.
- [09:05]
A
Right. And you can imagine that there is a mechanism for these types of security and standards to actually accelerate adoption. Right. If you think about the caution that companies have or the concern that customers might have in adopting these kinds of technologies, that concern can slow adoption. So to the extent that there is kind of a government stamp of approval of what constitutes best practices for safety, security, that sort of thing, if that's widely known, if that's widely disseminated, if, and if, if American brands have that kind of reputation and connotation that can give customers peace of mind, then that can actually accelerate adoption. I mean, you saw that in other general purpose technologies like electricity. Electric started getting adopted way, way faster when the risk of fire went way, way down. And can there be sort of a replicating of that story when it comes to AI standards? And I think that's what Kasey is going to focus on, on the innovation side of the equation.
- [10:10]
B
That's a really good way of looking at it. When you look to electricity and its development and its widespread use, we forget that starting fires was a big part of it at the very beginning. It's.
- [10:20]
A
Yeah, I mean, like if you put a candle next to a light bulb and then you walk out and the building burns down. Right. Everybody's going to blame today. But in 1890, that was not at all clear cut. Right. Like, both of them were totally at risk of starting.
- [10:32]
B
Well, it just puts this innovation into a bit of context. And I, and I appreciate you doing that, Craig. Let's move on and talk about the Bureau of Industry and Security. Bis. This is your favorite government entity, I believe.
- [10:45]
A
Yeah, it's. It's certainly up there. Yeah, it's up there.
- [10:49]
B
So the House Foreign affairs south and Central Asia subcommittee recently held a hearing with Undersecretary Jeffrey Kessler, the top official of the Bureau of Industry and Security, also known as BIS. The hearing was entitled Bureau of Industry and Security FY26 Budget, Export Controls and the AI Arms Race. And it included Extensive discussion of AI chip export controls. We've discussed the BIS on this podcast many times before. Can you remind us of their role in AI policy?
- [11:19]
A
Sure. So, to begin, you know, you might wonder, like, why was this happening in the south and Central Asia?
- [11:24]
B
That's my question, for sure.
- [11:25]
A
Yeah. For whatever reason, that is the Subcommittee on House Foreign affairs with Responsibility for Dual use Export Controls and Oversight of bis. So this is the right place to be having this conversation. And BIS has been at the center of U.S. china AI competition because of the AI chip export controls and the associated controls on things like semiconductor manufacturing equipment. So BIS has been in the hot seat for years now on the AI topic. And this hearing really just put it point blank, like, we want to talk to you about export controls and their connection to the AI arms race. Now, we've gotten statements, you know, from the Trump administration. They put out a statement when they repealed, for example, the Biden administration's AI diffusion rule. We got statements, you know, when they banned the export of Nvidia's H20 chips to China. And this hearing, I think, was like, kind of the first opportunity since those big muscle movements for bis's leadership. Now, Kessler, as the undersecretary, to sort of explain what is your grand unified theory of export controls? Because you can point to various moves by the Trump administration where here it looks like we're loosening export controls by selling more chips to, for example, Gulf states such as the UAE and Saudi Arabia, and getting rid of the fusion rule. Other places it looks like we're tightening export controls by banning the H20. And so this was the opportunity for the Trump administration personified by Kessler, to sort of explain, here is what we're trying to accomplish. Here is how you connect the dots between all of our various actions.
- [12:59]
B
Okay, so were there any specific comments, either by the members themselves or by Undersecretary Kessler that really stood out to you during his testimony?
- [13:09]
A
Sure. I mean, there was a bunch. I thought it was a really worthwhile and meaningful and meaty testimony. So one of the first things that Kessler said, and this was in his opening statement, was the importance of fulfilling killing President Trump's skinny budget request as it pertains to BIS. He wanted $303 million in funding for BIS, which is, you know, about 50% budget increase for BIS. So I've been screaming for this for years. And. And, you know, the Trump administration finally requested it. Kessler has actually now put numbers on what that would translate into in terms of additional capacity. So for domestic Enforcement, that would be 200 Export Enforcement Special agents across the United States. And then the overseas contingent of enforcement officials would go from 12 to 30. That's a really big increase in the overall manpower for. And this is a law enforcement agency, you know, prosecuting these cases against this kind of criminal activity and detecting criminal activity. One other thing is interesting is they want to bring in more specialized engineers and technical experts. So when we talk about export controls, you know, if it's a, if it's an item based control that's really tied to sometimes very nitty gritty technical specifications, right? Like, are you laying down, you know, 5 nanometer copper wire, for example? And so if you have like a special agent, you know, somebody with a gun and a badge, this individual, it might not be obvious to them, right? Like, how do you tell when a machine is or is not involved in like 5 nanometer copper wire lay down? So, so I think this is BIS acknowledging that for certain areas they do need in house technical experts who can kind of go toe to toe with industry and not, you know, take their word for it, know what to look for, know how to assess documents that are obtained under subpoena. And I think that's just a wonderful thing. Now one other thing I think here is just like that's the technical expertise, there's also the technical enabling capacity, right? BIS is overseeing like trillions of dollars in trade activity. A lot of that, you know, shows up in these databases. And you want those different databases to talk to each other in a way that connects the dots, right? You don't want, you don't want the answer to be somewhere there, lurking in a PDF, right, that nobody even knows to go read. You want these databases to sort of say, like, hey, this license application is coming from this company, but actually that company's secret owner, via a shell company, is this other company, right? And you want as much of that to be automated and surfaced to the analyst in a way that is rapid and well, well sourced. So I hope and expect that some of this money is going to go not just to hiring more people, but to giving these people more technical tools to make them more productive, to make them more effective. A couple other things that came out in this testimony from Kessler, the first official statement from bis, their internal projection of Huawei's AI chip production capacity. So this is the Ascend chip product line. And Kessler said, quote, huawei's Ascend chip Production capacity for 2025 will be at or below 200,000. We project that most or all of that will be delivered to companies in China. So what's really interesting here is my assessment is actually that they could make more than that. The reason why they would not make more than that is the Ascend chips come off the 7 nanometer wafer production line. And the 7 nanometer wafer production Line can also be used to make phone chips. It can also be used to make laptop chips. So it's kind of a choice that Huawei has to make strategically. How, how much of that wafer production capacity do we want to allocate to AI chips given the defect rate, customers interest in actually buying these things, et cetera, et cetera, et cetera. So, and then there's a second quote from Kessler that I thought was great, which was quote, we shouldn't take too much comfort from that fact. China is investing huge amounts to increase its AI chip production as well as the capabilities of the chips it produces. China is catching up quickly. So I certainly agree there. And I think one way to think about this is that, you know, as Nvidia keeps moving forward with the state of global chip design and the state of global chip manufacturing. Manufacturing, right, they're going to go from 4nm to 3nm to 2nm to 1nm. Like all of that is going to deliver performance improvement. They're going to go from great to greater and even greater in terms of performance. But Huawei is arguably going to be making a more strategic transition, which is from what is effectively a non viable chip, the Ascend 910B. It just doesn't work reliably enough or is even, you know, acceptable choice for a lot of the applications that are most important to AI right now, like large language models. That's why they have so many of these chips in China and they're in data centers where nobody uses them. So while Nvidia is going to go from the best chip in the world to an even more amazing chip, Huawei is going to go from a chip that literally no one wants because it doesn't work to a chip that maybe is still nowhere near as good as Nvidia, but at least it works. And I think that sort of off on kind of transition actually really matters strategically. And so I think, you know, the problems that Huawei has with their Ascend 910B line, their Ascend 910C line, with their surrounding software ecosystem, I do expect them to make progress on all of these things and that is going to be strategically meaningful. And BIS is acknowledging that. And I think that's completely fair.
- [19:03]
B
So what does all of this mean for China's, you know, grand AI ambitions?
- [19:09]
A
Well, I think one thing here is that we haven't. We do have statements from BIS that indicates that they really care about this policy. AI chip export controls. They're bringing on more enforcement capacity. At least they're requesting Congress give them more enforcement capacity. They're openly talking about the problem of chip smuggling. There was one exchange between a member of Congress in Kessler where, you know, the congressperson said there are some that deny this smuggling is happening at all. Kessler said it's happening. It's a fact, which, thank goodness he said that. I think that really needs to be better understood by more people, given that there are some companies out there putting out an alternative message that I think does need to be rebutted. But overall, I think what we've got here is a pretty good understanding that the Trump administration still cares about this policy. They still want this policy to work. And I think that really matters. If you're looking at these trade negotiations where there's been statements that because of China's export controls on rare earths, you know, these specialized metals that are important to manufacturing in so many industries, that maybe as part of the grand bargain between the United States and China, the United States would get rid of its AI chip export controls in exchange for China getting rid of its rare earth export controls. I think my. I mean, they didn't answer it explicitly, but my assessment is that that is not going to happen. I don't think that's consistent with what Kessler has articulated here. I think there's some parts of the story that might be rolled back, like export controls on, you know, EDA software. And we've seen reporting that that was like, rolled back somewhat after the Geneva negotiations and the subsequent negotiations in London. But overall, I think the Trump administration, you know, is sort of settling at this overall point here, which is they're going with the policy in a modified form. That 200,000 chip figure, which again, is for 2025. We're halfway through 2025. Huawei is going to continue making progress. Leonard Heim, who's a friend of mine and doing good work over at the Rand Corporation, he put out a pretty interesting analysis out on X, which was, If China produces 200,000 Ascend 910Cs annually, filling the UAE's planned 5 gigawatt data center would require 15 years of, for 3.1 million chips while delivering only half the performance. So, and to match performance, they would have to have 7 million chips consuming 2.2 times more energy. So the basic point here is like one of the things that we're, you know, hearing in the discourse on AI chip export controls is, you know, we need to sell abroad to prevent China from backfilling us. I do think that is a concern that should be on policymakers mind. But just in terms of where we are in the story in 2025, Huawei's level of production right now does not put them in a position to credibly backfill. I mean, that could change, you know, in three years and five years and 10 years. But at least for right now, they're not in a position.
- [22:22]
B
And it's something you'll be watching and others will be watching.
- [22:24]
A
Yes, absolutely.
- [22:25]
B
Okay, now let's talk about Meta and the new Super Intelligence Lab. Last week The Times, New York Times wrote about Meta's CEO Mark Zuckerberg's plan to build an AI superintelligence lab that's led by Scale AI CEO Alexander Wang. Meta is reportedly investing 15 billion in Scale AI and offering up to nine figure salaries to recruit staff from other AI labs. They're out there poaching. Why is Meta spending billions on this initiative? And I think I know the answer to this, but I really want to know what you think.
- [22:57]
A
Yeah, well, first I just got to promote the fact that we had former Scale AI CEO Alexander Wang here at csis. I got the chance to interview him. If you haven't listened to that event, you know, it's on YouTube. We actually re released a version of it on this podcast and you should definitely do it. He's a very smart, very interesting guy. And the fact that Meta is, I think I've seen it reported as an acquire hire, so it's close to an acquisition of the company, although it doesn't technically meet that threshold because they only bought 49% of the company. But Alexander Wang has now left Scale AI as CEO. He's going to go lead this Super Intelligence Lab lab at Meta. And presumably, I don't know this for a fact, but presumably he still has some meaningful ownership of Scale AI shares. So, you know, they have what they need to, to do what they want. I think this is really coming about the, like, the really accelerating pace of competition. One thing that I think is really important is this report from Business Insider that said in May that, you know, Meta's best family of large language models is called Llama, which is an open source thing. And on that first ll paper that came out in 2023, which was a landmark. There were 14 authors. Only three still work at Meta. Right. So a lot of their best talent, you know, has left the company. They feel this very ferocious competition for talent. One of the ways that they're looking to improve that is by acquire, hiring, you know, making this new transaction with scale AI and bringing on Alexander Wang. And then another thing that they're doing is again, according to reporting out there, Mark Zuckerberg is personally calling people and offering them, you know, what was it, like nine figure salaries. So that's like ballpark, $10 million. Wow. Is there a lot of money, right, going after the best talent in this field. And that's because of the moment we're at. Just last week, Sam Altman, OpenAI's CEO, published a new blog that said, quote, we are past the event horizon. The takeoff has started. Humanity is close to building digital superintelligence. So we're talking AIs that are not just, you know, useful in doing your work. We're talking AIs that are like, smarter than Albert Einstein, buy a lot at everything. That's where these companies assess themselves to be in the AI innovation story. And they think winning this competition, just like, you know, Congress and the White House and the Department of Defense think it really matters in the US vs China competition. Companies like Meta, companies like Google, companies like OpenAI, they're assessing that this, like, this one's for all the marbles folks, and they need to be willing to make jaw dropping bets to go after it. I mean, just, just thinking about like the size of this acquisition. $19 billion. Remember the. Or, sorry, $15 billion. Remember the chips act? Right, the CHIPS act, which was supposed to be a once in a generation investment. According to then Commerce Secretary Gina Raimondo, that was $51 billion over four years. Right? So like just meta dropping. This chunk of change is larger than the annual spend of the CHIPS act. So just jaw dropping amounts of money.
- [26:11]
B
Greg. So when we talk about super intelligence, what do they exactly mean? I mean, you just said smarter than Albert Einstein by a lot. How different is it from AGI or artificial general intelligence? Sometimes we think of this stuff as science fiction. Maybe too often we think of it as science fiction. But when people like Sam Altman and Mark Zuckerberg are talking about superintelligence, they really mean it and they really believe in it. Tell me what the distinction is.
- [26:39]
A
So there's not widespread agreement on what these terms mean. Part of the reason why there's not widespread agreement on this is because a lot of money is at stake. Depending on what these terms mean. I mean literally the terms of the OpenAI deal with Microsoft, where Microsoft invested a lot in. OpenAI says that certain things happen in that transaction once a level of artificial general intelligence is reached, right? So like as soon as there's money on the line, depending on what that word means, you can bet people are going to fight over what that word means. I'll just say that for my part, when you think about, you know, a human being, we're not just very smart at things, we're also incredibly flexibly smart, right? We can be good at chess, we can be good at horseback riding, we can be good at translating languages, languages. So that general intelligence refers to, you know, we're not just some like chess playing supercomputer. We're also an individual that can engage in language, write novels and do a bunch of other things. So it's that flexible generality and I think that sort of, it's, it's useful. Obviously there's types of intelligences that humanity is not flexible to understand. We have a really hard time imagining seven dimensional space, for example. So it's not like, like we're perfectly generalized, but we're broadly generalized. And so folks like OpenAI, they describe AGI as AI systems that are generally smarter than humans. Now, superintelligence is when we're not just generally smarter than humans, we're smarter than the smartest humans who have ever lived in basically every category of intelligence that humans have ever been good at. And, and note that, you know, in that quote I read earlier from the Sam Altman blog post, he talked about the word event horizon. And the point here is that one of the things that AIs and super intelligent AIs could be good at is AI research and making AIs ever smarter, ever more capable, et cetera. And so there's this, you know, hypothesis out there that's very widely believed in the AI community, right, that we could have something that is as smart as Albert Einstein one year year, and then have something that's 10 times smarter than Albert Einstein the next year, and then have something that's 100 or a thousand or 10,000 times smarter than Albert Einstein after that. So that's what that takeoff kind of whoa scenario is for AI. And that's really the story that is really tied up with that word superintelligence. So you can understand why companies are willing to make big strategic moves. And it's worth pointing out that there's been, you know, some strategic responses. So Scale AI was, in some ways they kind of occupied the same role in the ecosystem for AI. This is not a perfect analogy, but, you know, the same role that AWS occupied in computing. So Amazon has Prime Video, for example, so, you know, you can go watch movies on Amazon Prime Video streaming. But AWS is a cloud computing provider and one of their largest customers is Netflix. So literally, Netflix is a competitor of Amazon, but they're also a, a customer of Amazon and Scale AI. Because so many other firms like, like Meta, like OpenAI, like Xai, like Google, were all customers of Scale AI. The fact that Meta is now in the driver's seat for what Scale AI is doing is causing, you know, what Time Magazine is calling like a flurry of deal making activity. We've already heard that Google plans to cut ties with Scale AI, XAI plans to cut ties with Scale AI. So they're no longer comfortable being both a customer and a competitor of Scale AI, given its new relationship with Meta. And then just think about that for one second, right? Meta had to anticipate that this was going to happen. They knew that this was going to happen. So that means when they decided that 49% of scale AI was worth $15 billion, that was even after they took into account the fact that they were about to lose many of their biggest customers. It's just a remarkable time.
- [30:55]
B
It's remarkable. Well, we could do a whole podcast on superintelligence and we will in the days to come. Greg, thanks so much for all this today. Great insight as always. We will be back in a couple weeks with another great episode of the AI Policy Podcast.
- [31:10]
A
Thanks a lot, Andrew. Thanks for listening to this week's episode of the AI Policy Podcast. If you like what you heard, there's an easy way for you to help us. Please give us a five star review on your favorite podcast platform and subscribe and tell your friends. It really helps spread the word. This podcast was produced by Sarah Baker, Isaac Goldston and Sadie McCullough. See you next time.