An AI safety expert’s plea in Paris

Summary

POLITICO Tech Podcast Summary: "An AI Safety Expert’s Plea in Paris"

Release Date: February 7, 2025

---

Introduction

In the February 7, 2025 episode of POLITICO Tech, host Stephen Overle engages in a profound conversation with Yoshua Bengio, a renowned AI safety expert often hailed as one of the "godfathers of AI." The discussion centers around the urgent need for balancing rapid AI development with robust safety measures, especially in the context of the upcoming AI Action Summit in Paris. This summit gathers global leaders, including American Vice President J.D. Vance, French President Emmanuel Macron, Indian Prime Minister Narendra Modi, and OpenAI CEO Sam Allman, reflecting the heightened stakes in the global AI race.

---

The AI Action Summit: Shifting Global Agendas

Stephen Overle opens the dialogue by highlighting the significance of the AI Action Summit in Paris. He notes a marked shift in the global AI agenda since November 2023. While the first AI summit in the UK emphasized AI safety and risk mitigation, the Paris summit is predominantly focused on accelerating AI deployment and fostering international collaboration amidst intense competition.

Yoshua Bengio underscores this shift, explaining, "[The report] does an important job of documenting the Scientific evidence, including about the uncertainty that scientists have regarding those risks" (03:40). He emphasizes the divergence in views among experts regarding the pace of AI advancements and the associated risks, highlighting the unpredictable trajectory influenced by recent breakthroughs like O3 and DeepSeq.

---

AI Safety vs. Rapid Development: The Policymaker's Dilemma

The conversation delves into the evidence dilemma faced by policymakers. With experts divided on AI's development speed and potential risks, creating informed regulations becomes challenging.

Bengio articulates the dilemma:

"There's a global race to accelerate the development of AI... the risk is that we do these things unnecessarily and it could have consequences on the rate of that innovation" (04:10).

He draws parallels to historical scientific advancements, such as geoengineering, where the potential for both salvation and catastrophe necessitates cautious progression despite limited data.

---

The Balance Between Regulation and Innovation

When pressed about whether policymakers are shifting away from AI safety towards unfettered development, Bengio expresses concern over strong corporate lobbies opposing government intervention. He states:

"There is a very strong lobby that has been successful... very strong lobby against any kind of Government intervention" (06:22).

Despite these pressures, he warns that the relentless pursuit of more powerful AI systems without adequate safety measures could magnify existing risks. Bengio emphasizes the importance of not ignoring these challenges, highlighting growing civil society concerns and the need for collective action to steer AI development safely.

---

Diverse Government Approaches and the Need for Global Cooperation

Overle points out the differing focuses of the AI Action Summit compared to prior summits, reflecting varied governmental priorities. Bengio acknowledges the diversity in national approaches:

"Different governments... have different understanding, different level of awareness" (08:05).

He underscores the necessity of international dialogues to bridge these differences, advocating for cooperation over competition. Discussing the concept of agency in AI, Bengio warns against autonomous AI systems that may pursue goals misaligned with human interests, advocating instead for non-agentic systems that prioritize beneficial applications like scientific discoveries and healthcare advancements.

---

AI Governance: Market Mechanisms vs. Regulation

Addressing the current U.S. political landscape, Bengio discusses potential pathways for AI governance in the absence of stringent regulations. He proposes leveraging market mechanisms, such as liability frameworks and mandatory insurance, to encourage responsible AI development:

"If governments clarify liability of software systems, it would go a long way to create more responsible behavior" (14:36).

This approach allows for self-regulation within the industry, incentivizing companies to mitigate risks without imposing direct governmental restrictions.

---

Global Competition and the Need for Cooperative Safety Measures

The episode touches upon the intensified rivalry between the U.S. and China in AI development, especially following the release of China's DeepSeq model. Bengio cautions against an uncontrolled race, drawing historical lessons from the Cold War nuclear treaties:

"There's not enough understanding of those risks, enough awareness... pleasuring to find a way to make sure no one... does something stupid that we all pay for" (17:05).

He advocates for recognizing shared global interests in AI safety to foster agreements and treaties that prevent catastrophic outcomes, emphasizing that unchecked advancements could lead to universally detrimental consequences.

---

Conclusion: A Call for Balanced and Cooperative AI Advancement

As the conversation wraps up, Bengio reinforces the urgency of managing AI risks while maintaining developmental momentum:

"We need to step back and look at the bigger picture" (12:54).

He envisions a future where democratic nations lead in both AI capability and safety, ensuring that advancements are aligned with human interests and societal well-being. The episode concludes with a mutual acknowledgment of the complexities involved in navigating the AI landscape, highlighting the critical role of informed policymaking and international collaboration in shaping a safe and prosperous AI-driven future.

---

Key Takeaways

• Global AI Agenda Shift: From safety-focused discussions in the UK to rapid deployment and collaboration amid competition in Paris.

• Policymaker's Evidence Dilemma: Balancing regulation with innovation amidst uncertain AI development trajectories.

• Regulation vs. Innovation: Strong corporate lobbies resist government intervention, potentially exacerbating AI risks.

• Diverse National Approaches: Varied governmental priorities necessitate international cooperation for effective AI governance.

• Market Mechanisms for AI Governance: Leveraging liability and insurance frameworks as alternatives to direct regulation.

• Global Competition Risks: Unchecked AI races, particularly between the U.S. and China, could lead to catastrophic outcomes without cooperative safety measures.

• Balanced Advancement: Emphasizing the need for democratic nations to lead in both AI development and safety protocols to ensure beneficial outcomes for society.

---

This episode of POLITICO Tech offers a comprehensive exploration of the current state of AI development, the inherent risks, and the imperative for balanced, cooperative approaches to ensure that technological advancements align with human safety and societal benefits.

Transcript

[Yoshua Bengio]

Exclusively on ESPN. UFC 312 Saturday Reigning middleweight champion Dricus du Plessis defends his title in a rematch against Sean Strickland. And Zhang Weili defends her strawweight title against undefeated tatiana Suarez. UFC 312 Saturday at 10pm Eastern.

[Stephen Overle]

Buy it on espnplus.com ppb hey, welcome to POLITICO Tech. Today's Friday, February 7th. I'm Stephen Overle. World leaders and tech luminaries will be flocking to Paris in the days ahead. American Vice President J.D. vance, French President Emmanuel Macron, Indian Prime Minister Narendra Modi and OpenAI CEO Sam Allman all taking part in the AI Action Summit. There have been a few of these gatherings since November 2023, but since then the global AI agenda has shifted dramatically. The first AI summit in the UK focused on safety and how to mitigate the risks of AI, and Paris is all about rolling it out fast and how countries can try to collaborate while still racing to beat one another in the AI race. But Yoshua Bengio isn't giving up on the safety conversation. You've heard him on the podcast before. He's considered one of the godfathers of AI, though his titles are technically professor at the University of Montreal and Scientific Director at the Montreal Institute for Learning Algorithms. Now Professor Bengio has led a panel of AI experts in publishing the first international AI safety report, and over nearly 300 pages they have many words of caution for AI hungry politicians and tech executives. On the show today, I called up Professor Bengio in Paris to find out whether they can have their AI and keep it safe too. Here's our conversation. Well, professor, welcome back to Politico Tech.

[Yoshua Bengio]

Pleasure.

[Stephen Overle]

So you've come together with other experts in artificial intelligence to try to determine the real risks around general purpose AI and whether they can be mitigated. How close did you all get to an answer?

[Yoshua Bengio]

Well, the answer is that there is a rich literature clarifying those risks. There's also a lot of unknown unknowns when especially we're talking about when we get to human level AI. There is also very different views about those risks, their probability according to different experts, usually because they think that the advances in AI are going to be very slow or very fast and that changes their perception. But there's also evidence that we see the trajectory of advances over the last decade, the last few years, and even in the last few months, as you know with O3 and Deep Seq that came out after the bulk of the report was written. The report does an important job of documenting the Scientific evidence, including about the uncertainty that scientists have regarding those risks.

[Stephen Overle]

That was one of my takeaways in reading it, is that, you know, even the experts disagree on some of the answers to these questions, including how fast, you know, general purpose AI will be developed. One of the points noted in the report was that for policymakers, this does create kind of an evidence dilemma, right? They're, they're trying to create laws and regulations without a whole lot of data that to back it up. Can you break down that dilemma for me and what it means for kind of global regulation of AI?

[Yoshua Bengio]

As you know, there's a global race to accelerate the development of AI, either because of the competition between corporations or the competition between countries. And so it's legitimate to ask the question, if we slow down that innovation by having more government oversight, is there a risk that we do these things unnecessarily and it could have consequences on the rate of that innovation? Or the other risk is the governments don't do enough and we end up with disruptive changes in our society, even potentially with catastrophic outcomes, malicious use that could have a big impact, potentially loss of human control. So these are very high severity risks for which we don't have consensus on their probability. But if they do happen, they could be severe. So it's a difficult choice, but decision makers will have to face it and accept that we live with this uncertainty. By the way, it is not the first time in history where we have scientific advances or scientific experiments that are being considered that could have severe consequences if the experiment fails, but for which we don't have a lot of hard data to make sure it's going to go one way or the other way. A good example is geoengineering. It could, you know, save us or maybe disrupt the atmosphere in ways that we don't want. And the severity is so high if we get it wrong that scientists have decided and countries, for now, have decided to not go forward for this moment.

[Stephen Overle]

We're in where policymakers are kind of at a fork in the road in some ways, right? Between really putting restrictions on AI in the name of safety versus kind of letting it develop as quickly as it can. Does it feel to you like they're moving away from AI safety, that they're moving toward kind of unfettered development and just get to AI as fast as we can?

[Yoshua Bengio]

I'm not sure. What I can clearly see is a very strong lobby that has been successful, for example, with the proposed bill in California a few months ago.

[Stephen Overle]

Right.

[Yoshua Bengio]

Very strong lobby against any kind of Government intervention. But the unfortunate reality is that we're continuing to advance towards more and more powerful AI. Since then, we've seen evaluation of the biorisks of O1, for example, being on the threshold of an acceptable risk. We've seen advances with O3 and DeepSeq in terms of reasoning abilities. We've seen companies invest massively to turn the current technology into agents that would be more autonomous. But that autonomy is going to magnify the risks, many of the risks that are discussed in the report. So I think putting our head in the sand unfortunately doesn't solve the problems. So what I'm trying to say is that yes, there are forces going one way, but the evidence is mounting and more people in civil society, for example, where I'm here in Paris, are becoming more concerned and wanting to do something about it.

[Stephen Overle]

You're there, as you said, at this summit in Paris, which is being called the AI Action Summit. And even that, you know, so much of the focus, it seems, is around deploying AI and how you put AI to work. I guess it just seems to me like the conversation has changed a lot even since the first AI Safety Summit in the uk.

[Yoshua Bengio]

Well, it's different governments. And if you had asked the same question to these two governments a year ago, you would have also obtained different answers. It's normal across the world you have individuals who are leading their country and they have different understanding, different level of awareness, different beliefs, different scientific advisors that, you know, means they will maybe put the priority in different places. That's human. And you know, the reality means we need to have these summits, we need to have these summits so as to engage in the conversation with those who don't always agree with us. I gave a talk a few hours ago about the notion of agency. In other words, companies are building the next generation of AI systems that are not just good at chatting, but are good at doing things and having their own goals and being autonomous in achieving those goals. And I can understand this has huge economic value, but it's also a very dangerous proposition. It's going to get us closer to dangerous AI systems that we don't control. Because almost by definition, if you have an autonomous entity, it wants to preserve itself, it may choose sub goals that we would not approve of. This is well studied in our report. And this acceleration towards agentic system actually is not necessary. We could build systems that help us advance what people want, which is better drugs, better cues for cures for major diseases. Right, without taking those risks. And these kind of non agentic AIs is already what scientists are using for their discoveries. You know, like the Alphafold, which, which won this Nobel Prize just a few weeks ago. A few months ago. In addition, non agentic systems could be used to guardrail agentic systems. So what I'm trying to say is AI is not one thing. There are many ways of doing AI, any types of AIs and we can choose, if we decide collectively to steer the boat in a direction that's going to be both beneficial and safe.

[Stephen Overle]

Have AI defending us from AI, that's one.

[Yoshua Bengio]

One particular positive application. But in the shorter term, it could advance science, medicine in ways that could be positively transformative. But we still need to make sure these AI systems are not going to be misused against us. That's to solve that problem.

[Susan Ettlinger]

The PC gave us computing power at home, the Internet connected us, and mobile let us do it pretty much anywhere. Now generative AI lets us communicate with technology in our own language, using our own senses. But figuring it all out when you're living through it is a totally different story. Welcome to Leading the Shift, a new podcast for Microsoft Azure. I'm your host, Susan Ettlinger. In each episode, leaders will share what they're learning to help you navigate all this change with confidence. Please join us, listen and subscribe wherever you get your podcasts.

[Stephen Overle]

What will your message be to those world leaders who are particularly driving at development of AI and maybe less concerned with what that means for, for safety or for humanity?

[Yoshua Bengio]

Even so, first of all, I understand that many governments don't perceive their own efforts as dangerous and, you know, may see the American companies as the ones that could potentially do something dangerous. But in most countries there is no such large scale effort as we seeing with the leading American companies. And so their main concern is how do we bridge the gap so that we're not going to be left in the dust in economic terms or even in military terms, if you think about adversaries. And so it's kind of a natural attitude to have. That being said, I think more countries would benefit from understanding that the mistakes that could happen in other countries like the US could affect them pretty radically, whether it is because of malicious use, because, you know, criminals and terrorists don't care about laws and frontier and borders.

[Stephen Overle]

Right.

[Yoshua Bengio]

Or because of loss of human control, which would affect, you know, all humans. Clearly it's not sufficient to consider this economic race, but we need to step back and, and look at the bigger picture.

[Stephen Overle]

Speaking of, you know, individual countries, and you mentioned that a lot of the focus is on American companies. You know, the political landscape in the US has shifted pretty dramatically since President Donald Trump has returned to the White House. And as you know, he has reversed many of the Biden administration's policies on AI safety. I wonder if that worries you and, and what role you see for the US going forward here.

[Yoshua Bengio]

I don't think we've seen yet exactly what the policy is going to be in terms of AI. We've seen some signs which are not surprising that the Trump administration is concerned about the possibility of China starting to lead the way. I don't think, in spite of Deep Seek, that we're there, but it is a legitimate worry. I also think that there are opportunities for governments who don't want to see very strong regulation to use market mechanisms in order to obtain a sort of reliable, more reliable self regulation. So you can think about the mechanisms of liability. For example, if governments clarify liability of software systems, it would go a long way to create more responsible behavior. And similarly, if governments, without having to actually impose regulation, just say you have to have insurance in case of liability lawsuit responsibility that may come to the developers or the deployers, then we have the interesting market mechanism that from the point of view of the insurers, they have to estimate the risks honestly because otherwise they're going to lose money.

[Stephen Overle]

Right. This administration doesn't seem so keen on regulation, I guess, or putting any sort of restrictions when they talk about AI. It seems very focused on dominating the world, particularly China, and that if we don't do that, it's bad for our economy and bad for national security.

[Yoshua Bengio]

I understand that. I think that there's a real national security concern here to keep in mind. But my opinion is it's not either or because there are risks on both sides. So the wise thing to do, in my opinion, is to both manage the risks and make sure the democratic countries are in the lead. So it's not a matter of like putting less money on capability. And there is enough capital, whether it comes from governments who want to protect the public or it comes from companies who are, you know, getting the right incentives in one form or another. There's enough capital going around to invest in safety. Right now safety is getting like a tiny, tiny fraction of the total investment. If we put the right incentives, it could become something that, that attracts enough research to actually develop the guardrails that we need before these machines become too dangerous.

[Stephen Overle]

You know, you mentioned Deep Seq, which obviously is a Chinese company that released this new AI model that caught a lot of folks, I think by surprise. That has kind of ignited an urgency here in the US I wonder if you worry about that sort of tilting the balance here in favor of just investing in development. Is this kind of a significant moment that we're in right now?

[Yoshua Bengio]

Actually, this has been part of the discussion already for a while. Maybe Deep Seek is bringing it up more into the collective discussion, but this is an argument for some time. As I said, I think. I think this is something to consider seriously because we do want to make sure AI is not used against us. That being said, if we go into a race, say, between the US and China, that is completely uncontrolled. The right way to think about this is what is it that the Chinese and the Americans have in common that could give both of them the incentive to do the right thing and even to sign treaties?

[Stephen Overle]

What is that?

[Yoshua Bengio]

You have to remember what happened after the Second World War and until the sixties, when we were in the middle of the Cold War between the USSR and the west, and the US and the USSR ended up signing treaties and actually stopping to stockpile the dangerous nuclear weapons. Why? Because they saw their, you know, individual interests, that they have something in common, which is no party wanted to see a nuclear winter that would make everyone lose. And so what is the equivalent here? If we just focus on, oh, the other guys are going to use AI against us, then we're doomed. But if we think, okay, yeah, that's a possibility, but there's also the possibility that the other guy makes a mistake, that we make a mistake because we are going so quickly and we want to stay in the lead and we all lose because we create an AI that doesn't have human interests as its primary objective. I don't think there's enough understanding of those risks, enough awareness. And so right now, it's not something that dominates a discussion, but as we approach more and more capable, eventually super intelligence systems, hopefully governments realize that we are all in that same boat. And it's in our best interest to find a way to make sure no one else, including our own companies, does something stupid that we all pay for.

[Stephen Overle]

Well, Professor, I appreciate you being here on Politico Tech.

[Yoshua Bengio]

Thanks for having me and asking all these great questions. I really appreciate.

[Stephen Overle]

That's all for today's Politico Tech. If you enjoy Politico Tech, please subscribe and recommend it to a friend or colleague. And for more tech news, subscribe to our newsletters, Digital Future Daily and Morning Tech. Music in our show comes from the mysterious Breakmaster Cylinder. Our managing producer is Annie Reese. Our editors are Steve Heuser, Daniela Cheslo and Louisa Savage. I'm Stephen Overle. See you back here on Monday.