← 80,000 Hours Podcast
80,000 Hours Podcast
How AI could create the world’s biggest problems (article by Zershaaneh Qureshi)
01:29:46
Loading summary
Transcript1 lines
- [00:00]
A
How Artificial Intelligence Could Create the World's Biggest Problems an article for the 80,000 Hours website, written by Zoshane Qureshi, published in February 2026 and read by the author in March 2026. Introduction Imagine you're living 15,000 years ago. Your people are hunter gatherers and you sleep under the stars. If someone told you humans would one day build cities with millions of people, fly through the air, or carry all human knowledge in their pockets, you couldn't even begin to picture what they meant. Yet here we are. So how did our lives change so far beyond recognition? The story is complex, but there's a rough pattern. A few times in history, some radical breakthrough in technology, like the development of the plow and the steam engine, has led to a wave of productivity, innovation and social change that ultimately reshaped the world. Now we're on the cusp of a huge new breakthrough, artificial intelligence that can meet or exceed human capabilities across a wide range of tasks. This could bring another era of transformation. There could be an explosion of intelligence and innovation and a whole new population of digital beings. And with this civilization could see changes at least as profound as those brought about by industrialization or the rise of agriculture. But unlike the industrial and agricultural revolutions, a transformation driven by advanced AI might not take hundreds or thousands of years to unfold. This time around the world could become unrecognizable over the span of decades, or possibly even less than that. This period of transformation could bring astonishing prosperity, with AI enabling life saving medical breakthroughs and innovations for tackling the climate crisis. But it could also throw us unprepared into an alien world of challenges. Just imagine those hunter gatherers suddenly finding themselves in crowded settlements where diseases spread like wildfire, while also facing warfare between organized armies for the first time. Or imagine pre industrial humans forced to contend with enormous factories pumping out pollutants and mysterious new weap called nuclear missiles that can wipe out entire cities. This article will explain why we think advanced AI could be this transformative and why working to address the risks and broadly make the future of AI go well may be your best opportunity to have a positive impact on the world. So why do we think advanced AI poses the world's most pressing problems? For over a decade, we've looked into the biggest problems in the world, as well as approaches to solving them. We think the cluster of challenges raised by advanced AI are the most pressing problems facing humanity today because of their scale and the promising but neglected opportunities to address them. Our concern about AI risks isn't a reaction to the surge of interest in AI since the 2022 release of ChatGPT. We've argued that AI could pose catastrophic risks since 2016, and some thinkers had raised concerns long before then. In short, we think it's plausible that advanced AI could radically transform the world. This could pose extreme challenges for humanity, and it presents a potentially unique opportunity for having a positive impact. We go through the specific challenges we think are most pressing in our problem profiles. To find these, check out the links in the article and in the description below. This article explains why advanced AI gives rise to such important issues in general. There are lots of arguments you could make here, like the argument that advanced AI will constitute a second species, or that AI will make the 21st century the most important century for humanity. But here's an argument that makes the risks feel especially compelling to us. One AI could replace human labor in some of the most economically valuable fields. 2 Replacing human labor in these fields could trigger the next radical transformation of society. 3 this transformation could be extremely rapid and dramatic, especially if there are fast feedback loops in AI research and development. 4 A rapid AI driven transformation would raise a range of major challenges, including existential risks. And five work on these challenges is tractable but neglected. We'll argue for each of these claims in this article. But to be clear, you don't have to believe in this particular argument to think that AI poses existential threats. Even if AI doesn't automate lots of human labor, malicious actors could still use AI to design novel bioweapons or carry out sophisticated cyberattacks. And even without society being broadly transformed, advanced AI systems could still engage in deceptive behavior and undermine humans. These issues in themselves could be enough reason to prioritize working on specific AI risks. But the story we tell here, with the world rapidly being transformed through widespread automation, is the backdrop against which we expect all these risks to play out. A backdrop that makes them more likely to happen and potentially much harder to deal with. After all, AI systems have a better chance of mobilizing the resources needed to disempower humans if we've already deployed them in crucial roles in the economy. And the risks of malicious actors misusing AI are much greater if, through automating scientific research and technology development, powerful new weapons can be built faster than we can create robust protections against them being misused. With this rough story in mind, it's easier to understand why so many unprecedented challenges could emerge around the same time and have unusually serious consequences. Section 1. AI could replace human labor in the most economically valuable fields Many technologies like cryptocurrency, NFTs, the Internet of Things, fusion, and quantum computing have been overhyped. People often have high expectations of how much a new innovation will change the world, and reality sometimes falls short. But we think AI is going to be different. That's because, unlike other technologies, AI has the potential to compete with and even go beyond human intelligence. And that means it could replace and reproduce the main driver of progress in our history, flexible human labor. Some technologies, like ATMs, mimic extremely limited forms of human labor. Others, like steam engines and computers, also amplify what humans can do. But the idea behind artificial intelligence is that it'll be able to do almost any work humans can do, and do so mostly autonomously. ATMs didn't make all bank tellers unemployed because there were other tasks the humans could easily shift into. But imagine an ATM that could not only hand out cash, but also manage the bank's IT systems, contribute to company strategy, and give customers tailored financial advice. Imagine it could do this mostly without our help and more cheaply than human workers would. If that were the case, it's not clear why the bank would keep humans employed at all. Now, suppose the same system that did all this for the bank could also do equivalent work for tech companies, scientific research labs, consultancy firms, think tanks, the New York Times, the US government, and so on. That's the prospect raised by AI. We're already seeing glimpses of AI's increasingly general ability to do human work. AI systems today can do things that just a decade ago would have been astonishing. For example, consider the rapid progress language models have made on the GPQA benchmark, which asks challenging PhD level questions about chemistry, physics and biology. In mid 2023, Frontier AI Performance was just slightly better than random guesswork on these questions. But since early 2025, many models have been outperforming human experts, and sometimes by a large margin. They've also shown impressive improvement on software engineering tasks. For example, Anthropic's agentic coding tool, CLAUDE code, enables users to build applications by just describing what they want, even if they have no coding experience. A senior engineer at Google reported that CLAUDE code took one hour to generate a prototype of a system her team had spent a year exploring approaches to building. And Anthropic built its cowork product, which is basically a more user friendly version of CLAUDE code for non developers, in under two weeks. By getting CLAUDE code to write most of the code. Here's a few more examples of what current AI systems can do. They can predict complex biomolecular structures and interactions. Google DeepMind's AlphaFold 3, which is a successor to a Nobel Prize winning system, can predict how proteins interact with DNA, RNA and other structures. At the molecular level, they can solve hard maths problems competitively. Multiple AI models have reportedly achieved gold medal performance in the International Mathematical olympiad. Separately, when 30 top mathematicians were challenged to devise problems they believed AI wouldn't be able to solve, OpenAI's O4 mini thwarted many of their best attempts. Even solving one PhD level question in about 10 minutes. They could improve robotics so many leading robotics models are now AI driven. For example, Boston Dynamics is enhancing its AI Atlas robots with Google DeepMind AI to help them better understand and manipulate their environments. And these robots will be deployed for industrial work at Hyundai factories. They can carry out extended tasks independently on your computer. Unlike earlier models that could only generate text, new agentic AIs like Claude Code and OpenAI's codecs can now use many tools on your computer, execute code, search the web and chain multiple steps together, allowing them to complete extended real world tasks with far less human involvement. They can help with AI development itself. There's evidence that AI systems can outperform humans in AI R and D tasks, at least when limited to a two hour time window. And the list goes on. There are still plenty of things AI systems can't reliably do, especially most work that takes days or longer to complete. But the list of things these systems can't do is diminishing. And the pace of AI progress has been impressive, even with the range of capabilities they have today. It seems clear that AI systems could have considerable effects on society. At the very least, automating the specific tasks that AI is already good at, for example in software engineering, biochemistry and robotics, will speed up some areas of scientific progress and contribute to economic growth. But we expect that AIs will become much more widely capable than they are today and have far more transformative effects. A common saying in the industry is today's AI is the worst AI you will ever use. In fact, many people in the field think that AI will get good enough to do essentially anything that humans can do and more. One milestone here would be developing Artificial General Intelligence, or AGI. People use this term in many different ways, but we'll use it to describe AI systems that can compete with humans on almost all cognitive tasks, or at least the most economically valuable ones. Think advanced scientific research, or designing new technologies and products, or running businesses, consulting, and so on. This is the kind of system leading AI companies are actively trying to build, and they're funneling billions of dollars into being the first to get there. Looking at recent trends in AI development, we think it's surprisingly plausible, though far from guaranteed, that we'll get this sort of AGI within the next decade. But it probably won't stop there. There's no reason to think that humans represent the ceiling of mental ability. So eventually, AI could greatly exceed human performance on many, if not all, cognitive tasks. Plausibly, they could even do work that's as far beyond human abilities as calculus is beyond chimpanzee abilities. It also might not take long before society makes giant advances in robotics. Although today's robots are very rudimentary, they're improving. And as our AIs get cognitively smarter, they'll also get better at both controlling robotic limbs and designing them. This means AI systems might quickly become able to outperform humans on many physical tasks as well. Over the next few sections, we'll explain how the advanced AIs of the future could transform society and present serious risks. Our argument focuses on the prospect that humanity develops AGI or something similar. This isn't the only important milestone, as we'll explain in a moment. But we think that if AI can match human abilities at the cognitive tasks that most drive innovation and economic production, that's likely to be enough to enable the rapid progress we describe in the following sections. And if AI becomes even more impressive than this, which we think is probable, the effects could be even more dramatic. So there's an aside here where we ask the question, could less advanced AI systems still pose existential risks? And our answer is, yeah, we think so. In the main argument of this article, we're focusing on AI systems that are very skilled at a wide range of tasks. And that's because we think systems like this pose the highest and most obvious chance of transforming society and throwing up many extremely serious risks. But we don't think this is the only milestone in AI capabilities progress worth worrying about. Even narrowly capable AI tools could be used to cause serious harm. An AI that excels at biotechnology research could, for example, make it easier for people to develop dangerous pathogens, regardless of whether it can also trade stocks or carry out business strategies. An AI that's only useful for launching powerful cyber attacks could still shift the global balance of power, and so on. Plus, we might face rapid destabilizing changes to society in the lead up to developing AGI, not just after it arrives. As AI gradually automates more and more tasks, we could see increasing levels of disruption across the Economy, including increased risks of AI systems acting against human interests, being used dangerously, or concentrating power in the hands of the few. As we'll Discuss later on, AI systems starting to automate AI R&D itself could be especially disruptive, introducing dramatic feedback loops in AI progress. And this could be enough reason to prioritize working on AI risks now, even if you don't think we'll get AGI anytime soon. Okay, back to the main argument. And remember, we're focusing here on AI systems that are skilled at a wide range of economically valuable tasks. This could be AGI or something similar. Section 2. Replacing human labor in the most economically valuable fields could trigger the next radical transformation of society. So what would it mean if AI systems could outperform humans on such a wide range of tasks? The first thing people often think of here is widespread unemployment. This is a serious possibility and would have severe consequences for society. But we think focusing on it is actually missing an even bigger story. A world with machines that can replace this much human labor would look so dramatically different that it can be hard to imagine. For some sense of comparison, think of how different the world is today to how it was for our ancestors 200 years ago, 2,000 years ago, or 20,000 years ago. The worlds before electricity or the printing press or agriculture literally looked very different, and they had entirely different ways of life. With each of these major breakthroughs in technology, the world has been transformed. Take the first agricultural revolution. Before agriculture, humans were mostly hunter gatherers and often lived in small bands. The development of farming technologies like ploughs allowed us to produce far more food per person, leading to the first cities. And to an increasing extent, some people could specialize in tasks other than finding food, which allowed humans to invent metalwork, writing, and early governance systems. The Industrial Revolution followed a similar pattern. The arrival of technologies like the steam engine dramatically increased productivity and sparked innovations in manufacturing and communication. And once again, this led to radical changes in how humans live. Goods that were once luxury items became available to ordinary people. Railways connected distant cities, and huge swathes of people shifted from rural to urban life. What's going on here? Each period of transformation in history has its own complex story, and there are competing theories about what drove them. But one popular explanation says we keep seeing the same rough pattern. Powerful new technology both enables us to sustain larger populations and lets people do more with the same bodies and minds. This means more human labor and greater productivity, which has compounding effects as it leads to a wave of even further innovation. Since innovation often feeds economic growth, humanity has also become much wealthier in this process. In fact, since the late stages of the Industrial revolution, We've seen roughly exponential growth in gdp, Though this is only an imperfect indicator of just how much change has happened qualitatively. A common thread in all these stories is that it seems growth has always been reliant on human labor. Society has only been able to progress as fast as humans can produce and implement new ideas. By that we mean new theories, inventions, ways of working, and so on. But we're now on the brink of a new breakthrough. If future AIs can replace human workers in the most economically valuable fields, we'll no longer be so reliant on human labor to sustain these cycles of compounding innovation and wealth. Instead, AI could become the primary driver of progress. And we think this could lead to another transformation of society. Like other technological breakthroughs, it could enable society to produce far more ideas and perhaps far greater economic output, fundamentally changing what the world looks like. But Unlike previous technologies, AIs could actually take over the processes that most drive innovation and economic production, including the process of designing better AIs. And as we'll discuss next, these AI workers could also have huge advantages over their human counterparts. This could mean the transformation brought about by AI is extremely rapid and more dramatic than anything we've seen before. Section 3. This transformation could be extremely rapid and dramatic. So what could happen as AIs automate more and more of the economy? At the very least, we expect to see the total amount of labor quickly increase. Since, unlike humans, AI systems can be easily copied at scale. Given enough hardware, let's say we build an AI that could replace a human engineer. Estimates suggest huge uncertainty here, but depending on the circumstances, running anywhere between thousands and hundreds of millions of copies of this AI at once may be feasible. And this number could grow fast. With efficiency improvements to the algorithms behind these AI workers, We'll be able to run a greater number of copies with the same amount of computer. We might also be able to allocate more compute to running copies through buying more chips or designing more efficient ones. Soon we could have an AI workforce the size of a significant fraction of the world's working age population. AI workers could also have other advantages over human workers. For example, AIs can work much faster than humans, often compressing several days of information processing into minutes. AIs may be able to coordinate far more efficiently between themselves than humans do, perhaps at lower costs and greater scales. And AIs can become specialized very quickly with different versions fine tuned to be exceptionally good at specific tasks. Based on these advantages alone, we could soon be seeing unprecedented levels of innovation and production as more work is performed by AIs. This could transform society for the same reasons automating physical labor did during the Industrial Revolution. And we think things could actually be even faster and more dramatic than you might expect from what we've said so far. That's because at some point in this story, we expect AIs will be used to automate AI research and development itself. And this might trigger an intelligence explosion. That is a period of rapid technological progress driven by AI systems that can create even better AI systems. Here's how an intelligence explosion could unfold. First, AI systems become good enough to automate all or most work in AI research and development. These AI workers then help us build better AI systems much faster. These better systems are then even more useful for automating AI, R&D, which lets us build even better systems, and so on. Recursively. If this happens, it could create a positive feedback loop in which AI systems get better and better, possibly over a very short period of time. And this wouldn't just mean building AI systems that are better and better at AI, R and D. It would mean speeding up improvements to AI capabilities more broadly, giving us increasingly capable and general AI workers to deploy across the wider economy. And that, in turn, could accelerate progress in most areas of society. What would accelerated progress look like? As we've said, previous periods of transformation in history were ultimately limited by the pace at which humans could produce and implement new ideas, by which we mean new theories and inventions and ways of working. But now imagine having a vast workforce of AIs that can produce far more brilliant ideas than us much faster than we ever could, and act on them more efficiently. We think this could transform society over a shorter time frame than we've ever seen. What would this even look like? For one thing, scientific discoveries could be made at an unprecedented speed. The market could suddenly be flooded with new technologies that would otherwise have taken decades to develop. Infrastructure and manufacturing could expand to scales we can barely imagine. More speculatively, if AI workers are deployed more widely, we could see a surge of fresh ideas, not just in science and technology, but in art, politics, philosophy, entertainment that fundamentally change how we even think about the world. The world could get much richer, too, since many innovations could increase economic production. In fact, some researchers think an influx of new ideas from AI workers would lead to explosive economic growth. And in turn, some of this new wealth could be used to accelerate idea production even further. We're not sure we'll actually see an explosion in economic growth, since there could be bottlenecks on turning innovation into increased gdp. But we think there's at least a decent chance this will happen. And more importantly, even without explosive economic growth, there could still be a radical qualitative transformation of society and our ways of life, which is what matters most to our story. How quickly could society be transformed exactly? There will ultimately be some constraints on the trajectory of human progress. For example, at some point, we'll hit bottlenecks on AI development, say in the availability of compute energy or high quality data that limit how much better AI workers can actually get over a short period of time. And in every field. Making progress could get increasingly difficult as AI workers quickly exhaust the low hanging fruit of discoveries and new ideas. Plus, some types of work could be particularly resistant to being automated, like complex physical tasks or certain legal or political processes. And this could also slow down the flywheel. But even after accounting for these effects, some researchers still argue that AI automation could compress a century worth of progress into a decade. This level of progress couldn't be sustained forever, but the world could already have been radically reshaped by the point things slowed down, like how the Industrial Revolution eventually came to an end, but left behind a world that was totally unrecognizable. Section 4. A rapid AI driven transformation would raise a range of major challenges, including existential risks. The idea that AGI could supercharge innovation and economic output could be worth celebrating. The world could become extraordinarily rich and we could rapidly develop new technologies that help us tackle the climate crisis or eradicate diseases. Indeed, the promise of the technology is one reason why we expect some people to be excited about developing advanced AI systems. As Anthropic CEO Dario Amadei puts it, a big motivator of AGI development is, quote, a genuinely inspiring vision of the future. End quote. Generally speaking, fears of emerging technology are often unjustified. Many innovations that have been viewed with suspicion, like vaccines and railways, have ended up being hugely beneficial for humanity. But in this case, things seem different. For the first time, we're designing a whole new population of highly intelligent beings, agents that can do the most economically valuable things that human minds can do and might not rely on humans to do them. This introduces complex dynamics we don't seem prepared to deal with and don't even fully understand. Humans navigating advanced AI could be like toddlers trying to navigate a world of adults with changes to everything we know in science, the economy, geopolitics, and even our ways of Life happening faster than we can get to grips with the difficult new concepts behind them. Given the uncertainty around how AI development will unfold, it's hard to predict exactly what challenges we'll face. But here are some examples of the things we're most worried about. Firstly, we'll encounter agents that could be much smarter than humans and might have goals of their own. Those goals might lead them to undermine human interests or even disempower humanity if we can't control them. Secondly, small groups could gain unprecedented power. If elite groups can control powerful AI, they'll be far less reliant on humans to get things done. With a vast AI workforce, they could amass previously unseen levels of economic and political influence, or even seize power. And they probably wouldn't have strong incentives to represent the interests of the broader population. Thirdly, dangerous technologies like bioweapons could could become much more accessible. Access to highly capable AIs could make it much easier to design or get hold of dangerous weapons, significantly lowering the bar for people to cause devastating harm to humanity. Fourthly, we may create a large new population of beings whose welfare and interests matter, raising complicated questions about how to coexist with them. And finally, all these factors may also drive conflict and unrest, possibly culminating in a great power war or creating other unforeseen challenges. How we navigate these dynamics could determine whether the future goes well or badly. If we handle things wisely, we could create a flourishing future with unprecedented prosperity for all sentient beings and could even spread to the stars. But if we lose control of advanced AI, or if bad actors use it to undermine the rest of the world's interests, we could face a catastrophe like humans permanently losing our ability to shape the future or going extinct. In other words, we think these issues have existential stakes, making them among the most pressing problems in the world. And although we're hopeful that these issues are tractable, we can't just assume our institutions will navigate them well by default. After all, this is confusing unprecedented territory, and we've seen society stumble into disaster before when facing new challenges we haven't sufficiently planned for. Just think about the slow institutional responses to early COVID 19 warnings, or the numerous close calls we've seen with nuclear weapons. There are two ways speed can matter critically to this transition. One, it matters how much time we have from now until we get extremely capable and general AI systems. And two, it matters how quickly the world is transformed by these systems once they arrive. If there's only a few years until we get AGI or something similar, then we have limited time to avert the risks. And if advanced AI changes the world very quickly, we might not have time to adapt to the changing circumstances and make wise decisions. Even now, our institutions sometimes act too slowly. For example, it took around 50 years from the initial scientific warnings about climate change for the milestone Paris Climate Agreement to be signed. Unless we make big changes to how our institutions work, if AI becomes rapidly more capable and more productive, it seems it will be extremely difficult for society to keep up. There's lively debate over how soon advanced AI systems might arrive and how quickly they might change the world. But there's at least a decent chance that they'll be here within the next decade and that things will change very fast indeed. The level of expert concern suggests we need to take this possibility seriously. And given the stakes here, we think it's important to prepare for this possibility, even if there's only a small likelihood, like a 10% chance of it coming true. This means we can't just ignore the risks or delay acting on them. We need to find robust solutions before it's too late. Section 5 work on these problems is tractable, but neglected. We've been helping people who want to work on this problem for over a decade. In this time, the field has grown substantially. A 2025 analysis put the total number of people working on existential risks from AI at just over 1000. And we think even this might be an undercount, since it only includes organizations that explicitly brand themselves as working on AI safety. We'd estimate that there are actually a few thousand people focusing their work on the most important risks raised by AGI. But to put that into perspective, Nature conservancy alone has 3,000 to 4,000 employees, and it's just one of many organizations working on environmental protection and climate change. Other global issues, like public health, also receive a lot of attention. For example, the World Health organization employs over 8,000 people. This all means that AI risks are severely neglected in comparison to many other world problems, so each additional person working to address them can make a bigger difference. We're also optimistic that we can make progress on these problems. After all, humans are choosing to design and deploy these technologies, which means we have some influence over how things go. Part of the challenge here is that people who currently have the most influence over AI development aren't necessarily incentivized to prioritize safety. AI companies want to make money and face pressures to develop technologies quickly without fully accounting for the risks they impose on society. Political leaders care about public opinion and election cycles, which gives them less time and motivation to focus on serving broader or longer term interests. So we need people who want to prioritize using their careers to help others to work on the major challenges that might otherwise be ignored. Fortunately, there are lots of ways you can help to tackle these challenges. Check out our hub of AI career resources at 80,000 hours.org AI for more information. Now we've laid out our main argument. Let's move on to discussing some objections and our replies to them. Objection one. You're overestimating how fast and how dramatically AI would transform the world. We've argued that automating human labor could transform the world at an unprecedented pace. But there are several ways our argument could be wrong. First, an intelligence explosion might not actually happen. We might deploy a generation of AI workers to automate some fields, but fail to get them to create even better or more general AI workers. Perhaps because we hit the ceiling of what current AI approaches can achieve, or we fall into another AI winter, we'd still get a one time increase in the size and efficiency of our workforce, making society much more productive, but we probably wouldn't see the dramatic compounding improvements we described earlier. Second, the constraints to progress could be stronger than we were expecting. So even if AIs do help us build increasingly capable workers, the feedback loop this creates might not be quite as explosive as we've described. For example, bottlenecks in AI R and D like the availability of compute energy and high quality data could mean developing the next generation of AI workers is just a slow process. And in every field we try to automate, the returns to effort could sharply diminish as AI workers quickly exhaust the low hanging fruit, caugh causing the effects of an intelligence explosion to fizzle out. Human dependent tasks could also turn out to be critical bottlenecks. Some economically valuable tasks, for example ones that require complex interaction with the physical world, or managing projects over weeks or months could just take an especially long time to automate. At least in the early stages of automation, the speed of AI driven progress could be seriously constrained by the pace at which humans can do those remaining tasks. On top of all that, our model of human progress could be missing key components. We've argued that increased labor and new ideas can drive rapid progress, pointing to historical examples like the Industrial Revolution. But other drivers we haven't explicitly considered here, like institutional or cultural changes, could be crucial. And at the time we get AIs capable of replacing human workers, these drivers could just be weaker than would be necessary to support something like a century of progress in a decade. In any of these scenarios, we still think AI could change the world and pose the serious risks we described earlier. But these changes probably wouldn't happen as rapidly as we've imagined. And as we argued, speed does matter. It affects how much time we have to adapt to the changing circumstances and make wise decisions. If any of the objections we've just listed are correct, it might also be really hard to sustain a period of supercharged innovation and economic production. In that case, progress could fizzle out quickly, perhaps even before we see changes as dramatic as we saw during the Industrial Revolution. But given the scale of the risks here, we think it's important to be prepared for a scenario where AI does transform the world rapidly and dramatically, even if there's a relatively small chance, say 10% of this happening. Still, the uncertainty here does make it harder to weigh up working on AI risks against other pressing problems like factory farming. If advanced AI doesn't change the world very much or very fast, we'd expect that to mean the risks are lower and that it will be easier to adapt to them, even if it doesn't eliminate those risks. Objection 2. It's hard to believe that AI could really pose existential risks. So this all sounds pretty wild. Could AI really cause outcomes as bad as human extinction? The argument we made earlier that the transformative effects of AI could create unprecedented challenges that threaten humanity's survival feels convincing to us, but it's always worth doing a sanity check on bold and provocative arguments. One way to do that is to look at what people in the field and other leaders say about a topic. So what do they say? Well, several leading institutions are already treating frontier AI as part of posing catastrophic risks. More than 1,000 AI scientists and industry leaders, including Geoffrey Hinton, Yoshua Bengio, Sam Altman, and Demis Hassabis, signed the center for AI Safety's one sentence warning that, quote, mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. End quote. At the UK government's AI safety summit, 28 countries, including the US and China, issued the Bletchley Declaration, which acknowledged the, quote, unquote, potential for serious, even catastrophic, harm from frontier models and pledging joint risk mitigation work in the US President Biden's executive order from 30 October 2023 compelled US AI companies to share safety test results with the government before releasing powerful systems, a measure that was unprecedented outside of biosecurity or nuclear security. President Donald Trump's administration has also decided to treat AI as a potential national security. Despite appearing to be skeptical of the idea that AI could pose catastrophic risks, some leaders disagree. Though meta's chief scientist Yann Lecun, for example, has called extinction worries preposterous, arguing that AI can be engineered to be safe. Other influential scientists, such as Gary Marcus, Andrew Ng and Melanie Mitchell, have expressed skepticism about the potential for AI to have transformative effects and pose existential risks. Surveys of AI researchers point to non trivial extinction odds Katja Grace of AI Impacts surveyed nearly 3,000 AI researchers on a range of key questions in the field. The median survey respondent assigned at least a 5% probability that advanced AI could result in human extinction or a comparable disaster. And roughly a third to a half of participants put the risk at 10% or higher. It's possible researchers in their own field are exaggerating the danger or they could be underestimating it. Still, this level of concern should prompt us to take the risk very seriously. Forecasters take note, but seem to doubt the risks. The Forecasting Research Institute conducted the Existential Risk persuasion tournament in 2022 to investigate disagreements on this topic. Overall, they found that AI raised the biggest concern about existential risk of all the topics covered. But among participants, there was a big split in opinion on the risks between domain experts in AI and people with a strong track record in superforecasting. On average, domain experts in AI estimated a 3% chance of AI caused human extinction by the year 2000, while superforecasters put it at just 0.38%. But both groups did agree on a high likelihood of powerful AI being developed by the year 2100. They put this at around 90%. And even skeptics of AI risk saw a 30% chance of catastrophic AI outcomes when looking at a thousand year timeframe. We should note here that given the developments in AI since 2022, we'd expect both groups would now predict timelines to powerful AI to be significantly shorter. And we think this would likely raise their estimates of the risks. So what should we make of this? Overall, it seems many leaders and experts recognize the potential of AI to pose major risks. Risks including at the level of human extinction. But unlike other problems that humanity faces, such as climate change, this isn't a matter of scientific consensus. There's ongoing disagreement, and many credible people think the risks are lower than we do. Still, given the stakes, we think it would be reckless to dismiss the idea that AI could cause outcomes like human extinction. Objection 3. Isn't all this talk of AI changing the world just A fad. Some people think arguments like those in this article are just a response to the current wave of AI hype and won't stand the test of time. It's possible we've updated our beliefs too strongly on the basis of the latest AI development, and our predictions could turn out to be wrong. But it's worth noting that the basic ideas of this article are not especially novel or unique to our particular time period. Prominent thinkers have been warning us about the dangers and transformative potential of AI since the 1800s. Here's a quick timeline. So in 1863, English novelist Samuel Butler speculated in a letter that machines would eventually surpass humanity, with humans becoming the inferior species. In 1920, playwright Karel Czapek, who coined the word robot, wrote a play in which artificial workers rebellious and eventually cause human extinction. In the 1940s, Isaac Asimov wrote a series of stories about AI and robots which highlighted the need to ensure their safety for humanity and suggested they develop the ability to steer humanity's future. In 1950, John von Neumann, a prolific and highly influential physicist and mathematician, reportedly said, quote, the ever accelerating progress of technology and changes in the mode of human life gives the appearance of approaching some essential singularity in the history of the race. End quote. In 1951, Alan Turing, considered the father of theoretical computer science, wrote the it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers. At some stage, therefore, we should have to expect the machines to take control. End quote. And in 1965, mathematician I.J. goode said, quote, since the design of machines is one of these intellectual activities, an ultra intelligent machine could design even better machines. There would then unquestionably be an intelligence explosion and the intelligence of man would be left far behind. End quote. Okay, we don't take any of these claims as strong evidence for the case that AI poses existential risks. After all, many historical figures, even extremely smart and influential scientists, have had really erroneous beliefs about the future. But they do show us that the argument that this is all just a fad just doesn't hold up to scrutiny. Objection 4. Isn't AI going to be just like every other technology? In some senses, yes, we think AI will be like other technologies, but this doesn't mean we shouldn't be worried about it. Like many other technologies, AI has had its own cycles of hype and bust. Some people think the current trajectory of high investment and fast progress could fizzle out and even be followed by another AI Winter. But that's Not a good reason to ignore the risks. Although it's likely some of the drivers of AI progress will eventually slow down, we think there's a good chance we'll already have AGI or something similar by the time this happens. And at that point, we could already be facing major challenges we'd wish we were more prepared for. It's also worth noting that AI doesn't need to be fundamentally different to previous technologies to change the world and pose really serious risks. After all, other general purpose technologies like the steam engine have done this before. The Industrial Revolution fueled huge growth, but also precipitated climate change and laid the groundwork for the invention of nuclear weapons. Even if AI were just another general purpose technology, it could be as impactful as this, and that alone would be a big deal. But we do think there are ways in which AI might be genuinely different from anything humanity has previously seen, making it potentially more transformative and more risky than previous technologies. For one thing, we argued across sections 1, 2, and 3 that AI systems could effectively take over the processes of innovation and economic production, meaning progress would no longer be reliant on human labor and could happen much faster than ever before. And even if you're skeptical of that particular story, it still seems hard to deny that something unprecedented is happening here. For the first time, we're designing a new form of intelligence that will potentially surpass ours. We could encounter a whole new population of highly capable agents with their own interests and perhaps even the capacity for welfare and suffering. In some senses, they could be our competitors. And the dynamics this introduces could be unlike anything humanity has ever had to navigate before. Objection 5. Is it even possible to produce artificial general intelligence? People have been saying since the 1950s that artificial intelligence smarter than humans is just around the corner, but it hasn't happened yet. Some have argued that producing artificial general intelligence is fundamentally impossible. Others think it's possible in theory, but unlikely to actually happen, especially not with current deep learning methods. However, we think there are compelling reasons to believe AGI is achievable. First, the existence of human intelligence demonstrates that general intelligence is at least possible in principle, and human brains are made out of ordinary matter following the same physical laws as computers. Second, while past predictions were overly optimistic about how long it would take, they weren't necessarily wrong about the fundamental possibility of AGI. The field ran into blockers early on, but researchers found ways around them using creative new methods, and they now have access to vastly more computational power to run experiments and train new AIs than we could have imagined. A few decades ago. Third, in recent years, we've seen progress we don't think would have been predicted by those who believed powerful general AI would never be developed. For example, large language models have demonstrated emergent behaviors that weren't explicitly programmed, like few shot learning, analogical reasoning, and cross domain transfer. Fourth, though some argue current AI methods will never grasp certain forms of intelligent reasoning, these critiques have often been proved wrong. For example, Yann lecun claimed in 2022 that deep learning based models like ChatGPT would never be able to tell you what would happen if you placed an object on a table and then pushed that table, because such a basic situation was never described explicitly in text. But models like ChatGPT can now walk you through scenarios like this with ease. In the words of AI researcher Leopold Aschenbrenner, if there's one lesson we've learned from the past decade of AI, it's that you should never bet against deep learning. End quote. Look, there's real uncertainty here, and the skeptics might be right that there are some things advanced AI systems will just never achieve. But for AI to transform the world, the important question isn't whether we'll replicate every aspect of human cognition exactly. It's whether we can create systems that can. One, match or exceed human performance across the tasks that matter most for scientific research, economic productivity, and the other domains where intelligence is most valuable. 2 Perform those tasks faster or more cheaply than human workers can, and three work autonomously enough that progress in the fields they automate is no longer bottlenecked on the speed of human labor. All three of these things seem quite possible, and even this much may not be necessary for AI to pose serious or even existential scale risks. For example, our argument that people could catastrophically misuse AI mostly depends on AI systems becoming useful tools for designing weapons. An AI that's great at assisting humans with biotechnology research could make it far easier for people to develop dangerous pathogens, regardless of how well it performs at other types of research or how much oversight it needs from humans. So even if you think we'll never build AIs that are fully general or completely autonomous, the risks could still be extremely serious. Objection 6. Even if AGI is achievable, what if we're really far away from building it? There's lively debate over when we'll build AGI or something similar. We think there's a decent chance this will happen soon, perhaps within the next decade, and we're not alone. But it's worth considering other possibilities for example, researcher Ege Erdil has made an influential argument for AGI being multiple decades away, and some people think it's even further out than that. Plus, even people who think there's a good chance that AGI or something like it will arrive soon tend to also think that there's a good chance it will take a while. Even if AGI is many decades away, we still think it will transform the world when it arrives and create unprecedented challenges. But on this longer time frame, work to address these challenges would be less urgent because we'd have more time to prepare. Despite this, we still think it makes sense for many people to focus on AI risks now. There's a few reasons for this. Firstly, there's huge uncertainty around how long it will take for AGI to be developed. We need to prepare for the chance that it happens very soon so that we're covered in the worst case scenarios. Secondly, some issues with advanced AI might just take a long time to solve. Deep technical challenges could take many years of research to untangle. And some governance issues might require us to redesign how our institutions work, which won't happen overnight. Putting more work in now will give us a better chance of navigating the risks competently when they start to emerge. And thirdly, many people who could help a lot in a decade's time should start now, especially if they're early in their career. It takes time to build up expertise and career capital. So we still have years isn't a reason to not get started. Objection. 7 isn't the real danger from actual current AI and not some sort of futuristic AGI? There are definitely dangers from current artificial intelligence. For example, AI has frequently been linked to child safety concerns, with reports of AI chatbots generating sexualized images of children, engaging minors in sexual conversations, and in some cases even encouraging emotionally dependent teenage users to commit suicide. In addition, data used to train neural networks often contains hidden biases. This means that AI systems can learn these biases, and this can lead to racist or sexist behavior. AI models are also trained on copyrighted material without permission or compensation, raising serious questions about intellectual property rights and threatening the livelihoods of artists, writers and creators. And AI tools make it easier to run sophisticated scams at scale like deepfake videos, impersonating senior employees of companies to authorize fraudulent money transfers. These dangers are real and serious, and lots of people should focus on addressing them. But we still think that the amount of work going towards longer term AI risks needs to significantly increase. The good news is that there isn't always A big trade off between addressing shorter term or longer term AI risks. Lots of work that's geared towards existential threats from AI systems is also relevant to solving problems with existing AI systems. For example, some AI safety research focuses on ensuring that machine learning models do what we want them to and will still do this as their size and capabilities increase. Other research tries to work out how and why existing models are taking the actions that they're taking. Both of these things would help us prevent future AI systems from taking power, but they'd probably also help us prevent current AI systems from discriminating against marginalized groups or exploiting vulnerable users. We also think the current dangers are just the tip of the iceberg. As AI systems get more capable, the risks could get increasingly serious. As we've argued, future systems seem like they could pose threats not only to individual humans, but also to the very existence of humanity, say by enabling a catastrophic pandemic that wipes out much of the population or helping a small group establish a long lasting authoritarian regime. Ultimately, not all work on future risks will translate neatly into progress on today's issues. But we have limited time in our careers, and choosing which problem to focus on could be a huge way of increasing your impact. And it seems important for many people, though not all, to focus on addressing the worst case possibilities. Objection 8 Technological progress is a good thing for humanity Technological optimists point out that past technologies have generally made life better, not worse. Why should AI be different? Well, while technology has indeed brought many benefits, it's also created new risks and challenges. Developing nuclear weapons gave us both nuclear power and the threat of nuclear war. Advanced biomedical science has cured many diseases, but it also raises the risk of bioweapons and disastrous leaks of dangerous pathogens. Industrial factory farming has made for cheaper meat, but it's also a moral catastrophe for the animals themselves and has many negative side effects for humans. We agree that technology has usually benefited humanity overall, but the question is whether it will benefit us overall in this case. There are enough precedents of dangerous technological developments to be cautious. And there are specific reasons for concern in this case. As we discussed earlier, and given the potential scale and speed of AI development, the margin for error may be smaller than with previous technologies. Objection 9. This all just sounds too sci fi. The idea that something sounds like science fiction isn't a reason in itself to dismiss it outright. There are lots of examples of things first mentioned in sci fi that then went on to actually happen. There are even a few such cases involving technology that are genuine existential threats Today. For example, in his 1914 novel The World Set Free, H.G. wells predicted atomic energy fueling powerful explosives 20 years before we realized there could in theory be nuclear fission chain reactions. And 30 years before nuclear weapons were actually produced. In the 1920s and 1930s, Nobel Prize winning physicists Millikan, Rutherford and Einstein all predicted that we would never be able to use nuclear power. So nuclear weapons were literal science fiction before they were reality. And then in the 1964 film Dr. Strangelove, the USSR builds a doomsday machine that would automatically trigger an extinction level nuclear event in response to a nuclear strike. But they keep it secret. Dr. Strangelove points out that keeping it secret reduces its deterrence effect. And we now know that in the 1980s, the USSR built an extremely similar system. And yes, they kept it secret. When you hear something that sounds like science fiction, it's reasonable to want to investigate it thoroughly before acting on it. But having investigated it, if the arguments are solid, then simply sounding like science fiction is not a reason to dismiss them. Objection 10. Can it really make sense to dedicate my career to solving an issue that's based on a speculative story about something that may or may not ever happen? We never know for sure what's going to happen in the future. So unfortunately for us, if we're trying to have a positive impact on the world, that means we're always having to deal with at least some degree of uncertainty. We also think there's an important distinction between guaranteeing that you've achieved some amount of good and doing the very best you can to achieve the former. You can't take any risks at all, and that could mean missing out on the best opportunities to do good. When you're dealing with uncertainty, it makes sense to roughly think about the expected value of your actions. The sum of all the good and bad potential consequences of your actions weighted by their probability. Expected value isn't the only framework to use. We also think it's important to temper your estimates of expected value using common sense and other heuristics. But it's a really useful indicator of how important a certain course of action is. Given the stakes are so high and the probabilities of the risks from AI aren't that low. This makes the expected value of helping with this problem very high. We're sympathetic to the concern that if you work on AI, you might end up doing not much at all when you might have done a tremendous amount of good working on something that's more certain. But we think the world will be better off if we decide that some of us should work on solving these problems so that together we have the best chance of successfully navigating advanced AI rather than risking an existential crisis. Objection 11. Okay, AI might pose existential risks, but isn't issue X an even bigger problem? You might think it doesn't make sense to focus on the risks from future AI systems when the world faces so many other challenges. For example, you might want to do whatever you can to prevent the most death and suffering that's happening now. This would probably lead you to prioritise addressing factory farming or even wild animal suffering, since these issues concern present harms and are also incredibly neglected relative to their scale. Even if you do want to focus on making humanity's future go well, you might feel the risks from future AI systems are just too uncertain. In that case, you'd probably choose to work on threats that feel more concrete at this stage, like catastrophic wars. It's certainly reasonable to prioritize working on something else over AI risks. It would be arrogant to claim we've figured out all of the world's problems well enough to know the most pressing ones are definitely all downstream of powerful AI. But we still think focusing on the risks from advanced AI is often a bet worth making, because, as we've explained, there's a material possibility future AI systems could cause humans to go extinct or permanently lose control of the future. And as time goes on, more and more of the theoretical reasons for concern, like the potential for deceptive behavior in AI systems, are being borne out in practice. Plus, if AI does transform the world, this would probably shape all the other challenges society faces and dictate how they can or should be addressed. For example, what happens with AI might determine the military capabilities of the world's greatest powers, as well as the diplomatic tools they use to handle conflicts. So making sure powerful AI is handled responsibly could be a big component of addressing many other world problems. We don't think everyone listening to this should drop what they're doing to work on AI risks. And we're still excited to see people make progress on other pressing problems. But if you can find a role focused on AI risks that really suits you, we think there's a very good chance that that's the highest expected impact thing you could do. Learn more Want to learn how you can contribute to making AI go well? Advanced AI poses many different challenges, like the chance AI systems will disempower humanity, enable small groups to gain unprecedented levels of control, or lead to the development of devastating new weapons. You can find full explanations of issues like these and what you can do about them in our series on the world's most pressing problems. Just go to 80,000 hours.org and search for problem profiles. Or you can check out our hub of AI career resources, 80,000 hours.org AI. If you're interested in working on any of these problems, we'd be especially excited to advise you on your next steps. One on one. We can help you consider your options, make connections with others working on reducing risks from AI, and possibly even help you find jobs or funding opportunities, all for free. You can apply to speak with our team on our website. Many thanks to Cody Fenwick who drafted an earlier version of this article, much of which was incorporated here. Thanks also to Arden Kaler, Adam Bales, Andreas Mogensen, Benjamin Todd, Neil Bowerman and Aaron Gertler for their input. Please share this article with others who might find it helpful or interesting. Thank. You.