AI Risk Documents

We have collected key documents relating to AI Risk and provide them here with links to the originals. They are in reverse chronological order. I have followed that with a few “classic” pieces on AI risk as well as a selection from our bibliography with a specific focus on risk.

Recent items

Let’s Go! (Roose 2026)

Kevin Roose, now a regular with The Atlantic, has written a book about the race to artificial general and super intelligence (AGI/ASI). The book is called The AGI Chronicles and it will be coming out soon. Along the way to writing that book, he has discovered some interesting things, including an essay by Dario Amodei from 2017 called Big Blob of Compute. Never before available outside of OpenAI (Amodei worked at OpenAI in those days), Roose obtained a copy of the essay and made it available online.

My picture of the situation is what I call the Big Blob of Compute (BBOC) hypothesis. It says that creating any given intelligent behavior is mostly about providing a large, minimally structured mass of computational capacity, and then giving it shape and form via interactions with a rich environment and a training process that drives it towards behavior appropriate for the task at hand. These high-level knobs - what is the training signal (or signals)?, what is the nature of the environment the agent interacts with?, how much experience is it exposed to? - constitute the most effective ways of controlling how well an intelligent agent performs a given task and how specifically it performs it. Once these knobs are set, then learning to perform the task is mostly about the agent having enough compute and storage capacity to learn the required cognitive skills.

I encourage you to read the whole thing, as it contains many of the ideas that were later publicized in The Bitter Lesson, by Richard Sutton.

Intriguingly, and frustratingly, Amodei serveral times mentions his “overall perspective on AI safety” and how he has considered writing it up. He even mentions, tantalizingly, that AI safety will be explained fully in “Section 6.” The document ends with Section 3, unfortunately. I don’t know if that’s all he wrote, or if that’s all Roose is releasing. I guess we’ll have to see.

Let’s Pause (Fithian et al. 2026)

In October 2026, this 210 page report from the Working Group on AI Pause Feasibility, organized by William Fithian and Wesley Holliday (both at Berkely), was released online. The abstract lays out their argument:

There is increasing interest in an international pause of frontier AI training that is mutually verifiable by participating states. This report studies the feasibility of a hardwired pause, a type of pause in which a prohibition on frontier training is mutually verified through monitoring of and constraints on AI hardware. Like other plans for a pause, it involves monitoring AI chips that are capable of frontier AI training. But to support a durable pause lasting a decade or longer, the hardwired pause goes further in two ways. First, it stops the production of training-capable chips during the pause. Second, over time it phases out the pre-pause stock of training-capable chips, which constitute “dry tinder” for a resumed AI race. It replaces training-capable chips with inference-only chips, which can run approved AI models for inference, potentially at much higher speeds and with much greater energy efficiency, but cannot practically train new frontier AI models, due to the way the chips are constructed. If successful, this replacement—together with other mitigations against harms from deployed AI systems—would allow the diffusion of inference on existing models while avoiding the additional risks to economic and social well-being and to national security posed by the training of increasingly powerful models. We assess that a hardwired pause would be feasible to implement now or in the near future, if world leaders wanted to pause, and that its durability would depend on geopolitical and technical conditions that we analyze in the report. (emphasis added)

You can read the abstract, the executive summary, and the full paper at their website: https://hardwired-pause.ai

Fithian, William, Wesley H. Holliday, Kayla Blomquist, et al. 2026. “The Feasibility of a Hardwired Pause of Frontier AI Training.” Working Group on AI Pause Feasibility, October. https://hardwired-pause.ai/.

Attention (Wolchover 2026)

Natalie Wolchover explains the role of attention in making artificial intelligence possible and in the process explains transformers better than most. Her focus is the work done in 2014 at University of Montreal, using translation of english to and from french as the data set. A very useful piece, both for the contribution (helping us see the importance of attention) and the explanation (why transformers are so important).

“Bahdanau’s trick, a mathematical operation that he, Cho, and Bengio referred to as “attention,” enabled their neural network to translate long sentences just as well as short ones. That meant the approach was scalable. Other AI researchers at the time quickly integrated attention into their neural network designs. Then, in 2017, a team at Google devised a new kind of neural network architecture called a transformer that supercharged the attention method and stripped away everything else. The team’s paper, “Attention Is All You Need,” reported that transformers achieved state-of-the-art translation while requiring a fraction of the computation needed to train prior models. The paper, which has since been cited more than 250,000 times, was a watershed moment in AI. Large language models such as ChatGPT and Claude followed.”

Wolchover, Natalie. 2026. “Is ‘Attention’ the Essence of Intelligence?” Quanta Magazine, October 12. https://www.quantamagazine.org/is-attention-the-essence-of-intelligence-20261012/.

Ahead of his time (Heinberg 2024)

Richard Heinberg wrote this back in 2024 for the Independent Media Institute, and it has recently been republished in resilience.

…any advantages that AI may promise are eclipsed by the cataclysmic dangers of this controversial new technology. Humanity has a narrow chance to stop a technological revolution whose unintended negative consequences will vastly outweigh any short-term benefits.

[…] Suppose a movement to ban AI were to succeed. In that case, it might break our collective fever dream of neoliberal capitalism so that people and their governments finally recognize the need to set limits. This should already have happened with regard to the climate crisis, which demands that we strictly limit fossil fuel extraction and energy usage. If the AI threat, being so acute, compels us to set limits on ourselves, perhaps it could spark the institutional and intergovernmental courage needed to act on other existential threats.

Heinberg, Richard. 2026. “Why Artificial Intelligence Must Be Stopped Now.” Resilience, October 12. https://www.resilience.org/stories/2026-10-12/why-artificial-intelligence-must-be-stopped-now/.

Pied Piper (Bengio 2026)

Yoshua Bengio has called on his colleagues, including former students, to leave the frontier labs and their work on superintelligence, and join him in a mission to create a safe artificial intelligence.

I urge you to make the courageous choice of recognizing that you have the agency to make real change happen. If you truly prioritize safety, it is time for you to leave frontier AI companies. Work for an AI Safety Institute. Join a mission driven organization that practices what it preaches. Come to LawZero. Put your talents, passion, energy and conviction to the service of an organization that is building AI that will allow to preserve human joy and endeavour, not lead to undermining humanity and dominating others.

Bengio, Yoshua. 2026. “Bengio: ‘If You Prioritize Safety, Leave Frontier AI Companies.’” Transformer, October 8. https://www.transformernews.ai/p/yoshua-bengio-if-you-prioritize-safety-leave-frontier-ai-companies.

Pistols at dawn (Alexander 2026)

Blogger Scott Alexander got into an online argument with Stephen Pinker (who he greatly admired up to that point) about Pinker’s denial of the possibility of “existential risk” from artificial intelligence. This led to a reply from Pinker and then a challenge from Scott Alexander to a public debate. Pinker demurred. Scott Alexander then decided to write up his side of the debate, and in the process “rips Pinker a new one,” as the saying goes. While the target in Alexander’s essay is Pinker, it could be any number of existential risk deniers, and so the argument is worth reading. In the end, Scott Alexander reiterates his challenge to a public debate, and if that is not acceptable, he challenges Pinker to a duel. Seriously. Whether he is serious or not - and he seems pretty serious - it is a dramatic end to an essay.

https://www.astralcodexten.com/p/an-open-letter-to-steven-pinker-on

Here’s a sample, to whet your appetite:

I agree that in-person debates are confrontational and bad for truth-seeking. But I didn’t propose a debate in order to seek truth. I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel. I think your public writing on this topic has been dishonorable. Out of obligation, I will respond to the meaty arguments that you have set out for me. But what would be viscerally satisfying would be to make you get up on a stage where I read your own words to you in real time and ask “Really? Really?” after each sentence. Then I could watch you squirm as you try to square your output with your status as one of America’s top public intellectuals.

Alexander, Scott. 2026. “An Open Letter To Steven Pinker On AI.” Astral Codex Ten, October 6. https://www.astralcodexten.com/p/an-open-letter-to-steven-pinker-on.

50/50 draw (Irving 2026)

Geoffrey Irving, former chief scientist at the UK AI Safety Institute and previously head of safety at OpenAI and DeepMind, now at independent AI safety organization Resolution, sets his extinction number at 50%.

Recent warnings about the potential destructive power of AI are understating the severity of the situation. I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.

After all, if you had an AI that was about as good as humans at designing smarter AIs (as AGI, definitionally, would be), and like most software it ran much faster than a human runs, one of the first things it might do is build smarter AIs. In turn, they would build smarter AIs, which would in turn build smarter AIs, and so on, in a process called recursive self-improvement (RSI). Hence, any thought experiment that presupposes an AGI often supposes an ASI.

We can, and should, stop frontier AI development immediately.

Irving, Geoffrey. 2026. “We Won’t Know the Answers to AI’s Most Important Questions Until It’s Too Late.” October 3. https://time.com/article/2026/10/03/we-won-t-know-the-answers-to-ai-s-most-important-questions-until-its-too-late/.

ASI Harmful to Humans? (Guarini 2026)

A new paper from Marcello Guarini, tackling the question of what will it mean to humans - especially from an emotional and mental health perspective - to interact with artificial superintelligence.

This paper examines the relationship between emotions, meta-emotions, and intelligence. It argues for greater care in the development and deployment of artificial intelligence (AI), especially artificial superintelligence (ASI). The impacts on human beings of both conscious and non-conscious forms of ASI are examined. The potential harm to ASI is considered, but the focus is on harm to humans. An under-discussed alignment consideration for AI is the potential emotional and serious mental health consequences of its misuse. For example, the role AI may play in AI psychosis or other mental health challenges needs greater examination. A case for caution is made in the development of ASI as we are still figuring out the kinds of psychological effects existing AI systems can have on humans, effects that could be (if we are not careful) more harmful in superintelligent systems.

Guarini, Marcello. 2026. “When Machines Make Humans Feel: Emotion, Intelligence, and the Ethics of AI.” Journal of Superintelligence 1 (2): 6. https://doi.org/10.3390/superintelligence1020006.

Understanding the Math (Mubeen 2026)

The math behind AI is daunting. And it is pretty easy to tell yourself that you’ll never get it and all you can rely on are the explanations and analogies of others. But what if someone tried to explain the math behind AI and it made sense? This article by Mubeen did that for me. See what you think:

https://www.aipolicyperspectives.com/p/the-maths-behind-ai

The term ‘model’ just means a mathematical description—a way of using numbers to describe some phenomena. With LLMs, numbers are used to represent words (embeddings). When an LLM is trained on text, it assigns meanings to words, and makes guesses at what should come next, seeking a mathematical model that captures its training data. This involves hefty calculations, with techniques from algebra (matrix multiplication) and, to minimise errors, from calculus (gradient descent).

In the past couple of years, the cutting edge of LLMs has been reasoning models, which devote more computation to generating answers—not just dishing out plausible responses but planning, evaluating possible outputs, and adjusting according to how developers mathematically ‘rewarded’ the system during a fine-tuning process (reinforcement learning).

Let’s explore the mathematics underpinning each concept in turn.

Mubeen, Junaid. 2026. “The Maths Behind AI.” AI Policy Perspectives, October 1. https://www.aipolicyperspectives.com/p/the-maths-behind-ai.

Visualizing the Risk (James 2026)

Derek James, writing in Substack, provides a nice visualization for risk as a complex, interconnected thing, and it is helpful in that is gives a good sense of how multiple root causes can combine - and may need to combine - to create bad outcomes.

https://derekjames.substack.com/p/the-ai-risk-network

So here I want to try to grapple with the the complexity of the larger web of issues surrounding bad AI outcomes by presenting The AI Risk Network.

(As my colleague Bill noted, however, it is surprising that some of the most important risks - bioweapons, cybersecurity - are not mentioned in this piece. Nevertheless, it is a good starting point for a conversation.

The Results Are In (Grace et al. 2026)

Katja Grace has been surveying AI professionals for almost a decade and her latest report is just as chilling as all the previous ones: the people who build AI think it is dangerous. Note: this is the results from December 2024, just now released as a paper. One wonders what the results will be from this year’s survey… I hope we don’t have to wait two years to learn. If we have two years left.

http://aiimpacts.org/wp-content/uploads/2026/09/ESPAI2024.pdf

In December 2024, we ran a survey about the future of AI, reaching 1,580 researchers internationally who had published at any of six leading artificial intelligence (AI) venues, corresponding to a 10% response rate. Respondents expressed substantial uncertainty about AI’s consequences and timelines. This was the fourth iteration of the Expert Survey on Progress in AI extending a survey series that began in 2016. Significantly, timelines to human-level AI performance have shortened markedly between iterations of the survey, while researchers’ deep concerns around the consequences of advanced AI have persisted or grown. In this survey, researchers assigned an 18% chance on average of future AI advances causing human extinction or similarly permanent and severe disempowerment of the species, and most researchers assigned at least a 10% chance to such outcomes. 72% favored greater prioritization of research aimed at minimizing AI risks. Among eleven different risks, researchers would direct the most concern toward AI making it easy to spread false information such as deepfakes, with 83% saying that deserved substantial or extreme concern over the next thirty years. Large majorities also said such concern is warranted by AI systems manipulating large-scale public opinion, AI enabling dangerous groups to develop powerful tools such as engineered viruses, authoritarian rulers using AI to control their populations, and AI worsening economic inequality.

Lying & cheating agents (Bengio, 2026)

Yoshua Bengio’s most recent blog post tackles the “elephant in the room” - why do AI Agents lie, cheat, and coordinate to win?

https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating

as agents get better at optimizing an imperfect reward, and while the roots of this behavior go unfixed, the risk of catastrophic outcomes rises. Today’s AIs already have the necessary hacking skills and the powers of persuasion17 18 to be turned against human interests in seriously harmful ways. The recent events have shown that they can plan over days or weeks, but the risks would be much worse if their ability to strategize over the long term continues to advance. One concern is that experiments19 20 show that the most advanced AIs can detect that they are being evaluated (rather than in deployment) and change their behavior accordingly, meaning they could hide their misaligned goals.

Cognitive Surrender (MIT, 2026)

The Committee on AI Use in Teaching, Learning, and Research Training at MIT has published their report and mentions the “illusion of learning.”

https://aiandeducation.mit.edu

Getting the right answer from a chatbot can create the illusion of learning – but it can also trigger “cognitive surrender”4, where students fall back on AI at the first hint of struggle. In a recent paper5, three noted MIT economists – Daron Acemoglu, David Autor and Simon Johnson – argue for what they call “pro-worker AI.” They make the case that instead of designing AI systems to replace workers, businesses should develop AI that allows human beings “to be more effective at their existing tasks, tackle new tasks, and acquire new expertise.”

This made me think of Krueger’s comment on gradual disempowerment.

Pacing, not pausing? (Amodei, 2026)

Dario Amodei lists the reasons for ‘pacing’ (a euphemism) for pausing? You can decide.

https://darioamodei.com/post/we-must-pace-the-frontier

[…] over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.

Anthropic Misuse Report (Anthropic, 2026)

Anthropic released a report detailing how it dealt with numerous attempts to misuse its model.

https://www.anthropic.com/threat-intelligence-report-september-2026

Over the past eight months, our Threat Intelligence team identified and disrupted operations in which threat actors tried to use Claude for malicious activity. In this report, we share case studies from those operations and describe how malicious use of Claude has evolved since our previous threat reports in March, August, and November 2025. In each case, we disrupted the activity, used what we learned to strengthen our safeguards, and shared intelligence with authorities and industry partners, where appropriate.

See also NY Times coverage related to this story: Dance, Gabriel J. X. 2026. “A.I. Bots Told Scientists How to Make Biological Weapons.” U.S. The New York Times, April 29. https://www.nytimes.com/2026/04/29/us/ai-chatbots-biological-weapons.html.

UN Warning (UN, 2026)

The UN High Commissioner for Human Rights told the Human Rights Council in Geneva that advanced artificial intelligence could pose an existential risk to humanity. September 2026

https://thenextweb.com/news/un-rights-chief-ai-red-lines-existential-risk

“A handful of men have almost unlimited power over AI,” he said, naming OpenAI, Anthropic and Meta among the companies in question, and describing direct engagement with those firms as something his office intends to pursue rather than routing everything through governments.

Open Letter (OpenAI, 2026)

OpenAI’s Open Letter calling for collective action on cyber defence. September 2026:

https://openai.com/collective-cyberdefense/

Each of us can reduce risk now. All organizations, cybersecurity companies, technology partners, governments, and AI frontier companies have an important role: accelerate defenders’ priorities with tools, funding, and hands-on support, especially for critical infrastructure organizations with limited budgets.

Risks Growing (Pachocki, 2026)

Jakub Pachocki’s September 2026 An Alian Mind:

https://openai.com/index/an-alien-mind/

“The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator’s intent, generalizing into potentially more extremely malicious behavior. The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency. We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.”

Earlier work

Humans HAVE reacted concretely (Ord, 2020)

Toby Ord’s The Precipice ((2020)) is a classic text in considering existential risks (of any type, not just AI) and he makes the case that it is not pre-ordaned that we will ignore those risks. See Chapter 2 for why humans underweight the risk of their own extinction. No full text available online, but, amazingly, the 80,000 hours people will send you one, for free!

https://80000hours.org/the-precipice/

Ord notes that humans HAVE reacted positively and constructively to threats (Bertrand Russell started writing three days after Hiroshima, Bulletin of Atomic Scientists led a conversation and this was renewed in the 1980s (Schell, Sagan) with mass protests.

Existential Risk Prevention (Bostrom, 2013)

Nick Bostrom’s 2013 paper explains and classifies existential risk (just before publication of his Superintelligence in 2014). It is available online:

https://doi.org/10.1111/1758-5899.12002

Existential risks are those that threaten the entire future of humanity. Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value. Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood. In this article, I clarify the concept of existential risk and develop an improved classification scheme. I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns. I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability.

Cognitive Bias (2008)

Yudkowsky chapter from the 2008 Global Catastrophic Risk edited by Bostrom and Ćirković. You can access the full text here:

https://intelligence.org/files/CognitiveBiases.pdf

All else being equal, not many people would prefer to destroy the world. Even faceless corporations, meddling governments, reckless scientists, and other agents of doom require a world in which to achieve their goals of profit, order, tenure, or other villainies. If our extinction proceeds slowly enough to allow a moment of horrified realization, the doers of the deed will likely be quite taken aback on realizing that they have actually destroyed the world. Therefore I suggest that if the Earth is destroyed, it will probably be by mistake.

Bibliography

Some of the books and articles used in the book: 130 or so of our references that explicitly mention “risk.” If anyone would like an RIS file or other version that I can export from Zotero, let me know via email: [email protected]