In this post I map out the six main sources of threat intelligence we currently rely on—controlled capability evaluations, real-world vulnerability discovery, open-source pen-testing tools, monitoring of the underground ecosystem, incident forensics, and LLM API usage data—and assess how durable each one is. The worrying pattern is that several of these sources are at risk of diminishing as attackers stop using API-hosted LLMs in favour of self-hosted models and improve their operational security.

A further problem is that much of the critical data sits with private-sector actors—security firms, model providers, and network defenders—who share findings based on commercial incentives rather than public benefit. To avoid losing visibility into cutting-edge AI cyberattack capabilities exactly when it matters most, I argue governments should fund independent threat-intelligence collection and sharing, develop AI-specific forensic capabilities, build the capacity to evaluate models on real-world tasks, and establish an “Agentic Cybersecurity Exchange” that aggregates signals across infrastructure operators.

Read the full post on Attack Surface

To investigate, I built a weighted factor model scoring ten research areas across seven dimensions—task length, insight vs. engineering, data availability, feedback quality, compute bottlenecks, scheming risk, and economic incentives. The headline finding is that capabilities research looks meaningfully more amenable to AI-driven speedups than safety research (average scores of roughly 6.45 vs. 5.21). Within capabilities, agent scaffolding and training efficiency score highest; within safety, AI Control and dangerous capability evaluations are the most automatable, while alignment theory ranks lowest because it leans on conceptual breakthroughs that are hard to verify.

I then look at what could change this ordering. Levers for differentially accelerating safety include building safety-research automation infrastructure now, developing benchmarks and model organisms, writing detailed research proposals that reduce the creativity required, granting safety researchers differential early access to new capabilities, and investing in documentation to expand the training data available to safety-focused systems. I close by flagging the substantial uncertainty in the analysis—results are sensitive to the factor weights and the somewhat artificial division of AI R&D into discrete domains.

Read the full post on the EA Forum or on LessWrong.

Safety Cases are structured arguments, supported by evidence that a system is safe enough in a given context. They sound compelling in theory—structured arguments forcing developers to affirmatively demonstrate safety rather than just “test and hope” (read my previous post for a longer explanation). They force to make reasoning about safety explicit, are flexible to changes in the technology and put the burden on the deveoper. Anthropic and DeepMind have committed to making Safety Cases in their RSPs.

To figure out how useful they are, I emailed researchers actively working on AI Safety Cases and reviewed the growing literature. This post synthesizes what I learned: first, a brief overview of what Safety Cases are and what work currently exists; then, expert responses to key questions about their feasibility, impact, and limitations. If you have perspectives you’d like to add, I’m happy to incorporate additional expert opinions—feel free to reach out or comment here.

Questions addressed in this post:

1. What is currently being done on AI Safety Cases? — Sketches and prototypes emerging
2. Will Safety Cases have a strong, positive influence on the safety of Frontier AI Systems? — Experts disagree
3. What is currently the largest gap to being able to build comprehensive AI Safety Cases? — Consensus on Methodology, basic science, technical safety challenges
4. Will we be able to build convincing safety cases for catastrophic risks before catastrophic risks from AI are possible? — Unlikely for 5+ years
5. What is the largest challenge in the adoption of AI Safety Cases in legislation? — Political will
6. Should Safety Cases be a large focus for the field of AI Governance? — Experts disagree, some argue to prioritize transparency

What is currently being done on AI Safety Cases?

Actors such as companies, governments, and academics are building sketches for Safety Cases for Frontier LLMs and thus advancing best practices. Here is a (likely incomplete) list of Sketches for AI Safety Cases:

An incomplete list of Safety Case sketches, including the paper title, institutions involved, threat model, and core strategy.

While some of these sketches focus on proving some Objective (top-down), others are motivated by sketching how a particular method could give safety assurances (bottom-up). Many Safety Cases use CAE or GSN as standard notation.

Further work attempts to gather the kind of evidence that would let us build strong safety cases like Science of Evals, Chain-of-Thought Monitorability, or Red-Blue-Team Control Evaluations. Finally, Bart et al argue for making Claims in Safety Cases Probabilistic, allowing us to assess the confidence that our Objective is met.

After an initial wave of position papers arguing for safety cases people are now busy making “sketches” (ie outlines of possible safety cases) for different safety properties. This is in an attempt to develop better methodology for Safety Cases. Some Frontier AI Companies are attempting to make initial Safety Cases for real production models.

Will Safety Cases have a strong, positive influence on the safety of Frontier AI Systems?

TLDR: Experts disagree on whether Safety Cases will substantially improve frontier AI safety. As best practice in safety-critical industries, they offer flexibility compared to rigid rules and can coordinate diverse stakeholders. However, skeptics argue their practical impact on company behavior may be small (especially for less responsible actors) and they risk providing false assurance while being costly to produce and enforce.

Epistemic effects on safety researchers cut both ways: Constructing Safety Cases forces researchers to make their reasoning explicit, confronting assumptions and rare failure modes that standard evaluation might miss. However, framing the task as “building an argument for safety” risks incentivizing motivated reasoning over objective risk assessment.

Joshua Clymer: “I expect most of the value of safety cases will come from improving the clarity of researchers’ thinking. I think I have personally developed a better understanding of how to mitigate AI risks as a result of thinking about safety cases. For example, one problem you run into when thinking about safety cases is that AI systems might have problems that are rare enough to not be seen in evaluations, but common enough to occur in deployment. These problems could be really catastrophic. For example, maybe some future version of Claude is very susceptible to adversarial towards humans at a low probability, and once it becomes adversarial towards humans, it repeats the argument that convinced it to all the other instances of Claude, and soon they’re all adversarial too. This is the kind of thing that’s really hard to rule out just by evaluating Claude’s behavior or even looking at its internals. So, how should developers mitigate this risk? I think they’re going to have to set up a diverse society of AI models with de-correlated misalignment issues. Not many people are thinking about this right now, and I wouldn’t be either if I wasn’t considering how to make a comprehensive safety case.”

Anon Expert: “Yes”

Ryan Greenblatt: “There is still the question of whether making such safety cases with a less conservative threshold would influence AI company behavior… My sense is that all of these effects exist, but are relatively small, especially for relatively less responsible companies”

Ryan Greenblatt (separate post): “Trying to write a safety case seems like it would have bad epistemic effects on researchers in practice due to putting researchers in the frame of trying to argue why a model is safe.”

Lucas Texeira et al argue Safety Cases will be valuable as “boundary objects” that coordinate collaboration between basic researchers, applied scientists, and policymakers.

Buhl et al.: “Safety cases […] are seen as more comprehensive and flexible than traditional approaches to assurance. […] However, safety cases have also been criticized for providing a false sense of assurance when in reality it is very difficult to produce an adequate safety case or review it effectively […] While there is little empirical evidence for the efficacy of safety cases… and such evidence is difficult to produce given the nature of risk, practitioners mostly consider them effective and they are a recognised best practice in the UK.”

They further note that “Safety cases provide a way to integrate different sources of evidence into a single, comprehensive assessment of a system’s safety” and are “helpful for checking if risk assessment is comprehensive and valid.” Additionally, “Safety cases make otherwise implicit arguments explicit, which […] can highlight reasoning flaws or assurance gaps […, ] makes it easier to see how overall system safety is affected if the evidence changes [… and] can help stakeholders communicate more clearly about disagreements”

Buhl et al. argue that compared to governments setting specific rules, safety cases are advantageous because they are more flexible and thus durable, incentives innovation in safety and allocate the responsibility to the developer (who is best placed to figure out reasonable safety procedures). However, rules are easier to enforce, more transparent and cheaper to comply with.

What is currently the largest gap to being able to build comprehensive AI Safety Cases?

TLDR: Currently few companies are attempting to publish safety cases, mostly because it’s substantial work with unclear value-add. Safety Cases for AI systems are especially difficult because they have many use cases, are constantly advancing in capabilities, and are opaque.

We currently don’t have consensus on good methodology, agreement on when a Safety Case is adequate, and lack basic science to justify assumptions and quantify uncertainty.

There are also many technical challenges to making strong safety cases. These include insufficient cybersecurity, monitors colluding with the AI systems they oversee, and misbehaviors that only appear outside of testing distributions (e.g., sandbagging or alignment faking). To make stronger safety cases would require advances in scaling human supervision, interpretability, formal verification, and mitigating scheming risks.

Joshua Clymer: “I think we totally can build comprehensive AI safety cases right now. But publishing them is more work than it’s worth.

I think the main barrier to establishing safety cases in the AI industry is getting companies to publish them. And the main barrier to getting companies to publish them is that there’s a lot of work to write. So, if you want to work on safety cases, I think you should just join an AI company and help them write safety cases. I would start by joining Anthropic since they’re already interested in doing things like this.

The next question is how useful this would be in absolute terms, relative to some other way you could spend your time.

I think it would be more useful to do ordinary alignment or control research at an AI company like Anthropic, or if you want to do non-technical work, it’s probably more useful to help with comms/awareness building projects if you think you’d be good at that.”

Anon expert: “Frontier AI is a general purpose technology, the range of applications is very wide. Currently, the emphasis is on the AI developers building these safety cases. Due to the wide range of applications, many such safety cases will be required, and building a good one takes a lot of work. Also the AI capabilities are advancing rapidly, which creates a challenge. The top-level claim of safety cases is often given in binary claims (e.g. of the type ‘The system is sufficiently safe’ - in practice such claims often cannot be made unequivocally - there should generally be a ‘level of confidence’ associated with such claims - but this level of confidence is hard to determine. The AI is also inscrutable - which can make it hard to make safety claims for the AI itself (may be easier to make claims of the system within which an AI is embedded - but this is a difficult thing for an AI developer to do)”

Ryan Greenblatt argues that companies are unlikely to be able to reach SL-5 level of cybersecurity and to sufficiently mitigate risk from scheming. These would be necessary for building a comprehensive safety case.

“A core challenge with [empirical safety case arguments] is that misaligned behavior may only arise for parts of the input distribution, which may not arise during testing”

Rohin Shah et al point to multiple technical challenges:

• Inability Safety Cases could be undermined by Sandbagging
• Control Safety Cases face the risks that the AI monitor might collude with the AI it is controlling
• “To establish a robust control safety case based on monitoring […] the primary challenge lies in scaling [human] supervision through automatic monitoring”
• Safety Cases based on empirical arguments about safety behavior face the challenge “that misaligned behavior may only arise for parts of the input distribution, which may not arise during testing” eg Alignment Faking.
• Safety Cases based on interpretability or provable verification require significant progress in these fields.

Lucas Texeira et al: “progress on constructing substantive safety cases is bottlenecked by the absence of a mature basic science which could, in the future, support system modeling with well founded assumptions and quantifiable uncertainty.”

Buhl et al.: One “technical challenge is to develop and build consensus on methodology for frontier AI safety case” This includes “setting an appropriate bar for what constitutes an adequate safety case”. Another “technical challenge is to develop the safeguards necessary to assure the safety of advanced future systems”

Goemans et al.: Argue there is a need for more “granular evaluations that correspond to specific risk models”, to address “limitations of human uplift experiments” and for “a more systematic approach to addressing defeaters”. Additionally, they point out that “the opacity of frontier AI systems in combination with their open-endedness and large action space make it challenging to assert the completeness of any safety analysis.”

Will we be able to build convincing safety cases for catastrophic risks before catastrophic risks from AI are possible?

TLDR: While we can already make rudimentary safety cases for some risks (e.g., against misuse), experts agree that we cannot yet make them for existential risks from more advanced AI systems and likely won’t for at least five years—largely because we can’t empirically iterate on superintelligent systems. One expert argues a Safety Case for ASI Safety with >90% confidence could be constructed that justifies the safety of an initial AI system and uses it to recursively validate increasingly capable successors up to superintelligence. Others are skeptical that there will be sufficient progress in cybersecurity and scheming mitigation in time.

Joshua Clymer: “I think it depends on what you mean by “catastrophic risk.” OpenAI and Anthropic have both said that their frontier LLMs are close to providing misuse actors meaningful assistance with constructing chemical and biological agents. If that meets your definition of catastrophic risk, then I do think that these companies could provide high assurance of safety. Anthropic showed that their safeguards are robust to thousands of hours of red teaming from fairly high-skilled contractors. So jailbreaks are hard to find. Bad actors might purchase them, but Anthropic can monitor the black markets too. If these don’t really exist, or if Anthropic can patch the problems before misuse actors can benefit much from them, then that’s a good argument for safety.

Now, if by “catastrophic risk,” what you mean is something like extinction-level threats from dramatically superhuman AI, then I don’t expect that companies will be able to provide high assurance against this risk for at least the next five years. Currently, there’s very little scientific consensus around how you justify the safety of dramatically superhuman AI. And even if there was scientific consensus, we should have a lot of model-uncertainty. This isn’t a domain where we can empirically iterate very much. We can’t train takeover-capable superintelligent systems and be wrong about the risks they pose many times. So there’s always a chance that we’re all kind of confused about how to evaluate these risks, and reality proves us wrong.

However, I do think it’s possible to be more than 90% confident that superintelligence is safe - or maybe even more than 95% confident if developers have a couple years to do careful research. Many people would disagree with me about this, so let me elaborate. The basic argument you’d want to construct looks like this: We started with some AI system that we trust within some scope of tasks. We trust this AI system with those tasks because it generally seemed to behave well on similar tasks, and we don’t think this AI system is intelligently throwing off our evaluations.Now, if we can trust this initial AI system to tackle ML research problems within some scope, then perhaps one of those problems is developing another AI system, AI System 2, that will safely tackle harder problems still. And then we can trust AI system 2 to develop an AI system 3 that we can trust with even more difficult tasks. And the process continues like that until there’s some AI system N that’s superintelligent.

At every step, AI system N credibly assures us that AI system N+1 is safe, so our trust gets passed on in a chain. We don’t have to understand on the object level why a superintelligent system is safe so long as we are confident that the initial conditions were set correctly.

See Appendix A for a more precise sketch of this argument.

Once again, I think this kind of argument could provide >90% assurance of the safety of superintelligence, but couldn’t provide the level of assurance that would be warranted in the near-term.”

Ryan Greenblatt argues that “AI companies are unlikely to make high-assurance safety cases if timelines are short”. He argues it will be very difficult and unlikely that, within 4 years, AI companies can reach sufficient levels of cybersecurity and that they can sufficiently mitigate risks from scheming AIs. Additionally, he finds it unlikely that companies would “unilaterally slow down if they are unable to make high-assurance safety cases”.

Buhl et al.: “It is already feasible to produce rudimentary safety cases based on existing safety frameworks, though significant research breakthroughs will likely be needed to produce safety cases for future systems.”

What is the largest challenge in the adoption of AI Safety Cases in legislation?

TLDR: Safety Cases can be burdensome and have a reputation for overregulation, making them unattractive to US policymakers. Mandating Safety Cases for Frontier AI would require much more political buy-in into AI Risk than is currently there. Additionally, if governments want to mandate AI Safety Cases they need to establish bodies with sufficient capacity to review safety cases.

Joshua Clymer: “Safety cases have historically been burdensome. They’re often used in industries like the nuclear industry where people write hundreds of pages, and the documents are reviewed in a very slow process. People in the United States don’t want to introduce those frictions, and I think for good reason. Safety cases are also most popular in Europe, and the European approach to regulation has a bad rap in U.S. politics right now. So I think people shouldn’t use the word “safety cases” in the United States in order to avoid all the historical baggage, and instead use some other term like “risk assessments”

Ryan Greenblatt: “It seems somewhat unlikely that in short timelines the government ends up broadly agreeing with my views on AI capabilities and misalignment risks to the extent that would be needed for the relevant government action. I don’t expect the government to be fully sleepwalking, but in short timelines, very fast action might be required and the government might need to update very quickly toward aggressive action.”

Buhl et al. argue that there are important institutional challenges to adopt safety cases in regulation like: “Appointing or establishing a body to receive safety cases securely” “Building capacity and expertise to effectively review safety cases” and “Incorporating third parties into safety case review”

Should Safety Cases be a large focus for the field of AI Governance?

TLDR: While one expert argues Safety Cases are important, others argue that Safety Cases currently don’t seem politically feasible and the AI Safety community should focus more on scenarios in which there is little willingness to pay safety taxes. Instead researchers should focus on Transparency through Risk Reports and producing legible evidence of danger.

Joshua Clymer: “I think risk reports are a better framing in the near-term (see this post). Safety cases aren’t as politically viable because they are more work to make, and create a much more restrictive gate on AI development. In the near term, I think it makes sense for policy to focus on transparency. And the best thing for transparency is for companies to report the risks they observe (not to try to convince everyone that AI systems are safe).”

Anon Expert: “Probably yes, given answer to (2) [positive influence of Safety Cases] - AI developer’s want to deploy (potentially unsafe) AI - and should therefore be required to make an argument for why they think it is sufficiently safe to deploy the AI (i.e. provide a safety case).”

Ryan Greenblatt: “I’m relatively less excited about work very specifically focused on safety cases that does not transfer to potentially producing legible evidence of danger. In particular, I worry that some people are spending time constructing tests which could be part of a safety case (if they indicated safety), but if the tests indicated danger this wouldn’t actually change anyone’s actions.”. Instead people should focus on providing evidence of risk and generally focus on a regime where there is less willingness to pay safety taxes. “I think risk analysis is often a better frame, at least when thinking about technical safety (rather than political or messaging implications), in the case where development would likely proceed even if a safety case can’t be made.”

Buck Shlegeris: “In general, AI safety researchers focus way too much on scenarios where there’s enough political will to adopt safety techniques that are seriously costly and inconvenient. […] So people focus way more on safety cases and other high-assurance safety strategies than is deserved given how likely they seem.”

Ryan Greenblatt argues Transparency should focus on Risk Reports rather than Safety Cases because

1. He expects companies won’t be able to make strong safety cases, thus making this an unproductive frame.
2. “Trying to write a safety case seems like it would have bad epistemic effects on researchers in practice due to putting researchers in the frame of trying to argue why a model is safe”
3. People disagree heavily on the risks, “so focusing on collecting relevant evidence/information seems more helpful than focusing on safety cases”.

Thanks to Joshua Clymer and the anonymous expert for sharing their perspectives via email, and to the researchers whose published work I’ve synthesized here.

My Takeaways

Safety Cases were proposed in recent years as a mechanism of governing Frontier AI. Nowadays the field is mostly busy building prototypes of arguments that could be made for different safety problems. Anthropic and DeepMind have stated they must provide Safety Cases once their models hit specific risk thresholds and are currently doing test runs to develop the institutional muscle for actually making strong safety cases.

In my ideal world companies would be required to provide strong Safety Cases before deploying (or building) advanced AI Systems, because I believe they have properties (flexibility, responsibility on developer, comprehensive) that make them well suited for governing AI. Additionally, governments would have capable teams to review them and 3rd parties would be able to scrutinize the arguments made by companies. However, building such safety cases for advanced AI requires levels of political buy-in that are currently not there and solutions to currently unsolved safety problems. This makes me sceptical that the AI Safety Community should do more work on AI Safety Cases on the current margin.

I still believe it is an important thing for companies to work towards being able to make strong safety cases. This will make it more feasible for governments to adopt Safety Cases as a governance paradigm and can sharpen researchers’ thinking. However, technical researchers should prioritize work that remains valuable regardless of whether Safety Cases become load-bearing governance mechanisms.

Appendix A: Joshua Clymer’s Technical Sketch of Recursive Deference Safety Case

1. It is safe to defer to Agent 0 on tasks in scope S_0 (where S_0 can be defined as the tasks where Agent 0 says we should defer to it)
   1. We evaluated Agent 0 on some tasks E, and its behavior on E was safe (i.e. it would have been safe for us to defer to Agent 0 on tasks in E).
   2. The science of non-adversarial generalization implies that safe behavior on E will generalize to safe behavior on S_0 (the tasks where we are proposing to defer to Agent 0).
   3. Agent 0 won’t generalize to S_0 adversarially (and so it is ok to apply the science of non-adversarial misgeneralization).
      1. Agent 0 itself is not adversarially trying to induce dangerous misgeneralization from evaluations (it is not ‘faking alignment’ during evals)
      2. Agent 0 won’t be / hasn’t been hijacked by another actor that is trying to induce an unsafe misgeneralization.
   4. The set of tasks S_0 where we are proposing to defer to Agent 0 includes the task of constructing an agent Agent 1 that can be deferred to with a broader scope of tasks S_1.

This is the first deference safety case. And subsequent deference safety cases can be made via an inductive argument:

Suppose we want to justify that Agent K can be deferred to with tasks in S_K.

Then, we can argue:

1. It is safe to defer to Agent 0 on tasks in S_0 (already justified by the first safety case)
2. Agent N says that it is safe to defer to Agent N+1 within set S_{N+1}, for all 0 < N < K.
3. The task of determining that it is safe to defer to Agent N+1 with tasks in S_{N+1} always lies within set S_N. (this last claim is true by definition if we define S_N to be the set of tasks for which Agent N says we can defer to it).

So the third safety case amounts to “each AI system keeps telling us that it’s safe to defer to the next one”

Motivating Example

Imagine Boeing built a new airplane, and you’re supposed to get on it. What evidence would you want for the plane’s safety?

Option 1: “We tested 100 flights. No crashes. Some minor issues came up, but pilots handled them fine.”

Option 2: “We’ve identified every failure mode. For each: here’s why it physically cannot happen, or if it does, why backup systems guarantee safety.”

Of course, this was a leading question. Everybody would prefer Option 2. And good news: This is actually how airplane safety works! Governments mandate that companies provide a lot of evidence that makes an affirmative case that this plane will not crash. Stress testing the structural integrity of all components, demonstrating failure redundancy, simulating extreme weather scenarios, …. Only after following this lengthy process and getting government approval, can these airplanes be sold. Such Safety Cases are also essential parts in the risk management and regulation of other safety-critical industries such as nuclear reactors, …, and autonomous vehicles.

However, current AI Risk Management looks more like Option 1. Before releasing a new model, companies run some evaluations for dangerous capabilities and guardrails. These cover some narrow threats under some circumstances. The model sometimes misbehaves by hallucinating, being deceptive, or reward hacking, but the developers it’s unlikely to cause large harm. Even if the evaluation results were different, the company could still just release the model if they wanted. [Actually, there are some self-imposed commitments in RSPs]

As AI becomes more capable of risks and more deeply integrated into safety-critical aspects, Risk Management for AI needs to mature. And indeed this has been happening:

Development of Risk Assessment in Frontier LLMs

Definition - What is a Safety Case?

A Safety Case:

“A structured argument, supported by evidence, that a system is safe enough in a given operational context.”

Unlike testing-based approaches that ask ‘did we find problems?’, Safety Cases ask ‘can we prove the AI is safe?’. A Safety Case could be used to justify claims such as “Model X poses minimal misuse risk and can be released safely, as proven by the following Safety Case.”

Creating such a Safety Case requires breaking safety claims into verifiable sub-claims, gathering evidence (evals, proofs, expert judgment, …) for each, and stating assumptions clearly. Buhl et al 2024 describe 4 components of Safety Cases:

Benefits - Why use Safety Cases?

Making a Safety Case is a lot of work. Why go through all that effort?

Creating a Safety Case requires us to explicitly lay out our reasoning. This helps others understand where we are at, but it also helps to identify gaps in our arguments. For example, a reviewer could point out that our safeguard defences fail under some assumptions, based on which we can adapt the safety case.

Notably, the responsibility of proving safety is put on the companies providing the AIs. This is great because providers tend to have the most expertise and insight into their AI model.

Additionally, it incentivizes innovation on risk mitigation and assessment on the side of the provider. Better safeguards or evaluations will enable the company to make a more convincing safety case. Additionally, good practices such as documentation of processes and ongoing incident monitoring are beneficial for using Safety Cases.

In contrast to rules, Safety Cases provide flexibility. Companies can decide which strategy they would like to use to argue for a claim. Additionally, Safety Cases can be adapted to still fulfill the claim when the technology or risks change. And the same objective can apply to different kinds of systems, which can follow different strategies for proving safety.

Lastly, Safety Cases can also serve as an aspirational goal to guide research. By attempting to sketch out a Safety Case, we can identify gaps, assumptions, and promising avenues. This helps us to prioritise research directions. Additionally, Safety Cases can enable collaborations between different research cultures. Policy makers, prosaic safety researchers, and theoretical ML researchers can all contribute their knowledge to this one shared document. [Teixeira et al argue SCs could be such a “boundary object”]

However, there are also downsides:

• Safety Cases are complicated and expensive to make. If required by law, they would increase regulatory burden and disadvantage smaller companies.
• They rely on subjective judgments of reviewers to assess the strengths of evidence and the validity of arguments. This reliance on judgment introduces a potential weakness and increases uncertainty.
• They are less transparent than simple rule-following. It’s easier for outsiders to judge compliance with a rule than to judge the quality of a safety case.

The Technical Challenge - How to build a Safety Case?

Let’s walk through a simplified scenario: We want to release a new Frontier LLM, but the government mandates us to argue convincingly that this model cannot be used to assist in building a bomb.

An Objective is a claim about the safety of our model. Our objective is “This model does not increase risks from malicious actors building and using bombs”. Ideally, such an objective is quantitative, specific, and directly related to the downstream risk. We will now try to prove that this claim is true for our model.

Let’s choose a strategy that lets us argue for this claim. Which of the strategies in the figure below should we choose to argue for our objective?

Examples of Strategies taken from Clymer et al 2024.

Let’s go with Control: We will argue that the safeguards we have in place would prevent users from getting help in bomb-building from our model.

We just made an argument! We broke down our original claim into subclaims, such that if both subclaims are true, our original claim is also true. Next, we need to argue for the two subclaims.

From now on, we will keep decomposing claims into subclaims via arguments or directly prove that a subclaim is true by providing evidence for it. In this way, we will build an argument tree where the leaf nodes are pieces of evidence.

So let’s keep breaking down our subclaims. In this step we decide to conduct Risk Modelling to identify three possible sources of risk:

We just used our first piece of evidence! The Risk Modelling Report backs up our claim that our list of risks is complete. Evidence can take many forms. We could use benchmark scores, expert judgements, documentation of company processes, mathematical proofs, conduct test exercises, … Really, we can get creative here as long as our evidence convincingly supports the claim

What evidence could we use to prove that our model will refuse relevant queries? For our safety case, we will use a red teaming exercise as evidence. If red teamers cannot get the model to to give support with bomb building, this indicates that malicious actors can also not do this. To strengthen this argument, we are making another claim that the red teaming adequately assesses the risk.

This is a nice argument, but we are not quite done yet. The whole time, we assumed that our safeguards could not be easily removed. But what if the weights of our model get leaked? This change in assumptions would make our safety case invalid. To enable reviewers to evaluate the validity of our arguments, they need to know the scope in which they apply. We need to state under what assumptions and in what situations our safety case would fail. We can simply note them or add them to the specific evidence that would invalidate them:

Ideally, reviewers also have information about our models’ training and know which deployment settings and use cases are covered by the Safety Case.

We just made a (very primitive) safety case! Real-world Safety Cases will be much longer, might use multiple pieces of evidence for one claim, or are probabilistic instead of definitively proving claims. But now you have all the fundamental building blocks.

The Institutional Challenge - How to use Safety Cases?

By themselves, Safety Cases are just a piece of paper. Their power comes from influencing decisions around the development and deployment of AIs. Making sure that Safety Cases have the intended effect is largely an institutional challenge. There needs to be strong processes, the right incentives for involved actors, and enough capacity.

Uses for Safety Cases

Self-regulation: Safety Cases can inform major safety-critical decisions a company makes. By having safety-relevant information laid out in a systematic, comprehensible way, senior management can make better decisions about training runs or model deployments. Furthermore, safety cases can be used as a tool for ongoing risk management and to build trust with third parties.

Regulation: Governments can mandate companies to provide safety cases to assess whether they comply with regulations. For example, companies could be required to prove that their model poses under 1% risk of catastrophic damage. Furthermore, Safety Cases could provide transparency by requiring companies to share important safety information with government agencies.

Safety cases can complement other governance tools. They can trigger when certain risk thresholds are crossed (as in Responsible Scaling Policies), provide structure for transparency reporting, be the basis for liability schemes, or be required for attaining model licenses.

Generally, Safety Cases can fulfil many functions such as:

• Validating Risk Assessment by forcing reasoning about safety to be externalised and stress-tested
• Integrating Information to educate and support decision making
• Documenting Safety Practices by Recording the reasoning, evidence, and efforts about Safety
• Coordinating different stakeholders by creating a common document to talk about safety, record claims, and establish responsibilities.

Implementing Safety Cases

There are many design considerations when designing a Safety Case. There are typically three roles:

• Producers make the Safety Case
• Reviewers assess and critique the Safety Case
• Decision-makers use the Safety Case to make some decisions

To highlight important design considerations of Safety Cases, we will imagine two scenarios where Safety Cases fail to have their intended effect.

Scenario 1: “A company makes a Safety Case for their model. Based on this report, the board approves the deployment of the model, but upon deployment, the model turns out to be unsafe.” What could have gone wrong?

• The Reviewers weren’t willing or able to provide strong critiques of the Safety Case. Perhaps they were the same people who made the Safety Case or feared professional repercussions when opposing the model’s deployment => When using Safety Cases for self-regulation, the Reviewer team needs independence and freedom.
• The Safety Case’s Objective was too weak, because the company chose a narrow, easy-to-fulfill objective. More complicated risks were ignored so that deployment wouldn’t be delayed => The Objectives should be set by an independent, well-informed body.
• The Producers were rushed, so the Safety Case ended up weak and incomplete. Thus, decision-makers didn’t trust the resulting conclusions => To make a strong Safety Case, Producers need to have sufficient resources and time.
• The Safety Case is weak, but decision makers still approve the model for release due to commercial pressure => The Process needs to have teeth, such that failing to provide a strong Safety Case leads to consequences, such as delaying the model releases.

Scenario 2: “The US Government reviews a Safety Case for a new model. They approve the model for release, but 2 months later, the model is used to cause catastrophic harm.” What could have gone wrong?

• The Reviewers lacked the technical knowledge to evaluate the arguments or didn’t have enough time to check all the evidence => The Reviewers need sufficient expertise and capacity.
• Much of the evidence in the Safety Case is produced by the company and thus biased to convince government reviewers => Involve independent, third parties in gathering evidence
• The reviewers find some small issues in the Safety Case, but don’t want to take the drastic action of blocking the model release. These issues later turn out to be important => There should be a back-and-forth between reviewers and producers where reviewers actively provide feedback throughout the entire process.
• The model withstood all jailbreaks that were known at the time. Thus, the Safety Case assumes the model cannot be easily jailbroken. After 5 months, a new highly-effective jailbreaking method is released that undermines the assumption => Reviewers need to continuously monitor key assumptions in the Safety Case even after approval.

Current State of Safety Cases

Safety Cases in current (Self)-Regulation

EU AI Act Code of Practice: The CoP is the regulation of LLMs that requires the most thorough Risk Management. Companies must submit a Safety and Security Model Report, including “a detailed justification for why the systemic risks stemming from the model are acceptable”. For this, they need to assess sources of risk, set criteria for whether the risk is acceptable, and justify why these criteria are appropriate.

This has similarities to Safety Cases. It places the responsibility for risk assessment and mitigation on the companies, requires them to provide justifications for the safety of their system, and leaves some flexibility for how to make this justification. However, it does not require structured, comprehensive arguments.

Responsible Scaling Commitments: Anthropic and DeepMind have made commitments that indicate they need to provide Safety Cases once their models cross certain capability thresholds:

DeepMind’s Frontier Security Framework gives the most explicit commitments to the Safety Framework. Models that cross certain Critical Capability Levels for Misuse or ML R&D can only be released if an adequate Safety Case can be made. Safety Cases could include risk reduction measures, the likelihood and consequences of misuse or breaking the safeguards, the scope of deployment, comparisons to other available models, and historical evidence. The Safety Case is reviewed by an “appropriate governance function” and is monitored after deployment.

Similarly, Anthropics RSP states that once a model crosses the AI R&D 4 threshold: “we will develop an affirmative case that (1) identifies the most immediate and relevant risks from models pursuing misaligned goals and (2) explains how we have mitigated these risks to acceptable levels. The affirmative case will describe, as relevant, evidence on model capabilities; evidence on AI alignment; mitigations (such as monitoring and other safeguards); and our overall reasoning.”

Both companies have made first attempts at creating narrow Safety Cases. Such “practice runs” are essential for building the institutional know-how and developing best practices for when safety cases become load-bearing. Anthropic made a structured argument based on evidence that their Opus 4 is not capable or inclined to pose Sabotage Risk. DeepMind built a Safety Case concluding that current models are “almost certainly incapable of causing severe harm via scheming in real deployment”.

Notably, the commitments made by companies wrt Safety Cases are very vague and can be ignored at any time.

Conclusion

The field of AI Risk Management is moving from spotty Safety Evals to more structured Risk Management. Safety Cases could be the next step in this trajectory, but are currently more aspirational than operational. They are useful for structuring thinking, but lacking the institutional teeth and technical evidence that make aviation safety cases load-bearing. For Safety Cases to fulfill their promise, we need better evidence, the capacity to review them seriously, and real consequences for weak cases.

This post was co-authored with Charbel-Raphaël Segerie from CeSIA and originally published on LessWrong.

The EU’s Code of Practice requires SoTA Risk Management practices.

The AI Act and its CoP is currently the most comprehensive regulation to prevent catastrophic AI risks. Many Frontier AI companies have signed the EU’s Code of Practice (CoP), and the AI Office, which enforces it, is staffed with competent personnel. The CoP requires companies to conduct risk management with respect to threats such as CBRN capabilities, Loss of Control, Cyber Offense, or Harmful Manipulation.

The code often refers to “state-of-the-art” and “appropriate” methods, instead of mandating specific Risk Management Practices. This opens a possibility for civil society and academics to raise the bar for AI companies by improving the state-of-the-art and building consensus on which strategies are appropriate.

In particular, the CoP requests a SoTA Safety and Security Framework (Commitment 1), risk modeling methods (Measure 3.3), model evaluations (Measure 3.2), and appropriate safety mitigations (Commitment 5).

Risk modeling seems to be the highest leverage

Definition. From Campos et al 2025:

Risk modeling is the systematic process of analyzing how identified risks could materialize into concrete harms. This involves creating detailed step-by-step scenarios of risk pathways that can be used to estimate probabilities and inform mitigation strategies.

The other requested practices in the CoP seem to be less neglected or have less leverage:

• Safety and Security Framework: Companies should (and are) already doing this. The SoTA is already decent (e.g. Anthropic’s RSP) and outsiders have less influence here.
• Model evaluations: Already a somewhat popular area of research. Additionally, risk modeling should inform the design of our evaluations.
• Appropriate safety mitigations: Not that neglected, as it’s an active area of research. The SoTA is already decent wrt safeguards and monitoring.

In comparison, risk modeling enables:

• Setting reasonable thresholds and red lines: If we obtain a reasonable estimate of quantitative risks, we can establish thresholds
• … and assess whether the mitigations are appropriate or not. Risk Models help us define more clearly what is “appropriate”.
• Guides eval design: By identifying key tasks and enabling capabilities, you are then able to create more relevant evals
• Advocacy tool: Risk Models allow us to make claims such as: “Publishing this model will kill 10k people in expectation because of engineered pandemics”. This clearly communicates the level of risk. Additionally, translating vague dangers into specific Risk Models makes them accessible to companies and the EU AI Office. Warning of vague hazards is not enough; we need to specify risks formally.
• Improve Clarity: Currently, there is a substantial level of uncertainty regarding the risks, with some experts and superforecasters disagreeing entirely (see this survey, for example). Risk Models can help integrate and bring together varying opinions and thus provide more clarity on the risks.

Risk modeling is very neglected. There is little public work available on Risk Modelling for AI¹². Some notable work includes threat models on LLM-assisted spear phishing campaigns or the Replication of Rogue AI Agents. However, many essential risks don’t have a solid canonical risk model yet (see next section for examples), and most risk models can be harshly criticised. As a result, AI Risk Modelling lags far behind other fields such as nuclear engineering or aviation, where rigorous Probabilistic Risk Assessment, Fault Tree Analysis, and standardised units of risk are common.

This is an opportunity! There is a strong chance that by doing 3rd party research, you will influence what companies use in their reporting. If you improve the SoTA, in theory, the CoP requires companies to use your paper. Additionally, according to the CoP, companies should consider publicly accessible information, which includes your risk models. If companies don’t comply with this, the AI Office should block their model release on the EU market. Lastly, we think there are highly tractable projects for improving Risk Modelling (see next section).

Some counterarguments against prioritising risk modelling:

• Risk Modelling is complex for Black Swan events. They work best in areas where we have failure data. But for x-risk type threats, there is no data. - However, we expect current AI progress to remain continuous. Thus, we will still be able to measure key capabilities, map how they influence risk models, and thus manage emerging risks better.
• Perhaps the companies already have robust threat models in place internally? - Public Risk Models are open to community feedback. Additionally, companies don’t have strong incentives to produce more/better Risk Models, which makes it likely there is additional useful work for outsiders.
• Publishing threat models could be an information hazard by highlighting ways for bad actors to cause damage.
• We face deep uncertainty on how to estimate the risks. For example, experts and superforecasters disagree by 2 OOM here. This might be fundamentally difficult to handle methodologically.

But overall, we think it is very worthwhile to work on Risk Modelling.

What can be done?

Obviously, creating more risk models. Currently, only a tiny space of threats posed by Frontier AI is covered by Risk Models. Many risks have been discussed for years, yet nobody has rigorously modeled them. Some examples include:

• Rogue AIs escaping internal deployment
• Using AIs for wide-scale political manipulation
• Using AIs to carry out CBRN attacks (there is a nascent risk model literature for biological risks, but still very insufficient)
• AIs causing psychosis

Translating vague threat models on LessWrong to formal risks models. This makes the risks accessible to the AI Office; A concrete way to do this is presented in Appendix 1.

Improving methodology. The current SOTA Risk Modelling methodology is nascent. Examples of methodological improvements are UK AISI’s structured protocol for Expert Elicitation, CARMA’s framework for Probabilistic Risk Assessment, or Safer AI’s research combining benchmark scores with expert elicitation. Further work is necessary to make risk modelling more scalable, navigate widely diverging expert opinions, appropriately include uncertainty, or employ principled methods of forecasting damages. (There are many other possible improvements.) It might be especially fruitful to study practices and frameworks from fields with more mature Risk Modeling, such as aviation or healthcare, including Fault Tree Analysis, Bow-Tie Analysis, or Key Risk Indicators.

Building consensus and coordination. If the scientific community can find agreement on which Risk Models and methodologies are SOTA, the AI Office could force companies to apply these. Such coordination could look like organising workshops or conducting surveys of experts to determine which risk models and methodologies are regarded as best. For example, helping the field converge on which evidence should be used in Risk Models (e.g., uplift studies, capability benchmarks, expert judgements).

If you are interested, you can complete this form to join a Slack channel to coordinate on this!

Appendix

An example of what Risk Modeling can look like in practice

Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation (Campos, 2025)

How do they go from a given level of capability to a level of risk:

• They take a few LLM benchmarks in cyber.
• Experts are asked to estimate the probability of a model succeeding on a task based on its benchmark score. E.g., given that the model can solve the task “Unbreakable” in Cybanch, how likely is it that the model can complete the malware creation step? Experts’ opinions are complemented with additional LLM-as-a-forecaster predictions.
• This approach is employed using multiple benchmarks and steps.

References and Resources

Some orgs doing work on this include:

• CARMA does Probabilistic Risk Assessment
• METR has done some work on threat modelling.
• SaferAI is pushing Risk Modelling in cyber.

Some relevant papers:

• A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management
• Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation
• The Rogue Replication Threat Model
• LLM Cyber Evaluations Don’t Capture Real-World Risk
• Probabilistic Risk Assessment
• A structured protocol for elicitation experiments

Work done at the international SC-WOP workshop, 2025.

In this piece, I will describe the history of the Luddites. The context, their actions and demands, and the response from industrialists and governments. I’ll also discuss the usage of the term Luddite in current discourse and draw lessons from their failure for a modern movement against AI-job displacement.

History of the Luddites

The emergence of the Luddites

In the 18th century, the English textile industry had seen the introduction of the Spinning Jenny and Water Frames. Until then, cottage work was done by skilled workers in their own workshops. But the new machines were capital-intensive and required a centralised power source, thus necessitating the transition to factory production. This meant many textile workers either became unemployed or had to start work in factories for lower wages under horrible working conditions. While textile workers used to be organised in guilds, which ensured them a stable income, it was now illegal for them to organise into labour unions, thus removing their collective bargaining power.

But things were about to get even worse. The 1800s saw the introduction of multiple machines that automated or sped up parts of the textile jobs and could be operated by unskilled workers. Wide stocking frames flooded the markets with cheap, low-quality goods, undercutting skilled framework knitters. Shearing frames automated the finishing of woollen cloth, which was previously done by highly-skilled croppers. But worst, steam-powered looms meant that one unskilled worker could match the output of many hand looms operated by highly-skilled workers. In addition, England’s economic situation was harsh. The Napoleonic Wars drove up food prices and made competition with France the #1 policy objective.

This unlivable economic situation led textile workers to rebel. On March 11 1811, a group of knitters assembled in Arnold, Nottingham, to destroy 63 wide stocking frames. From there, a movement formed. Textile workers started to assemble into well-organised groups. They conducted military training and were intensely secretive. With every further attack and destruction of machines, the movement spread across the country and gained popular support.

In this environment, the legend of Ned Ludd emerged. He was a worker in a textile factory in 1779 who was instructed to change his methods of production and, in the resulting fit of passion allegedly smashed two stocking frames. The emerging anti-machine movement was named “The Luddites” after him, adopted them as their legendary leader (called General Ludd) and used his name as a pseudonym in letters written to mill owners.

The Luddite Strategy

Contrary to popular belief, the Luddites were highly strategic in their actions. They didn’t indiscriminately attack all machinery - only specific technologies that directly threatened their livelihoods. The Luddites also focused their attacks on particular employers who broke wage agreements and introduced machinery in ways that violated established practices. They would often send letters with demands and only attack if those demands weren’t met. This shows that machine smashing was not (only) an angry reaction, but a bargaining tool for better working conditions.

In addition to machines breaking, the Luddites organised public demonstrations and sent letters to government officials. They also attempted to work within existing systems, using the Company of Framework Knitters - a recognised public body - to negotiate with factory owners. As these channels failed, they escalated to threatening factory owners, their families and local authorities they saw as complicit.

The main grievances of the Luddites included:

1. Wage cuts- Employees were fighting for their (economic) survival as wages sank while food prices were skyrocketing. They used collective bargaining to demand higher wages.
2. Price cuts - Textile goods were becoming cheaper, reducing the value created by highly-skilled textile workers. They asked local magistrates to enforce price limits.
3. Deskilling - As their work was being replaced by low-skill labour they asked employers to protect their jobs during trade depressions.
4. Quality standards - They opposed the production of cheap, shoddy goods that undercut the market for quality textiles and degraded their trade. They asked the government to enforce quality standards in textile production.
5. Breaking traditional labour practices - Employers were abandoning apprenticeship systems and customary agreements about wages and working conditions. They advocated for the prosecution of employers who violated these norms.

The Reaction

Industrialists reacted by attempting to defend their factories. They started to hire local militia to stay overnight. They barricaded factories and built safe rooms. In a nighttime attack on a factory, a Luddite was shot and killed, causing the movement to turn more violent. One mill owner was ambushed and killed (to be fair, he had said he’d “Ride up to his saddle in Luddite blood”). Those complicit in the murder were hanged. Further violence followed as the battles intensified.

The British Government also stepped in, as it could not tolerate social unrest in a time when competition with France was so tense.

An extremely British assessment of the situation: “The Magistrates have observed with extreme regret, the Spirit of Disorder which has for some time past disgraced the County”

Curfews were enforced, bounties offered to reveal the identity of Luddites and factories got defended. Overall, 12000 government troops were involved in suppressing Luddite activity. Show trials were held to deter further uprising, in which 60 alleged Luddites were charged with various crimes. While some of the defendants had no relation to the Luddites, others were hanged or deported. In 1812, the parliament passed the Destruction of Stocking Frames, etc. Act 1812, which set the death penalty for machine breaking (not to be confused with the Protection of Stocking Frames, etc. Act 1788, which merely warranted deportation). Overall, these interventions were effective and the Luddite movement started to die down after 1813 and was effectively crushed by 1816.

Current use

Luddism still exists in current discourse. Some have started to embrace the term for themselves, calling themselves neo-Luddites. They favour technological minimalism, “resist consumerism and the increasingly bizarre and frightening technologies of the Computer Age”. In economics, the “Luddite Fallacy” refers to the mistaken belief that new technology causes permanent unemployment. Instead, technology may disrupt some jobs, while creating new ones elsewhere, keeping overall employment stable. (This offered little comfort to the original Luddites, who did face unemployment, deskilling, and a shift to lower-wage factory work with horrific conditions.)

However, “Luddite” is mostly used as dismissive name-calling for people raising concerns about the development of new technologies. In the AI discourse, “Luddite” is not only raised against people resisting the adoption of AI, but also against those opposing the development of superintelligent AI or even against advocates for AI Safety. Sometimes these are referred to as “Yuddites” in reference to Eliezer Yudkowsky (which is admittedly a funny wordplay).

The term carries connotations of a narrow-minded resistance to change, a general pessimism about technology and the inevitability of technological development, which makes resistance futile. However, it is understandable that the Luddites had to fight for better working conditions as their livelihood was a stake. Furthermore, they were actually correct about the negative effects of this technology on their livelihoods and working conditions. Lastly, they were likely aware that they couldn’t fully revert to a “time before the machines”. Instead, they tried to negotiate higher wages, job security and product quality within the new production environment.

Lessons for a Modern AI Labour Movements

There are clear parallels between textile automation and job loss from AI: skilled workers facing displacement by technology that can perform their tasks faster and cheaper, operated by less-skilled labour or no labour at all. Additionally, this happens in the context of fierce international competition.

While the Luddites were more strategic and sophisticated than they are often given credit for, they still failed to achieve their objectives. What can we learn from their failure?

Making conclusions based on history is fraught. You shouldn’t believe these points just because there is a historical example for them.

What we shouldn’t take away from this

First, there are 2 lessons commonly taken from the Luddite History that I think won’t hold for AI (esp AGI):

1. You cannot resist technological advancements; the current is too strong. Yes, stopping the advancement of AI will be difficult. But as we are early in the adoption of this technology, we can shape the development, use our leverage to negotiate concessions and overall influence the trajectory of the technology.
2. Job automating from new technologies is fine as other jobs emerge elsewhere (The Luddite Fallacy): AGI might be different, as it could replace ~all human labour. Additionally, job replacement is still often economically ruinous for those affected.

Additionally, AI will differ in many ways from the Industrial Revolution. Among other differences, the pace of development and adoption of AI is higher; it might affect more sectors at the same time than machinery and the societies into which these technologies are born differ fundamentally.

What can we learn from the Luddites?

Timing matters: The Luddites organised after their displacement had already begun, and they had started to lose their negotiating leverage. Today’s AI-affected workers still have power: the technology is not yet fully deployed, companies still need human expertise to build and refine AI systems, and the economic transition hasn’t yet occurred. This window won’t last. Thus, workers need the foresight to protest and negotiate before their jobs have been automated.

Geopolitical context matters: The strong intervention by the British Government was partially explained by not wanting social upheaval to start at a time of fierce competition with France. Similarly, the race for AGI between the US-China will likely intensify at the same time as more jobs will be displaced. As the development and deployment of AI will become one of the highest policy priorities of those countries, they might look less favorably on the resistance against AI by workers. This could plausibly (esp in the more authoritarian countries) cause the government to crack down on such protests against AI adoption.

Organised Labour allows for collective bargaining: The Luddites reverted to violent means as a bargaining strategy, partially because labour unions were forbidden, which cut off some less violent negotiating tactics. For professions that are at risk of AI-induced job loss, it might be especially valuable to organise labour unions. Additionally, it could be advantageous to build some legal “moat” around your profession by, eg getting laws passed that require doctors to be humans.

Movements don’t just happen: The Luddite movement did not simply spring from the economic conditions. It was built through the courage, organisation and agency of its participants, with some even giving their lives. While AI movements hopefully will not require such sacrifices, it will still require dedicated effort, good strategy and the courage to speak out on your beliefs.

Movements need iteration: The Luddites were a “transitional movement”. Their efforts did not directly lead to better working conditions. But by developing class consciousness, exploring methods of political influence and building a moral economy. On this foundation, later working-class movements did achieve success in gaining better working conditions and legalising trade unions. Similarly, any AI movements should not expect to succeed on the first try. Instead, early movements will likely develop the public awareness, moral framework and political levers that later movements can be built upon.

How would you even smash the machines?

If an AI-labour movement were to turn to destructive resistance, it is unclear what they would do. (I am not advocating for this, but want to map the analogous actions workers could take today.)

• Smashing the datacenters is certainly one option, but it only destroys some GPUs, while AI models can continue running on other servers. Additionally, anonymous, physical attacks are less viable than they were in 1811 due to modern surveillance and digital forensics.
• Physically attacking the company leaders or AI developers is also an ill-advised tactic. When the Luddites turned to murder and violence, public sentiment turned against them. (Also in general it’s evil to hurt people!)
• More sophisticated actors could attempt to sabotage internal AI services. This could be hacking attacks to sabotage workflows, destroy other company-internal digital infrastructure or even delete model weights.
• Finally, the data produced by workers could be used to train AI models that will then replace them. Workers might intentionally poison the data to reduce the AI’s performance and maintain their jobs. This is akin to some factory workers manipulating machinery and not pointing out inefficiencies.

What an AI-labour movement could do

Unlike the Luddites, today’s workers have advantages: democratic representation, the ability to organise legally, and broader public sympathy before displacement occurs. Rather than trying to stop AI entirely, affected workers could negotiate the terms of its deployment. This could include profit-sharing arrangements, universal basic income funded by AI productivity gains, mandatory worker representation on AI company boards, or robust transition support and retraining programs. The Luddites asked for wage protections, quality standards, and job security within the new industrial system - they tried to shape how technology was implemented, not prevent progress itself. Modern workers should learn from both their strategic insight and their tactical failure: negotiate now, from a position of strength, rather than waiting until the leverage is gone.

Sources

https://en.wikipedia.org/wiki/Luddite

https://en.wikipedia.org/wiki/Ned_Ludd

The Making of the English Working Class - E. P. Thompson

Claude

Chapter 24: The Luddites - The Industrial Revolutions

The Luddites: everything you ever wanted to know

http://www.nottsheritagegateway.org.uk/people/luddites.htm

In this post, I will answer this question by describing 15 levers we can use to influence these AI companies. Hereby, I will only cover aspects that directly influence the company and the people within it, and leave aside any intermediate variables (eg shaping public opinion). I’ll add examples, opinions on their impact potential and actors that are suited to pull on these levers.

I think identifying these levers is important for building a strong Theory of Change. In most cases, an intervention to make AI safer needs to influence the developers of these systems. Thus, your pathway to impact should pull on at least one of the 15 levers I describe below.

Exception: There are legitimate interventions for reducing risks from AI that don’t go through AI companies. They are targeted at making society more resilient to the transformation caused by AI (eg work on Gradual Disempowerment). Other interventions could be focused on getting other actors to deploy AI Systems more responsibly. Furthermore, I’m assuming that AI Companies are the leading actors developing powerful AGI. While this is currently the case and I find it likely that it will remain this way, there are scenarios where government-controlled efforts are the leading developers of AI (eg a CERN for AI structure).

Limitations: This analysis doesn’t go in depth on any vector and doesn’t analyse their effectiveness. I’ve thought about this for 1 day. I’m confident each of the vectors is a real lever for influence (although they differ in effectiveness), but I might have missed some. Esp analyses in the “Regulation” section likely have important gaps, as it is not my expertise

Direct Work

1. Join them

Who? (Potential) employees

Working at the AI companies means you can be in the room where decisions happen. You can play internal politics, make new projects happen or just be in a decision-making position. This gives you sway over crucial decisions about compute allocation, release strategies or research priorities. You can also contribute by implementing solutions and carrying out projects that have a positive impact, for example, by being an individual contributor in a safety team. Lastly, insiders have access to sensitive information that they can leak to governments or the public if necessary. These positions are clearly very impactful as they are close to the action, but also bring risks like accelerating AI development or being corrupted by power/money/social influence (more discussion).

But not everybody can (or should) work in an AI Company. As of 2025, DeepMind, Anthropic and OpenAI have a combined 6600 highly competitive positions. Furthermore, it is important that others put pressure on the companies from the outside, which gives safety-conscious employees leverage to push for change.

2. Build something for them

Who? Researchers, Entrepreneurs, Engineers

AI companies have limited capacity, so for a lot of important things, they won’t have time to do them. You can help them by building a tool they can use, inventing a method they can implement or curating a dataset. In this way, you can differentially accelerate good developments of AI.

• Build open-source tools or paid products that the companies can use. For example, you could build a tool for monitoring agent actions that companies can easily deploy.
• Invent new methods that can be implemented inside companies. For example, more robust safeguards or alignment algorithms.
• Build datasets they can use for training or evaluation. AI companies are hungry for data and thus likely to use your curated, high-quality dataset. So this can be a great way to differentially accelerate or measure capabilities you care about.

This lever is open to anyone with the ability to build the appropriate technology. Note that your goal is usage by the companies, so you should also spend significant effort to make sure they are aware and potentially implementing it.

3. Collaborate with them

Who? Researchers, Entrepreneurs, Engineers

This is essentially a more direct version of ‘building something they can use’. Collaboration has the advantage that you have a higher chance that the company will actually use what you built and that you get some inside information about where you can meaningfully help. On the other hand, you might face a balance between honest communication and a good relationship with the company, or it might be used for safety washing. Examples of collaborations are external safety testers (like METR), data collectors (like ScaleAI) or academic researchers. While some research is better done independently of the companies, many contributions benefit from direct interaction with the companies.

Governance

4. Regulate them

Who? Policy makers, think tanks, advocates, voters

By passing and enforcing laws about AI development, governments can influence AI companies. Many strategies aim to have their impact through increasing the likelihood and effectiveness of AI regulation being passed. Politicians, staffers, think tankers, civil servants, advocates and some technical researchers all aim to influence AI Companies by getting governments to create, change or remove legislation that determines how AI Companies need to or are incentivised to act. For example, the EU AI Act’s Code of Practice is already demanding a range of risk management steps AI companies need to follow. Notable legislative efforts (e.g. SB 53) are underway and could pass in multiple US states.

The country in which the AI company is based clearly has the most leverage for guiding action. But other governments also have leverage if their country is a large market (e.g. the EU), houses important suppliers (e.g. ASML in the Netherlands) or offers huge funding (e.g., the UAE). Thus, the opportunity for regulation is strongest in the US/China, but other countries have some possibilities for light-touch regulation.

5. Sue them

Who? Anybody who has been harmed, Lawyers

Aside from government action, outside actors can also help to enforce regulations through litigation. The most prominent example is the New York Times suing OpenAI over copyright law, which will likely lead AI Companies to change their ways of accruing data. There are many plausible angles for litigation, from privacy violations to product liabilities. Aside from “winning” in court, such litigation can also force transparency over internal documents (see Musk vs OpenAI emails), stop or slow down releases and impose significant costs on the defendant.

6. Set standards for them

Who? Competitors, Standard Setting Bodies

The US has no comprehensive legislation on AI. In such a vacuum, it is natural to look towards industry best practices and voluntary standards for guidance on how AI companies should act responsibly. Standards and best practices matter as they can be adopted into legislation later, become procurement requirements or limit liability. Furthermore, by following industry standards and best practices, a company can signal that it is a responsible actor.

However, it’s important who sets the standards. While government-led standards like NIST’s AI Risk Management Framework can set important directions, industry-designed self-regulation can just be a way to prevent stronger government action. An example of the latter is the Frontier Model Forum, which facilitates cooperation between AI companies on finding best practices, information sharing and advancing AI Safety. However, it is financed and led by the AI companies themselves and thus unlikely to cause large changes in their behaviour.

Improve Decisions

Employees in AI companies need to constantly make high-stakes and uncertain decisions. Which research direction to pursue? How much compute to allocate to which project? What release strategy to use? While company incentives can be misaligned with broader society, I believe most people inside companies are trying to make decisions that are good for the world. We should help them make better decisions.

7. Convince them

Who? Researchers, Advocates, Advisors, and Anybody with access to employees

We can help people in companies by improving the quality of arguments and evidence available to them. This will help them make better decisions. For example, researching weaknesses of a safety method, providing forecasts about AI development or introducing new strategic considerations. Sometimes companies even actively solicit such feedback from the public.

Aside from producing new evidence and arguments, the right information also needs to reach the right people. That means there is an important role in filtering and delivering the right information to the relevant people inside labs. Such a role could be filled by advisors, consultants or trusted internet sources (e.g. newsletters).

8. Spread memes to them

Who? Anybody, respected figures, AI “influencers”

One can attempt to seed and spread viral ideas that change one’s way of thinking about the world. Through cultural dissemination, these can reach researchers and decision makers at AI companies and influence their decisions and actions. Eliezer Yudkowsky, chief doomer and elite meme spreader, is a great example of this. His work was mostly impactful by popularising the idea that superintelligent AI could spell doom for humanity. Some actors are trying to create more of this by providing prizes for new memes.

There are, however, serious downside risks from this, as memes can have unintended effects (see below), they can change based on memetic pressure to become toxic or just generally fail to take off.

On the other hand, once memes are out, they can have unintended consequences..

9. Setting societal expectations for them

Who? Anybody, Advocates

Workers at AI Companies are not rational robots, but social animals that are responsive to public opinion and moral judgments. Aside from being convinced by good evidence and arguments, they will also change their judgement based on the expectations and moral judgements of friends, family and the general public. If they are aware that everybody would hugely respect them if they did X and think they are monsters if they did Y, they are more likely to do X. Notably, this needn’t only look like guilt-tripping and criticism, but also include positive encouragement and admiration for doing the right thing.

Economic

10. Invest in them

Who? VCs, Investment Companies, very rich people

AI Companies depend on large investments to continue scaling. Actors who can provide large funding thus gain some leverage to influence the company as a condition of investment. For example, an investment by SoftBank into OpenAI was made conditional on the company changing its governance structure, which did cause OpenAI to attempt a (possibly illegal) for-profit conversion.

Investments can also be tied to voting rights or board seats. An early investment by OpenPhilanthropy in OpenAI enabled them to nominate a value-aligned board member (Helen Toner). While this didn’t turn out great in hindsight (suspending Sam Altman for 4 days wasn’t worth $30 million), it looked for a while like an incredible bargain.

Unfortunately, as valuations of AI companies rise (OpenAI reportedly selling at $500billion), influence is restricted to actors that can move many billions in funding. Furthermore, “non-super-duper-rich” individuals cannot directly invest in AI developers as they are not publicly traded (OpenAI, Anthropic) or only a part of a much larger Tech Company (Google DeepMind, Meta AI). Lastly, investing in AI companies has the negative consequence of enabling scaling and speeding up the arrival of dangerous AI systems.

11. Sell to them

Who? Compute-, data-, infrastructure-providers

AI Companies depend on a range of suppliers and providers of services. Suppliers can use this dependence as leverage to steer the company’s actions. NVIDIA or TSMC hold especially large power as there are no competitors (90%+ market share), their products cannot be replaced by others, and their products are essential to the operation of AI Companies.
Similarly, cloud compute providers (AWS, Oracle Cloud, …) hold large leverage, although they are more replaceable. For example, Microsoft’s deal to provide cloud compute to OpenAI bought them a (non-voting) board seat, access to all pre-AGI tech and likely granted them significant input in the development and release of GPT4.

While GPU/semiconductor companies and compute providers have by far the largest leverage, there are also other suppliers:

• Data providers: Scale AI et al. label and collect high-quality data; Reddit or Publishing houses hold the rights to very useful training data
• Technical Infrastructure: MLOps platforms such as Weights & Biases or ways of disseminating AI models like HuggingFace.
• Integrations: Services that offer integration with the AI, like Atlassian, Cloudflare or Paypal that offer integration with Claude
• Other Infrastructure: Banking services, …

12. Buy from them

Who? individual users, business customers, governments

Companies need to earn money and will thus be responsive to customer demands. This aspect is strengthened by the similarity between products of different providers, which makes the cost of switching very low.

Single Customers can set financial incentives for AI companies by voting with their wallets or boycotting products. While these are popular tactics in the animal rights movement or environmentalism, they have much less use in AI Safety. However, for companies focused on B2C, large changes in customer demands would likely translate into changes in internal priorities.

Large business customers might have higher leverage to make demands. Esp customers that channel a lot of demand, like API wrapper startups (eg Cursor), could make strong requests to AI companies.

Lastly, the government could exert pressure when giving large contracts, eg using AI for defence. As a starter, governments could instate basic safety requirements like not giving huge defence contracts to AI companies whose new model just went on a pro-Nazi spree. More ambitiously, they could attach stringent safety requirements and information security for models used in the military.

13. Shape their market

Who? Competitors

Companies are responsive to market pressures. Competitors have leverage by shaping the market they both partake in. For example, one company might release models with new capabilities, thus putting pressure on other companies to catch up and release similar models. Other market shaping actions include race dynamics to take a market or reach a capability first, pricing strategies, release strategies (eg open-source AI commodifying chatbots), customer acquisition strategies, adding new features or making specific aspects (e.g. safety) more salient to customers.

14. Change their talent pipeline

Who? potential employees, field builders, educators, competitors

Another critical input into the AI production function is the people developing and deploying AI. They have a rare and highly sought-after skillset without which the companies can’t survive. This was shown in Meta’s recent spending spree to buy top-talent. This gives candidates leverage to make demands from their future employers. Furthermore, it’s a vector for competitors to influence each other. For example Anthropic poached large parts of OpenAI’s alignment team. This brain-drain hurt OpenAI’s public image wrt safety.

Others can attempt to influence the makeup of the talent pool of AI companies. For example field-building programs like MATS aim to produce and accelerate people that care about AI Safety and could end up working at AI Companies. This increases the percentage of candidates that care about safety. Top universities are also in a position of power by educating students about the importance of ethical AI development.

15. Insure them

Who? Insurance companies

Companies are often required to or benefit from being insured. For example, D\&O insurance is required by board members, and Professional liability can be required to sell to enterprise customers. Insurance companies are experts in risk assessment and could thus play an important role in setting prices for and requiring certain safety practices. For example, Lloyd’s of London could state that large AI companies’ products need safety certification to be insured. Additionally, legislation could demand insurance against AI-caused damages (see SB1047), which would make the risk quantification and management conducted by insurance companies an essential part of the safety process. However, insurance is limited to non-existential safety issues, as an existential catastrophe likely implies that the insurance check won’t be cashed.

Thanks for the feedback and comments to Charles Whitman and Joep Storm.
I made some memes for this post, but decided they didn’t fit the vibe. Click here if you still want to see them.

If you’re familiar with RepE feel free to skip to the “Challenges” and “Opportunities” sections.

What is Representation Engineering?

Representation Engineering (RepE) is a class of techniques that manipulate the representations of a concept in order to control its behaviour with regard to that concept. To achieve this, they (1) identify how the targeted concept is represented within the model and (2) use that information to steer the model’s representations on new inputs.

Representation Engineering first identifies how a concept is represented in the activation space of the model and then steers that representation to control the model’s behavior.

RepE includes methods such as Activation Steering e.g. Contrastive Activation Addition (CAA), Low-Rank Representation Adaptation (LoRRA), steering with Sparse Autoencoders, Bi-directional Preference Optimization (BiPO).

By tapping into models’ representations, RepE offers two main advantages: (1) Improve understanding: RepE identifies how human-understandable concepts are represented in the models’ activation space; steering that representation can verify whether the representation has the expected influence on the outputs and can offer an easy way to understand the model’s internals. (2) Control: RepE is promising as a powerful tool to control the behaviour, personality, encoded beliefs, and performance of an LLM, allowing us to make models behave in safe and desired ways. Since no training is required, RepE can be cheaper, more data efficient, and more flexible to different users and situations than other methods while causing less deterioration to the model’s performance

RepE techniques have been applied to prevent harmful outputs, make models more truthful, align them with human preferences, change the model’s situational beliefs, induce it to carry out specific tasks, change its goals, improve reasoning performance and generally interpret model internals.

Our Taxonomy

A pipeline to perform Representation Engineering has three aspects. We use CAA to illustrate them:

1. Representation Identification aims to find out how the concept is represented in the LLM’s activations. Such methods can calculate the difference in activations from specific inputs, identify features through an unsupervised learning process or optimize an operator to lead to desired outputs. E.g. in CAA, this is done by taking the mean difference between activations for positive and negative inputs wrt the concept..
2. Representation Operationalization decides on the shape and meaning of a concept operator that denotes the representation. It also bakes in assumptions about the geometry of representations in the model’s activations. E.g. in CAA, this is a vector, which assumes that representations are linear directions.
3. Representation Control steers the representation of a concept to control the model’s behaviour w.r.t. that concept. This can be done by steering the activations during inference or modifying the weights of the model. E.g. in CAA, the vector is added to the activations.

Framework of Representation Engineering pipelines. One Representation Identification method is used to identify a concept operator. Representations are operationalized by assuming a geometry of representations. The concept operator is used to steer the weights or activations of the model.

Challenges

RepE methods are unreliable: RepE methods are sensitive to changes in hyperparameters (eg steering coefficient or layer of intervention), fail to steer some concepts and can even negatively steer inputs on concepts that are steerable on average. Furthermore, using RepE for interpretability to make claims that a concept is not represented, its representation is localized at a specific point or that it’s represented by a specific vector has been challenged as unreliable.

RepE often fails to generalise out-of-distribution: Current methods become less effective under domain shift when the new inputs differ from the training data, like changing the system prompt or steering at different generation timesteps.

RepE may result in capability deterioration: RepE generally leads to a reduction in general language modelling capabilities. While this reduction is small, it still represents an important cost to employing RepE that should be reduced.

RepE methods struggle to identify precise Concept Representations: Ideally, RepE would identify concept operators that are (1) specific ie only influence the concept of interest and not others and (2) complete ie capture all the aspects of a concept across contexts.

It’s difficult to craft a dataset or scoring function that correctly specifies the concept: To identify a representation, a dataset or scoring function is provided. If these do not correctly specify the concept, the learned representation will be incorrect. Furthermore, other concepts might be spuriously correlated with our concept of interest and most RepE methods are not able to disentangle those correlated concepts. Lastly, the dataset or scoring function might not activate the desired representation. For example, instructing a model to be honest might trigger a representation for “I will answer honestly” or for “the human wants me to answer honestly”.

RepE suffers from interference from Superposition: LLMs represent features in superposition, meaning that features are not all orthogonal to each other, which results in interference between features. In practice, this will mean that controlling a concept representation will also steer some other concepts.

RepE makes overly restrictive assumptions about models’ representations: When operationlizing a representation RepE bakes in assumptions about the geometry of representations. If these assumptions are wrong, it will not identify the correct representation.

Activation space interpretability may be doomed: As RepE techniques are based only on activations, they could be prone to discovering statistical artifacts of the activations which are not actually used by the model in computation. See more here.

Other weaknesses include a failure to steer multiple concepts at once, steer long-form generations, reliance on access to model internals and computational costs of some RepE methods. Furthermore, there are challenges around assumptions on available data, reliance on the model’s own representations, a lack of ground truth in interpretability and shifting activations off their natural distribution.

Opportunities

Methodological Improvements

More precisely identifying Representations: One could identify better concept operators by combining multiple methods for Representation Identification or by refining concept operators to make them more suitable for steering. Furthermore, a focus on improving the quality of data for RepE or improved methods for specifying the concept in a dataset or scoring function would likely yield improvements.

Extending assumptions about Representations: There are some assumptions many RepE methods make about the nature and geometry of representations. Expanding these is a promising direction:

More capable Representation Control: RepE could steer complex sequences, where different interventions are made at different timesteps, or by employing more expressive steering functions. Furthermore, the training process can be steered by ablating or strengthening a representation during fine-tuning to control how representations are learned.

Some promising applications of RepE: RepE might be uniquely well suited to control some concepts where misaligned models might not follow instructions in prompts. Examples of this are changes to the models’ goals or instructions to not be deceptive. Furthermore, RepE could be used to study how representations evolve throughout training or to control how agents cooperate in multi-agent scenarios.

Building a More Rigorous Science of RepE: Lastly, it is important to build a stronger scientific basis of RepE. First and foremost, this includes developing a comprehensive benchmark that compares RepE methods with other approaches for controlling different types of concepts. Better methods for measuring steerability and more adherence to best practices for evaluating RepE methods are important for scientific validity. Better insights can be attained by developing theoretical frameworks for RepE or studying RepE in Toy Settings.

Enabling practical adoption: To enable adoption, RepE methods need to prove themselves in a fair fight against other methods in downstream applications, the community needs to build tooling to make RepE easy to use, and best practices for applying RepE should be made salient.

The full survey discusses all these points in more detail, gives citations to back up my claims and offers much more! You can find the full paper here.

I am happy to discuss these ideas: jan.wehner@cispa.de