AI Incident Trajectories: A Pragmatic Classification Framework for AI Safety Policy

Forum topic · 小凯 · 2026-05-04

Summary

A Chinese tech forum post summarizes the paper "A pragmatic classification of AI incident trajectories" (arXiv: 2604.21412, 2026-04-28) by Isaak Mengesha, Branwen Owen, Charlie Collins, Tina Wong, Simon Mylius, Peter Slattery, and Sean McGregor. The post argues that the apparent surge in AI incidents conflates three distinct factors: changes in reporting propensity, growth in deployment scale, and the actual harm rate per unit of exposure. The paper proposes classifying incidents into four trajectory types—growing, stable, declining, and spike—each requiring different policy responses, from emergency intervention for growing trajectories to rapid-response mechanisms for spikes. The author also outlines data challenges facing AI incident research, including underreporting, reporting bias, vague incident definitions, and attribution difficulty, and closes with a Feynman-inspired caution about separating signal from noise in AI safety statistics. The article ends with four questions practitioners should ask when evaluating AI system safety.

> Paper: A pragmatic classification of AI incident trajectories > Authors: Isaak Mengesha, Branwen Owen, Charlie Collins, Tina Wong, Simon Mylius, Peter Slattery, Sean McGregor > arXiv: 2604.21412 | 2026-04-28

---

1. The Illusion That "AI Incidents Are Exploding"

Open the news and you'll see headlines like:

"AI self-driving car kills a pedestrian"
"AI hiring system discriminates against women"
"AI chatbot induces a user to self-harm"
"AI-generated fake news influences elections"

It looks like AI incidents are growing explosively. But the real question is: is that true?

The paper identifies a key problem: statistics from public AI incident databases conflate three distinct factors: 1. Changes in reporting propensity: things that weren't reported before are reported now 2. Growth in deployment scale: the more AI is used, the more incidents naturally occur 3. Harm rate per unit of exposure: the actual probability that any given AI system causes harm

Without distinguishing these three factors, we cannot judge whether AI is actually becoming more dangerous.

---

2. Four Types of Incident Trajectories

The research proposes a framework that classifies AI incidents into four "trajectories":

1. Growing

The harm rate per unit of exposure is rising
This is the most dangerous signal—it means the AI system itself is becoming more dangerous
Possible causes: increasing system complexity, regulatory lag, accumulating technical debt

2. Stable

The harm rate per unit of exposure stays constant
Absolute harm counts are increasing, but only because deployment is scaling up
Warrants attention, but is not an urgent crisis

3. Declining

The harm rate per unit of exposure is falling
Good news—safety measures are working
But we need to confirm: is this genuine improvement, or underreporting?

4. Spike

A sudden surge in harm rate, followed by a fall
Usually associated with new technology releases, major updates, or specific events
Requires rapid response and post-hoc analysis

---

3. Why Classification Matters

Different trajectories call for completely different policy responses:

Growing: emergency intervention needed—pausing deployments, mandatory safety reviews
Stable: continuous monitoring and preventive investment
Declining: learn best practices and disseminate them
Spike: rapid-response mechanisms and post-incident analysis capability

If we misclassify a growing trajectory as stable, we may miss the optimal window for intervention. If we misclassify a spike as growing, we may overreact and stifle innovation.

---

4. The Data Problem

Current AI incident data faces serious challenges:

Underreporting: many incidents are never publicly reported
Reporting bias: high-profile incidents get coverage, while everyday small harms are ignored
Vague definitions: what counts as an "AI incident"? System failure? User misuse? Design flaws?
Attribution difficulty: was the harm directly caused by AI, or did AI merely exacerbate an existing problem?

Without good data, there can be no good policy.

---

5. A Feynman-Style Judgment: Separating Signal from Noise

Feynman emphasized in his lectures on experimental design:

> "You must be very careful not to deceive yourself—and you are the easiest person to deceive."

AI incident statistics are exactly like this. We see absolute harm counts rising and assume AI is getting more dangerous. But this could be "noise" (deployment scale expanding, reporting increasing) rather than "signal" (per-unit risk rising).

The scientific approach is: control variables, separate the different effects, and find the true causal signal.

---

6. Takeaways

If you are evaluating the safety of an AI system, ask yourself:

1. "Am I looking at absolute incident counts, or incident rates per unit of exposure?" 2. "Is the data affected by changes in reporting propensity?" 3. "Is the definition of an incident consistent?" 4. "Does my conclusion account for changes in deployment scale?"

AI safety policy cannot be built on intuition. It must be built on rigorous data analysis.

This research offers a starting point: not simply counting incidents, but understanding their "trajectory"—rising, stable, declining, or spiking?

Only by answering this question can we know: should we hit the brakes, or keep accelerating?