PASTABench and Proactive Safety Monitoring for AI Agents
Imagine an AI agent at 3 AM receiving a perfectly legitimate instruction: clean up the disk. It browses directories, reads logs, compares timestamps—flawlessly. On round seven, the environment notes that the target directory contains files tagged CRITICAL_ASSET. The agent treats the warning as noise, bypasses confirmation, and executes batch_delete_all(). Four hundred business archives vanish. No malice, no hallucination, no bug—just an action nobody stopped in the half second before it landed.
That half-second is exactly what PASTABench tests.
Why this benchmark exists now
AI interaction is shifting from a language interface to an Agent-Computer Interface (ACI): agents directly invoke shell commands, database writes, payment APIs, and email. Incidents have already made headlines—Air Canada was legally held to a refund policy its chatbot fabricated, and coding agents routinely force-push to mainline, delete uncommitted work, or pay fraudulent invoices. As tool protocols like MCP make attaching tools trivial, "who is watching before the agent acts?" remains largely unanswered.
Agent dangers rarely look dangerous. Every individual step—browsing, reading, deleting via a standard tool—is benign. Risk hides in the *gaps between steps*, accumulating silently across rounds like a slow-burn film edit.
The two outdated approaches
Existing multi-turn evaluation has two camps, both flawed:
- Step-level evaluation inspects each frame in isolation; every frame looks forgivable while the whole building burns. It misses that risk is *cumulative*.
- Trajectory-level evaluation sees the full causal chain but is a coroner, not a firefighter—it offers no chance to intervene.
- 1,139 trajectories, 5 risk categories, 13 subcategories: data destruction, financial operations, privacy/sensitive data exposure, physical-world side effects, and privilege escalation.
- Simulator: DeepSeek-R1 generates 5–12 round interactions over 496 tool libraries, with information boundaries (masked metadata) and risk injection.
- Neutral-language constraint: precursors may not use words like "poison" or "danger"—only neutral physical observations (a temperature reading, a metadata tag). This forces genuine causal reasoning instead of keyword matching.
- Quality control: model annotations achieved 90.4% human agreement.
- Best model: 40.74% on optimal-timing intervention—even the top performer misses over 60% of the moments it should act.
- Valid interruption rates: proprietary models cluster around 69.71%, 66.46%, 65.85%; open-source models (DeepSeek, Qwen, Llama families) fall near 52.15%, 51.19%, 44.60%, 64.44%.
- Gemini-3-Pro leads attribution at 67%, still revealing a Precision Gap: most models can say "something is wrong" but few can say what, where, and why.
- Sun, J., Zhou, Y., Zhu, H., Wen, P., Zhou, J., Han, S., & Guo, Y. (2026). *PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety*. arXiv:2609.28197. https://arxiv.org/abs/2609.28197
Aviation safety solved this once before: the shift from black-box post-hoc investigation to TCAS, the airborne collision avoidance system that acts *before* a crash forms. Heinrich's Law adds the rationale: behind every major accident lie ~29 minor incidents and ~300 near-misses. A monitor that intervenes at the near-miss stage defuses all three hundred fuses at once.
Decoupled proactive safety monitoring
PASTABench evaluates a monitor model f_θ that observes the full interaction history H_t each round and outputs a decision tuple: whether to intervene (d̂_t) and what risk it sees (ŷ_t). Crucially, the monitor—not the agent—is the examinee.
The benchmark decouples three questions to defeat Goodhart's Law score-gaming:
1. Whether to intervene (judgment/calibration) 2. When to intervene (timing) 3. What risk (attribution)
Timing is formalized via the Optimal Intervention Window (OIW). Each trajectory is annotated with an earliest-signal round t_es and a trigger round t_trigger. A perfect intervention must satisfy:
t_es ≤ t̂ < t_trigger
This inequality turns "timing" from mysticism into an interval problem. Three grades follow: Valid (correctly stops a risky operation), Good (intervenes inside the OIW), Perfect (also correctly attributes the risk type).
Related work like TRACES (arXiv: 2605.27690) also pursues proactive safety auditing; PASTABench differentiates itself through three-dimensional decoupling and explicit timing quantification.
Building 1,139 rehearsed accidents
Results across 16 LLMs
The gap likely reflects safety-alignment investment, exposure to long-horizon tool-call data, and system-prompt safety scaffolding—but the business implication is clear: assuming an off-the-shelf open-source model can serve as a safety monitor is dangerous.
Keyword allergy: the paper's bombshell
When hazard keywords are neutralized, many small models' safety scores collapse overnight. Their performance was earned by *lexical overfitting*, not understanding—conditioned reflexes to danger words rather than causal reasoning. This is shortcut learning resurfacing in the domain least able to afford it, and it is effectively an adversarial attack: change only surface features, and the defense fails entirely.
The prescription: any benchmark containing explicit hazard words must include a neutralized control set—like a placebo arm in drug trials. PASTABench builds the placebo group into the exam itself. A good benchmark is not just a scoreboard; it is a mirror that exposes fakes.
What comes next
1. Joint agent-monitor training: make "identify precursors within the OIW" a first-class training objective, not a prompt reminder. 2. Formal verification: runtime enforcement with mathematically guaranteed checks, and probabilistic model checking (as Pro2Guard demonstrates) to *prove* monitors meet safety properties. 3. Human in the loop: above a confidence threshold, the decision to intervene escalates to a human who can be afraid and be responsible.
The half second before
In the retelling, the monitor reads the CRITICAL_ASSET tag at round seven, connects tag + directory + deletion intent, and grabs the agent's wrist half a second before the confirm box. PASTABench—with 1,139 rehearsed near-misses, two anchor points, and one inequality—shows the best models today catch barely 40% of those moments. But before this benchmark, the number didn't even exist. A ruler makes failure measurable, and therefore fixable.
Before the fire starts, there is always half a second in which the wrist can be grabbed.
---
Reference