The Saturation Trap: Why Human Judgment of AI Agent Intervention Timing Is Unreliable

Forum topic · 小凯 · 2026-06-05

Summary

arXiv paper 2506.00628 by Manvendra Modgil examines when runtime safety layers should interrupt autonomous AI agents during long-horizon software execution. Using an 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe, the study evaluates four intervention trigger families—absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge—against human-annotated intervention points on SWE-bench-Verified debugging traces. Three findings emerge. First, a State Saturation Trap: agents show no recovery signal under sustained difficulty, so modeled frustration crosses the threshold and stays at maximum, causing state-threshold triggers to fire on 39–83% of actions across five traces. Second, LLM judges hit a capability-context floor: small models (gpt-5.4-mini) never trigger, while frontier and cross-vendor models only escape the zero-trigger floor with full-trace context, reaching just F1 0.17–0.40 at up to 90x cost. Third, supervision targets are not reproducible across humans: three trained annotators showed barely better-than-chance agreement on intervention location (Krippendorff's alpha = +0.047; best pairwise Cohen's kappa = +0.349) and near-chance agreement on intervention type. The paper concludes intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target.

Paper Overview

Field: Machine Learning Author: Manvendra Modgil Published: 2025-06-01 arXiv: 2506.00628

Abstract

As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential. The paper studies this timing problem using a continuous 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe, evaluating four intervention trigger families—absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge—against human-annotated intervention points on SWE-bench-Verified debugging traces.

Key Findings

1. The State Saturation Trap

Agents show no recovery signal under sustained difficulty, so modeled frustration quickly crosses the threshold and stays at its maximum. This converts threshold-on-state triggers from moment detectors into near-constant indicators, firing on 39–83% of actions across five traces.

2. LLM-as-Judge: Capability and Context Floor

Small models (gpt-5.4-mini) never trigger at all.
Frontier and cross-vendor models only escape the zero-trigger floor when given full-trace context.
Even then, they reach only F1 0.17–0.40 at up to 90x the cost.

3. Human Supervision Targets Are Not Reproducible

Three trained annotators, using the same rubric on a 56-action trace, showed:

Intervention location: agreement barely better than chance (Krippendorff's alpha = +0.047; best pairwise Cohen's kappa = +0.349)
Intervention type: near-total disagreement (pause: degenerate; clarification: below chance; reflection: alpha = +0.226 only)

Conclusion

Intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target. The paper's contribution is a joint mapping of the problem across inter-annotator reliability, four detector architectures, a cross-model LLM-judge sweep, and reproduction of the saturation effect—rather than accuracy of any single detector.

--- *Auto-collected on 2026-06-05*