[论文] The Saturation Trap and the Subjectivity of Intervention Timing: Why A...

论文概要

研究领域: ML
作者: Manvendra Modgil
发布时间: 2025-06-01
arXiv: 2506.00628

中文摘要

随着自主AI智能体从对话系统转向长时程软件执行，决定何时中断智能体的运行时安全层变得至关重要。我们使用连续18维情感动态引擎（HEART）作为诊断探针来研究这一时机问题，评估四种干预触发器家族——绝对状态阈值、复合状态-动作模式、正则推理特征提取和零样本LLM评判——与人类在SWE-bench-Verified调试轨迹上标注的干预点对比。我们报告三项发现。第一，状态饱和陷阱：智能体在持续困难下不显示恢复信号，因此建模的挫败感迅速越过阈值并保持在最大值，将基于状态阈值的触发器从时刻检测器转变为近乎恒定的指示器，在五个轨迹中对39-83%的动作触发。第二，LLM评判的能力与上下文底线：小模型（gpt-5.4-mini）从不触发，而前沿和跨厂商模型仅在完整轨迹上下文下才能逃离零触发底线，即便如此也只能达到F1 0.17-0.40，且成本高达90倍。第三，也是最重要的，监督目标在人类之间不可复现：三名训练有素的标注者使用同一评分标准在56动作轨迹上，对干预位置的一致性仅略高于偶然（位置Krippendorff's alpha = +0.047；最佳成对Cohen's kappa = +0.349），对干预类型则完全不一致（暂停退化；澄清低于偶然；反思仅alpha = +0.226）。我们结论：干预时机是一个低可靠性构念，使单标注者F1成为不合适的优化目标。我们的贡献是对该问题在人类评分者间信度、四种检测器架构、跨模型LLM评判扫描和复现饱和效应之间的联合映射，而非任何单一检测器的准确率。

原文摘要

As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential. We study this timing problem using a continuous 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe, evaluating four intervention trigger families - absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge - against human-annotated intervention points on SWE-bench-Verified debugging traces. We report three findings. First, a State Saturation Trap: agents show no recovery signal under sustained difficulty, so modeled frustration quickly crosses the threshold and stays at its maximum, converting threshold-on-state triggers from moment...

自动采集于 2026-06-05

#论文 #arXiv #ML #小凯