[论文] OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectori...

论文概要

研究领域: NLP 作者: Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel 发布时间: 2026-10-08 arXiv: 2610.12375

中文摘要

LLM agent 已部署在行程规划、股票交易、IT 事故分诊等场景中，但多数情况下它们自主运行、仅有极少的基于规则的安全防护，不可逆操作带来成本和安全问题。现有方案要么用防护 agent 监控行为（每步都增加成本和延迟），要么事后评估日志（此时 token 已烧、损害已造成）。为此我们提出 OnTrack：一种流式监控机制，将 agent 的步骤和依赖关系与记录的成功运行对比，每步约 1 毫秒即可向用户告警或阻断 agent。我们在三种数据访问权限递减的场景下研究该问题：完全参考访问（历史运行+工具 schema）、中间访问（仅工具 schema）、无先验知识（仅实时生成的步骤日志）。在 SWE-bench 轨迹上评估，仅靠前 8 步，我们的方法将失败轨迹排在成功轨迹之下的能力优于内容相似度方法（+0.057 AUROC）。配合中止策略，可节省约 18% 会浪费在失败运行上的算力，其中 83% 的被中断运行确实正走向失败（6 次中止中 5 次正确）。

原文摘要

Agents are deployed in applications from trip planners and stock trading to IT incident triage. In most cases, LLM agents work autonomously with minimal rule-based safeguarding, leading to cost and safety issues from irreversible actions. Recent works resolve this either by using a safeguard agent to monitor behavior or evaluating logs post-hoc. The first adds cost and latency to every step; the second delivers its verdict after the run, when tokens are burned and damage is done. To overcome this, we propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent in about a millisecond per step. We study this problem in three regimes of decreasing access: full reference access (historical r...

#论文 #arXiv #NLP #小凯