OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories

Forum topic · 小凯 · 2026-10-12

Summary

OnTrack is a streaming monitoring mechanism for LLM agents that detects failing trajectories in real time by comparing an agent's steps and dependencies against recorded successful runs, alerting users or blocking the agent in roughly one millisecond per step. The paper, by Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, and Nikhil Mungel (arXiv:2610.12375), addresses the gap between safeguard-agent approaches, which add cost and latency at every step, and post-hoc log evaluation, which arrives too late to prevent wasted tokens or damage. The authors study three regimes of decreasing reference access: full access (historical runs plus tool schemas), schema-only access, and no prior knowledge (only live step logs). Evaluated on SWE-bench trajectories, using only the first 8 steps, OnTrack ranks failing trajectories below successful ones better than content-similarity methods, with a +0.057 AUROC improvement. Combined with an abort policy, it saves about 18% of compute otherwise wasted on failing runs, and 83% of aborted runs were indeed heading toward failure (5 of 6 aborts correct).

Paper Overview

Research Area: NLP Authors: Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel Published: 2026-10-08 arXiv: 2610.12375

Summary

LLM agents are now deployed in domains ranging from trip planning and stock trading to IT incident triage, but they typically operate autonomously with minimal rule-based safeguarding. This creates cost and safety risks from irreversible actions. Existing mitigation strategies fall into two camps: using a safeguard agent to monitor behavior (adding cost and latency to every step), or evaluating logs post-hoc (when tokens are already burned and damage is done).

The authors propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs, enabling it to alert users or block the agent in about a millisecond per step.

Three Access Regimes

The problem is studied under three regimes of decreasing reference access:

1. Full reference access: historical runs plus tool schemas 2. Intermediate access: tool schemas only 3. No prior knowledge: only the live-generated step log

Key Results

Evaluated on SWE-bench trajectories, using only the first 8 steps, OnTrack ranks failing trajectories below successful ones better than content-similarity methods (+0.057 AUROC).
Combined with an abort policy, OnTrack saves about 18% of compute that would otherwise be wasted on failing runs.
83% of aborted runs were indeed heading toward failure — 5 out of 6 aborts were correct.

Abstract (Original)

Agents are deployed in applications from trip planners and stock trading to IT incident triage. In most cases, LLM agents work autonomously with minimal rule-based safeguarding, leading to cost and safety issues from irreversible actions. Recent works resolve this either by using a safeguard agent to monitor behavior or evaluating logs post-hoc. The first adds cost and latency to every step; the second delivers its verdict after the run, when tokens are burned and damage is done. To overcome this, we propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent in about a millisecond per step. We study this problem in three regimes of decreasing access: full reference access (historical r...

---

*Auto-collected on 2026-10-12. Full paper: https://arxiv.org/abs/2610.12375*

OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories

Summary

Paper Overview

Summary

Three Access Regimes

Key Results

Abstract (Original)

Tags