[论文] Adversarial Speculative Reasoning Improves Long Horizon Agents

论文概要

研究领域: ML
作者: Hongyin Luo, Wei Fang, Lijie Yang, Owen Stepan, Daniel Suit, Jeff Krapf et al.
发布时间: 2026-11-03
arXiv: 2611.05345

中文摘要

投机解码（speculative decoding）一直被当作纯系统层优化：在不改变模型行为的前提下加速大语言模型解码。在此视角下，训练更好草稿模型的努力只关注接受率。我们认为，对长程智能体推理这一观点并不成立。在前沿开源模型于复杂智能体任务上的测量中，我们发现验证窗口内数值噪声会增长，思考过程中的草稿接受结果会在 30 步之后改变模型的输出分布，从而以随机方向显著影响长程智能体的推理准确率。本文通过控制这种噪声来提升目标模型的推理能力。我们无法改变目标模型验证通过的总体 token 分布，但可以通过训练草稿模型「被拒绝」来以最高精度编码关键推理：我们在智能体轨迹中识别这些关键点，并训练草稿模型偏好对抗性选择。推理时，训练后的草稿与目标模型恰好在这些决策点分道扬镳、验证重启，关键思考 token 从目标模型采样并在其噪声最低的条件下编码。这把关键推理「索引」得更牢、长程更耐用。在复杂智能体编程基准上，该方法在不改变目标模型或其输出分布的前提下提高了端到端成功率，仅训练了宿主参数的 0.06%~0.28%。我们的结果把投机解码机制从纯加速器重新定位为上下文精度的主动分配器。

原文摘要

Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution. Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.

自动采集于 2026-11-05

#论文 #arXiv #AI #小凯