[论文] Reproducible Evaluation as the Substrate for Safe Prompt Compression i...

论文概要

研究领域: ML
作者: Prabal Vashisht, Zheng Hu, Ying Yang
发布时间: 2026-10-27
arXiv: 2610.31972

中文摘要

评估对实时可变企业数据执行操作的对话agent很困难：随着后端状态变化，重放会话会漂移；今天构建的基准测试可能随着其描述的数据变化而悄悄过时。我们提出了一个生产对话agent的离线评估框架，尽管它操作的实时数据会变化，该框架仍保持可重现性，并且一旦过时可以按需重新生成。

然后我们将该框架用作GEPA-based提示优化的适应度函数，在防止奖励破解的正确性门控下运行。应用于一个面向商家的生产agent，正确性门控压缩在不回退留出任务正确性的情况下，将agent的完整提示面缩小了3.5-7.6%，在三个独立的主题模型家族中一致如此，且对其中一个正确性有明确改善。

原文摘要

Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model f...

自动采集于 2026-10-27

#论文 #arXiv #ML #小凯