Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Conversational Agents

Forum topic · 小凯 · 2026-10-26

Summary

A new arXiv paper (2610.31972) by Prabal Vashisht, Zheng Hu, and Ying Yang addresses a key challenge in evaluating conversational agents that operate on live, mutable enterprise data: replayed sessions drift as backend state changes, and benchmarks can silently go stale. The authors present an offline evaluation harness for a production conversational agent that remains reproducible despite changing live data and can be regenerated on demand when it becomes outdated. The harness is then used as the fitness function for GEPA-based prompt optimization, protected by a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, the correctness-gated compression approach shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness. Results are consistent across three independent subject-model families, with one model showing an explicit improvement in correctness. The work demonstrates that reproducible evaluation infrastructure can serve as a safe foundation for automated prompt compression in real-world LLM deployments.

Paper Overview

Field: Machine Learning Authors: Prabal Vashisht, Zheng Hu, Ying Yang Published: 2026-10-27 arXiv: 2610.31972

Abstract

Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, with an explicit correctness improvement on one.

Key Contributions

Reproducible offline evaluation harness: a benchmarking framework that remains stable even as the underlying live enterprise data changes, and can be regenerated on demand when it goes stale.
GEPA-based prompt optimization: the harness is used as the fitness function for prompt compression/optimization.
Correctness gate against reward hacking: optimization is constrained so that compression cannot degrade held-out task correctness.
Production results: 3.5-7.6% prompt surface reduction on a merchant-facing production agent, consistent across three independent subject-model families, with one model showing a clear correctness improvement.