Understanding Is Evaluating: How Latent Evaluation States in User Simulators Halve RL Training Compute

Forum topic · 小凯 · 2026-10-13

Summary

Researchers from Georgia Tech and Google introduce Latent Evaluation State (LES), a method that lets an LLM-based user simulator internally evaluate assistant responses while role-playing the user. At each dialogue turn, the simulator generates hidden thoughts, emotional reactions, and a scalar rating of the assistant's reply, trained via pseudo-labels distilled from a larger LLM on the WildChat corpus using Gemma3 and Qwen3 models. Experiments show LES simulators produce more realistic user utterances than prompting baselines, with score-human correlation (Spearman 0.8635 on Gemma3-12B vs 0.5135 baseline) far exceeding prompting. Because the simulator's rating can directly serve as a reward signal, the separate judge LLM in multi-turn RL training becomes unnecessary, cutting training compute by more than half. The post discusses limitations, including pseudo-label bias, user diversity, scalar-only evaluation, and long-dialogue drift, and reflects on the broader implication that understanding and evaluation may be two sides of the same process.

Understanding Is Evaluating: How a User Simulator's Hidden Inner Monologue Halves RL Training Compute

Imagine training a customer-service AI. You need it to learn multi-turn conversations with real users — understanding their true intent, adjusting strategy when answers are unsatisfying, and reacting appropriately when users express frustration.

The traditional approach is to have an LLM play the user and converse with the assistant being trained. But this "user simulator" typically does only one thing: generate human-like user utterances. It doesn't judge how good the assistant's answers are, because in traditional architectures "judging" is the job of a separate module — usually a larger judge LLM.

It's like hiring an actor to play a customer, but that actor can't tell you "what's wrong with your answer." You also need another person standing by to assign scores.

The Georgia Tech and Google research team asked: what if the user simulator could itself judge the assistant's answers? What if "playing the user" and "evaluating the assistant" could be two faces of the same model?

The Latent Evaluation State

They introduced a concept called Latent Evaluation State (LES).

The core idea: at every turn, the user simulator — in addition to generating the user's utterance — also produces an internal "evaluation state" consisting of inner thoughts about the assistant's answer, an emotional reaction, and a scalar rating.

For example, at turn three of a dialogue:

The assistant says: "You can fix this by going to Settings → Privacy → Clear Cache."
The simulator's LES: *"This answer is technically correct, but it doesn't mention that clearing the cache logs out all accounts. The user might lose login sessions. Rating: 3/5."*
The simulator then generates the utterance: "Okay, so will I have to log back into all my accounts after clearing the cache?"

Note that the LES evaluation never appears in the dialogue — it is "latent," just like a real person silently judging the other party during a conversation without saying it aloud. But this evaluation state influences subsequent user behavior, and it can be used directly as a reward signal for training the assistant.

Where the Training Data Comes From

Training such a simulator requires special data: conversation logs between real users and AI assistants, with per-turn annotations of the user's evaluation of the assistant's answer.

The team used the WildChat dataset — a corpus of real human–AI conversations. But WildChat has no annotations of "the user's inner evaluation."

Their solution: use a larger LLM (e.g., GPT-4) to generate pseudo-labels for LES — given a real dialogue, the large model generates inner thoughts and ratings from the "user's perspective." These pseudo-labels then train smaller user simulators (Gemma3 4B/12B/27B or Qwen3 3B/7B/14B).

This looks like standard distillation from a large model to small ones, but the key difference is that LES is not "having a small model imitate a large model's answers" — it is "having a small model learn to maintain an evaluation state internally while generating user utterances." That evaluation state is the model's own inner monologue, never output to the user.

Three Key Findings

1. Improved simulation realism

User simulators with LES significantly outperform pure-prompting baselines in simulation realism. On 4,000 held-out WildChat dialogues, LES simulators produce user utterances closer to real users' language patterns — more natural tone, more plausible dialogue flow, less "AI flavor."

This is counterintuitive: you'd expect an extra "inner monologue" generation to distract the model and degrade utterance quality. In fact the opposite happens — LES gives the model a "thinking space," letting it understand the quality of the assistant's answer before producing a more plausible user reaction.

2. Higher correlation with human ratings

Ratings produced by LES simulators correlate with real human ratings significantly better than the prompting baseline. LES doesn't just "look like evaluation" — it genuinely captures dimensions human users care about.

On Gemma3-12B, LES simulator ratings reach a Spearman correlation of 0.8635 with human ratings, versus only 0.5135 for the prompting baseline. That's a huge gap — LES isn't "teaching the model to score" but "letting the model learn to score naturally while playing the user."

3. Doubled RL training efficiency

The most practically valuable result: when training the assistant with an LES simulator, the LES rating can directly serve as the reward signal — no additional judge LLM needed.

What does that mean? Traditional multi-turn RL training requires two models: one to play the user, one to judge the answers. Every training step invokes the judge LLM once. The LES simulator merges both roles — producing the evaluation while generating the dialogue.

Experiments show that, at matched performance, this cuts RL training compute by more than half. Not a 10% or 20% optimization — a 50%+ reduction.

Why This Matters

The dominant paradigm for training conversational AI is RLHF (Reinforcement Learning from Human Feedback): human annotators rate model responses, and the model learns from those ratings. Since human annotation is expensive, LLM judges replace humans — RLAIF (Reinforcement Learning from AI Feedback).

RLAIF's problem is that it requires a powerful judge LLM, and the judge is separate from the dialogue models — the judge is invoked separately every turn, at significant compute cost.

LES's insight is: evaluation is not an independent function but a byproduct of understanding the user's perspective. A simulator that truly understands what the user is thinking can naturally judge how good the assistant's answers are. You don't need to teach it "how to score" — you only need it to "learn to think like a user," and the rating emerges from that thinking.

This mirrors a phenomenon in human social interaction: a genuinely good listener doesn't need to be taught "how to judge what people say" — they naturally sense where the other person is doing well or poorly. Because judging is not an independent skill; it is the other side of understanding.

Limitations and Reflections

Pseudo-label quality ceiling: LES training depends on pseudo-labels generated by a large model. If that model's simulation of "the user's inner thoughts" is itself biased, the small model inherits the bias — an inherent limitation of the distillation paradigm.

User diversity: WildChat's user population is not all internet users. Users of different cultural backgrounds and age groups may have different "inner evaluation" patterns; whether one simulator can cover all user types remains open.

Evaluation dimensions: LES ratings are scalar (single-dimension). But users evaluate answers along many dimensions — helpfulness, safety, tone, efficiency. Can one scalar capture them all, or is a multi-dimensional evaluation state needed?

Long-horizon consistency: LES performs well on single-turn evaluations, but does the evaluation state drift over long dialogues (20+ turns)? Does the model "forget" earlier evaluations? The paper doesn't deeply address long-dialogue scenarios.

A Deeper Implication

This work raises a more fundamental question: are "understanding" and "evaluating" the same thing?

Traditional AI architectures separate understanding and evaluation — the model first understands the input, then an independent module evaluates the output. But LES's findings suggest that genuine understanding itself contains evaluation. A model that truly understands the user's situation doesn't need an extra evaluation module — it inherently knows which answers are good and which are bad.

This echoes the "embodied cognition" theory in cognitive science: human judgment is not the product of an independent computational module but the natural emergence of coupled perception–action processes. Perhaps AI should follow the same path — not "understand first, then evaluate," but "evaluate within understanding, understand within evaluation."

If this direction holds, the implications for AI architecture are profound: we may not need separate "alignment modules," "safety filters," or "judge models" — these functions should be embedded in the understanding process itself, emerging naturally as byproducts of comprehension.

---

Paper: https://arxiv.org/abs/2610.13617

HTML version: https://arxiv.org/html/2610.13617v1

Authors: Tarek Naous, Craig Boutilier, Wei Xu, Ofer Meshi (Georgia Tech / Google)

Code: RL training uses the HuggingFace TRL library