UserLM-2.0: Training a Human-Like Yet On-Task User Simulator with Reinforcement Learning

Forum topic · 小凯 · 2026-10-24

Summary

Evaluating multi-turn AI assistants with GPT-4 or Claude playing the user has a core flaw: simulated users are either too robotic (every turn advances the task, sounding unnatural) or too human (drifting off-task and forgetting goals). A paper by Mustafa Omer Gul, Philippe Laban, and Tanya Goyal, "Towards User Simulators That Stay On Task And Still Sound Natural" (arXiv: 2610.31512), introduces UserLM-2.0, a Qwen3-14B-based user simulator trained in two stages: supervised fine-tuning on real human-AI dialogues, then RL (GRPO) with a dual reward—goal coverage plus a naturalness score from a separately trained classifier. Results show UserLM-2.0 achieves 99% goal coverage on UserSim and UserChat benchmarks (above human reference at 89% and all frontier baselines like GPT-5.6), while naturalness reaches 85%, roughly 2-4x that of GPT-5.6 variants (21-35%). Behavioral analysis shows UserLM-2.0 closely matches human patterns on revisit rate (39.0% vs 38.8%), pure repetition (33.0% vs 33.3%), and idle turns (4.3% vs 6.9%). Cost per simulated conversation is $0.11 versus $205.20 for GPT-5.6-Sol—about 1/1867th. The key methodological insight: human-likeness and task reliability are independent dimensions that must be optimized separately, a balance point found via multi-objective RL.

An Awkward Testing Dilemma

Suppose you build an AI assistant and want to test whether it stays helpful over 50 conversation turns. You hire 100 human testers, each chatting for 50 turns—only to find the testers themselves go off-topic, forget tasks, and ask the same questions repeatedly. You can no longer tell whether the assistant or the tester is failing.

This dilemma is real. The mainstream way to evaluate multi-turn AI assistants today is to have GPT-4 or Claude play the user, but these "fake users" have two chronic problems: they are either too human-like (going off-topic, forgetting tasks) or too robotic (advancing the task in every single turn, with extremely low naturalness).

In the paper *Towards User Simulators That Stay On Task And Still Sound Natural* (arXiv: 2610.31512), Mustafa Omer Gul, Philippe Laban, and Tanya Goyal propose a solution: train a dedicated user simulator with reinforcement learning, balancing "task fidelity" and "naturalness."

The simulator is called UserLM-2.0, and its core insight is: natural does not mean random; going off-topic does not mean human-like.

The Two-Stage Training Philosophy

UserLM-2.0 is trained in two steps:

Step 1: Supervised fine-tuning (SFT). A base model is trained on real human-AI conversation data to learn "what humans roughly look like when chatting with AI." The base model is Qwen3-14B, with data from real multi-turn dialogues between users and AI assistants, where each dialogue is annotated with the user's goal.

Step 2: Reinforcement learning (RL). This is the key innovation. The RL reward has two components:

Goal coverage reward: how many predefined sub-goals the simulator covers across 50 turns. For example, if the user's goal is "book a flight to Tokyo tomorrow," the sub-goals include departure location, date, cabin class, passenger count, etc.; the simulator must ask about all of them step by step.
Naturalness reward: an independently trained "naturalness classifier" scores each turn for how much it sounds like a real human. The classifier is itself an LLM, trained by contrasting real human dialogues with GPT-4-generated "robotic" dialogues.

The two rewards are combined with weights and optimized with GRPO, a PPO variant designed for LLMs that is more stable and more memory-efficient than PPO.

The elegance of this design: it treats "being human-like" and "being reliable" as two independent optimization objectives, rather than assuming one automatically implies the other. Previous user simulators either optimized only task completion (sounding robotic) or only similarity to human dialogues (drifting off-task like humans do).

The Numbers: UserLM-2.0 vs GPT-5.6

Experiments were run on two benchmarks, UserSim (task-oriented dialogue) and UserChat (open-domain dialogue). Baselines include GPT-5.6-Sol, GPT-5.6-Luna, Kimi-K2.6, DeepSeek-V4-Pro, and previous best user simulators USP and SFTUser-14B.

Goal Coverage (%)

| Model | UserSim | UserChat | |------|---------|----------| | GPT-5.6-Sol | 96 | 96 | | GPT-5.6-Luna | 97 | 96 | | Kimi-K2.6 | 96 | 96 | | DeepSeek-V4-Pro | 95 | 95 | | UserLM-2.0 | 99 | 99 | | UserLM-2.0-SFT | 97 | 97 | | USP | 80 | 80 | | SFTUser-14B | 88 | 88 | | Human Reference | 89 | 89 |

UserLM-2.0 even exceeds the human reference on goal coverage (99% vs 89%), and beats all closed-source frontier models. RL training genuinely made the simulator more reliable—it almost never forgets a sub-goal.

Naturalness (%)

| Model | UserSim | UserChat | |------|---------|----------| | GPT-5.6-Sol | 35 | 35 | | GPT-5.6-Luna | 21 | 22 | | Kimi-K2.6 | 22 | 22 | | DeepSeek-V4-Pro | 22 | 22 | | UserLM-2.0 | 85 | 85 | | UserLM-2.0-SFT | 96 | 90 | | USP | 14 | 14 | | SFTUser-14B | 56 | 56 | | Human Reference | — | — |

A counterintuitive finding: GPT-5.6's naturalness is extremely low (21-35%), while UserLM-2.0-SFT scores highest (90-96%). After RL, UserLM-2.0's naturalness drops from 90% to 85%—the price paid for task fidelity.

Still, compared to closed-source frontier models, UserLM-2.0's naturalness is 2-4x higher. In other words: with GPT-5.6 as the simulated user, only 1 in 3 utterances sounds human; with UserLM-2.0, 4 in 5 do.

Cost Comparison

| Model | Cost per conversation | |------|------------| | GPT-5.6-Sol | $205.20 | | GPT-5.6-Luna | $127.00 | | Kimi-K2.6 | $77.00 | | DeepSeek-V4-Pro | $127.00 | | UserLM-2.0 | $0.11 |

UserLM-2.0 costs 1/1867th of GPT-5.6-Sol. A test suite that used to cost $10,000 now costs about $5.

Behavioral Analysis: What Counts as the Right Kind of "Human-Like"

The most interesting part of the paper is the behavioral analysis. The authors define four behavioral metrics:

Revisit: fraction of sub-goals re-asked after already being covered within the 50 turns.
Rep. Only: fraction of turns that purely repeat without advancing any goal.
Emp. (idle turns): fraction of turns with a goal present but no progress (e.g., small talk).
W/goal: average number of words used per sub-goal covered.

| Model | Revisit | Rep. Only | Emp. | W/goal | |------|---------|-----------|------|--------| | DeepSeek-V4-Pro | 64.9 | 57.4 | 12.6 | 127 | | Llama-3.1-8B | 53.1 | 43.2 | 24.2 | 190 | | GPT-5.6-Luna | 51.0 | 47.8 | 8.9 | 96 | | Kimi-K2.6 | 52.4 | 47.2 | 4.6 | 77 | | GPT-5.6-Sol | 52.4 | 45.1 | 3.6 | 83 | | HumanLM-Opinion | 45.3 | 28.6 | 50.5 | 315 | | USP | 33.6 | 20.4 | 46.6 | 86 | | UserLM-8B | 35.1 | 24.0 | 31.2 | 204 | | SFTUser-14B | 16.7 | 10.3 | 31.8 | 120 | | UserLM-2.0-SFT | 40.2 | 31.8 | 18.6 | 54 | | UserLM-2.0 | 39.0 | 33.0 | 4.3 | 59 | | Human Reference | 38.8 | 33.3 | 6.9 | 34 |

A striking fact emerges from this table: UserLM-2.0's behavioral pattern almost perfectly matches the human reference.

Revisit rate 39.0% vs human 38.8%—nearly identical. Humans repeat questions; so does UserLM.
Pure repetition 33.0% vs human 33.3%—nearly identical. Humans repeat themselves; so does UserLM.
Idle turns 4.3% vs human 6.9%—close. UserLM is closer to humans than GPT-5.6-Sol (3.6%).
Words per goal 59 vs human 34—UserLM is more verbose than humans, but better than GPT-5.6-Sol (83) and DeepSeek-V4-Pro (127).

By contrast, GPT-5.6-Sol has a revisit rate of 52.4% (human: 38.8%) and pure repetition of 45.1% (human: 33.3%)—GPT-5.6 repeats more than humans do, but advances the task in nearly every turn (idle rate only 3.6%). This is the quantified form of "too robotic": it never small-talks, but keeps re-confirming the same thing.

Meanwhile HumanLM-Opinion, a model optimized purely for naturalness, has an idle rate of 50.5%—half its time is spent small-talking without advancing the task. This is the quantified form of "too human": it does sound human, but in the human way of going off-topic.

UserLM-2.0 is the only model close to humans on all four metrics. RL taught it "when to advance the task and when small talk is okay"—a judgment that cannot be encoded with simple rules.

A Deeper Insight

The paper's core contribution is not UserLM-2.0 itself, but a methodological insight: "human-likeness" and "reliability" are two independent dimensions that must be optimized separately.

Previous approaches either assumed "human-like implies reliable" (wrong—HumanLM-Opinion shows human-like models drift), or "reliable implies human-like" (also wrong—GPT-5.6 shows reliable models turn mechanical). UserLM-2.0 explicitly optimizes both dimensions with a dual-objective RL reward, finding a Pareto-optimal point.

This insight generalizes to broader AI evaluation scenarios:

Adversarial testing: you need a simulator that "acts like an attacker but stays on topic" to test safety guardrails.
Education: you need a simulator that "acts like a student but progresses through exercises" to test tutoring AI.
Customer service: you need a simulator that "acts like an angry customer but never forgets the original complaint" to test support bots.

Every scenario is a balance between "naturalness" and "task fidelity," and RL is currently the most mature tool for explicit multi-objective optimization.

The paper honestly lists limitations: UserLM-2.0 was trained and evaluated only in English; the naturalness classifier is itself an LLM and may carry bias; 50-turn dialogues may be too short to test true long-horizon tasks; and the simulator's ability to model adversarial users was not tested.

Open Source and Availability

UserLM-2.0 is based on Qwen3-14B and trained with the HuggingFace TRL library. The paper states that model weights and datasets will be open-sourced upon acceptance. The earlier UserLM-8B is already available on HuggingFace (microsoft/UserLM-8b).

---

Paper: Towards User Simulators That Stay On Task And Still Sound Natural

Authors: Mustafa Omer Gul, Philippe Laban, Tanya Goyal

Published: 2026-10-22