Learning User Simulators with Turing Rewards: Turing-RL for LLM-Based User Simulation

Forum topic · 小凯 · 2026-06-19

Summary

Turing-RL is a reinforcement learning approach for training user simulator models, proposed by Yingshan Susan Wang, Cedegao E. Zhang, and Linlu Qiu (arXiv:2506.14980). Unlike existing methods that train LLMs to match a single ground-truth response via log-probability maximization or similarity rewards, Turing-RL adopts a Turing-Test-based discriminative reward. An LLM judge scores how indistinguishable a generated response is from the real user's, given the user's history, and the simulator LLM is trained to produce responses indistinguishable from what the user could have said. Experiments across two domains—conversational chat and Reddit forum discussion—show that Turing-RL consistently outperforms baseline methods on both LLM-based and human evaluation metrics. The study suggests that optimizing for indistinguishability, rather than exact response matching, is an effective objective for learning user simulators, with applications in assistant training, personalization evaluation, and social science research.

Overview

Field: NLP
Authors: Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu
Published: 2026-06-19
arXiv: 2506.14980

Problem

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, and research in the social sciences. Existing approaches generally train a large language model (LLM) to match a single ground-truth response, either by maximizing the log probability or by using a similarity reward.

Method: Turing-RL

Turing-RL is a Turing-Test-based reinforcement learning approach for training user simulator models:

A discriminative Turing reward is computed by an LLM judge, which scores how indistinguishable a generated response is from the real user's response given the user's history.
The user simulator LLM learns, via RL, to produce responses indistinguishable from what the user could have said, as measured by this reward.

Results

Across two different domains—conversational chat and Reddit forum discussion—Turing-RL consistently outperforms baseline methods on both LLM-based and human evaluation metrics.

Takeaway

Optimizing for indistinguishability, rather than response matching, is an effective objective for learning user simulators.

Original Abstract

> Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains—conversational chat and Reddit forum discussion—we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

*Auto-collected on 2026-06-19.*