AdvRole: Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents

Forum topic · 小凯 · 2026-09-26

Summary

AdvRole is an adversarial context rewriting framework that turns reinforcement learning for LLM-based role-playing agents into a closed-loop curriculum. Addressing the distributional bottleneck of RL methods that train on a fixed, pre-collected scenario pool, AdvRole alternates between an Actor that learns role-playing and a Rewriter that edits character settings and dialogue contexts into actor-specific hard scenarios. The Rewriter is trained with a performance-gap reward that prefers rewrites lowering the current Actor's score, so the scenario pool evolves alongside the Actor and continuously targets unmastered regions of the character-context space. Evaluated on three role-playing benchmarks covering English and Chinese plus a newly released multilingual benchmark, AdvRole consistently outperforms baselines. Authors: Zheng Zhang, Liu Liu, Qi Chai (arXiv:2609.21994).

Paper Overview

Field: Machine Learning Authors: Zheng Zhang, Liu Liu, Qi Chai Published: 2026-09-25 arXiv: 2609.21994

Key Points

LLM-based role-playing agents are widely used in personalized assistance and social simulation.
Recent RL methods typically train on a fixed scenario pool collected before learning begins, creating a distributional bottleneck: as the agent improves, the scenarios where it underperforms change, while the training distribution remains static.
AdvRole is an adversarial context rewriting framework that turns role-playing RL into a closed-loop curriculum.
It alternates between an Actor that learns to role-play and a Rewriter that edits character settings and dialogue contexts into actor-specific hard scenarios.
The Rewriter is trained with a performance-gap reward, preferring rewrites that reduce the current Actor's score.
The scenario pool co-evolves with the Actor, continuously targeting unmastered regions of the character-context space.
On three role-playing benchmarks covering English and Chinese, plus a newly released multilingual benchmark, AdvRole consistently outperforms baselines.

Original Abstract (excerpt)

Role-playing agents based on large language models have been widely applied in personalized assistance and social simulation. Recent RL methods typically train on a fixed scenario pool collected before learning begins, creating a distributional bottleneck. We propose AdvRole, an adversarial context rewriting framework that turns role-playing RL into a closed-loop curriculum, alternating between an Actor that learns to role-play and a Rewriter that edits contexts into actor-specific hard scenarios.

--- *Auto-collected on 2026-09-26*