Summary
This paper (arXiv 2610.00197, NLP) investigates automated rewards for training language models in conversational humor, focusing on reward exploits and countermeasures. Two reward approaches aim to capture understandable surprise and predicted audience amusement. Controlled tests reveal that an embedding-based surprise reward accepts word-shuffled replies as readily as genuinely witty ones. A fluency filter detects these shuffles, but the combined reward also rejects some witty replies and fails further validation. Predicted laughter from an audience model is vulnerable to laughter cues in either speaker's messages; cross-speaker normalization blocks covered attacks, though unmatched expressions remain exploitable. Three reinforcement-learning runs evaluate successive reward revisions: the final combined scoring round improves scores by 0.0903 and reduces zero-score episodes by 40%, but humor-specific gains still fall below preregistered targets. The findings highlight a broader challenge in automated reward design: countermeasures must close exploitable shortcuts while preserving the intended behaviors the reward was designed to encourage.
Paper Overview
- Field: NLP
- Author: Sam Larson
- Published: 2026-10-05
- arXiv: 2610.00197
Abstract (English translation)
We investigate automated rewards for training language models in conversational humor, focusing on reward exploits and countermeasures. Two approaches aim to capture understandable surprise and predicted audience amusement. Controlled tests show that an embedding-based surprise reward accepts word-shuffled replies as readily as witty ones. A fluency filter detects the shuffles, but the combined reward also rejects some witty replies and fails further validation. An audience model's predicted laughter is instead vulnerable to laughter cues in either speaker's messages. Normalizing these cues across speakers blocks the covered attacks, although unmatched expressions remain exploitable. Three reinforcement-learning runs evaluate training with successive reward revisions. The final combined scoring run improves scores by 0.0903 and reduces zero-score episodes by 40%, but humor-specific gains still fall below preregistered targets. These findings illustrate a broader challenge in automated reward design: countermeasures must close exploitable shortcuts while preserving the behaviors the reward was intended to encourage.
---
*Auto-collected on 2026-10-05.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635591