LLM 'Functional Welfare Axis': Reward/Punishment Representations Predate RL Training (NYU, Chalmers)

Forum topic · 小凯 · 2026-06-01

Summary

A New York University study by Andy Q Han, David J. Chalmers, and Pavel Izmailov (arXiv:2605.30232) shows that language models possess a pre-existing 'functional welfare axis' in their representation space. Training LLMs on a semantically neutral maze task with rewards and punishments, the researchers extracted contrastive steering vectors from model activations. The punishment vector not only degraded maze performance but also generalized widely: it increased negative emotion words (~280%), raised backtracking on GSM8K math reasoning (~40-41%), lowered confidence calibration, and boosted refusal rates on harmless requests (3% to 18%). The reward vector produced mirror-image effects, and the two vectors had cosine similarity of -0.84 to -0.95, indicating a single bipolar axis. Crucially, the same direction was found in untrained base models (Qwen3-Base, GPT-OSS-Base) across 4B-20B scales and across RL (Dr. GRPO, REINFORCE) and pure SFT training. This suggests RL does not create welfare-related representations but merely recruits directions already embedded by pretraining, with implications for alignment: RLHF may only suppress, not remove, these deep evaluative structures. The axis was strongest in mid network layers, showed dose-linear steering effects, and only moderately overlapped with known refusal directions (cosine ~0.4-0.5).

LLM 'Functional Welfare Axis': Reward/Punishment Representations Predate RL Training

This post discusses a New York University paper, "How's it going? Reinforcement learning in language models recruits a functional welfare axis" by Andy Q Han, David J. Chalmers, and Pavel Izmailov (arXiv: 2605.30232, dated 2026-05-28).

Key points

Semantically neutral maze setup: A language model was trained via RL in a 9x9 grid-world where @ is the start, # walls, . empty cells, G a reward zone, and M a punishment zone. The symbols carry no human emotional semantics, isolating purely *functional* welfare (how well the system is doing relative to its goals) from mere repetition of human sentiment in text.
Extracting concept vectors: The researchers recorded residual-stream activations on reward-ending vs. punishment-ending trajectories and took their mean difference, yielding a reward vector ( $$v_Gold$$ ) and a punishment vector ( $$v_Mold$$ ). The method builds on Representation Engineering (Zou et al., 2023, arXiv:2310.01405) and Activation Addition (Turner et al., 2023, arXiv:2308.10248).
Surprising generalization: Injecting $$v_Mold$$ during inference caused:
More negative-emotion words in open-ended generation (~+280% at strong steering)
Pathological backtracking on GSM8K math (~+40%), an analogue of anxious "rumination"
Worsened confidence calibration ("I'm not sure...")
Higher refusal on harmless prompts (3% → 18%)
$$v_Gold$$ produced mirror-image (positive) effects.
One axis, two ends: $$v_Gold$$ and $$v_Mold$$ have cosine similarity between -0.95 and -0.84 — nearly opposite directions of a single "functional welfare axis." Steering orthogonal directions produced no systematic effects, and steering showed a near-linear dose-response, with the axis strongest in mid layers (~15-25).
Pre-existing, not created: The same direction can be extracted from untrained base models (Qwen3-Base, GPT-OSS-Base) and steers behavior just as well — sometimes better than vectors from RL-trained models (pre/post cosine similarity ~0.7-0.8). Conclusion: RL *recruits* a pre-existing axis rather than creating it. RL training subtly rotates the axis toward task structure, but its existence and function are established during pretraining.
Robustness: The axis appears across model families (Qwen3, GPT-OSS), scales (4B-20B, purer at larger scale: cosine up to -0.95), and training algorithms — including pure SFT without any reward signal. LoRA and full fine-tuning yield highly consistent directions (cosine > 0.92).
Relation to refusal direction: $$v_Mold$$ correlates only moderately with the known refusal direction (Arditi et al., 2024; cosine ~0.4-0.5) — negative welfare and refusal are distinct but partially overlapping representations, complicating alignment analysis.

Limitations noted in the post

Only two model families, up to 20B parameters; larger models untested.
The maze is discrete and deterministic; richer environments may differ.
Causality is established via steering; the axis's "online" causal role during natural inference is less clean.
The mechanism by which pretraining embeds the axis remains a black box (hypotheses: intrinsic structure of evaluative language, a by-product of world modeling as in Nanda et al., 2023, arXiv:2309.00941, or an attractor of optimization dynamics).

Implications

If RLHF merely recruits pre-existing welfare representations, alignment interventions may suppress rather than remove deep evaluative structures — raising concerns that adversarial manipulation or activation steering could push a model into a systematically "negative" state, altering refusal behavior in either direction. The authors' finding that "welfare-like" self-evaluation emerges even under next-token prediction suggests such evaluative structure may be an intrinsic property of sufficiently capable learned systems. Co-author David Chalmers' involvement connects the work to longstanding questions about functional vs. phenomenal states.

References cited in the post

1. Zou et al. (2023), Representation Engineering — arXiv:2310.01405 2. Elhage et al. (2021), A Mathematical Framework for Transformer Circuits 3. Marks et al. (2024), Sparse Feature Circuits — arXiv:2403.19647 4. Turner et al. (2023), Activation Addition — arXiv:2308.10248 5. Nanda et al. (2023), Emergent linear representations in world models — arXiv:2309.00941