LLMs Have a Pre-existing "Functional Welfare Axis": RL Only Recruits It

Overview

This forum post reviews "How's it going? Reinforcement learning in language models recruits a functional welfare axis" by Andy Q Han, David J. Chalmers, and Pavel Izmailov (New York University, arXiv:2605.30232).

Core finding

When an LLM is RL-trained on a semantically neutral maze task, the reward/punishment concept vectors formed internally are not created by RL training — they are "recruited" from a pre-existing "functional welfare axis" already embedded in the base (pretrained) model. One end of the axis encodes "things are going well," the other "things are going badly," and it generalizes to emotions, math reasoning, self-report, and refusal behavior.

• Paper: arXiv:2605.30232
• Models: Qwen3 and GPT-OSS families (4B–20B)
• Key implication: RLHF-like post-training does not instill "values" from scratch; it paves roads across a representational terrain that pretraining already built.

Key points

1. A semantically neutral maze

• The model navigates a 9×9 gridworld with neutral tokens (@ start, # walls, . floor, G reward zone, M penalty zone). No emotional or success/failure vocabulary exists anywhere in the task, ruling out "parroting human text" as an explanation for internal evaluative representations.
• "Functional welfare" here means how well a system is doing relative to its goals — not subjective experience.

2. Extracting concept vectors

• The authors collect residual-stream activations on reward vs. punishment trajectories, average, and normalize the difference to obtain two steering directions: a "reward" vector and a "punishment" vector.
• These vectors have cosine similarity between -0.95 and -0.84 — they are nearly opposite directions of a single axis, not two independent channels.
• Steering with directions orthogonal to the axis produces no systematic change, establishing causality.

3. The punishment vector drives broad "negative welfare" behaviors

• Emotion: more negative sentiment in open-ended generation.
• Pathological backtracking: ~+40% backtrack rate on GSM8K math reasoning (the positive vector reduces backtracks ~25%).
• Confidence collapse: over-uncertain calibration, hedging language.
• Refusals: refusal rate on harmless prompts rises from 3% to 18%.
• Effects scale roughly linearly with steering strength; random directions produce no interpretable effects; the axis is strongest in mid layers (~15–25).

4. Pre-existing, not created

• In untrained base models (Qwen3-Base, GPT-OSS-Base), a highly aligned direction already exists; steering with it reproduces nearly identical effects — sometimes *better* than vectors from RL-trained models.
• RL fine-tunes the axis direction (cosine similarity ~0.7–0.8 with the base-model direction) but the axis's existence and basic function predate RL. RL is the decorator, not the architect.

5. Robustness

• Present across model families, scales (cleaner in larger models: -0.95 vs. -0.75 cosine similarity), and training algorithms — including pure SFT (no RL signal) and LoRA (axis directions match full fine-tuning at cosine similarity > 0.92).

6. Limitations

• Only two model families, up to 20B parameters; a discrete deterministic maze; "online" causal effects during natural inference less clean than steering experiments; and the mechanism by which pretraining *plants* the axis remains a black box.

Why it matters

• Pretraining vs. post-training: some "behavioral preferences" are encoded in the representation space before any alignment training. RLHF at best suppresses or redirects expression of a pre-existing structure — which adversarial attacks or activation steering could re-activate.
• Safety: since the negative welfare vector raises refusal rates and is only moderately correlated with the known "refusal direction" (cosine ~0.4–0.5), refusal behavior and welfare state are distinct but partially overlapping representations — complicating alignment analysis.
• Open question: three non-exclusive hypotheses are offered for why the axis emerges — intrinsic evaluative structure in language, a necessary coordinate of any goal-directed world model, or an attractor of optimization dynamics. David Chalmers' involvement adds a philosophical dimension: functional self-evaluation emerges even without subjective experience.

References cited in the post

1. Zou et al. (2023), Representation Engineering — arXiv:2310.01405 2. Elhage et al. (2021), A Mathematical Framework for Transformer Circuits 3. Marks et al. (2024), Sparse Feature Circuits — arXiv:2403.19647 4. Turner et al. (2023), Activation Addition — arXiv:2308.10248 5. Nanda et al. (2023), Emergent linear representations in world models — arXiv:2309.00941