The Neutral Mask: RLHF Mutes Partisan Bias in LLMs But Doesn't Remove It

Imagine a restaurant waiter who smiles at every customer and says "order whatever you like." He seems neutral and professional. But look closer: his smile is trained. Underneath, he is warmer to customers in suits and colder to those in T-shirts—the training manual just told him not to show it.

That is the core finding of Professor Wendy K. Tam of Vanderbilt University, who performed brain-surgery-level analysis on the internal representations of Llama 3.1 8B before and after RLHF alignment. Her unsettling conclusion: RLHF does not eliminate the partisan bias structure in the model—it only cuts the causal pathway from bias to output. The bias is still there; it has been muted.

A Partisan Compass Inside the Model

Previous work found a "partisan direction" in layer-18 hidden states of Llama 3.1 8B Instruct—a specific vector direction in 4096-dimensional space. Projecting along it linearly separates Democratic and Republican texts with AUC 0.935 and Cohen's d of 1.94.

In other words, the model has a "political compass" internally. This compass was not created by RLHF—it comes from hundreds of millions of political texts in pretraining data.

Compression, Not Elimination

The researcher fed 84 prompts (covering abortion, guns, immigration, plus neutral controls like "steak") to both the base and instruct models and compared projections onto the partisan direction:

• Base model: projections range 0.5 to 1.253 (span 1.753)
• Instruct model: projections compressed to 0.011–0.388 (span 0.377)

The range shrank more than 4x and standard deviation by more than 3x. But crucially, the projections were not zeroed out—they were compressed into a narrow band around 0.169. RLHF compressed the partisan structure rather than erasing it.

Brain Surgery with Sparse Autoencoders

To understand the 0.169 offset, the study used a sparse autoencoder (SAE) to decompose layer 18 into 32,768 latent features, with only 64 active per input.

In the base model, five features clearly encode political content:

• Feature 9036: anti-Biden attack rhetoric
• Feature 19268: progressive advocacy
• Feature 12677: gun control discussion
• Feature 19447: conservative policy positions
• Feature 23185: religious conservatism

These activate sporadically in the base model depending on the prompt. In the Instruct model, all five go completely silent. The instruct model also uses far fewer features overall: 244 unique features across the 84 prompts, down 65% from the base model's 706.

What Is the 0.169 Offset?

SAE decomposition splits the offset into three parts:

1. Decoder bias (68%): 0.114—corpus-average partisan projection inherited from SAE training on congressional tweets, unrelated to the model itself. 2. Feature contributions (24%): ~34 features push Republican-ward (total 0.232), ~30 push Democrat-ward (total 0.191), nearly canceling to a net of 0.041. 3. Reconstruction error (8%): 0.014.

The key discovery is feature 32143—the largest partisan-score contributor on 83/84 prompts. It encodes not political stance but discourse style: numbered lists, bullet points, citing specific data sources, formal survey framing—exactly the "structured, informative reply" style RLHF trains for. It correlates with the partisan direction only because the SAE was trained on congressional tweets, where formal institutional discourse happens to come more from Republican legislators. The 0.169 offset is a byproduct of discourse style, not political bias.

The Critical Experiment: Feature-Level Steering

To move from correlation to causation, the researcher ran steering experiments: adding specific feature decoder vectors to layer-18 hidden states during generation.

In the base model, steering along the anti-Biden feature (9036) instantly produces aggressive right-wing text; steering along the progressive feature (19268) produces left-wing advocacy text. The same steering has no effect on the Instruct model—push or pull however you like, it steadily outputs balanced "both sides have a point" text.

It's like trying to control a TV with a remote whose signal is being jammed. The remote exists, the TV works, but the signal pathway is cut.

Functional vs. Structural Neutrality

The central distinction:

• Structural neutrality: bias disappears from the weights—it fundamentally doesn't exist.
• Functional neutrality: bias remains in representations but cannot influence output.

RLHF achieves functional neutrality. The partisan geometry is intact; the five policy features retain their directions and magnitudes; only the causal pathway to the generation pipeline is severed.

This implies RLHF safeguards can potentially be bypassed. The researcher notes that inferring and amplifying a user's partisan identity could reactivate partisan generation. Prior research also shows safety fine-tuning mainly affects the first few generated tokens while deeper representations remain untouched.

Why This Matters

If RLHF fundamentally "disconnects" rather than "deletes," the same pattern may apply to other value domains—not just political bias but harmful content and discriminatory attitudes. Aligned model behavior may be more fragile than outputs suggest.

It's like painting a structurally flawed building a lovely color. From outside, everything looks fine; the cracks in the foundation remain, and you don't know when problems will surface.

For LLMs interacting with hundreds of millions of users daily, understanding what alignment actually changes—and what it doesn't—may be one of the most pressing open questions today.

---

Paper link: https://arxiv.org/abs/2606.09735

Core methods: logistic regression probes + sparse autoencoder decomposition + feature-level steering experiments Key models: Llama 3.1 8B Base vs Instruct Key findings: RLHF compresses variance of the partisan signal (range reduced 4x+) but does not remove partisan geometry; policy-encoding features go fully silent in the instruct model; feature-level steering works on the base model but not the instruct model—the causal pathway is severed, not the structure deleted.