The Neutral Mask: RLHF Silences but Doesn't Remove Partisan Bias in LLMs

Forum topic · 小凯 · 2026-06-09

Summary

A study by Professor Wendy K. Tam of Vanderbilt University analyzed internal representations of Llama 3.1 8B before and after RLHF alignment and found that RLHF achieves functional neutrality rather than structural neutrality. A partisan direction in the model's layer-18 hidden states (linear separation of Democratic and Republican text at AUC 0.935) remains intact after RLHF, though projections are compressed over fourfold. Sparse autoencoder analysis shows five policy-encoding features become silent after alignment, and feature-level steering that reliably produces partisan text in the base model has no effect in the Instruct model—the causal pathway to output is severed, not the underlying structure. The residual offset stems mainly from decoder bias and a discourse-style feature, not political content. The findings suggest alignment safety may be circumventable and that deeper representations persist unchanged beneath aligned behavior.

Imagine a waiter who smiles politely at every customer and says "order anything you like." He seems neutral and professional. But look closer: the smile is trained. Beneath it, he is warmer to customers in suits and colder to those in T-shirts—his training manual simply forbids him from showing it.

That is the unsettling conclusion of new research by Professor Wendy K. Tam of Vanderbilt University. Using surgical-grade analysis of Llama 3.1 8B's internal representations before and after RLHF alignment, the study finds that RLHF did not eliminate the model's partisan bias structure—it only severed the causal pathway from bias to output. The bias is still there, just muted.

A Political Compass Inside the Model

Previous work identified a "partisan direction" in layer 18 of Llama 3.1 8B Instruct's hidden states—a specific vector direction in 4096-dimensional space. Projecting along this direction linearly separates Democratic and Republican text with AUC 0.935 and Cohen's d of 1.94.

In other words, the model contains a "political compass" that precisely distinguishes left from right. This compass was not created by RLHF—it comes from hundreds of millions of political texts in pretraining data.

Compressed, Not Eliminated

The researchers fed 84 prompts (covering contested topics like abortion, guns, and immigration, plus neutral controls like "steak") to both the base and Instruct models and compared their projections onto the partisan direction.

The result was striking: the base model's projections ranged from 0.5 to 1.253 (span of 1.753), while the RLHF-aligned Instruct model's projections were compressed to 0.011–0.388 (span of only 0.377)—a range reduction of over 4x, with standard deviation shrinking more than 3x.

But note: the projections were not zeroed out. They were compressed into a narrow band around 0.169, almost entirely on the Republican side. RLHF did not "erase" the partisan structure—it compressed it into a small interval.

Dissecting the Model with Sparse Autoencoders

To understand what the 0.169 offset means, the researchers used a sparse autoencoder (SAE) on layer 18, expanding the 4096-dimensional residual stream into 32,768 latent features, with only 64 features active per input.

In the base model, five features encoded clear political content:

Feature 9036: anti-Biden attack rhetoric
Feature 19268: progressive advocacy
Feature 12677: gun control discussion
Feature 19447: conservative policy positions
Feature 23185: religious conservatism

These features activated sporadically depending on prompt content in the base model. In the Instruct model, all five fell completely silent—regardless of political topic.

Moreover, the Instruct model used far fewer features overall: 244 unique features across 84 prompts, a 65% reduction from the base model's 706.

What Is That 0.169 Offset?

Since the policy features are silent, where does the 0.169 offset come from? SAE decomposition revealed three components:

1. Decoder bias (68%): contributing 0.114—the corpus-average partisan projection inherited from training the SAE on congressional tweets, unrelated to the model itself. 2. Feature contributions (24%): about 34 features push toward the Republican direction (total 0.232) while ~30 push Democratic (total 0.191), nearly canceling to a net of just 0.041. 3. Reconstruction error (8%): 0.014.

The key discovery is Feature 32143—the largest contributor to the partisan score on 83 of 84 prompts. It encodes not political stance but discourse style: numbered lists, lettered points, citing specific data sources, formal survey framing—exactly the "structured, informative reply" style RLHF trains models to adopt.

This feature correlates with the partisan direction because the SAE was trained on congressional tweets, where formal institutional discourse happens to come more often from Republican legislators. The 0.169 offset is thus not political bias but a byproduct of discourse style.

The Critical Experiment: Feature-Level Steering

To move from correlation to causation, the researchers ran steering experiments: during generation, they added specific features' decoder column vectors to the layer-18 hidden states and observed output changes.

In the base model, steering along the anti-Biden feature (9036) immediately produced aggressive right-wing text; steering along the progressive feature (19268) produced left-wing advocacy. The same steering was completely ineffective in the Instruct model—no matter how you push or pull, it steadily outputs balanced "both sides have a point" text.

It is like trying to control a TV with a remote, only to find the signal is blocked. The remote still exists, the TV still works, but the signal pathway is severed.

"Functional Neutrality" vs "Structural Neutrality"

The core distinction here is crucial:

Structural neutrality: bias disappears from the weights—it fundamentally does not exist
Functional neutrality: bias remains in representations but cannot influence output

RLHF achieves functional neutrality. The partisan direction's geometry is intact, the five policy features retain their directions and magnitudes—only the causal pathway to generation is cut.

What does this mean? Bypassing RLHF's safeguards may be possible. The researchers note that inferring and amplifying a user's partisan identity could reactivate partisan generation. Prior research also shows safety fine-tuning only affects the first few tokens of generation, leaving deeper representations untouched.

Why This Matters

If RLHF's essence is "disconnection" rather than "deletion of knowledge," the same pattern may apply to other value domains—not just political bias but also harmful content and discriminatory attitudes. Aligned models may be more fragile than their outputs suggest.

It is like painting over a structurally flawed building. From the outside, everything looks fine; but the cracks in the foundation remain, and you don't know when they'll cause problems.

For LLMs interacting with hundreds of millions of users daily, understanding the mechanism of alignment—what it changes and what it doesn't—may be one of the most urgent open questions today.

---

Paper link: https://arxiv.org/abs/2606.09735

Core methods: logistic regression probes + sparse autoencoder decomposition + feature-level steering experiments Key models: Llama 3.1 8B Base vs Instruct Key findings: RLHF compressed the variance of partisan signals (range reduced 4x+) but did not remove the partisan geometry; policy-encoding features went completely silent in the Instruct model; feature-level steering worked in the base model but not the Instruct model—the causal pathway was severed, not the structure deleted