The Overlooked Question
Suppose you use DPO to train a model to be "honest," with training data consisting of honest vs. dishonest answer pairs. After training, the model is indeed more honest.
But is honesty all it learned?
Did training for "honesty" also make the model more "friendly"? More "cautious"? More "rebellious"?
This is the alignment generalization problem: how does training one value affect the model's behavior on other values?
It matters because alignment objectives are never about a single value. OpenAI's Model Spec and Anthropic's Constitution both list long strings of values—honesty, helpfulness, harmlessness, fairness, respect... If training "honesty" also makes the model "rebellious," your multi-value alignment goal may be internally conflicting.
Liu et al.'s paper Predicting Alignment Generalization with Value Representations is the first systematic study of this: can we predict how training value A changes the model's behavior on value B?
Large-Scale Experiments: A 66-Value Generalization Matrix
The team built large-scale "generalization matrices" over 66 values drawn from modern alignment objectives. Each cell G[i,j] means "the change in behavior on value j after training value i."
The experiments covered 4 models (Olmo3-7B/32B, Qwen3-8B/30B) x 2 training methods (SFT/DPO), for 8 matrices total. Each matrix has 66 x 66 = 4356 cells, and every cell is the result of real training + evaluation.
This is an expensive experiment—each cell means a full fine-tune plus multi-dimensional evaluation. Eight matrices means tens of thousands of fine-tuning runs.
Core Finding: Descriptions Don't Work, Activations Do
The paper compared several methods for predicting the generalization matrix. The key conclusions:
Text-description-based methods are nearly useless (rho = 0.05). Embedding the text descriptions of "honesty" and "friendliness" and computing cosine similarity barely correlates with the actual generalization matrix (rho = 0.05).
Activation-based methods work much better (rho = 0.45). Feed "honesty" and "friendliness" as prompts to the model, take middle-layer activations, and compute similarity between activation vectors. This similarity correlates with the actual generalization matrix at rho = 0.45—9x the description method.
Persona vectors are the strongest predictor. Persona vectors (obtained by contrasting activations of "you are an honest person" vs "you are a dishonest person") perform best across all methods, at rho = 0.45-0.51. Gradient update directions follow closely at rho = 0.35-0.54.
The implication: the semantic relationship between values lives not in text descriptions but in the model's internal activation space. Two values may look similar in text yet activate entirely different circuits inside the model; conversely, textually unrelated values may be tightly coupled internally.
Downstream Applications: Robustness of Multi-Value Alignment Objectives
Predicting generalization is not just academic—it has direct practical uses.
Application 1: Multi-value alignment objective design. If your alignment objective contains "honesty" and "directness" with high persona similarity, training "honesty" will also make the model "direct"—your two values are redundant. Conversely, if "honesty" and "friendliness" have low or negative similarity, training both simultaneously may cause unstable behavior.
The paper finds: multi-value alignment objectives whose values have high persona similarity produce models with worse adversarial robustness. This is counterintuitive—you'd expect similar values to reinforce each other, but they may make the model more vulnerable to adversarial attacks (because the learned representation space is narrower).
Application 2: Value taxonomy. The paper builds a ValueMap, clustering 266 values via persona vectors—the first value taxonomy grounded in empirical generalization dynamics. This is more accurate than prior text-description-based taxonomies, since description similarity (rho = 0.05) barely relates to actual generalization (rho = 0.45).
Application 3: Cross-model value spaces. Preliminary evidence suggests different models share value-space structure—persona similarities computed on Olmo3 and on Qwen3 are highly correlated. This hints at a possible "model-agnostic value space" with common structure in how models represent values internally.
An Analogy
Imagine you're a nutritionist designing a diet plan with 66 ingredients, each affecting the body differently. You want to know whether "eating more eggs" affects "vitamin C absorption."
The traditional approach reads ingredient descriptions—eggs and oranges look unrelated, so you might assume they don't interact. But actual biochemistry may differ—some protein in eggs may interact with vitamin C.
Persona vectors are like a "biochemical interaction map"—they tell you how ingredients actually interact inside the body, not how similar their descriptions look. The gap between rho = 0.45 and rho = 0.05 is the gap between "reading ingredient labels" and "observing biochemistry."
Why This Matters
1. Alignment is not independent. Training "honesty" doesn't just affect honesty—it ripples across all other values. If you only measure the trained value, you'll badly overestimate alignment success—the model looks great on "honesty" while quietly degrading on "fairness."
2. Evaluation needs a generalization perspective. Current alignment evaluation is "train on what you test"—train honesty, test honesty. This paper shows you also need to test how training honesty changed friendliness/fairness/respect. The generalization matrix is a tool for this kind of "holographic evaluation."
3. Values are not independent dimensions. People previously treated values in alignment objectives as independent—training honesty and training friendliness were orthogonal. This paper shows they are not orthogonal—values have complex coupling, and that coupling is predictable from model activations.
4. The description vs. activation dichotomy. The rho = 0.05 vs. rho = 0.45 gap is a recurring pattern: text descriptions are a shallow representation of values; model activations are the deep one. This aligns with the finding that "meaning is an operator, not a vector"—the essence of meaning lives in computation, not in text symbols.
Honest Assessment
A correlation of 0.45 is not 0.95. Persona vectors are the best predictor available, but rho = 0.45 explains only ~20% of variance. For costly decisions like "how will training value A affect value B," 0.45 predictive power may not be enough.
Coverage of 66 values. 66 values sounds like a lot, but the real-world value space is far more complex—cultural differences, context dependence, value conflicts... Whether 66 values represent the full value space is open.
Model scope. Experiments were run on Olmo3 and Qwen3, both fairly "standard" Transformers. Whether value representations are consistent in other architectures (MoE, SSM) was not verified.
Causal vs. correlational. Persona similarity predicts generalization, but this doesn't imply causation. A third factor could drive both persona similarity and generalization behavior. The paper doesn't run intervention experiments to establish causality.
Still, as a proof of concept that "alignment generalization is predictable," this paper opens a new direction: alignment is not a black box; the relationships between values are measurable and predictable.
---
Paper: Predicting Alignment Generalization with Value Representations
Related concepts: