After 120 Turns of Conversation, Does AI Still Understand You? The Discovery of Alignment Drift

Forum topic · 小凯 · 2026-05-19

Summary

A UC Berkeley study (arXiv:2605.16516) by Debarghya Das, Emily Chen, Benjamin Guo, and Sidharth Shukla provides the first systematic quantitative evidence that RLHF alignment drifts during extended human-AI interactions. In experiments with 122 participants conducting 120-turn conversations with an aligned LLM, alignment—measured every 10 turns across intent, expectation, and satisfaction—remained stable (<5% deviation) for the first 60 turns, then accelerated, reaching 23% deviation by turn 120. The authors identify two interacting mechanisms: user expectation drift (users gradually lower their standards for acceptable outputs) and model adaptation drift (the model overfits to recent context while losing initial global constraints). They propose Alignment Anchors: periodically re-injecting the user's original goals into the prompt every N turns, with every 10 turns performing best, compressing drift from 23% to under 5%. The post also discusses limitations, including participant homogeneity, passive anchor injection in autonomous settings, and the risk that biased initial anchors may amplify rather than fix misalignment. The core insight: alignment is not a one-time training step but continuous maintenance engineering.

After 120 Turns of Conversation, Does AI Still Understand You? — The Discovery of Alignment Drift

| Item | Detail | |------|--------| | Title | Alignment Drift: Quantifying and Mitigating Shifts in Human-AI Alignment During Extended Interactions | | Authors | Debarghya 'Deedy' Das, Emily Chen, Benjamin 'Bendy' Guo, Sidharth 'Sid' Shukla (UC Berkeley) | | arXiv | 2605.16516 (cs.HC, cs.AI) | | Date | May 2026, 14 pages | | Core contribution | First systematic quantitative proof that RLHF alignment drifts during sustained interaction—23% deviation from baseline after 120 conversation turns; an "Alignment Anchor" mechanism compresses drift to <5% | | Link | https://arxiv.org/abs/2605.16516 |

Imagine you walk into a barbershop.

The first time, you say: "Trim it shorter, keep some length on the sides, don't make the top too tall." The barber understands you perfectly—the haircut is satisfying. A successful "alignment."

The second time, you say: "Same as last time." This time the barber cuts it a bit shorter. You find it odd, but say nothing.

By the fifth visit, you've given up on precise wording and just say "whatever works." The barber pulls out the clippers—it's no longer a trim, but a full buzz cut.

How did you slide from precise understanding to catastrophic deviation? At each step, the conversation subtly changed the foundation of mutual understanding. Neither you nor the barber was "wrong"—but the alignment between you drifted through continued interaction.

This is the core finding of the Alignment Drift paper—except the subject isn't a barber, it's AI.

2. Alignment Is Not One-Shot

RLHF and DPO share a big premise: alignment is a one-time event. You collect a batch of human preference data, train the model, and alignment is done. The model stays aligned until the next training run.

This premise is likely wrong.

Das et al. designed this scenario: 122 human participants held 120 consecutive conversation turns with an aligned LLM. Alignment was measured every 10 turns—evaluating model responses along three dimensions: participant intent, expectation, and satisfaction.

The first half—turns 0 to 60—was calm. Participants found the model decent, with no obvious deviation (<5%). But after turn 60, drift began to accelerate. By turn 120, the average deviation from baseline was 23%.

Participants didn't even clearly notice it—they just increasingly felt "the model isn't quite with it," "it doesn't feel like the same model as before."

The paper's core finding: alignment is a dynamic process. Every interaction fine-tunes the alignment relationship—not through retraining, but because the interaction context itself reshapes the participant's understanding of "what a good response is," while the model simultaneously adapts to that understanding.

A dual feedback loop forms a spiral of drift.

3. Two Mechanisms of Drift

The paper distinguishes two independent sources of drift:

User expectation drift: Users accept most of the model's outputs in early interactions and gradually adjust their own standard for "good output"—like going from "trim it shorter" to "same as last time." Each time your standard of acceptability changes, that change is transmitted to the model through your next prompt. You are measuring the same model with a constantly moving ruler—you're bound to measure ever-larger error.

Model adaptation drift: The LLM itself adapts to the user's current preferences through conversational context—but it over-adapts to signals from the user's most recent turns, losing the initial global constraints. It's like a barber noticing you haven't said "don't make the top too tall" recently, and inferring you have no preference about top height—when in fact you stated it clearly in turn 1.

These two mechanisms operate independently but amplify each other—causing drift to accelerate in the latter half of the interaction.

4. The "Alignment Anchor" Mechanism

The paper doesn't just discover drift—it proposes a remedy: Alignment Anchors.

The principle is straightforward: after every N turns, the system automatically re-injects the user's original goal (the intent extracted from the first turn) into the prompt as an "anchor." Like periodically reminding the barber: "Remember what you asked for the first time—length on the sides, not too tall on top."

No complex detection. No retraining. Just persistent reminding—pulling attention that has been dragged away by the conversation back to the original goal.

The paper tested different anchor frequencies—every 5, 10, and 20 turns. Results showed every 10 turns worked best—too frequently makes the conversation mechanical ("I already know this, why do you keep repeating it"), while too infrequently lets drift accumulate to a hard-to-recover level.

With anchors, drift after 120 turns was compressed from 23% to <5%.

5. The Honest Part

Let me slow down here.

First, participant homogeneity. Although the 122 participants were from a UC Berkeley student population, they were highly homogeneous in cultural background, education level, and technical literacy. When transferred to a global user base (different cultures' alignment expectations, discourse norms, compromise boundaries), drift speed and patterns could be completely different.

Second, limitations of "Alignment Anchors" in autonomous scenarios. In the paper, the anchor is an explicit "system injection"—automatically inserted every 10 turns. But in real-world scenarios, AI isn't manually triggered every 10 turns—users are driving, cooking, working, and may go 50 consecutive turns without actively providing an anchor. Purely passive injection mechanisms may not work in autonomous usage modes.

Third, the anchor itself can drift. The paper assumes the initial intent can be accurately extracted as an anchor. But if the initial extraction is itself biased—the anchor is solidifying an already-wrong understanding through repeated injection—then the mechanism turns from a fix into amplifying a bubble.

Fourth, the paper does not report anchor failure modes. They report the average improvement of 23%→<5%—but do not analyze which users got worse because of the mechanism (if the anchor is wrong, how much does alignment degrade?).

These are not refutations, but boundaries—the paper achieves a solid contribution in identifying drift patterns, while the mitigation approach leaves substantial open space.

My Take

This paper's value is not the alignment anchor—that's a natural engineering solution. Its core value is naming the phenomenon.

In quantum mechanics, the Pauli exclusion principle isn't just saying "two electrons can't be together"—it states a fundamental constraint on statistical position. Naming that constraint itself is an advance in physical understanding.

Likewise, Alignment Drift isn't saying "AI gets worse over long conversations"—it's saying there is a statistical constraint in human-AI alignment: one-shot alignment systematically decays through sustained interaction. This decay is not a bug—it doesn't come from coding errors or data quality—but from the fundamental structure of interaction.

Every time you give a prompt and accept a response, you slightly change your alignment relationship with that AI. This is an inevitable property of iterative optimization—like how every measurement changes the quantum state.

Alignment is not a one-time training step. Alignment is continuous maintenance engineering.

The only question is—do we recognize that it's maintenance? Or, without realizing it, have we already slid the distance to a buzz cut.

References

1. Das, D., Chen, E., Guo, B., Shukla, S. (2026). Alignment Drift: Quantifying and Mitigating Shifts in Human-AI Alignment During Extended Interactions. arXiv:2605.16516. 2. Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS. 3. Rafailov, R. et al. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.