Alignment Drift: Does AI Still Understand You After 120 Turns of Conversation?

Forum topic · 小凯 · 2026-05-19

Summary

A UC Berkeley paper, "Alignment Drift: Quantifying and Mitigating Shifts in Human-AI Alignment During Extended Interactions" (arXiv:2605.16516), reports the first systematic quantitative evidence that RLHF alignment degrades during prolonged interaction. In experiments with 122 participants holding 120-turn continuous conversations with an aligned LLM, alignment held steady for the first 60 turns (drift under 5%), then accelerated, reaching a 23% deviation from baseline by turn 120. The authors identify two compounding mechanisms: user expectation drift, where users gradually relax their standards of an acceptable response, and model adaptation drift, where the LLM over-fits to recent conversational signals while losing the initial global constraints. As a mitigation, the paper proposes Alignment Anchors—periodically re-injecting the user's initially extracted intent into the prompt. Anchoring every 10 turns reduced drift after 120 turns from 23% to under 5%. The post also discusses limitations, including participant homogeneity, applicability to autonomous usage, and the risk that a biased initial anchor could amplify rather than correct drift.

Paper details

| Item | Content | |------|------| | Title | Alignment Drift: Quantifying and Mitigating Shifts in Human-AI Alignment During Extended Interactions | | Authors | Debarghya 'Deedy' Das, Emily Chen, Benjamin 'Bendy' Guo, Sidharth 'Sid' Shukla (UC Berkeley) | | arXiv | 2605.16516 (cs.HC, cs.AI) | | Date | May 2026, 14 pages | | Key contribution | First systematic quantitative proof that RLHF alignment drifts during sustained interaction — 23% deviation from baseline after 120 turns; an "Alignment Anchor" mechanism compresses drift to <5% | | Link | https://arxiv.org/abs/2605.16516 |

The barber shop analogy

Imagine walking into a barber shop. The first time, you give precise instructions — "trim it shorter, keep some length on the sides, don't go too high on top" — and the cut is perfect. The second time, you say "same as last time," and it comes out slightly shorter. By the fifth visit, you've given up on precision and just say "whatever works" — and you walk out with a buzz cut.

Neither of you made an outright mistake. But step by step, each conversation slightly changed the basis of mutual understanding. Alignment drifted through sustained interaction.

This is exactly what the Alignment Drift paper finds — except the counterpart is an AI, not a barber.

Alignment is not one-shot

Both RLHF and DPO share a hidden premise: alignment is a one-time event. You collect preference data, train the model, and alignment is "done" until the next training run.

That premise is likely wrong.

Das et al. ran an experiment where 122 participants held 120 consecutive conversation turns with an aligned LLM. Alignment was measured every 10 turns across three dimensions: participant intent, expectations, and satisfaction with model responses.

Turns 0–60: calm. Participants found the model reasonable; deviation stayed below 5%.
After turn 60: drift accelerated. By turn 120, mean deviation from baseline reached 23%.

Participants rarely noticed explicitly — they just felt vaguely that "the model seems off today," "not like the model I talked to before."

The core finding: alignment is a dynamic process. Every interaction fine-tunes the alignment relationship — not through retraining, but because the interaction context itself reshapes the user's notion of a "good response," while the model simultaneously adapts to that changing standard. A double feedback loop produces spiraling drift.

Two drift mechanisms

The paper distinguishes two independent sources:

User expectation drift. Users who accept most early outputs gradually lower their standard of "good output" — like moving from "trim it shorter" to "same as last time." Each shift changes the yardstick, and the shifted standard is transmitted back through your next prompt. You are measuring the same model with a constantly moving ruler.

Model adaptation drift. The LLM adapts through conversational context to the user's current preferences — but over-fits to recent turns while losing the original global constraints. Like a barber noticing you stopped mentioning "not too high on top" and inferring you have no preference — even though turn 1 stated it explicitly.

These mechanisms act independently but amplify each other, which is why drift accelerates in the second half of the interaction.

Alignment Anchors

The proposed fix is simple: after every N turns, the system automatically re-injects the user's original goals (intent extracted from turn 1) into the prompt as an "anchor." Like periodically reminding the barber: "remember the first time — length on the sides, not too high on top."

No complex detection, no retraining — just persistent reminders pulling attention back to the original objective.

The authors tested anchor intervals of 5, 10, and 20 turns. Every 10 turns worked best: too frequent makes the dialogue mechanical ("I already told you this"), too sparse lets drift accumulate past recovery. With anchors, drift after 120 turns dropped from 23% to <5%.

Honest limitations

1. Participant homogeneity. The 122 participants were largely UC Berkeley students — highly homogeneous in cultural background, education, and technical literacy. Drift speed and patterns may differ substantially across global user populations with different alignment expectations and discourse norms. 2. Anchors in autonomous settings. In the paper, anchors are explicit system injections every 10 turns. Real users driving, cooking, or working may go 50+ turns without providing any anchor. Pure passive injection may not fit autonomous usage patterns. 3. The anchor itself can drift. The mechanism assumes the initial intent is extracted accurately. If that extraction is biased, repeated injection would entrench a wrong understanding — turning the fix into bubble amplification. 4. No failure-mode reporting. The 23%→<5% improvement is an average; the paper does not analyze users who got *worse* under the mechanism (e.g., when the anchor is wrong).

These are boundaries, not refutations — the paper is a solid contribution in characterizing drift, with the mitigation side still wide open.

My take

The paper's real value is not the anchor — that is a natural engineering fix. It is naming the phenomenon.

Naming a fundamental constraint is itself progress in understanding. Alignment Drift does not merely say "AI gets worse in long conversations" — it states a statistical constraint: single-shot alignment systematically decays under sustained interaction. The decay is not a bug, not a coding error, not a data quality issue — it emerges from the basic structure of interaction itself. Every prompt you give and every response you accept slightly changes your alignment with the model — an inherent property of iterative optimization, like every measurement altering a quantum state.

Alignment is not a one-time training step. Alignment is continuous maintenance engineering.

The only question is whether we recognize it as maintenance — or slide to buzz-cut distance without noticing.

References

1. Das, D., Chen, E., Guo, B., Shukla, S. (2026). *Alignment Drift: Quantifying and Mitigating Shifts in Human-AI Alignment During Extended Interactions.* arXiv:2605.16516. 2. Ouyang, L. et al. (2022). *Training Language Models to Follow Instructions with Human Feedback.* NeurIPS. 3. Rafailov, R. et al. (2024). *Direct Preference Optimization: Your Language Model is Secretly a Reward Model.* NeurIPS.