User Model Extraction via Belief Self-Distillation: Reading and Writing LLM Beliefs About Users

Forum topic · 小凯 · 2026-09-29

Summary

This forum post introduces a recent arXiv paper (2609.31603) proposing Belief Self-Distillation (BSD), a unified read-write framework for inspecting and causally manipulating the implicit user beliefs that large language models form during conversations. BSD learns a compact user representation that can be decoded (read) and written back into the model, bridging linear probing and causal probing. The frozen LLM acts as its own teacher, distilling beliefs from natural dialogues without external annotations. Experiments across multiple model families show BSD faithfully recovers user beliefs and achieves substantially stronger interventions than matched hidden-state steering. Notably, the paper finds that model refusal behavior depends not only on the request itself but also on the model's inferred user intent: fixing the request and changing only the belief alters refusal. The authors also reveal a cross-model convergence: independently trained LLMs share a common geometric structure in their user representations. These findings show implicit user models are readable and causally writable internal states, with direct implications for AI safety.

Paper Overview

Field: NLP Authors: Ali Holmov, Yiran Huang, Kirill Bykov, Zeynep Akata Published: 2026-09-25 arXiv: 2609.31603

Abstract (Full Translation)

Large language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate. We introduce Belief Self-Distillation (BSD), a unified read-write framework that bridges linear and causal probing by learning a compact user representation that can be both decoded and written back into the model. The frozen LLM acts as its own teacher, distilling beliefs from natural conversations without external annotations. Unlike conventional probing, BSD isolates not only information present in activations, but a state whose causal role can be directly tested. Across multiple model families, BSD faithfully recovers user beliefs and enables substantially stronger interventions than matched hidden-state steering. Critically, we show that model refusal depends not only on the request itself but on the model's belief about the user's intent: fixing the request and changing only this belief changes refusal behavior. We further reveal a surprising cross-model regularity: independently trained LLMs converge to a shared geometric structure in their user representations. Together, these results show that implicit user models are readable and causally writable internal states, with direct implications for AI safety—governing how models modulate safety decisions based on who they believe they are talking to.

Key Takeaways

Unified read-write framework: BSD bridges linear and causal probing via a compact, decodable, and writable user representation.
Self-distillation: The frozen LLM serves as its own teacher, requiring no external annotations.
Causal validation: BSD isolates states whose causal roles can be directly tested, not just information present in activations.
Safety relevance: Refusal behavior depends on inferred user intent, not just the request content.
Cross-model convergence: Independently trained LLMs share a common geometry in user representations.