Measuring a Model's Hesitation: Belief Trajectory Energy (BTE)
When we use large language models, we only see the answer produced by the final layer. But a Transformer is constantly changing its mind internally: every layer offers a current judgment about "what comes next," corrections accumulate layer by layer, and the final output is merely the endpoint of a long revision process. A team from Fudan University, USTC, and the Shanghai Innovation Institute (paper v1 posted October 4) rescued this discarded revision trajectory and turned it into a measuring tool called Belief Trajectory Energy (BTE) (arXiv 2610.05114).
Method
The approach borrows from logit lens: each layer's intermediate residual state is projected back into vocabulary space through the model's own output head, yielding that layer's next-token prediction distribution—effectively peeking at the model's "current belief" at every layer. The Jensen-Shannon divergence between adjacent layers' beliefs measures how much the model changed its mind at that step. The paper offers an elegant theoretical anchor: JS divergence locally induces the Fisher-Rao geometry of the prediction distributions, so each inter-layer step size is not an arbitrary statistic but a geometrically meaningful "prediction update."
Averaging (pooling) across all tokens and all layers produces a fixed-length depth profile—the model's reasoning "ECG." For a single scalar, the middle quarter of layers performs best (layers 12–19 on a 32-layer Llama): perturbation experiments show that high values in later layers are largely artifacts from perturbations near the output layer, while the mid-section is stable.
What the Ruler Measures
1. Difficulty. Without any downstream fine-tuning, the mid-section BTE scalar alone distinguishes how hard math and reasoning tasks are for a given model—and captures difficulty complementary to human-annotated labels. Some problems humans find easy make the model's revision trajectory struggle, and vice versa.
2. Detection. On real peer reviews from eight conference-years spanning ICLR and NeurIPS 2021–2024, with paper-disjoint train/test splits, a BTE depth profile from Qwen fed to a lightweight classifier reaches a macro AUROC of 0.998. Transfer across generator families drops only to 0.995, and rewriting the review text yields 0.994—it recognizes not writing style but the internal structure of the writing process.
3. Attribution. Across eight different text generators, attribution accuracy is 95.6%.
4. The instrument has a growth history. On the OLMo-2-7B pretraining checkpoint sequence, BTE structure forms progressively during training; targeted fine-tuning can selectively reshape it. In other words, BTE reads out not only properties of the text under test but also what the scoring model itself has learned and seen—the same text shows different hesitation patterns in models trained on different data. The discussion section makes this explicit: trained models are not just problem solvers; they can serve as measurement instruments whose internal prediction dynamics characterize the data flowing through them.
Editorial Notes on Interpretation
Two claims circulating in commentary need calibration:
- "First-ever": the paper does not claim this; it only says "we introduce."
- "Unbiased metric": no such term appears in the paper. The original wording is "principled"—the JS divergence guarantees each revision step has a geometric interpretation, not statistical unbiasedness.
The Bigger Picture
The real selling point: the final output is a lossy interface, and BTE is an attempt to read the lost process information back. This family already includes recovering learning events from weight-update logs and history from KV caches; reading the reasoning process from inter-layer belief trajectories is the newest member. And once the "model as instrument" perspective holds, the boundary between ruler and measured object blurs: you use the model to measure the data, and you are also measuring how the data shaped the model.