From Jacobian Spectra to Boltzmann Distributions: Geometric Diagnosis and Correction of Diffusion Model Hallucinations
Structural hallucinations in diffusion models—such as anatomically implausible hands—have long been attributed to interpolation behavior in the training data distribution. A joint team from Warsaw University of Technology and Harvard Medical School (arXiv:2605.05026, May 2026) reframes this failure mode as geometric instability on the model-induced manifold. By establishing precise mathematical links between the Jacobian singular value spectrum, local intrinsic dimension (LID), and hallucinations, the authors propose Intrinsic Quenching (IQ)—a sampling-time correction mechanism grounded in a Boltzmann distribution perspective.
1. Mechanism: Geometric Mapping from Noise Space to Data Manifold
The generator \(G_\theta\) maps noise space \(\mathbb{R}^n\) to a data manifold \(M_\theta \subset \mathbb{R}^n\). Its local behavior at \(x_1\) is characterized by the Jacobian \(J = \nabla_{x_1} G_\theta(x_1)\) with SVD \(J = U \Sigma V^\top\).
Under the manifold hypothesis, real data points should exhibit spectral truncation: \(\sigma_i \approx 0\) for \(i > d\), where \(d \ll n\) is the intrinsic dimension. The team proves a quantitative relation between local manifold instability (LMI) and the spectrum:
Instability is thus proportional to the sum of squared singular values: inflated spectral energy amplifies small perturbations into significant output deviations. LID acts as an "effective dimension counter," marking the transition between significant and noise-level singular values.
2. Key Theory: DSM Loss Equals LID
A central theoretical finding (Proposition 2):
Geometrically, the DSM loss measures how well the model projects noise back onto the manifold, which depends on local manifold complexity—i.e., LID. Theorem 1 further shows LMI is strictly bottlenecked by LID: reducing LID linearly lowers the instability upper bound, providing the theoretical guarantee for IQ.
3. The Correction: Boltzmann Re-weighting with Dynamic Filtering
IQ defines an energy \(E(x_t) = \text{LID}_\theta(\hat{x}^\theta_0(x_t))\) and a corrected score:
Theorem 2 shows the corrected reverse process samples from a Boltzmann distribution \(p_t^{\theta,\lambda_t}(x_t) \propto p_t^\theta(x_t) \cdot \mathbb{E}[\exp(-\lambda_t \cdot \text{LID}_\theta(x_0))]\), performing posterior contraction: probability mass shifts from high-LID to low-LID regions.
Two design choices ensure stability and selectivity:
- Dynamic step-size scaling: \(\lambda_t\) is normalized by the natural update magnitude, keeping correction proportional to the state's evolution.
- Threshold filtering: correction activates only when LID exceeds a threshold \(q\), preserving tail diversity of rare-but-valid samples.
- Efficiency: per-step SVD needed; approximate LID estimators (randomized SVD, spectral methods) are a goal
- Threshold tuning: dataset-specific \(q\); a theoretical mapping to intrinsic dimension is open
- Hallucination types: structural only; semantic-level hallucinations unaddressed
- Model classes: diffusion-specific so far; GAN/flow/autoregressive applicability unverified
- Time steps: approximations assume small \(t\); large-\(t\) LID estimation remains open
4. Experiments
A two-stage human annotation protocol (calibration on 128 real images, then independent structural-hallucination labeling) was used. Hallucination Ratio (HR) results:
| Dataset | Baseline HR↓ | IQ HR↓ | Relative drop | Baseline UP↑ | IQ UP↑ | |---|---|---|---|---|---| | 11kHands | 29.3% | 9.0% | 69.3% | 39.8% | 68.0% | | MNIST | 37.3% | 10.2% | 72.7% | — | — | | SimpleShapes | 25.8% | 9.4% | 63.6% | — | — | | GaussianGrid | 20.2% | 8.9% | 55.9% | — | — | | FFHQ | 8.2% | 4.2% | 48.8% | 45.3% | 46.1% | | AFHQV2 | 6.9% | 5.9% | 14.5% | 41.8% | 42.6% |
On RSNA sparse-view CT reconstruction, IQ kept PSNR/SSIM/LPIPS essentially unchanged while improving ResNet50 observer diagnostics: mAP 0.27 → 0.31 (+14.8%), mROC 0.85 → 0.86—reducing anatomical artifacts that could mislead diagnosis.
5. Comparison with Baselines
Unlike discriminator guidance (DG), attention-map methods (AAM), or robust sampling schemes (RODSCAS/RODSSAS), IQ operates at the differential-geometric/manifold level rather than in probability or realization space. Low-probability samples may be rare-but-valid, whereas geometrically unstable samples are almost certainly hallucinations. Theorems 1–2 provide mathematical guarantees beyond empirical observation. The main limitation is computational cost (per-step SVD of the Jacobian).
6. Limitations and Open Directions
7. Takeaway
The work advances hallucination analysis from statistical description to geometric diagnosis, revealing that the standard training loss already implicitly encodes manifold geometry. Practically, it suggests monitoring the LID distribution of generated samples alongside FID/IS: a right-skewed LID histogram may signal subtle structural hallucinations even when FID looks healthy.
Paper metadata (verified): "Local Intrinsic Dimension Unveils Hallucinations in Diffusion Models" by Bartlomiej Sobieski, Matthew Tivnan, Dawid Płudowski, Michał Jan Włodarczyk, Pengfei Jin, Przemyslaw Biecek, Quanzheng Li (Warsaw University of Technology; University of Warsaw; Massachusetts General Hospital; Harvard Medical School), arXiv:2605.05026, 2026-05-06.