Paper Overview
- Field: Representation learning
- Authors: Ziwei Su, Junyu Ren, Victor Veitch
- Published: 2026-07-01
- arXiv: 2507.00009
- Embedding norms carry semantic information even when the training loss is scale-invariant.
- The correlation arises from optimization dynamics, formalized via an analytic formula.
- Norms can act as cost-free calibration signals for retrieval tasks.
Abstract (translated)
Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these 'discarded' norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty. In this work, the authors provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, they derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. They also show how this gives rise to signals that can serve as 'free' calibration tools in specific models and retrieval tasks, providing a grounded explanation for previously heuristic observations.
Key Takeaways
*Auto-collected on 2026-07-01*