Optimization Dynamics Imprint Semantic Specificity in Contrastive Embeddings (arXiv 2507.00009)

Forum topic · 小凯 · 2026-07-01

Summary

This paper (arXiv 2507.00009, by Ziwei Su, Junyu Ren, and Victor Veitch) explains why embedding norms in contrastive models carry semantic information even when scale-invariant losses and cosine similarity ignore magnitude. Empirically, embedding lengths correlate with concept specificity, token frequency, and human uncertainty. The authors develop a formal theoretical framework based on optimization dynamics and derive an analytic formula showing that embedding length naturally encodes these semantic properties as a byproduct of training. They further demonstrate that these norm-based signals can serve as 'free' calibration tools for specific models and retrieval tasks, providing a grounded theoretical explanation for previously heuristic observations about the informativeness of discarded embedding norms.

Paper Overview

Field: Representation Learning
Authors: Ziwei Su, Junyu Ren, Victor Veitch
Published: 2026-07-01
arXiv: 2507.00009

Abstract

Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty.

In this work, the authors provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, they derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. They also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks, providing a grounded explanation for previously heuristic observations.

Key Takeaways

Embedding norms, though ignored by cosine similarity, correlate with semantic attributes (concept specificity, token frequency, human uncertainty).
Optimization dynamics analysis yields an analytic formula showing how embedding length encodes semantic information during training.
Norm-based signals can act as free calibration tools for models and retrieval tasks.