Source Paper
| Item | Details | |------|---------| | Title | Scale-Invariant Repulsion for Contrastive Learning | | Authors | Jiawei Zhao, Simon S. Du, Jason D. Lee (University of Washington / Princeton) | | arXiv | 2605.16421 (cs.LG, stat.ML) | | Date | May 2026 | | Core contribution | Identifies the fundamental limitation of the temperature parameter in contrastive learning—fixed temperature creates scale-dependent repulsion leading to intra-class collapse—and proposes scale-invariant repulsion that adapts to embedding scale instead of a fixed softmax temperature | | Link | https://arxiv.org/abs/2605.16421 |
1. The Problem Nobody Asked About
In contrastive learning, an encoder assigns each data point a position in high-dimensional space—images, text, graph nodes—and a loss function pulls same-class items together and pushes different items apart. Think of marbles in a bowl: repulsion spreads them evenly.
But almost nobody asks: how strong should the repulsive force be? Should it scale with embedding distance?
This paper's answer: no. The repulsive force should adapt to the intrinsic scale of the embeddings. Otherwise, contrastive learning develops a strange but widespread problem—intra-class collapse.
2. Temperature Is a Rigid Ruler—and the World Isn't Rigid
Nearly all modern contrastive methods—SimCLR, SupCon, CLIP, DINO—contain a temperature parameter (τ), a fixed constant controlling repulsion strength. Low temperature pushes the most similar negatives hardest (approaching hardmax); high temperature pushes everything uniformly.
The paper's insight: because embedding scale changes dynamically during training—small at initialization, surging mid-training, stabilizing at convergence—while temperature is a constant, there is a fundamental mismatch. The failure modes:
- Early training: embedding scale is small, all distances are similar. Fixed temperature pushes all negatives apart too uniformly; the encoder learns nothing.
- Mid-training: embedding scale inflates. Softmax repulsion decays exponentially with distance, so distant negatives receive almost no repulsion—distant negative clusters emerge.
- Late training: embedding scale contracts. The fixed temperature suddenly becomes too strong, violently pushing samples apart *within* classes, triggering intra-class collapse.
3. Intra-Class Collapse—Your Model Falls Apart Inside Correct Categories
We usually worry about inter-class confusion: a cat embedded near dogs. But this paper reports the opposite failure—the model separates cats from dogs correctly while destroying the internal structure of the cat class.
An orange cat and a black cat should be close (same class) but not identical. Under fixed temperature, once the embedding scale falls below a critical threshold, repulsion starts a positive feedback loop within a class: push too hard → samples spread → the encoder learns meaningless intra-class partitions → backpropagation reinforces them → more pushing. The class shatters into fragments—cat A ends up farther from cat B than from dog C.
The paper visualizes this on a toy task, and a linear classifier trained on the collapsed distribution cannot see the problem.
4. The Fix: Bind Repulsion to Embedding Scale
The principle: apply a repulsive potential whose strength adjusts in real time to the current embedding scale. Concretely, the paper replaces fixed softmax gradients with normalized gradients, keeping the repulsion norm consistent at all distances. Not "push less when far away"—but "push with a constant unit magnitude; only the direction changes with distance."
The result: repulsion doesn't vanish when scale inflates, doesn't explode when scale contracts, and intra-class structure is protected. The authors call this scale-invariant repulsion.
5. Results—and Honest Caveats
On standard benchmarks (CIFAR-10, CIFAR-100, ImageNet-100), linear-probe scores improve by 0.5–2%, growing with training epochs. Significant, but not revolutionary.
Honest caveats:
1. This is primarily a theoretical contribution—identifying a fundamental flaw and proposing a principled fix, not a new SOTA method. The predicted failure mode (intra-class collapse at high epochs) genuinely occurs, which is the real value. 2. The method requires tracking second-order statistics of embedding scale during training, adding measurable memory and compute overhead in large-batch training. The paper doesn't quantify this in detail. 3. No validation on very large models—experiments use ResNet-18/50. The claim that scale-invariant repulsion should apply to any fixed-temperature contrastive method is not empirically supported on the small-to-large transfer path.
My Take
The core value is not engineering improvement but a beautiful observation. While contrastive learning research focuses on positive/negative pair selection and augmentation strategies, nobody asked: when embedding scale changes, is the repulsion still reasonable?
Every data point in contrastive learning doesn't just represent content—it defines the geometry of the entire space through repulsion. A fixed temperature imposing the same push at every scale is geometrically unreasonable. Just as you cannot measure bacteria and galaxies with the same ruler, you cannot repel embeddings at all scales with one fixed temperature.
References
1. Zhao, J., Du, S.S., Lee, J.D. (2026). Scale-Invariant Repulsion for Contrastive Learning. arXiv:2605.16421. 2. Chen, T. et al. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML. 3. Khosla, P. et al. (2020). Supervised Contrastive Learning. NeurIPS. 4. He, K. et al. (2020). Momentum Contrast for Unsupervised Visual Representation Learning. CVPR.