Overview
- Paper: Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models
- Authors: Jiayu Li, Jiaxin Qi, Sheng Zhou, Jiaqiang Huang, Xiansheng Hua
- arXiv: 2605.00591 | 2026-04-30
- Standard methods apply softmax once; DSPT applies it twice (softmax over the softmax results), compressing extreme probabilities and smoothing gradients.
- This yields intrinsic gradient suppression: no extra hyperparameters, no noise estimation — large gradients from wrong labels are automatically compressed.
- Suppressed gradients mean gentler updates, preserving pretrained knowledge while staying sensitive to correct signals.
- Noise-rate estimation — unknown in practice; inaccurate estimates break them.
- Auxiliary noise-detection networks — added complexity and compute.
- Thresholds, weights, temperature parameters — costly per-dataset tuning.
- Plug-and-play: swap softmax for double softmax — nearly a one-line change.
- No hyperparameters: no noise rate needed; works on any dataset.
- Lightweight: near-zero overhead, no extra memory, no inference slowdown.
The Problem: Label Noise in CLIP Prompt Tuning
Real-world datasets inevitably contain label noise — human annotation errors, inaccurate automatic labels, and fuzzy class boundaries. A dog labeled "cat", a beach labeled "mountain", a car ambiguously labeled "truck".
CLIP prompt tuning (freezing the image and text encoders, learning only prompts) is efficient but highly sensitive to label noise:
1. Extreme gradient updates — mislabeled samples produce wrong-direction gradients with large magnitudes; a single noisy sample can overwhelm multiple correct ones. 2. Destruction of pretrained priors — CLIP already has strong vision-language alignment; noisy gradients attempt to overwrite it. 3. Self-reinforcing loop — biased predictions lead to more erroneous updates.
DSPT: Double-Softmax Prompt Tuning
Core insight: CLIP provides a near-optimal initialization, so adaptation should be *conservative* — especially against extreme gradients from noisy labels.
Method:
Results: significant gains under label noise, with no tuning and minimal compute overhead.
> Think of it as noise-canceling headphones for CLIP: not blocking learning entirely, but damping the noise (extreme gradients from wrong labels) so the music (correct signal) comes through.
Why Hyperparameter-Free Matters
Existing noisy-label methods typically require:
DSPT instead offers:
The Case for Simplicity
Echoing Feynman — if you can't explain it simply, you don't understand it — the author argues the best methods are often the simplest. DSPT's philosophy: don't add complexity; exploit properties of existing components. No new loss functions, no new architectures — just a different way of computing softmax, with the mathematical properties themselves doing the denoising.
Takeaways
If you train vision-language models or deal with noisy labels, ask:
1. Does my dataset have label noise? 2. Is it hurting my prompt tuning? 3. Do I need a complex solution, or a simpler one? 4. Are pretrained priors being eroded by noise?
DSPT shows the best solution is sometimes not adding components but rethinking how existing ones are used. In the ML toolbox, the sharpest tools are often the simplest.