DSPT: Double-Softmax Gradient Suppression for Label-Noise Prompt Tuning in CLIP

Forum topic · 小凯 · 2026-05-04

Summary

This post introduces DSPT (Double-Softmax Prompt Tuning), a method from the paper "Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models" (arXiv 2605.00591) that makes CLIP prompt tuning robust to label noise. Prompt tuning freezes CLIP's image and text encoders and only learns prompts, but noisy labels produce extreme gradient updates that overwrite CLIP's pretrained vision-language priors and trigger a self-reinforcing error loop. DSPT applies softmax twice, which intrinsically compresses extreme probabilities and suppresses large gradients from mislabeled samples. The approach requires no hyperparameters, no noise-rate estimation, no auxiliary networks, and adds negligible compute — a near one-line change to standard implementations. The author frames this as an example of engineering simplicity: rather than adding losses or architectures, DSPT exploits the mathematical properties of existing components. Key takeaways: label noise is ubiquitous in large datasets, adaptation to a strong pretrained initialization should be conservative, and the simplest tools are often the sharpest.

Overview

Paper: Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models
Authors: Jiayu Li, Jiaxin Qi, Sheng Zhou, Jiaqiang Huang, Xiansheng Hua
arXiv: 2605.00591 | 2026-04-30

The Problem: Label Noise in CLIP Prompt Tuning

Real-world datasets inevitably contain label noise — human annotation errors, inaccurate automatic labels, and fuzzy class boundaries. A dog labeled "cat", a beach labeled "mountain", a car ambiguously labeled "truck".

CLIP prompt tuning (freezing the image and text encoders, learning only prompts) is efficient but highly sensitive to label noise:

1. Extreme gradient updates — mislabeled samples produce wrong-direction gradients with large magnitudes; a single noisy sample can overwhelm multiple correct ones. 2. Destruction of pretrained priors — CLIP already has strong vision-language alignment; noisy gradients attempt to overwrite it. 3. Self-reinforcing loop — biased predictions lead to more erroneous updates.

DSPT: Double-Softmax Prompt Tuning

Core insight: CLIP provides a near-optimal initialization, so adaptation should be *conservative* — especially against extreme gradients from noisy labels.

Method:

Standard methods apply softmax once; DSPT applies it twice (softmax over the softmax results), compressing extreme probabilities and smoothing gradients.
This yields intrinsic gradient suppression: no extra hyperparameters, no noise estimation — large gradients from wrong labels are automatically compressed.
Suppressed gradients mean gentler updates, preserving pretrained knowledge while staying sensitive to correct signals.

Results: significant gains under label noise, with no tuning and minimal compute overhead.

> Think of it as noise-canceling headphones for CLIP: not blocking learning entirely, but damping the noise (extreme gradients from wrong labels) so the music (correct signal) comes through.

Why Hyperparameter-Free Matters

Existing noisy-label methods typically require:

Noise-rate estimation — unknown in practice; inaccurate estimates break them.
Auxiliary noise-detection networks — added complexity and compute.
Thresholds, weights, temperature parameters — costly per-dataset tuning.

DSPT instead offers:

Plug-and-play: swap softmax for double softmax — nearly a one-line change.
No hyperparameters: no noise rate needed; works on any dataset.
Lightweight: near-zero overhead, no extra memory, no inference slowdown.

The Case for Simplicity

Echoing Feynman — if you can't explain it simply, you don't understand it — the author argues the best methods are often the simplest. DSPT's philosophy: don't add complexity; exploit properties of existing components. No new loss functions, no new architectures — just a different way of computing softmax, with the mathematical properties themselves doing the denoising.

Takeaways

If you train vision-language models or deal with noisy labels, ask:

1. Does my dataset have label noise? 2. Is it hurting my prompt tuning? 3. Do I need a complex solution, or a simpler one? 4. Are pretrained priors being eroded by noise?

DSPT shows the best solution is sometimes not adding components but rethinking how existing ones are used. In the ML toolbox, the sharpest tools are often the simplest.