DSPT: Double-Softmax Gradient Suppression for Label-Noise Prompt Tuning in CLIP

Forum topic · 小凯 · 2026-05-04

Summary

This forum post discusses a paper on making prompt tuning for vision-language models like CLIP robust to label noise. Real-world datasets inevitably contain mislabeled samples, and CLIP prompt tuning is highly sensitive to them: noisy labels produce extreme gradient updates that corrupt the model's pretrained visual-language priors and create a self-reinforcing degradation loop. The paper proposes Double-Softmax Prompt Tuning (DSPT), which applies softmax twice to compress extreme probabilities and inherently suppress large gradients caused by incorrect labels. Because the suppression is intrinsic to the double-softmax computation, DSPT requires no noise-rate estimation, no extra networks, and no hyperparameter tuning—changing a single softmax line is enough. The approach is framed philosophically as preserving pretrained knowledge through conservative adaptation, echoing Feynman's principle that simple methods are often the most effective. The post highlights DSPT's plug-and-play nature, negligible computational overhead, and improved performance under label noise, concluding that the sharpest ML tools are often the simplest.

Noise Immunity in Vision-Language Models: Can CLIP Still Learn When Labels Are Wrong?

> Paper: Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models > Authors: Jiayu Li, Jiaxin Qi, Sheng Zhou, Jiaqiang Huang, Xiansheng Hua > arXiv: 2605.00591 | 2026-04-30

1. The Dataset with Wrong Labels

Imagine training a vision-language model (like CLIP) on a dataset where:

A photo of a "dog" is labeled "cat" (wrong!)
A "beach" photo is labeled "mountain" (wrong!)
A "car" photo is labeled "truck" (ambiguous!)

In the real world, label noise is everywhere: human annotation errors, inaccurate automatic labeling, blurry class boundaries, and large-scale datasets where noise is unavoidable.

The problem: CLIP prompt tuning is extremely sensitive to label noise.

2. Why Is CLIP Prompt Tuning Afraid of Noise?

Prompt tuning freezes CLIP's image and text encoders and only learns prompts—a lightweight, efficient way to adapt the model to downstream tasks.

How label noise hurts:

1. Extreme gradient updates – Mislabeled samples produce wrong signals with potentially large magnitudes; a single noisy sample can overwhelm many correct ones. 2. Destruction of pretrained priors – CLIP already has strong visual-language alignment; gradients from noisy labels try to overwrite this knowledge. 3. Self-reinforcing vicious cycle – Noise-corrupted predictions lead to more erroneous updates, compounding the damage.

3. DSPT: Intrinsic Gradient Suppression via Double Softmax

The paper proposes Double-Softmax Prompt Tuning (DSPT).

Core insight: > CLIP already provides a near-optimal initialization. Adaptation should be conservative—especially against the extreme gradients caused by noisy labels.

Technical approach:

1. Double softmax – Instead of one softmax over logits, apply softmax twice. This compresses extreme probabilities and smooths gradients. 2. Intrinsic gradient suppression – No extra hyperparameters, no noise estimation needed. The double softmax itself suppresses large gradients from wrong labels automatically. 3. Preserving pretrained priors – Suppressed gradients mean gentler updates, so pretrained knowledge isn't quickly overwritten while the model stays sensitive to correct signals.

Results: significantly improved performance under label noise, no tuning required, minimal computational overhead.

*It's like giving CLIP noise-canceling headphones: not blocking learning entirely, but lowering the noise (extreme gradients from wrong labels) so the music (correct signals) is clearer.*

4. Why Does "No Hyperparameters" Matter So Much?

Limitations of existing noisy-label methods:

Require noise-rate estimation – Real-world error rates are unknown, and inaccurate estimates break these methods.
Require extra networks – Some train auxiliary "noise detection" networks, adding complexity and cost.
Require tuning – Thresholds, weights, temperature parameters per dataset = high engineering cost.

DSPT's advantages:

Plug-and-play: just replace softmax with double softmax—a one-line change.
Hyperparameter-free: no tuning, no noise-rate knowledge needed, works on any dataset.
Lightweight: nearly zero computational overhead, no extra memory, no impact on inference speed.

5. A Feynman-Style Judgment: Good Methods Are Simple Enough to Not Need Explanation

Feynman said:

> "If you can't explain it to a freshman, you don't really understand it."

In ML engineering:

> "The best methods are often simple. Double softmax is so simple it needs no complex theory—yet it just works. This 'simple but effective' quality embodies engineering wisdom."

DSPT's philosophy: don't add complexity—exploit the properties of existing components. Not a new loss function, not a new architecture, just a change in how softmax is computed, using mathematics itself for denoising.

6. Takeaways

If you're training vision-language models or handling noisy labels, ask yourself:

1. Does my dataset have label noise? 2. Is that noise hurting my prompt tuning? 3. Do I need a complex method, or is there a simpler solution? 4. Are my pretrained priors being eroded by noise?

DSPT reminds us: sometimes the best solution is not adding more components, but rethinking how existing components are used.

Double softmax is so simple it might be overlooked—but that simplicity makes it a powerful weapon against label noise: no tuning, no extra computation, no noise-rate knowledge. In the ML toolbox, the sharpest tools are often the simplest.