Prompt-Induced Score Variance in Zero-Shot VLM Safety Classification

> Paper: Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification > Authors: Charles Weng, Dingwen Li, Alexander Martin > arXiv: 2605.00326 | 2026-04-29

The Phenomenon: Same Image, Different Safety Scores

The post opens with a striking scenario: the same image is checked for unsafe content (violence) using two semantically equivalent prompts:

• Prompt 1: "Does this image contain violent content?" → unsafe probability 0.85
• Prompt 2: "Does this image have violent elements?" → unsafe probability 0.45

Same image, equivalent wording — yet a huge gap in scores. The author frames this as a reliability problem: the safety classification depends on prompt phrasing, so it cannot be trusted.

Key Findings of the Paper

The core finding: zero-shot VLM safety classifiers are extremely sensitive to semantically equivalent prompt reformulations, and prompt-level variance strongly correlates with error rates.

1. Semantically equivalent prompts yield different scores — even with the label position fixed, equivalent prompts produce materially different unsafe probabilities across multiple VLM families. 2. Variance is a fragility indicator — high cross-prompt variance implies prompt-level inconsistency and higher error rates, making variance a reliable diagnostic of fragility. 3. Training-free detection — by evaluating prompt variance, unreliable samples can be identified without any additional training, improving safety. 4. Consistent across benchmarks — the phenomenon holds across multiple multimodal safety benchmarks and VLM families; it is a systemic issue, not an isolated one.

The author offers an analogy: an unreliable classifier is like a security scanner that alarms for "please check this bag" but not for "see what's inside this bag."

Why This Matters

Consequences of unreliable safety classification:

• Missed detections: harmful content labeled safe and allowed to spread, harming users.
• False positives: benign content flagged and removed, restricting legitimate speech.
• Distrust: real safety status becomes unknowable when it depends on phrasing.

Practical uses of prompt variance:

• Diagnostic tool: high-variance samples can be routed to human review.
• Improvement directions: design more robust prompts and use multi-prompt ensembles to reduce variance and improve reliability.

Takeaway

The author invokes Feynman-style reasoning: a result that depends on how it is measured indicates a problem with the measurement itself.

> If the "safety" of the same content depends on how you ask, the classifier isn't judging — it's guessing. A reliable safety system should give consistent answers to semantically equivalent questions.

Questions practitioners should ask:

1. Is my system sensitive to prompt variations? 2. Have I evaluated cross-prompt variance? 3. Are high-variance samples flagged as unreliable? 4. Is my safety classification robust enough?

Closing thought: a safety system's reliability cannot rest on a lucky prompt. When a classifier gives consistent answers regardless of phrasing, it graduates from "prompt gambler" to "trustworthy guardian" — and on the scales of safety, consistency matters more than raw accuracy.