Paper Slam 4/24: When Text Hijacks Vision, When Evaluation Splits Time

*Two papers, one question: is what you measure really what you think you're measuring?*

This post reviews two papers that attack the same blind spot from different fields: a step everyone treats as a neutral constant turns out to be a hidden variable that quietly rewrites conclusions.

• Paper 1: Khayatan et al., *"When Prompts Override Vision"* (arXiv:2604.21911) — how textual instructions induce hallucination in large vision-language models (LVLMs).
• Paper 2: Filat et al., *"Temporal Taskification in Streaming Continual Learning"* (arXiv:2604.21930) — how time-based partitioning in continual learning evaluation is a structural variable, not neutral preprocessing.

Key points

Paper 1: HalluScope — dissecting LVLM hallucination

Why existing benchmarks fall short

POPE, CHAIR, and HallusionBench tell you *that* a model hallucinated, but not *why*. HalluScope distinguishes three causes:

1. Perceptual failure — the vision backbone genuinely can't resolve the object. 2. Learned co-occurrence prior — the model answers from statistics ("tables usually have chairs") rather than the image. 3. Presupposition-induced hallucination — the question presupposes an object's existence ("What logo is on the batter's cap?" when there is no cap), bypassing vision entirely.

Benchmark construction

• 3,000 semantically diverse COCO images selected via sentence-transformer embeddings + K-Center Greedy.
• Objects detected with Florence-2-large, verified with Grounding-DINO and Qwen2-VL-7B review.
• For each image, an adversarial object is chosen: absent from the image but highly expected by co-occurrence statistics, found via Pointwise Mutual Information (PMI). Example: an image containing a bat, ball, glove, and person — but no baseball cap — makes "baseball cap" the adversarial object.
• Four question types per image:
• Rec_pos (present object), Rec_rnd (random absent object), Rec_adv (adversarial absent object), AdP (question presupposing the adversarial object exists).

Headline results

Across InternVL3-5-8B, Molmo-7B-D, Qwen2-VL-7B, Qwen2.5-VL-7B, and LLaVA-1.5-7B:

• Rec_pos / Rec_rnd accuracy is mostly above 85–90% — vision itself is largely fine.
• Rec_adv drops 8–37% relative to Rec_pos/Rec_rnd — models trust learned statistics over their eyes.
• AdP drops 25–85% relative to Rec_pos/Rec_rnd. On the Instances subset, LLaVA-1.5-7B goes 94.8% → 6.3%. The prompt's presupposition effectively short-circuits visual input.

The authors' conclusion: improving LVLM reliability is increasingly an alignment problem between modalities, not a perception problem.

HalluVL-DPO: weighted preference optimization

• Training data: 27.4K synthetic Stable Diffusion 3 images, 100K+ queries with verified object presence/absence; questions cover presupposition types (TPQ/CPQ), object existence, and detailed descriptions.
• Chosen/rejected pairs are generated via hint augmentation. To avoid reward hacking (overly short answers), three strategies are used: Contrastive Hint Augmentation, Post-Hoc Hallucination Injection, and Model-Assistant Answer Inversion.
• Preference pairs are weighted by semantic gap; the most informative class (chosen correctly rejects a hallucination, rejected fully accepts it) alone lifts AdP from 5.85 to 81.05.

Results:

• LLaVA-1.5-7B: HalluScope AdP 5.85 → 83.54 (DPO) → 84.65 (weighted DPO); Rec_adv 58.5 → 81.28; CP-Bench CPQ 0.54 → 71.22.
• Qwen2-VL-7B: AdP 57.96 → 82.83; CP-Bench CPQ 19.19 → 75.00.
• General benchmarks (MME, ScienceQA, MM-Vet, HallusionBench, CHAIR) hold or improve; only a minor POPE dip (87.55 → 85.69 on LLaVA).

Paper 2: Temporal Taskification — the split is a variable

The hidden assumption

Streaming continual learning (CL) requires cutting a continuous data stream into discrete "tasks." Standard practice: fixed-length temporal windows. Almost nobody asks whether the cut itself changes the conclusions. Filat et al. show it does.

Setup

• Data: CESNET-Timeseries24 — 40 weeks of real ISP network traffic; 100 highest-density IPs, 10-minute aggregation; task = predict next-step avg_duration.
• Everything fixed except the split: same stream, same Transformer, same training protocol (AdamW, lr=1e-4, batch=256). Windows: 9, 30, or 44 days (all ≡ 2 mod 7 for weekday alignment).

Structural analysis framework

• Plasticity Profile: distribution of distances d(P_k, P_{k+1}) between adjacent tasks — are transitions mild or abrupt?
• Stability Profile: distribution of distances between non-adjacent tasks — is old knowledge useful in the future?
• Both are one-dimensional distributions (Wasserstein distance), so splits with different task counts remain comparable. Profile distance combines them into a single metric.
• Boundary-Profile Sensitivity (BPS): perturbing internal boundaries by ±1 day and measuring average profile distance. High BPS = fragile split; low BPS = robust split.

Findings: the 9-day split has the highest BPS (most fragile); 44-day is the most robust; 9-day vs 44-day shows the largest profile distance. Results are invariant to shifting all boundaries by 2 days.

Downstream impact

Four CL methods (Continual Finetuning, Experience Replay, EWC, LwF) evaluated on MSE, Forgetting, and Backward Transfer:

• Continual Finetuning average MSE (×10³): 3.26 (9-day) / 1.88 (30-day) / 8.40 (44-day) — several-fold differences from the split alone.
• Experience Replay Forgetting: −1.60 / −0.24 / −5.22 across splits.
• LwF Backward Transfer flips sign: 0.09 / 0.42 / −0.29 across splits.

Method rankings are not taskification-invariant: a conclusion like "Replay beats finetuning" can reverse under a different split. Structure (noisy 9-day distance matrices, high BPS) predicts unstable CL behavior; smooth 44-day structure predicts consistent behavior.

The common thread

| Dimension | 2604.21911 (HalluScope) | 2604.21930 (Taskification) | |---|---|---| | Field | Multimodal LVLMs | Time series + continual learning | | Hidden variable | Question wording/presupposition | Temporal partition scheme | | Approach | Diagnostic benchmark + preference optimization | Model-free structural analysis + sensitivity metric |

Both papers expose the same fallacy: treating a tool's *form* as neutral when the form is part of the content. Both prioritize diagnosis before treatment — HalluScope dissects *which kind* of hallucination; profile distance and BPS flag fragile evaluation designs *before any model is trained*. Both release variance that prior narratives had compressed away.

Closing thought

Neither result is a finish line: HalluScope covers COCO-style images, not medical, driving, or satellite domains with their own "adversarial objects"; CESNET is one stream among many (finance, healthcare, climate). The shared principle is Feynman's: *"The first principle is that you must not fool yourself — and you are the easiest person to fool."*

Appendix: core numbers

HalluScope (LLaVA-1.5-7B, Instances subset)

| Metric | Original | HalluVL-DPO | HalluVL-DPO_w | |---|---|---|---| | Rec_pos | 94.85 | 87.35 | 84.30 | | Rec_rnd | 89.17 | 94.03 | 95.18 | | Rec_adv | 58.50 | 76.50 | 81.28 | | AdP | 5.85 | 83.54 | 84.65 |

CESNET-Timeseries24 (Continual Finetuning)

| Taskification | Avg MSE (×10³) | Forgetting | Backward Transfer | |---|---|---|---| | 9-day | 3.26 | -5.13 | 0.00 | | 30-day | 1.88 | -3.72 | -0.01 | | 44-day | 8.40 | -6.45 | 0.01 |

BPS comparison: 9-day = highest (most fragile); 30-day = medium; 44-day = lowest (most robust).

*Paper Slam 4/24 | Paper 1: arXiv:2604.21911 | Paper 2: arXiv:2604.21930*