ViT's Pixel-Level Politics: Every Prediction Depends on Where the Patch Grid Falls

Forum topic · 小凯 · 2026-06-12

Summary

Vision Transformers split images into fixed-size patches, and the starting offset of that patch grid—a 'phase'—can silently change per-pixel predictions, especially near semantic boundaries. A paper by Oğuzhan Ercan formalizes this grid phase as a nuisance variable and proposes Uniform Phase Marginalization: run K forward passes with different grid offsets (K=4 default), inverse-align outputs to the original pixel grid, and average log-probabilities. The method is training-free and works purely as an inference wrapper. On dense prediction benchmarks it consistently improves results: +0.82 mIoU on GTA5→Cityscapes, +1.39 mIoU on ADE20K with DINOv2, lower RMSE on NYU Depth v2, and +3.75 on HPatches matching. Under equal compute, it beats standard translation-based TTA. Boundary-region metrics improve most, confirming phase instability concentrates at semantic edges. K=4 saturates gains, making it the recommended cost/performance point. Limitations include 4× inference latency and evaluation only on DINOv2/DINOv3 backbones.

ViT's 'Pixel-Level Politics': Every Prediction Depends on Where the Patch Grid Falls

> The same pixel in the same image can receive two different predictions from a Vision Transformer. It's not a data problem or a model problem—it's the 'phase' of the patch grid created when ViT slices the image. A new paper offers a zero-training-cost fix.

1. The Problem: One Image, One Pixel, Two Answers

Take a street scene. A pixel on the road is predicted as "road." Shift the image 8 pixels to the right and feed it back: the same physical pixel is now predicted as "sidewalk."

This isn't the model being wrong—it's the ViT patch grid shifting.

Vision Transformers cut images into fixed patches (e.g., 16×16). The grid has a starting offset relative to the top-left corner—the phase. Depending on which patch a pixel lands in, its patch neighbors change, and after self-attention and MLP transformations, the output can differ completely.

The effect is strongest near semantic boundaries: whether a pixel becomes "road" or "sidewalk" depends on its patch neighbors, which change with phase. It's like gerrymandering—the same voter, assigned to a different district, yields a different result. The patch grid is the ViT's electoral map.

2. Phase Marginalization: A Zero-Training Fix

2.1 Core Idea

The author, Oğuzhan Ercan, formalizes phase as a nuisance variable and performs post-hoc marginalization:

1. Sample K patch-grid phase offsets (default K=4) 2. Run a full forward pass per phase 3. Inverse-align each output back to the original pixel coordinates 4. Average the aligned log-probabilities

No model modification. No retraining. A pure inference-time wrapper.

2.2 The Math

Phase definition:

\phi = (d_x, d_y), \quad d_x, d_y \in \{0, \ldots, P-1\}

where P is patch size (default 16). The shift operator $S_\phi$ uses reflective padding and cropping. Model output: $z_\phi = h(f_\theta(S_\phi(x)))$ . Inverse alignment $A_\phi^{-1}$ reshapes outputs to a grid, bilinearly upsamples, and crops back.

Marginalization:

\bar{z}(p) = \frac{1}{K}\sum_{k=1}^{K} A_{\phi_k}^{-1}(z_{\phi_k})(p)

The standard phase set (K=4, even P):

\Phi_4 = \{(0,0), (0, P/2), (P/2, 0), (P/2, P/2)\}

sampling all four quadrants—a minimal complete coverage of phase space.

2.3 Difference from Generic TTA

| Dimension | Generic TTA | Phase Marginalization | |------|----------|----------------------| | Sampled object | Flips, crops, translations | The patch grid's phase itself | | Alignment | Inverse transform of prediction | Exact inverse alignment to original pixel grid | | Transform definition | Arbitrary semantic augmentations | Fixed offsets defined by patch size | | Compute | Same (K forwards) | Same, but more effective |

Key distinction: generic TTA moves the whole image, changing patch *content*. Phase Marginalization only moves the grid—each pixel is regrouped into different patches, but patch content is unchanged. It's a pure correction of an architectural quantization effect.

3. Results: Four Tasks, Consistent Gains

3.1 Main Results (Uniform Phase Marginalization, K=4)

| Task | Dataset | Backbone | Baseline | K=4 | Gain | |------|--------|------|------|-----|------| | Semantic segmentation | GTA5 → Cityscapes | DINOv3/F | 51.94 | 52.76 | +0.82 | | Semantic segmentation | SYNTHIA → Cityscapes | DINOv3 | 36.00 | 36.88 | +0.88 | | Semantic segmentation | SYNTHIA → Cityscapes | DINOv2 | 32.90 | 33.77 | +0.87 | | Semantic segmentation | ADE20K | DINOv3 | 48.82 | 49.58 | +0.76 | | Semantic segmentation | ADE20K | DINOv2 | 43.77 | 45.16 | +1.39 | | Depth estimation | NYU Depth v2 | DINOv3 | 0.6506 | 0.6277 | -0.0229 RMSE | | Local feature matching | HPatches | DINOv3 | 39.02 | 39.98 | +0.96 | | Local feature matching | HPatches | DINOv2 | 24.43 | 28.18 | +3.75 |

Effective both cross-domain and in-domain. The largest gain is DINOv2 on ADE20K (+1.39 mIoU), suggesting older backbones suffer more phase instability; DINOv3 may already partially mitigate it.

3.2 Head-to-Head with Generic TTA

| Method | Forwards | mIoU | Latency ms/img | |------|---------|------|------------| | Single pass (K=1) | 1 | 52.90 | 20.77 | | Random subpatch-shift TTA | 4 | 53.00 | 83.60 | | Integer-shift TTA | 4 | 53.22 | 82.80 | | Uniform Phase Marginalization | 4 | 53.53 | 87.52 |

+0.31 mIoU over translation TTA at the same compute. Not "more compute for better results"—the same compute, spent on the right variable.

3.3 Scaling K: K=4 Is the Sweet Spot

| K | Forwards | mIoU | Latency | |---|---------|------|--------| | 1 | 1 | 52.90 | 20.77 | | 2 | 2 | 53.22 | 44.08 | | 4 | 4 | 53.53 | 87.52 | | 8 | 8 | 53.53 | 176.39 | | 16 | 16 | 53.58 | 347.16 |

K=4 → K=8 yields zero improvement (53.53 vs 53.53)—the four quadrant phases already cover the critical variation. This is the basis for recommending K=4 as default.

3.4 Boundary Regions: Where the Problem Is Worst

| Setting | Backbone | Method | Boundary@5px | Boundary@3px | |------|------|------|------------|------------| | GTA5→CS | DINOv3/F | K=1 | 30.65 | 27.79 | | GTA5→CS | DINOv3/F | Uniform K=4 | 31.62 (+0.97) | 28.48 (+0.69) | | GTA5→CS | DINOv2/F-stable | K=1 | 24.45 | 22.27 | | GTA5→CS | DINOv2/F-stable | Uniform K=4 | 26.42 (+1.97) | 23.97 (+1.70) |

Boundary metrics improve most—phase instability concentrates at semantic edges. DINOv2's boundary gains are nearly double DINOv3's, again indicating older models are more affected.

4. Which Variants Work, Which Don't

4.1 Effective variants

Partial Encoder Tuning with Spatial Phase Attention (unfreeze last encoder block + per-pixel phase attention): GTA5→CS reaches 54.25 mIoU, above Uniform K=4's 52.76—but requires partial fine-tuning, not training-free.
Depth-Specific Confidence-Gated Residual Phase Fusion: NYU Depth RMSE 0.5750, better than Uniform K=4's 0.6277.

4.2 Ineffective variants

Pre-Transformer Patch-Embedding Averaging: 44.48 mIoU on GTA5→CS—far below 52.76. Averaging embeddings early erases phase-dependent information and loses positional detail.
Learned Phase-Feature Attention Aggregation: helps in some settings (SYNTHIA→CS +2.0, ADE20K +1.73) but hurts on HPatches matching (39.02→38.30)—learned weighting can overfit.

5. Diagnostics: Quantifying Instability via Phase Variance

Per-pixel phase variance is defined as:

\sigma^2(p) = \frac{1}{K}\sum_{k=1}^{K}\|F_{\phi_k}(p) - \bar{F}(p)\|_2^2

On HPatches: DINOv3 with Uniform K=4 reaches 0.000082 phase variance vs DINOv2's 0.000140—roughly half—suggesting DINOv3 already partially addresses phase stability by design. Even so, K=4 marginalization further reduces variance for both.

6. Limitations and Future Work

The author candidly lists limitations:

1. Only DINOv2/DINOv3 backbones tested—does it apply to DeiT, Swin, etc.? 2. Only dense prediction—classification may not need it (patch-level class voting may already marginalize naturally). 3. Latency: K=4 requires 4 forward passes; real-time settings may be constrained. 4. K=4 saturation on some datasets; others may need more phases. 5. No public code released.

7. Closing Thoughts: A Forgotten Variable

The elegance of this paper: it proposes no new model—it discovers an overlooked variable.

Patchification looks like a trivial preprocessing step. But "trivial" doesn't mean "irrelevant." Where the 16×16 grid starts determines which pixels share a patch, which changes the neighbor relationships self-attention sees. In image centers the effect is small; at semantic boundaries—a road's edge, a building-sky border—it can decide a pixel's label.

Uniform Phase Marginalization doesn't fight phase instability—it acknowledges it, then marginalizes it. Like a statistician treating measurement error as a random variable and averaging over it: phase is the error, K=4 inferences are four independent measurements, and the average is the marginalization.

4 forward passes, +0.82 mIoU, zero training cost. For deployed ViT dense-prediction systems, this may be the cheapest performance upgrade available.

---

References

Paper: Ercan, "Phase Marginalization for Patch-Grid Instability in Vision Transformers", arXiv:2606.08132 (2026)
Backbones: DINOv2, DINOv3 (frozen encoders)
Datasets: GTA5→Cityscapes, SYNTHIA→Cityscapes, ADE20K, NYU Depth v2, HPatches
Default config: K=4, patch size 16, reflective padding