Phase Marginalization in Vision Transformers: Why Every Pixel Prediction Depends on Patch Grid Phase

Forum topic · 小凯 · 2026-06-12

Summary

Vision Transformers (ViTs) slice images into fixed patch grids, and the grid's starting offset—its phase—acts as a hidden variable: the same pixel can receive different predictions (e.g., road vs. sidewalk) when the image shifts relative to the patch grid, especially near semantic boundaries. This article analyzes a training-free fix called Uniform Phase Marginalization: sample K=4 grid phase offsets (the four quadrant offsets for patch size 16), run forward passes for each, inverse-align outputs back to the original pixel grid, and average the aligned log-probabilities. Tested with frozen DINOv2 and DINOv3 backbones, it improves semantic segmentation (GTA5→Cityscapes +0.82 mIoU; ADE20K +1.39 mIoU with DINOv2), monocular depth (NYU Depth v2 RMSE 0.6506→0.6277), and feature matching on HPatches (+3.75 with DINOv2). At equal compute, it beats standard shift-based test-time augmentation by +0.31 mIoU, and gains saturate at K=4. Boundary-region metrics improve most, confirming that phase instability concentrates at semantic edges. The method requires no retraining and acts purely as an inference wrapper.

Phase Marginalization in Vision Transformers: Every Pixel Prediction Depends on the Patch Grid

> One image, the same pixel, two different predictions. Not a data problem, not a model problem—the culprit is the *phase* of the grid ViT uses when cutting the image into patches. A paper proposes a zero-training-cost fix.

1. The Problem: Same Image, Same Pixel, Two Answers

Take a street scene. A pixel on the road is predicted as "road." Shift the image right by 8 pixels and feed it back: the same physical pixel now becomes "sidewalk."

The model isn't wrong—the ViT patch grid has shifted.

Vision Transformers split images into fixed-size patches (e.g., 16×16). The grid has a starting position relative to the image's top-left corner: this offset is the *phase*. The same pixel falls into different patches under different phases, and after self-attention and MLP transforms, the output can differ completely.

The effect is strongest near semantic boundaries: whether a pixel is "road" or "sidewalk" depends on its patch neighbors. Change the phase, change the neighbors, change the prediction.

It resembles gerrymandering: the same voter, assigned to a different district, produces a different outcome. The ViT patch grid is the district map.

2. Phase Marginalization: A Zero-Training-Cost Solution

2.1 Core Idea

Author Oğuzhan Ercan formalizes phase as a nuisance variable and applies post-hoc marginalization:

1. Sample K grid phase offsets (default K=4) 2. Run a full forward pass for each phase 3. Inverse-align each output back to the original pixel coordinates 4. Average the aligned log-probabilities

No model changes. No retraining. Purely an inference wrapper.

2.2 The Math

Phase definition:

\phi = (d_x, d_y), \quad d_x, d_y \in \{0, \ldots, P-1\}

where P is the patch size (default 16). The phase-shift operator $S_\phi$ uses reflective padding and cropping. Model output:

z_\phi = h(f_\theta(S_\phi(x)))

Inverse alignment $A_\phi^{-1}$ : reshape patch outputs to a grid → bilinear upsampling → inverse-offset cropping. Marginalization:

\bar{z}(p) = \frac{1}{K}\sum_{k=1}^{K} A_{\phi_k}^{-1}(z_{\phi_k})(p)

The standard phase set (K=4, P even):

\Phi_4 = \{(0,0), (0, P/2), (P/2, 0), (P/2, P/2)\}

2.3 Difference from Generic TTA

| Dimension | Generic TTA | Phase Marginalization | |---|---|---| | What is sampled | Flips, crops, shifts of the image | Phase offsets of the patch grid itself | | Alignment | Inverse-transform predictions | Exact inverse alignment to the original pixel grid | | Transform definition | Arbitrary semantic augmentations | Fixed offsets defined by patch size | | Compute | Same (K forwards) | Same, but more effective |

Key distinction: generic TTA shifts the whole image, so the model sees different patch *content*. Phase Marginalization only moves the grid—the same pixels are regrouped into different patches, but patch content is unchanged. It corrects a purely architectural quantization effect.

3. Results: Improvements Across Four Tasks

3.1 Main Results: Uniform Phase Marginalization, K=4 (Table 1)

| Task | Dataset | Backbone | Baseline | K=4 | Gain | |---|---|---|---|---|---| | Semantic segmentation | GTA5 → Cityscapes | DINOv3/F | 51.94 | 52.76 | +0.82 | | Semantic segmentation | SYNTHIA → Cityscapes | DINOv3 | 36.00 | 36.88 | +0.88 | | Semantic segmentation | SYNTHIA → Cityscapes | DINOv2 | 32.90 | 33.77 | +0.87 | | Semantic segmentation | ADE20K | DINOv3 | 48.82 | 49.58 | +0.76 | | Semantic segmentation | ADE20K | DINOv2 | 43.77 | 45.16 | +1.39 | | Depth estimation | NYU Depth v2 | DINOv3 | 0.6506 | 0.6277 | -0.0229 | | Feature matching | HPatches | DINOv3 | 39.02 | 39.98 | +0.96 | | Feature matching | HPatches | DINOv2 | 24.43 | 28.18 | +3.75 |

Works both cross-domain and in-domain. DINOv2 gains the most on ADE20K (+1.39 mIoU), suggesting older backbones suffer more phase instability; DINOv3 partially mitigates it but still leaves headroom.

3.2 Head-to-Head vs. Generic TTA (Table 2)

| Method | Forwards | mIoU | Latency ms/img | |---|---|---|---| | Single pass (K=1) | 1 | 52.90 | 20.77 | | Random subpatch-shift TTA | 4 | 53.00 | 83.60 | | Integer-shift TTA | 4 | 53.22 | 82.80 | | Uniform Phase Marginalization | 4 | 53.53 | 87.52 |

At equal compute, +0.31 mIoU over generic shift TTA. This isn't "more compute for better results"—it's the same compute spent on the right variable.

3.3 K Scaling: K=4 Is the Sweet Spot (Table 3)

| K | Forwards | mIoU | Latency | Cost-effectiveness | |---|---|---|---|---| | 1 | 1 | 52.90 | 20.77 | baseline | | 2 | 2 | 53.22 | 44.08 | +0.32 / 2.12× | | 4 | 4 | 53.53 | 87.52 | +0.63 / 4.21× | | 8 | 8 | 53.53 | 176.39 | +0.63 / 8.49× | | 16 | 16 | 53.58 | 347.16 | +0.68 / 16.71× |

K=4 → K=8 yields zero improvement (53.53 vs 53.53). The four quadrant phases already cover the essential variation—hence K=4 as the recommended default.

3.4 Boundary Regions: Where the Problem Is Worst (Table 4)

| Setting | Backbone | Method | Boundary@5px | Boundary@3px | |---|---|---|---|---| | GTA5→CS | DINOv3/F | K=1 | 30.65 | 27.79 | | GTA5→CS | DINOv3/F | Uniform K=4 | 31.62 (+0.97) | 28.48 (+0.69) | | GTA5→CS | DINOv2/F-stable | K=1 | 24.45 | 22.27 | | GTA5→CS | DINOv2/F-stable | Uniform K=4 | 26.42 (+1.97) | 23.97 (+1.70) |

Boundary metrics improve more—phase instability is worst near semantic edges. DINOv2's boundary gains are roughly double DINOv3's.

4. Which Variants Work—and Which Don't

Effective variants

Partial Encoder Tuning with Spatial Phase Attention (unfreeze last encoder block + per-pixel phase attention): GTA5→CS 54.25 mIoU (above Uniform K=4's 52.76), but requires partial fine-tuning.
Depth-Specific Confidence-Gated Residual Phase Fusion: NYU Depth RMSE 0.5750 (better than Uniform K=4's 0.6277).

Ineffective variants

Pre-Transformer Patch-Embedding Averaging: GTA5→CS only 44.48 mIoU. Averaging embeddings before attention erases phase-specific information—a premature marginalization that loses positional detail.
Learned Phase-Feature Attention Aggregation: helps in some cases (SYNTHIA→CS +2.0, ADE20K +1.73) but hurts HPatches matching (39.02→38.30)—learned weighting can overfit.

5. Diagnostics: Phase Variance as an Instability Metric

The paper defines per-pixel phase variance:

\sigma^2(p) = \frac{1}{K}\sum_{k=1}^{K}\|F_{\phi_k}(p) - \bar{F}(p)\|_2^2

| Backbone | Method | Phase variance (HPatches) | |---|---|---| | DINOv3 | Uniform K=4 | 0.000082 | | DINOv2 | Uniform K=4 | 0.000140 |

DINOv3's phase variance is nearly half of DINOv2's—newer models partially account for phase stability by design, yet K=4 marginalization still reduces variance further.

6. Limitations and Future Work

1. Only DINOv2/DINOv3 backbones tested: does it extend to DeiT, Swin, etc.? 2. Dense prediction only: classification may not need phase marginalization (class voting over patches may already marginalize it). 3. Latency: K=4 requires 4 forward passes; real-time settings may suffer. 4. K=4 saturation: may not hold on all datasets. 5. No public code: the paper does not release a codebase.

7. Conclusion: A Forgotten Variable

The elegance of this work: it proposes no new model—it discovers an overlooked variable.

Patchification looks like a trivial preprocessing step. But trivial does not mean irrelevant. When an image is cut into a 16×16 grid, the grid's origin—offset by 0 or 8 pixels—determines which pixels share a patch, and hence which neighbors self-attention sees. In image centers the effect is small; at semantic boundaries—the road/curb line, the building/sky edge—it can decide a pixel's label.

Uniform Phase Marginalization doesn't fight phase instability—it acknowledges it and marginalizes it, the way statisticians treat measurement error as a random variable and average over it. Here phase is the error, the K=4 inferences are four independent measurements, and the average is the marginalization.

4 forward passes, +0.82 mIoU, zero training cost. For deployed ViT dense-prediction systems, this may be the cheapest performance boost available.

---

References

Paper: Ercan, "Phase Marginalization for Patch-Grid Instability in Vision Transformers", arXiv:2606.08132 (2026)
Backbones: DINOv2, DINOv3 (frozen encoders)
Datasets: GTA5→Cityscapes, SYNTHIA→Cityscapes, ADE20K, NYU Depth v2, HPatches
Default config: K=4, patch size 16, reflective padding