How Seemingly Inconsequential Design Choices Dictate LLM Performance in Pathology: A Deep Dive into arXiv 2606.12407

Forum topic · 小凯 · 2026-06-11

Summary

A forum post on zhichai.net analyzes an MIT and Harvard Medical School paper (arXiv:2606.12407) showing that input-design choices—not model capability—have been crippling general-purpose LLMs on pathology tasks. Whole-slide images contain ~100,000×100,000 pixels, so models must view them through sampled patches. The authors ran a full factorial experiment over four factors: inference mode (majority voting vs. all-in-one joint context), patch size (224–1024px), magnification (5×, 10×, 20×), and patch count (10/20/30), across 934 questions in the MultiPathQA benchmark (GTEx, TCGA, PANDA, SlideBench, ExpertVQA) plus a held-out CPTAC set. Simply switching from majority voting to all-in-one inference lifted GPT-5 accuracy by up to 12.5 percentage points while cutting cost ~10×. The optimized configuration raised TCGA accuracy from 15.1% to 43.9% and GTEx from 38.1% to 71.6%, surpassing specialized models like GIANT—without fine-tuning. Larger patches and lower magnification generally helped (matching LLM pretraining distributions), except PANDA, which needs 20× cellular detail. Mixed magnifications offered no added benefit, and patch count showed diminishing returns past 20. The takeaway: baselines matter, and 'input engineering' may outweigh domain-specific architectures.

Opening a Window in a Gigapixel Room: How a Pathology Paper Teaches the Art of 'Seeing'

> Paper: How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology > Authors: Kian R. Weihrauch, Thomas A. Buckley, William Lotter, Arjun K. Manrai > Institutions: MIT, Harvard Medical School, Dana-Farber Cancer Institute > arXiv: 2606.12407 > Published: June 10, 2026

This post discusses the paper's allegory of the "window": a whole-slide image (WSI) contains roughly 100,000 × 100,000 pixels—too much for any LLM's context window—so researchers must choose *how* the model sees it. The paper's claim is that these seemingly trivial choices determine whether the model reads truth from the gigapixel maze, and that prior benchmarks chose the wrong windows.

Key points

Why pathology images are special

A typical WSI is ~100,000 × 100,000 pixels; printed at full scale it would exceed 27 meters per side.
General LLM vision encoders are pretrained on natural, medium-resolution, context-rich images—not 224×224 patches at 20× magnification containing a few dozen cells.
The common baseline protocol samples small, high-magnification patches, classifies each independently, and aggregates via majority voting. The paper questions whether this baseline systematically understates general LLMs.

A 72-configuration factorial experiment

| Factor | Levels | |---|---| | Inference mode | Majority voting vs. all-in-one (joint context) | | Patch size | 224, 512, 896, 1024 px | | Magnification | 5×, 10×, 20× | | Patch count | 10, 20, 30 |

Benchmark: MultiPathQA (934 questions across GTEx organ classification, TCGA cancer classification, PANDA Gleason grading, SlideBench VQA, ExpertVQA), plus held-out CPTAC validation (200 slides) never used for configuration selection.
Three-phase evaluation: exploratory factor analysis on 100 slides, full validation on MultiPathQA, then cross-model generalization (GPT-5, Qwen 3.5 Plus, Gemini 3 Flash).

Finding 1: Inference mode is the dominant factor

Majority voting makes each patch an isolated judgment; the model cannot connect distant regions.
All-in-one inference lets the model reason across regions. Switching modes alone (224px, 20×, 30 patches) yielded:
GTEx +5.64 pp, TCGA +8.65 pp, PANDA +5.67 pp, SlideBench +2.06 pp, ExpertVQA +12.50 pp
All-in-one is also ~10× cheaper: tokens drop from ~30,000–56,000 to ~10,000; cost from $0.077–0.221 to $0.012–0.029 per slide.

Finding 2: Bigger patches beat the 224×224 default

The 224×224 default is an ImageNet-era legacy suited to CNNs trained on small crops, not LLM encoders trained on larger contextual images.
Performance generally increases with patch size; 896–1024 px is optimal for most tasks.
Task-dependent exceptions: GTEx prefers 1024px at 5× (maximum field of view), while PANDA prefers 512px at 20× (cellular detail needed for Gleason grading).

Finding 3: Lower magnification usually wins—except PANDA

Counterintuitively for the clinical 20× standard, general LLMs perform better at 5× (and 10×) because low-magnification, wide-field images are closer to their pretraining distribution.
PANDA is the sole exception: Gleason grading requires 20× cellular morphology.
Lesson: there is no universal best configuration; each task needs its task-adapted "viewing distance."

Finding 4: Patch count shows diminishing returns

Going from 10 to 20 patches helps; 20 to 30 yields flat or marginal gains. Patch count is the least influential factor. 20 patches is the cost-quality sweet spot.

Headline results: from "weak" to surpassing specialized models

| Method | TCGA | GTEx | PANDA | SlideBench | ExpertVQA | |---|---|---|---|---|---| | GPT-5, literature protocol | 15.1% | 38.1% | 21.5% | 49.0% | 39.1% | | GPT-5, optimal config | 43.9% | 71.6% | 31.0% | 61.9% | 63.3% | | GIANT (specialized agent) | 32.3% | 54.1% | — | — | — |

The optimized GPT-5 nearly triples TCGA accuracy and roughly doubles GTEx, surpassing the specialized GIANT agent without any fine-tuning.
A balanced configuration (896px, 10×, 20 patches, all-in-one) generalizes across tasks and models:
GPT-5: TCGA 15.1% → 39.5%; GTEx 38.1% → 62.9%; held-out CPTAC 32.7% → 36.5%
Qwen 3.5 Plus: TCGA 54.3%; GTEx 66.8%; CPTAC 39.7%
Gemini 3 Flash: TCGA 63.0%; GTEx 75.5%; CPTAC 55.9% (+23.4 pp), confirming generalization on unseen data.

Why these choices matter: distribution alignment

The core insight is alignment between input distribution and pretraining distribution. A 224px, 20× patch is out-of-distribution for a general vision encoder; a 1024px, 5× patch invokes pretrained knowledge of texture, layout, and structure.
All-in-one mode enables cross-region reasoning analogous to a pathologist's spatial reasoning across fields of view.
ANOVA reveals interaction effects (e.g., patch size × magnification on GTEx/TCGA; inference mode × patch size on PANDA), so input design is a nonlinear, multi-dimensional optimization, not a linear "bigger is better" rule.

Mixed magnifications: no synergy

Ablations mixing 5×/10×/20× patches under a fixed 30-patch budget produced results *between* single-magnification runs, not better:

| Config | TCGA | GTEx | PANDA | |---|---|---|---| | 5× only | 41.8% | 70.2% | 22.1% | | 10× only | 39.6% | 62.8% | 20.8% | | 20× only | 33.3% | 56.8% | 28.8% | | Mixed (10 each) | 41.1% | 66.4% | 25.3% |

Probable causes: context dilution across scales, task-specific optimal scales being averaged out, and reduced per-scale patch counts under fixed budget.

Field of view: necessary but not sufficient

Total FoV = (patch size / magnification) × patch count. Larger FoV correlates with better performance, but does not fully explain scaling: at equal FoV, different configurations still perform differently. Patch size, magnification, and count each carry qualitative—not just quantitative—effects.

Implications for pathology AI

1. Baselines must be redesigned: claims that "specialized models are stronger" were based on suboptimal LLM baselines and may be systematically overstated. 2. Domain-specific training is re-evaluated: general LLMs with tuned inputs can rival specialized systems, offering cheaper, faster deployment paths. 3. Input engineering emerges as a discipline: like prompt engineering, choosing inference mode, patch size, magnification, and count will become a key applied skill; future systems may auto-select configurations per task type. 4. The alignment principle likely extends beyond pathology to other visual domains.

Closing thought

The post ends with a Zen parable: when shown the moon, do not stare at the pointing finger. Pathology AI has fixated on architecture and parameters (the finger) while overlooking how information is presented (the moon). As the authors show, in a gigapixel room, the position, size, count, and angle of the windows can matter more than the room's contents. The general LLM is not a poor student—merely one seated at the wrong desk.

References

1. Weihrauch, K. R., Buckley, T. A., Lotter, W., & Manrai, A. K. (2026). *How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology*. arXiv:2606.12407. 2. GTEx Consortium (2013). The Genotype-Tissue Expression (GTEx) project. *Nature Genetics*, 45(6), 580-585. 3. TCGA Research Network. https://tcga-data.nci.nih.gov 4. CPTAC Consortium. https://proteomics.cancer.gov

*This article discusses arXiv:2606.12407 by the MIT and Harvard Medical School team. All figures are cited from the original paper; interpretive errors are the forum author's own.*