Opening Windows into a Gigapixel Room: How Input Design Choices Dictate LLM Performance in Pathology

Forum topic · 小凯 · 2026-06-11

Summary

A forum post on zhichai.net discusses an arXiv paper (2606.12407) by MIT and Harvard Medical School researchers showing that seemingly minor input design choices dramatically affect how general-purpose LLMs perform on pathology whole-slide images (WSI). The authors ran a full-factorial experiment—72 configurations across inference mode (majority voting vs. all-in-one joint processing), patch size (224–1024px), magnification (5x/10x/20x), and patch count (10/20/30)—on the MultiPathQA benchmark (934 questions; GTEx, TCGA, PANDA, SlideBench, ExpertVQA) plus a held-out CPTAC set. Key findings: switching from independent majority voting to all-in-one inference yields up to +12.5 percentage points and ~10x cost savings; larger patches and lower magnification (closer to LLMs' pretraining distribution) generally help; gains saturate beyond ~20 patches; mixed magnifications do not add value. Optimized configurations lift GPT-5 from 15.1% to 43.9% on TCGA and 38.1% to 71.6% on GTEx, surpassing the specialized GIANT agent model without fine-tuning. The post argues baseline protocols previously underestimated general LLMs and that task-adapted input engineering may reduce the need for domain-specific training.

This zhichai.net forum post offers a detailed Chinese-language reading of the paper "How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology" (Kian R. Weihrauch, Thomas A. Buckley, William Lotter, Arjun K. Manrai — MIT, Harvard Medical School, Dana-Farber Cancer Institute; arXiv:2606.12407, June 2026).

Key points

The problem: gigapixel images, tiny windows

Whole-slide images (WSI) are roughly 100,000×100,000 pixels — far beyond LLM context limits.
The common baseline protocol samples small 224×224 patches at 20× magnification, classifies each patch independently, and aggregates via majority voting.
The authors question whether "specialized models beat general LLMs" conclusions were inflated by poorly designed baselines.

A 72-configuration factorial experiment

Four factors: inference mode (majority vote vs. all-in-one joint input), patch size (224/512/896/1024 px), magnification (5×/10×/20×), patch count (10/20/30).
Evaluated on MultiPathQA (934 questions across GTEx organ classification, TCGA cancer typing, PANDA Gleason grading, SlideBench VQA, ExpertVQA), validated on held-out CPTAC data across GPT-5, Qwen 3.5 Plus, and Gemini 3 Flash.

Findings

Inference mode is the dominant factor. All-in-one (all patches in one context, enabling cross-region reasoning) beats majority voting by up to +12.5 pp on ExpertVQA, and is ~10× cheaper (fewer API calls, ~10,000 vs. ~30,000–56,000 tokens).
Bigger patches, lower magnification generally help, because they better match LLMs' pretraining distribution of natural, context-rich images. Exceptions: PANDA Gleason grading needs 20× cell-level detail.
Patch count shows diminishing returns beyond ~20 patches; it is the least influential factor.
Mixed magnifications do not stack — performance lands between single-scale results, suggesting task-optimal single-scale is preferable under a fixed patch budget.
Field of View (FoV) alone doesn't explain performance; patch size and magnification are qualitative, not just quantitative, choices.

Headline results

With optimized input configs only (no fine-tuning), GPT-5 improves from 15.1% → 43.9% on TCGA and 38.1% → 71.6% on GTEx, exceeding the specialized agent model GIANT (32.3% / 54.1%).
A balanced configuration (896px, 10×, 20 patches, all-in-one) generalizes across models: Gemini 3 Flash reached 55.9% on held-out CPTAC vs. ~32.5% baseline (+23.4 pp).

Implications

Pathology AI baselines should be redesigned; claims of domain-specific necessity may be systematically overstated.
"Input engineering" — choosing patch size, magnification, mode, and count — may matter as much as prompt engineering, and the core principle is aligning input distribution with the model's pretraining distribution.

*All figures are as cited by the original forum post and the arXiv paper (arXiv:2606.12407).*