Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

• Field: Computer Vision (CV)
• Authors: Siddharth Gupta, Jitin Singla
• Published: 2026-09-09
• arXiv: 2609.10495

Abstract

In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently. We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image. RBQE is evaluated on a standardized 1,223-image external benchmark drawn from four public datasets, using four referee configurations chosen to separate two design axes: referee independence and architectural diversity.

Using a common Agreement Dice descriptor, a same-architecture referee differing from the primary model only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), showing that independent training alone is sufficient. Cross-architecture referees improve further: SegFormer-B0 achieves the strongest performance (ROC-AUC = 0.960), significantly outperforming the same-architecture control and UNet++, and exceeding a representative Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol, whereas a prompt-coupled MedSAM referee underperforms despite maximal architectural diversity.

Because empty-mask agreement is trivially separable, the authors also report a restricted evaluation excluding such cases: ROC-AUC falls to 0.876 (SegFormer-B0, 1,046 images) and 0.783 (same-architecture control, 975 images), yet RBQE's margin over both baselines widens on this identical subset. RBQE additionally increases the mean Dice of retained predictions as low-agreement cases are progressively rejected, supporting selective prediction, and requires only one additional deterministic referee forward pass at inference. The study supports cross-model agreement as a practical, interpretable reliability framework for automated polyp segmentation.

Key Takeaways

• RBQE is a reference-free, deployment-time quality estimator based on agreement between primary and referee segmentation models.
• Independent training alone (same architecture, different initialization) already gives a strong signal (ROC-AUC = 0.923); adding architectural diversity helps (SegFormer-B0 referee: ROC-AUC = 0.960).
• Findings remain robust after excluding trivially separable empty-mask cases.
• Supports selective prediction with only one extra deterministic forward pass at inference.