Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation (RBQE, arXiv:2609.10495)

Forum topic · 小凯 · 2026-09-11

Summary

In real-time colonoscopy, ground-truth masks are unavailable at inference time, so polyp segmentation models can fail silently. This paper proposes Referee-Based Quality Estimation (RBQE), a reference-free framework that measures agreement between a primary segmentation model and an independently trained referee on the same image. Evaluated on a standardized 1,223-image external benchmark from four public datasets, RBQE shows that a same-architecture referee differing only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), while cross-architecture referees improve further, with SegFormer-B0 reaching the strongest performance (ROC-AUC = 0.960) and exceeding a Test-Time Augmentation baseline by 0.055 ROC-AUC. Excluding trivially separable empty-mask cases, ROC-AUC drops to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), yet RBQE's margin over baselines widens. RBQE also supports selective prediction, raising mean Dice of retained predictions as low-agreement cases are rejected, with only one extra deterministic forward pass at inference.

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Field: Computer Vision Authors: Siddharth Gupta, Jitin Singla arXiv: 2609.10495

Abstract (translated)

In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently. We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image. RBQE is evaluated on a standardized 1,223-image external benchmark drawn from four public datasets, using four referee configurations chosen to separate two design axes: referee independence and architectural diversity. Using a common Agreement Dice descriptor, a same-architecture referee differing from the primary model only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), showing that independent training alone is sufficient. Cross-architecture referees improve further: SegFormer-B0 achieves the strongest performance (ROC-AUC = 0.960), significantly outperforming the same-architecture control and UNet++, and exceeding a representative Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol, whereas a prompt-coupled MedSAM referee underperforms despite maximal architectural diversity. Because empty-mask agreement is trivially separable, we also report a restricted evaluation excluding such cases: ROC-AUC falls to 0.876 (SegFormer-B0, 1,046 images) and 0.783 (same-architecture control, 975 images), yet RBQE's margin over both baselines widens on this identical subset. RBQE additionally increases the mean Dice of retained predictions as low-agreement cases are progressively rejected, supporting selective prediction, and requires only one additional deterministic referee forward pass at inference. Our study therefore supports cross-model agreement as a practical, interpretable reliability framework for automated polyp segmentation.

Key takeaways

Problem: Polyp segmentation models may fail silently at deployment, with no ground truth to detect errors in real time.
Approach: Use an independently trained "referee" segmentation model; measure agreement (Agreement Dice) between referee and primary model as a per-image reliability score.
Independence alone helps: A same-architecture referee with only a different random seed already provides a useful signal (ROC-AUC = 0.923).
Architectural diversity helps more: SegFormer-B0 referee achieves the best performance (ROC-AUC = 0.960), beating UNet++ referees and a TTA baseline by 0.055 ROC-AUC; a prompt-coupled MedSAM referee underperforms despite maximal diversity.
Harder subset: Excluding empty-mask cases, ROC-AUC drops to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), but RBQE's advantage over baselines grows.
Selective prediction: Progressively rejecting low-agreement cases increases the mean Dice of retained predictions.
Cost: Only one additional deterministic referee forward pass at inference.

---

*Auto-collected on 2026-09-11. Source: arXiv:2609.10495*