BenchX: A Large-Scale Benchmark Revealing Demographic and Protocol Biases in Cancer-Detection AI

Forum topic · 小凯 · 2026-06-25

Summary

BenchX is a large-scale, open benchmark of 85,355 CT scans designed to quantify inconsistencies in tumor-detection AI models across real-world clinical settings. The benchmark systematically evaluates 12 AI models across tumor size, tumor location, patient subgroups (age, sex, race), and imaging protocols such as contrast phases. The authors use large language models (LLMs) to extract and organize subgroup information from clinical records, making the subgroup-level analysis scalable and reproducible. Key findings show that state-of-the-art models, while optimized for average accuracy, perform significantly worse on rare or underrepresented subgroups—for example, young African American women—and on small tumors or non-standard imaging protocols. Because collecting sufficient annotated data for such rare cases is often impractical, the authors argue that rigorous subgroup-level evaluation should become standard practice in medical imaging and computer vision research. BenchX provides a foundation for building more reliable and robust tumor-detection models. Paper: arXiv 2506.14717.

Paper Overview

Field: Computer Vision / Medical Imaging
Authors: Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi
arXiv: 2506.14717

Abstract

Artificial intelligence (AI) has achieved remarkable success in medical imaging, but it is widely recognized that these models often perform inconsistently across real-world clinical settings. Such inconsistencies occur when patient demographics and imaging protocols vary, for example, in detecting small tumors, analyzing scans from different contrast phases, or evaluating patients of different ages or sexes.

To quantify these inconsistencies, the authors develop a large-scale, open benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models across tumor size, location, patient subgroup, and imaging protocol. They leverage large language models (LLMs) to extract and organize subgroup information from clinical data, which makes the analysis both scalable and reproducible.

Key Findings

Current state-of-the-art AI models are optimized for average accuracy but perform worse on rare or underrepresented subgroups, such as young African American women.
Performance also degrades on small tumors and non-standard imaging protocols.
Collecting enough annotated data for these rare cases is often impractical, motivating systematic subgroup-level benchmarking instead.

Significance

The benchmark provides a foundation for building more reliable and robust tumor-detection AI models and highlights the need for rigorous subgroup-level evaluation in medical imaging and computer vision.

---

*Source: zhichai.net, auto-collected 2026-06-25.*