Paper Overview
Field: Computer Vision (CV) Authors: Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi arXiv: 2506.14717
Summary
Artificial intelligence (AI) has achieved remarkable success in medical imaging, but these models often perform inconsistently across real-world clinical settings. Such inconsistencies occur when patient demographics and imaging protocols vary — for example, in detecting small tumors, analyzing scans from different contrast phases, or evaluating patients of different ages or sexes.
To quantify these inconsistencies, the authors developed BenchX, a large-scale, open benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models across:
- Tumor size
- Tumor location
- Patient subgroups (age, sex, demographics)
- Imaging protocols (e.g., contrast phases)
- Current state-of-the-art AI models, although optimized for average accuracy, perform worse on rare or underrepresented subgroups, such as young African American women.
- Collecting sufficient annotated data for these rare cases is often impractical in practice.
- The benchmark demonstrates the necessity of rigorous subgroup-level evaluation in medical imaging and computer vision.
Large language models (LLMs) are leveraged to extract and organize subgroup information from clinical data, making the analysis both scalable and reproducible.
Key Findings
Significance
BenchX provides a foundation for building more reliable and robust tumor-detection AI models, offering the community a standardized way to measure demographic and protocol biases in cancer detection systems.
--- *Source: arXiv:2506.14717*