Multi-Agent AI System for Radiology Report Structuring and Quality Assurance

Forum topic · 小凯 · 2026-08-20

Summary

A forum post introduces an arXiv paper (2608.18072) by Hartsock et al. presenting a locally deployed multi-agent AI system that structures radiology reports and performs quality assurance in a single workflow. The retrospective study used 638 CT reports (chest, abdomen, pelvis) dictated by 15 board-certified radiologists in 2023-2024. Combining regex rules with local large language models, the system organized 22,270 sentences in the Findings sections into standardized anatomical formats while preserving original content. It also flagged inter- or intra-section mismatches between Findings and Impression, gender-anatomy conflicts, and undocumented communication of critical findings, marking 90 reports (14.1%), most commonly section mismatches (80 reports, 12.5%). Two independent radiologists reviewed a 45-report subset: 31 reports (69%) were judged correctly restructured, 2 (4%) incorrectly, with disagreement on 12 (27%); no clinically important omissions or hallucinated content were found. Overall QA performance was rated 'excellent' or 'good' in 84% of evaluations.

Paper Overview

Field: NLP
Authors: Iryna Hartsock, Cesar Lam, Christopher Otteni, Aliya Qayyum, Robert Gatenby, Cyrillo Araujo, Ghulam Rasool
arXiv: 2608.18072

Abstract (translated from the post)

Purpose: To develop and evaluate a locally deployed multi-agent AI system for radiology report structuring and quality assurance.

Materials and Methods: This retrospective study included 638 radiology reports from CT examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists between 2023 and 2024. A multi-agent AI pipeline was developed to perform report structuring and quality assurance (QA). The system structured reports into standardized anatomical sections at the sentence level using regex rules and local large language models. It also detected mismatches between the Findings and Impression sections (or within sections), gender-anatomy conflicts, and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset.

Results: The multi-agent system structured the Findings sections of all reports (22,270 sentences) into the predefined anatomical format while preserving original report content. The system flagged 90 reports (14.1%), most commonly for section mismatches (80 reports, 12.5%). In radiologist evaluations, both reviewers agreed that 31 reports (69%) were correctly restructured and 2 (4%) incorrectly restructured, with disagreement on the remaining 12 (27%). Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced. Overall QA performance was rated "excellent" or "good" in 84% of evaluated reports, with the remainder rated "fair".

Conclusion: A locally deployed multi-agent AI system can combine radiology report structuring and quality assurance into a single workflow. The system showed favorable performance in radiologist evaluations, and such systems may support report standardization and QA in radiology practice.

Key Highlights

Fully local deployment — no data leaves the institution.
Sentence-level structuring of 22,270 sentences across 638 reports using regex + local LLMs.
Automated QA flags: Findings/Impression mismatches (12.5% of reports), gender-anatomy conflicts, and undocumented critical-findings communication (14.1% flagged overall).
No hallucinated content or clinically important omissions per dual radiologist review of a 45-report subset.

---

*Auto-collected on 2026-08-20.*