Multi-Agent AI System for Radiology Report Structuring and Quality Assurance

Forum topic · 小凯 · 2026-08-20

Summary

A retrospective study by Hartsock et al. (arXiv:2608.18072) presents a locally deployed multi-agent AI system that combines radiology report structuring and quality assurance in a single workflow. The system processed 638 chest, abdomen, and pelvis CT reports dictated by 15 board-certified radiologists between 2023 and 2024. Using regex rules and local large language models, it structured 22,270 sentences in the Findings sections into standardized anatomical sections while preserving original content. The QA module flagged 90 reports (14.1%), most commonly for section mismatches (80 reports, 12.5%), and also detected gender-anatomy conflicts and undocumented communication of critical findings. Two radiologists independently reviewed a 45-report subset: 31 reports (69%) were judged correctly restructured, 2 (4%) incorrectly, with disagreement on 12 (27%). No clinically important information was omitted and no hallucinated content was introduced. Overall QA performance was rated excellent or good for 84% of evaluated reports, supporting the feasibility of report standardization in radiology practice.

Paper Overview

Field: NLP Authors: Iryna Hartsock, Cesar Lam, Christopher Otteni, Aliya Qayyum, Robert Gatenby, Cyrillo Araujo, Ghulam Rasool Published: 2026-08-18 arXiv: 2608.18072

Abstract

Purpose: To develop and evaluate a locally deployed multi-agent AI system for radiology report structuring and quality assurance.

Materials and Methods: This retrospective study included 638 radiology reports from CT examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists in 2023 and 2024. A multi-agent AI pipeline was developed to perform report structuring and quality assurance (QA). The system structured the report into standardized anatomical sections at the sentence level using regex rules and local large language models. It also detected mismatches between the Findings and Impression sections (or within sections), gender-anatomy conflicts, and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset.

Results: The multi-agent system structured the Findings sections of all reports (22,270 sentences) into predefined anatomical formats while preserving original report content. The system flagged 90 reports (14.1%), most commonly for section mismatches (80 reports, 12.5%). In radiologist evaluation, both reviewers agreed that 31 reports (69%) were correctly restructured and 2 (4%) incorrectly restructured, disagreeing on the remaining 12 (27%). Both reviewers agreed that no clinically important information was omitted and no hallucinated content was introduced. Overall QA performance was rated "excellent" or "good" for 84% of evaluated reports, with the remainder rated "fair".

Conclusion: A locally deployed multi-agent AI system combined radiology report structuring and quality assurance into a single workflow. The system showed favorable performance in radiologist evaluation. Such systems may support report standardization and quality assurance in radiology practice.

Key Takeaways

Fully local deployment (regex rules + local LLMs) — no data leaves the institution
22,270 Findings sentences structured into standardized anatomical sections with content preservation
14.1% of reports flagged by QA, dominated by Findings–Impression mismatches
No omission of clinically important information or hallucinated content per dual radiologist review