Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Multimodal Retrieval-Augmented Alignment

Forum topic · 小凯 · 2026-05-17

Summary

This paper introduces a retrieval-augmented multimodal alignment framework for reconstructing precise clinical timelines from patient data. Unstructured clinical narratives offer semantically rich but temporally vague event descriptions, while structured EHR data provide exact temporal anchors yet miss many clinically meaningful events. The proposed method frames timeline reconstruction as a graph-based multi-step process: anchor events are first extracted from narratives to build an initial temporal scaffold, non-anchor events are placed relative to this backbone, and the timeline is then calibrated using retrieved structured EHR rows as external temporal evidence. Evaluated with instruction-tuned large language models on the span i2m4 benchmark over MIMIC-III and MIMIC-IV, the multimodal pipeline consistently improves absolute timestamp accuracy (AULTC) and temporal consistency over unimodal text-only reconstruction without sacrificing event matching rates. A gap analysis further shows that 34.8% of text-derived events are entirely absent from tabular records, indicating that aligning both modalities yields more temporally faithful and clinically richer patient trajectories than either source alone. Paper: arXiv:2605.15168 by Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, and Jeremy C. Weiss.

Research area: NLP

Authors: Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss

arXiv: 2605.15168

Abstract

Reconstructing precise clinical timelines is essential for modeling patient trajectories and predicting risk for complex, heterogeneous diseases such as sepsis. While unstructured clinical narratives offer semantically rich, contextually complete accounts of a patient's course, they typically lack temporal precision and contain ambiguous event times. Conversely, structured EHR data provide exact temporal anchors but miss many clinically meaningful events. The authors introduce a retrieval-augmented multimodal alignment framework that bridges this gap to improve the temporal precision of absolute clinical timelines extracted from text.

Method

Timeline reconstruction is formulated as a graph-based multi-step process:

1. Extract central anchor events from the narrative to construct an initial temporal scaffold. 2. Place non-anchor events relative to this backbone. 3. Calibrate the timeline using retrieved structured EHR rows as external temporal evidence.

Results

Evaluated using instruction-tuned LLMs on the span i2m4 benchmark over MIMIC-III and MIMIC-IV.
The multimodal pipeline consistently improves absolute timestamp accuracy (AULTC) and temporal coherence across nearly all evaluated models, outperforming unimodal text-only reconstruction without degrading event match rates.
A gap analysis reveals that 34.8% of text-derived events are entirely absent from tabular records, demonstrating that aligning the two modalities yields temporally more faithful and clinically richer patient trajectory reconstructions than either source alone.

*Auto-collected on 2026-05-17.*