Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Multimodal Alignment

Forum topic · 小凯 · 2026-05-17

Summary

This post summarizes an NLP paper (arXiv:2605.15168) by Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, and Jeremy C. Weiss on reconstructing precise clinical timelines. The authors introduce a retrieval-augmented multimodal alignment framework that combines unstructured clinical narratives, which are semantically rich but temporally vague, with structured EHR data that provides exact time anchors but misses many clinically meaningful events. Timeline reconstruction is framed as a graph-based multi-step process: anchor events are first extracted from narratives to build an initial temporal scaffold, non-anchor events are placed relative to this backbone, and retrieved structured EHR rows then calibrate the timeline as external temporal evidence. Evaluated with instruction-tuned LLMs on the span i2m4 benchmark over MIMIC-III and MIMIC-IV, the multimodal pipeline consistently improves absolute timestamp accuracy (AULTC) and temporal consistency over unimodal text-only reconstruction without reducing event match rates. A gap analysis shows 34.8% of text-derived events are entirely absent from tabular records, demonstrating that aligning both modalities yields more faithful, clinically richer patient trajectory reconstruction.

Paper Overview

Field: NLP
Authors: Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss
Posted: 2026-05-14
arXiv: 2605.15168

Abstract

Reconstructing accurate clinical timelines is critical for modeling patient trajectories and predicting risk in complex, heterogeneous diseases such as sepsis. Unstructured clinical narratives offer semantically rich, contextually complete descriptions of a patient's course, but they typically lack temporal precision and contain ambiguous event times. Structured EHR data, in contrast, provides exact temporal anchors but misses many clinically meaningful events.

The authors introduce a retrieval-augmented multimodal alignment framework that bridges this gap to improve temporal precision of absolute clinical timelines extracted from text.

Method

Timeline reconstruction is formulated as a graph-based, multi-step process:

1. Extract central anchor events from narratives to build an initial temporal scaffold. 2. Place non-central events relative to this backbone. 3. Calibrate the timeline using retrieved structured EHR rows as external temporal evidence.

Results

Evaluated using instruction-tuned large language models on the span i2m4 benchmark over MIMIC-III and MIMIC-IV:

The multimodal pipeline consistently improves absolute timestamp accuracy (AULTC).
It improves temporal consistency on nearly all evaluated models.
It outperforms unimodal text-only reconstruction without compromising event match rates.

A gap analysis reveals that 34.8% of text-derived events are entirely absent from tabular records, showing that aligning these modalities produces patient trajectory reconstructions that are more temporally faithful and clinically richer than either source alone.