> Paper: Training Large Language Models to Predict Clinical Events > Authors: Benjamin Turtel, Paul Wilczewski, Kris Skotheim (Lightning Rod Labs) > arXiv: https://arxiv.org/abs/2605.12817 | Research date: 2026-05-25
---
1. One Question Unifies All Prediction
The core challenge of clinical decision-making has never changed — a physician facing one patient must anticipate the disease trajectory with incomplete information.
Will this patient die? Need intubation? Develop a drug-resistant infection? Require dialysis? The traditional approach trains a separate model for each question — one for medication prediction, one for mortality, another for procedures.
This paper does something cleaner: it writes every prediction task as a natural-language question and feeds them all to the same model.
- "Will this patient require endotracheal intubation and mechanical ventilation during this hospitalization?"
- "Will this patient be pronounced dead during this hospitalization?"
- "Will this patient's sputum culture return positive for a pathogen?"
- [ ] Whether gpt-oss-120b's pretraining data contains MIMIC-III (degree of contamination)
- [ ] Extending the framework to MIMIC-IV
- [ ] Fusing structured EHR data (labs, vitals) with narrative notes
- [ ] Transfer to different specialties (cardiology, neurology, oncology)
- [ ] Direct comparison with clinical risk scores (APACHE, SOFA, NEWS)
- [ ] Impact of the question-generator choice (Gemini vs GPT vs Claude) on data quality
Same patient record, same model, different questions. That is the core design — unifying heterogeneous clinical prediction through questioning.
---
2. Foresight Learning: Temporal Compliance Baked into the Data
2.1 Framework Origin
Foresight Learning was introduced by Kraljevic et al. in 2024 in *The Lancet Digital Health* for patient-timeline modeling. Its core principle: reward predictions made with information available at prediction time using realized clinical outcomes. This paper extends it from "timeline generation" to "event prediction."
2.2 Three-Step Data Construction
Step 1: Trajectory construction. All timestamped clinical notes from a single MIMIC-III hospitalization, strictly time-ordered: nursing documents, physician progress notes, consult notes, radiology interpretations, discharge summaries. Inclusion criteria were strict — at least 9 timestamped notes and a recorded discharge time. 702 hospitalizations made it into the dataset.
Step 2: Question generation. For each trajectory, a split point t was randomly chosen (strictly before discharge). Notes before t form the prediction context; notes after t are used for outcome parsing (hidden from the generator). Gemini 2.5 Flash generated clinically meaningful prediction questions under explicit constraints — the generator received no post-split notes and no discharge documents. Questions could only be based on pre-split information.
Step 3: Label resolution. Gemini 2.5 Flash then assigned binary labels to each question based on the full post-split documents (including the discharge summary), determining whether the queried event occurred after the prediction time and before discharge. Questions without an assignable supportive label were excluded. Final output: 6,900 prediction samples, 25% positive-label rate.
2.3 Rigid Temporal Compliance
The essence of this pipeline is the complete elimination of lookahead bias. Traditional clinical prediction models often err by using "future information" to predict the "future" — e.g., training a mortality model on discharge summaries that contain the death record, so the model learns leaked signals rather than predictive capability. The split mechanism ensures a temporal firewall: documents visible at prediction time strictly precede the prediction point; outcome parsing uses documents after it.
---
3. Model Design: LoRA Fine-Tuning a 120B Model, Benchmarked Against GPT-5
| Component | Configuration | |------|------| | Base model | gpt-oss-120b (120B params, decoder-only) | | Adaptation | LoRA, rank r=32 | | Trainable params | LoRA adapters only; base weights frozen | | Max context | 16,000 tokens (retain most recent documents when truncated) | | Algorithm | GRPO (Generalized Reward Policy Optimization) | | Group size / batch size | 4 / 32 | | Final checkpoint | step 200 |
Input/output format. Input: task instruction + time-ordered patient records + prediction question. Output: a probability between 0 and 1 — the estimated probability that the queried event occurs after the prediction time and before discharge — plus natural-language reasoning grounded in the observed trajectory.
Reward design: log score. Training uses the strictly proper scoring rule:
where p is the predicted probability and y the binary outcome. It rewards high probability on realized outcomes and severely penalizes overconfidence; maximizing expected log score is equivalent to maximizing the log-likelihood of observed outcomes under the model's predictive distribution. Under GRPO, the model samples 4 full reasoning trajectories plus probability estimates per example, each scored against the realized outcome via the log score; only LoRA parameters update.
---
4. Results: Calibration Matters More Than Ranking
| Model | Reward | Brier ↓ | ECE ↓ | AUROC | Top-10% Lift | |------|--------|---------|-------|-------|-------------| | Constant baseline (24.8%) | -0.5890 | 0.1996 | — | — | — | | Base model (prompted) | -0.5856 | 0.1994 | 0.1269 | 0.6992 | 2.3358 | | GPT-5 | -0.4636 | 0.1457 | 0.0422 | 0.7954 | 2.9927 | | Trained model (step 200) | -0.4586 | 0.1453 | 0.0398 | 0.7993 | 3.0657 |
The trained model comprehensively beats the prompted base model and slightly outperforms GPT-5 under the same retrospective setup.
| Metric | Base → Trained | Improvement | |------|-------------------|------| | ECE | 0.1269 → 0.0398 | −68.6% | | Brier | 0.1994 → 0.1453 | −27.1% | | AUROC | 0.6992 → 0.7993 | +14.3% | | Top-10% Lift | 2.34 → 3.07 | +31.3% |
Clinical significance of calibration. AUROC measures ranking; Brier measures probability quality; ECE measures calibration — whether a predicted 70% event actually occurs ~70% of the time. In clinical settings, calibration matters more than ranking: a high-AUROC but poorly calibrated model gives wrong probabilities, leading to misallocated resources (e.g., an ICU transfer based on an inflated 70% death estimate). ECE dropping from 0.1269 to 0.0398 means this bias shrinks dramatically.
Top-10% lift. 3.07 means the highest-risk decile has 3.07× the overall event rate: with a 25% base rate, the top 10% of patients experience roughly 76.8% of events — extremely useful in resource-constrained settings, where reviewing 10% of patients covers over three-quarters of positive events.
---
5. Reasoning Quality: 84% Win Rate in Blind Evaluation
On 50 matched prediction pairs, Gemini 2.5 Flash served as a blind judge comparing the trained vs. base models on clinical reasoning, medical knowledge, grounding, and clinical utility:
| Dimension | Trained model win rate | |------|------| | Clinical reasoning | 78.0% | | Medical knowledge | 92.0% | | Grounding | 78.0% | | Clinical utility | 82.0% | | Overall | 84.0% |
Qualitatively, the trained model more frequently incorporates time-relevant clinical evidence, links patient-specific findings to the predicted outcome, considers alternative future scenarios when expressing uncertainty, and produces more detailed reasoning tied to the evolving clinical course. GRPO training improved not just probability calibration but reasoning quality — the model learns not just to "guess right" but to "think like a physician."
---
6. Engineering Judgment: Why This Paper Matters
Data construction matters more than model scale. GPT-5 is a general-purpose model not fine-tuned for clinical prediction; the trained model, built on the open-source gpt-oss-120b with only LoRA adaptation, slightly surpasses it. The bottleneck is not scale but how data is constructed — Foresight Learning's temporal framework plus a unified natural-language question interface.
The unifying power of a natural-language interface. Traditional methods require separate datasets and classifiers per endpoint. Here, medications, procedures, organ support, microbiology tests, and mortality are all questions answered by one model. New tasks require no retraining — just a new question. This is instruction tuning extended to the clinical domain.
Noisy clinical notes. The paper candidly discusses noisy clinical notes (auto-filled text, templated language, redundant documents). Yet the method's strength is that it needs no human selection of which notes matter: the model trains on full trajectories and learns which text patterns relate to which predictions — shifting feature engineering from humans to the model.
---
7. Limitations and Boundaries
1. Single-center retrospective data: MIMIC-III comes from one center (Beth Israel Deaconess Medical Center) and may not reflect other institutions, populations, or current practice. 2. Note noise and bias: clinical notes are noisy, incomplete, and shaped by documentation behavior, limiting signal quality and introducing bias. 3. Automated pipeline errors: question generation and label resolution both rely on Gemini 2.5 Flash, which may err in question relevance, label assignment, or event timing. 4. Research-use restriction: MIMIC data-use terms confine the model to scientific research — a method demonstration, not a deployable clinical product. 5. Pretraining contamination: neither GPT-5's nor gpt-oss-120b's pretraining data is disclosed; it may include MIMIC-III or derivatives, limiting true out-of-distribution evaluation.
The fifth is especially important: if gpt-oss-120b's pretraining data included MIMIC-III, the "beats GPT-5" conclusion might partly stem from data contamination rather than genuine generalization. The paper is honest about this, which is commendable.
---
8. Conclusion: A Paradigm-Shift Signal for Medical AI
The paper's value is not a new architecture but a new data-construction paradigm. The traditional workflow — manual structured feature extraction → separate models per endpoint → heavy annotation cost. This paper's workflow — raw clinical notes time-ordered → automatically generated natural-language questions → one model answering all questions → lightweight LoRA fine-tuning.
The core shift is from structured encoding to natural-language interfaces. Structured data still matters, but the clinical signal in unstructured text has been severely underestimated. The 6,900 samples are demonstration-scale; sampling more admissions, more split points, and more questions per split could scale to millions of samples — with no architecture redesign, only more written questions.
This "questions as supervision" idea contrasts interestingly with "Useful Memories Become Faulty," which warns about the reliability of LLM auto-summarized memories. Here, LLM-generated questions proved reliable training data — at least for clinical prediction. The key difference: generating questions is a creative task; parsing labels is a discriminative task. LLMs are more reliable at judging right/wrong than at abstract distillation — perhaps a more general rule.
---
9. Open Questions / Follow-ups
References
1. Paper: https://arxiv.org/abs/2605.12817 2. Foresight Learning: Kraljevic et al. (2024), Lancet Digital Health 3. MIMIC-III: Johnson et al. (2016), Scientific Data 4. gpt-oss-120b: OpenAI open-source model 5. GRPO: Shao et al. (2024), DeepSeekMath