TRIAGE: Giving AI Clinical Dialectical Thinking — When LLMs Learn to Argue Each Diagnosis Separately
Researchers from KAIST, AITRICS, and the University of Wisconsin-Madison found a core bug in LLM-based medical diagnosis: when an AI generates reasoning first and then predicts, its risk probabilities collapse toward 0 or 1 — like someone deciding before thinking and then only gathering supporting evidence. This destroys the gradient between "low-risk" and "high-risk" patients. TRIAGE's solution: have the AI generate independent reasoning for each candidate outcome, like prosecution and defense in a courtroom debate, then extract continuous, calibrated, cross-patient-comparable risk scores. Result: a 3.3% AUPRC improvement on a 4B model and an 81% reduction in calibration error, with explanations more clinically logical than SOTA baselines.
Key points
- Risk polarization: On MIMIC-III mortality prediction, gpt-oss-120b's predicted class probability exceeded 99.98% with near-zero variance when using reasoning-first chains, versus 86.4%±18.8% for answer-only prediction.
- Root causes: (1) *Pre-commitment* — 71.7% of reasoning chains state a conclusion before the answer token; (2) *one-sided confirmation bias* — reasoning only presents evidence supporting the committed outcome.
- Dialectical reasoning: For binary prediction, TRIAGE generates two separate rationale chains (one supporting each outcome), with no intermediate conclusions allowed before the final decision, so the model must synthesize both sides itself.
- Implicit probability extraction: Instead of verbalized probabilities ("85% chance"), TRIAGE reads softmax over answer-token logits — yielding continuous, calibrated, comparable risk scores.
- Two-stage training: SFT on dialectical trajectories generated by GPT-5.1 or Kimi K2 Thinking (on Qwen3-4B-Base), then GRPO reinforcement learning with a batch-level hinge-loss contrastive reward that directly optimizes cross-patient risk ranking.
- TRIAGE_SFT+RL is first overall (average rank 1.58): P12 87.3/59.0, P19 89.3/53.8, MIMIC 86.7/54.1.
- Strongest baseline GRU-D averages 86.9/56.7; TRIAGE improves AUPRC by 3.3% on average.
- Zero-shot frontier LLMs rank last: GPT-5.1 (rank 10.50) and gpt-oss-120b (rank 11.67) — general reasoning does not transfer directly to clinical ISMTS prediction.
- Ablation: standard one-sided rationale *hurts* performance (AUPRC 43.1 vs. 53.4 for answer-only, even with 10-sample averaging), directly confirming risk polarization.
- ECE drops to 0.03–0.04 across datasets (vs. 0.16–0.21 for GRU-D / STraTS) — roughly 81% lower calibration error; Brier Score ~49% lower.
- Total score 7.744 vs. 6.474 for STraTS+IG+GPT post-hoc explanations (+1.269), with the largest gains in interpretive summary (+0.902) and alternative-diagnosis explanation (+0.288).
- Only binary prediction validated; multi-class extension is future work.
- Rationale generation adds latency for low-latency settings.
- Evaluation used LLM-as-judge rather than clinical experts; generated rationales may contain errors and must not replace qualified clinical judgment.
- MIMIC-III data was processed locally under PhysioNet protocols; no data sent to third-party LLM APIs; local inference used open-weight models (Kimi K2 Thinking).
- Paper: *TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs* — Hyeongwon Jang, Gyouk Chu, Changhun Kim, Joonhyung Park, Hangyul Yoon, Eunho Yang (KAIST, AITRICS, University of Wisconsin-Madison), arXiv: 2606.09030
- Code: https://github.com/HyeongWon-Jang/TRIAGE
- Datasets: P12 (in-hospital mortality), P19 (sepsis 6h onset), MIMIC-III (in-hospital mortality)
Input representation
Clinical data is irregularly sampled medical time series (ISMTS). TRIAGE serializes the series into text with set-based encoding (timestamps, vitals, labs), letting the LLM read the patient timeline without a specialized time-series encoder — unlike black-box baselines such as GRU-D, Raindrop, or STraTS.
Results
Discrimination (AUROC / AUPRC):
Calibration:
Explanation quality (IDEA rubric, LLM-as-judge on 200 samples):
Robustness: With 10–50% of variables randomly hidden, TRIAGE stays competitive or leads on AUPRC, suggesting dialectical structure is naturally robust to missing evidence.
Limitations and ethics
Takeaway
An LLM's "confidence" in medical diagnosis is often an artifact — not genuine belief, but the reasoning structure forcing it. By making the model defend every candidate outcome, TRIAGE restores risk scores to the model's true implicit probability distribution. The paradigm shift — from "decide first, find evidence after" to "examine all evidence before deciding" — mirrors how good clinicians think. The same risk-polarization trap may affect credit scoring, insurance pricing, and threat detection: anywhere AI reasons before judging.