MedGame: Turning Medical Records into Interactive Narrative Games with LLM Directors
A Medical Student's Dilemma
Imagine you are a medical student preparing for clinical rotation exams. You receive a case record:
> 55-year-old male, chest pain for 3 hours, radiating to the left arm, with sweating. History of hypertension. ECG shows ST elevation...
The traditional approach is to read, memorize, and answer questions. But real clinical reasoning doesn't work that way. Doctors face progressively unfolding information: what to ask first, what to examine next, when to order tests, when to adjust treatment—each step influences the next. A medical record is static; reasoning is dynamic. This mismatch is exactly what MedGame aims to solve.
Core Idea: A Two-Engine Architecture
MedGame's design philosophy is separation of concerns: one engine transforms medical records into branching narratives, and another turns those narratives into executable multimodal games.
Engine 1: Medical Narrative Designer
The input is a static case summary; the output is a structured narrative graph with state nodes and decision points. Each node is a clinical scenario ("ER initial diagnosis," "lab results return," "patient suddenly deteriorates"), and each decision point offers multiple branches ("immediate thrombolysis," "coronary angiography first," "conservative treatment"). The model doesn't generate concrete visuals—only functional descriptions such as "consultation room," "resuscitation room," "family meeting room."
Engine 2: Story Director
Given the narrative graph, this engine expands each node into concrete multimodal generation tasks: scene images, character images, dialogue, vitals panels. The key design is organizing these tasks into a directed acyclic graph (DAG)—some tasks must wait for prerequisites (e.g., the "resuscitation room scene" depends on the "patient deterioration" node), while others can run in parallel. This turns multimodal generation from a linear pipeline into a dependency-aware scheduling problem.
Separating the two engines has engineering significance: narrative design requires medical knowledge, while story direction requires visual orchestration. Optimizing each with different models or fine-tuning data is more efficient than one giant model doing everything.
Data and Scale
The team built 5,000 clinical cases across eight specialties: cardiology, endocrinology, gastroenterology, hematology/oncology, nephrology, neurology, respiratory and critical care, and rheumatology—4,000 for training and 1,000 for testing. Each case was converted into a branching narrative game where players (medical students) make choices at each decision point, and the story advances accordingly.
Results: Commercial Models Crush Open Source
Evaluation has two layers: automatic structural validation (JSON format compliance, business logic consistency) and GPT-5.2-as-judge content quality scoring.
Strict Validation:
- Claude-Sonnet-4.5: 99.5%
- Gemini-3-Pro: 100.0%
- Qwen3-32B: 79.4%
- Gemma-3-27B / MedGemma-27B: below 60%
- Claude-Sonnet-4.5 / Gemini-3-Pro: over 99%
- Open-source models: 56.50%–80.30%
- The two-engine decoupling is an engineering-sound decision, letting medical reasoning and visual orchestration optimize independently
- DAG scheduling turns multimodal generation from a linear pipeline into a dependency graph, with clear efficiency gains
- 5,000 cases across eight specialties is a substantial data scale
- Evaluation relies mainly on GPT-5.2 as judge; despite human expert calibration, LLM-as-Judge bias remains an open problem
- Case sources have limited representativeness
- No reported learning outcomes from actual student use—the game was built, but the core question "does playing the game improve clinical reasoning more than reading records?" remains unanswered
- The large open-source gap means the system is currently affordable only for users of commercial LLMs
- Paper: https://arxiv.org/abs/2607.21570
- Code: https://github.com/med-air/MedGame
Story Director task (Sample-wise Validation):
The gap is clear: commercial frontier models can reliably turn records into executable games, while open-source models fall behind even on structural compliance—and the medical accuracy gap is larger. Fine-tuning narrows it—fine-tuned Qwen3.5-27B rose from 79.4% to 99.1% on Strict Validation—but requires separate fine-tuning with dedicated narrative and director data.
An Honest Assessment
Strengths:
Limitations:
A noteworthy detail: the Story Director's multimodal rendering supposedly "adds perceptual value beyond pure text interaction"—but the paper doesn't quantify this gain. Is gamified learning actually more effective than text, or just flashier? Follow-up work must answer this.
Conceptual Takeaway
MedGame's essence is converting a static knowledge artifact into a dynamic decision trajectory. The idea isn't limited to medicine: legal cases, historical events, engineering fault diagnosis—any scenario where "the conclusion is known but the reasoning process matters" applies. The two-engine architecture (one for narrative structure, one for multimodal orchestration) is a transferable template.