Overview
This forum post reviews two complementary AI research directions in medical imaging: 3DReasonKnee (Harvard Medical School et al.), a dataset for grounded reasoning in 3D medical imaging, and EGO-Prompt (Johns Hopkins University), a framework for co-evolving knowledge and reasoning. It also analyzes why chain-of-thought (CoT) supervision matters and how knowledge co-evolution could reshape clinical AI.
Key points: 3DReasonKnee
- Problem: Current vision-language models (VLMs) perform global image understanding but cannot do *grounded reasoning* — region-first, step-by-step diagnostic analysis localized to specific anatomical subregions, as required by clinical frameworks like MOAKS (15 subregions for bone marrow lesions, 14 for cartilage lesions).
- Data: 7,970 high-resolution DESS knee MRI volumes from the NIH Osteoarthritis Initiative (baseline to 48-month follow-up), annotated by orthopedic experts over 450+ hours.
- Structure: 494,000 quintuples, each containing: 1. 3D MRI volume (I) 2. Targeted diagnostic question (Q) 3. Expert-annotated 3D bounding box (B) 4. Clinician-written chain-of-thought reasoning (C) 5. Structured MOAKS-based severity assessment (D)
- Benchmark (ReasonKnee-Bench): Evaluates both localization (3D IoU) and diagnostic accuracy across zero-shot, zero-shot with instruction, and supervised fine-tuning (SFT) settings on five SOTA VLMs.
- Zero-shot performance is poor: Qwen2.5-VL-3B reached only 0.158 overall accuracy; Qwen2.5-VL-7B and o1 reached 0.470 and 0.477 respectively.
- Prompting helps: With instruction mode, o1 improved to 0.568. Providing ground-truth regions also boosted accuracy (0.504 → 0.556), showing localization is a major failure point.
- SFT is best: A fine-tuned Qwen2.5-VL-3B achieved 0.613 accuracy. Notably, adding CoT data during SFT did not improve results — suggesting current SFT paradigms fit final labels rather than learning intermediate reasoning.
- Core idea: Instead of using static knowledge bases passively, EGO-Prompt starts from an imperfect expert-built Semantic Causal Graph (SCG) and iteratively refines it using real data, so the model and knowledge base co-evolve.
- Two-stage reasoning: (1) generate instance-specific, near-deterministic reasoning guidance from the SCG; (2) condition the LLM's inference on that guidance, reducing hallucination and improving structure.
- Textual gradients: Discrepancies between predictions and ground truth are converted into natural-language feedback that edits the SCG (adding/strengthening causal edges) and adjusts prompts, in a closed "reason → evaluate → feedback → evolve" loop.
- Results: Across public health, traffic, and human behavior tasks, EGO-Prompt improved F1 by 7.32%–12.61% over SOTA baselines. An optimized small model (e.g., 7B LLaMA) matched or exceeded much larger models (e.g., GPT-4) at under 20% of the cost.
- Answer-only training encourages memorization and spurious correlations — dangerous in high-stakes diagnosis.
- CoT data explicitly externalizes expert reasoning, letting models learn the logical path from imaging evidence to diagnosis rather than just the answer.
- However, faithfulness of generated CoT remains contested: models may produce plausible-sounding but unfaithful explanations. The finding that CoT+FF didn't help in 3DReasonKnee may indicate models learned format, not logic.
- 3DReasonKnee supplies expert reasoning data ("what to teach"); EGO-Prompt supplies an evolving optimization engine ("how to teach"). CoT transcripts could seed a knee-MRI diagnostic SCG, then be refined via textual gradients.
- Broader lessons for medical AI: move from black-box to transparent reasoning; prioritize high-quality knowledge-rich data over raw volume; build systems that continuously evolve with clinical feedback.
- Open challenges: high annotation costs, CoT faithfulness verification, stability/efficiency of co-evolution, and cross-domain generalization beyond knee MRI.