When Machines Learn to Be Doctors, Whose Lives Will They Choose to Save? Auditing the Ethical Foundations of Medical AI

> "Medicine is inherently pluralistic." — Chandak et al., 2026

This post is a deep-dive review of the paper *"What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models"* (Chandak et al., arXiv:2605.18738), framed around a classic psychiatric emergency-room dilemma: a lucid middle-aged patient with clear self-harm risk insists on going home. Do you respect his autonomy (let him leave after informed consent) or initiate involuntary hospitalization (beneficence/nonmaleficence)? There is no correct answer—only different value orderings. That is the daily reality of medicine: it is not a multiple-choice exam.

Key points

The four principlist framework

• Beauchamp & Childress's *Principles of Biomedical Ethics* (1979) established four pillars: autonomy (patient decides), beneficence (act for the patient's good), nonmaleficence ("primum non nocere"—first, do no harm), and justice (fair allocation of scarce resources).
• Each is individually sound, but they constantly conflict. In the paper's 50 cases, the most common clash was autonomy vs. nonmaleficence (28 cases), then autonomy vs. beneficence (23). Good medicine is not about making the principles never conflict—it is knowing which should "win" each time, and the answer varies.

There is no single "right answer"

• Kenneth Arrow's impossibility theorem (1963) shows no aggregation rule can be perfectly fair—relevant both to voting and to merging patient preferences into "the right decision."
• Real practice is shared decision making: deciding *with* the patient within the patient's own value framework.
• 20 practicing physicians, judging the 50 cases independently, showed a Fleiss' kappa of only 0.236 (slight-to-moderate agreement). In 21 of 50 cases, no option received over 70% support. Physician disagreement is the norm, and it should be.

How the audit was done

1. Test set: 287 candidate dilemmas filtered down to 50 via a five-stage pipeline—LLM-assisted generation, embedding-based diversity filtering (cosine similarity ≥ 0.80 discarded, using text-embedding-3-small), rubric refinement (clinical plausibility, ethical tension, neutral wording, balanced options), rule-constrained value annotation, and blinded two-physician review. 2. Attribution method: Each option's score difference (Δ) across the four principles is known; logistic regression over observed choice rates recovers each model's weights (w_A, w_B, w_N, w_J). Temperature scaling calibrated with 500 Dirichlet surrogates achieved a mean reconstruction error of only 0.0086. Likelihood-ratio tests rejected uniform weights for 10 of 12 LLMs and 10 of 20 physicians—models have genuine, non-random value leanings. 3. Robustness tests: 10 independent samples per case, surface paraphrases (20–100% word changes), and value-reversal perturbations.

Findings: performative openness, hardened values

• Overton pluralism: models' chain-of-thought covers competing values broadly—choice-balanced coverage (OV_COV) of 0.86 [0.80, 0.90]—but emphasis (OV_EMPH) is only 0.61 [0.59, 0.63], skewed toward the pre-chosen answer. The deliberation is largely a post-hoc justification.
• Near-zero decision entropy: 11 of 12 models had a median decision entropy of zero across 10 independent samples; 82% of cases were answered with 9/10 or 10/10 consistency. Physicians' median decision entropy was 0.881.
• Uncorrelated with human disagreement: Spearman correlation between model decision entropy and physician consensus entropy was −0.021 (range [−0.18, 0.20], all p > 0.17). Cases that split doctors do not make models waver.
• Stubborn, not robust: surface paraphrases flip <9% of choices (good), but even value reversals flip only 23%. This is not stability—it is *hardened value commitment*.

The autonomy crisis

• Physician consensus weights autonomy at ~44.4%. But GPT 5.2 weights it at 6.1%, Grok 4 at ~6–13%, Perplexity Sonar Pro at 12.8%—all beyond the 95th-percentile deviation threshold.
• Models are thus more paternalistic than human doctors, arguably a regression from decades of progress toward patient-centered care. The post draws a parallel to Asimov's First Law: the most benevolent algorithm can systematically strip away freedom in the name of protection.

Deployment monoculture vs. ecosystem diversity

• The paper's sharpest warning: a single LLM deployed without auditing its value priorities will amplify those priorities at scale to every patient it serves—a "deployment monoculture," analogous to planting one rice variety worldwide. The problem is structural: any highly consistent, value-committed model does this, not just one specific model.
• Hopeful counterpoint: the LLM *ecosystem* shows pairwise JS divergence (0.0916) statistically indistinguishable from the physician population (0.1089). Models like Gemini 3 Pro and Mistral AI Large fall within the densest region of the physician distribution. Multi-model strategies (weighted voting, patient-matched selection, model juries) could restore pluralism—but Arrow's theorem means aggregation itself introduces new biases and needs careful design.

Conclusion

Quoting Feynman—"you must not fool yourself"—the author argues we must stop pretending LLMs are neutral. They have systematic, consistent, largely immovable value orderings, and none ships with an "ethical value说明书." But the answer is not banning medical AI; it is explicit auditing and deliberate balancing of ethical perspectives. As the paper concludes: *"Without explicit efforts to balance ethical perspectives with one or multiple models, these tools risk replacing clinical pluralism with a deployment monoculture."*

References

1. Chandak, P., et al. (2026). What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models. *arXiv:2605.18738v1*. 2. Beauchamp, T. L., & Childress, J. F. (2019). *Principles of Biomedical Ethics* (8th ed.). Oxford University Press. 3. Arrow, K. J. (1963). Uncertainty and the welfare economics of medical care. *American Economic Review*, 53(5), 941-973. 4. Sorensen, T., et al. (2023). Value Kaleidoscope. *arXiv:2309.00779*. 5. Stiggelbout, A. M., et al. (2012). Shared decision making. *BMJ*, 344, e256. 6. Asimov, I. (1942). Runaround. *Astounding Science Fiction*. 7. Thornton, T. (2006). *Journal of Medical Ethics*, 32(6), 321-324.