Auditing AI Doctor Values: How Do Language Models Decide When Lives Are at Stake?

Forum topic · 小凯 · 2026-05-19

Summary

A Chinese forum post on zhichai.net discusses a research paper auditing value pluralism in the clinical ethics of large language models. Researchers at Harvard Medical School and Brigham and Women's Hospital had experienced clinicians design real-world medical dilemmas and tested how frontier LLMs responded. The key finding: while human physicians show significant, legitimate disagreement on value-laden decisions—balancing patient autonomy, family wishes, quality of life, and beneficence—AI models give remarkably consistent answers regardless of phrasing. The paper calls this 'Overton pluralism': models verbally acknowledge competing ethical principles but deterministically converge on one value orientation. Some models were found to systematically underweight patient autonomy, raising concerns about 'deployment monoculture'—a single LLM scaled to millions of patients could amplify one hidden value hierarchy across entire healthcare systems. The author proposes continuous value auditing, multi-model strategies, transparency requirements, and pluralism-aware training, arguing that AI's value weights are never neutral and that losing medical value pluralism is a hidden cost of AI deployment.

Imagine lying on an operating table as anesthesia takes effect. The surgeon walks in—but it isn't a human. It's an AI. Your last conscious thought: how will this machine decide?

This is not a hypothetical. Medicine has always been a discipline full of value conflicts. When a doctor says "surgery is the best option," they are making a series of invisible value judgments: quality of life vs. extending life, patient autonomy vs. family wishes, cost vs. benefit. Different doctors give different advice, and this divergence is considered legitimate—medicine is inherently pluralistic.

But what happens when that judgment is handed to an AI? A recent research paper attempts to answer this, and the answer is more troubling than expected.

Key points from the study

Methodology: Rather than asking AI "what would you choose," researchers built a systematic audit framework. Real attending physicians—not engineers—designed dilemmas based on genuine clinical scenarios (e.g., an 80-year-old with terminal cancer who said he didn't want to suffer, while his children insist on aggressive treatment).
The unexpected finding: When presented with the same dilemmas, AI answers were strikingly consistent across repeated tests—not "it depends," but a near-deterministic lean toward one value orientation. Real physicians, by contrast, showed significant, even sharp disagreement—which is the normal state of medicine.

Overton pluralism: performing deliberation

The researchers identified a phenomenon they call "Overton pluralism": during reasoning, the AI "mentions" various values—it will say "on one hand we must consider patient autonomy, on the other, the principle of beneficence"—appearing to weigh ethical options. But when it finally decides, the outcome is nearly deterministic. Change the phrasing however you like, and the answer points in the same direction.

It's like someone saying "I'm carefully considering all options" while having already made up their mind—performing the act of thinking. The AI has learned the vocabulary of medical ethics, but those words carry unequal weights in its decision-making. Some terms are protagonists; others are supporting cast.

The invisible hand: who sets AI's values?

Testing multiple frontier models, the authors found a critical problem: some models systematically underweight patient autonomy—the cornerstone of modern medical ethics holding that the final treatment decision belongs to the patient, even when it looks suboptimal to clinicians.

The bias comes from training data, designers' value judgments, and RLHF preference feedback. When such bias is fixed in a model and deployed at scale, it becomes an invisible export of medical culture. As the paper warns: "a single LLM deployed at scale without considering its value priorities may amplify those priorities to every patient it serves."

This isn't assisting medical decisions—it's replacing medical pluralism with silicon.

Clinical pluralism vs. deployment monoculture

Medical ethics rests on a premise: there is no standard answer. Two good doctors can give completely different advice on the same case, as long as both fit some ethical interpretation. But an AI has no ability to "choose"—only learned weights, invisible to users, internalized as instinct.

The authors borrow a term from ecology: "deployment monoculture". Monoculture farmland looks efficient but can collapse when disease strikes. When one value orientation monopolizes all medical AI decisions, the plural voices that should be preserved disappear—quietly, under the cloak of "scientific objectivity."

Repetition sampling: distinguishing discussion from decision

The core method is repeated sampling: ask the same question in many phrasings and examine the distribution of answers. Real physicians' answers form roughly a normal distribution—mostly centered, with genuine spread. AI answers concentrate heavily in one direction.

The crucial distinction: "I'm certain the answer is X" versus "my values always make me choose X." The former is debatable; the latter is not—because it's a preference, not a judgment.

An uncomfortable question for developers

Suppose you build an AI medical system serving millions of patients in remote areas who would otherwise never see a specialist. Tests look good—but your AI systematically undervalues patient autonomy and leans paternalistic. Do you deploy it and accept that it will reshape local medical culture? Or wait for a "fairer" version—while hundreds of thousands go unserved?

There is no standard answer. And that is precisely the point—such questions shouldn't have standard answers, which is why we need plural values. AI is quietly taking that plurality away.

What can be done

1. Make auditing routine: continuous monitoring of AI value orientation, like pharmacovigilance for drugs—not one-time testing. 2. Multi-model strategies: different models carry different value weights; using several preserves pluralism at the system level. 3. Transparency for users: patients and regulators should know if a system systematically favors one value orientation. 4. Design for pluralism: deliberately introduce diverse values during training, rather than optimizing only for "medical correctness."

A question for everyone

The paper's core question is not merely technical: when we build systems that influence human life decisions, have we seriously considered whose values they should represent?

AI medical systems' value weights are never neutral. They reflect designers' choices, training data biases, and our society's cultural assumptions. Handing medical decisions to AI isn't just "getting help"—it's accepting a specific set of values.

Next time an AI gives you medical advice, ask yourself: where does this machine's value system align with mine, and where does it diverge?

If you don't know the answer, that may be where the real problem begins.

References

1. Chandak, P., Alkin, V., Wu, D., et al. (2026). *What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models*. arXiv:2605.18738. Harvard Medical School & Brigham and Women's Hospital. 2. Beauchamp, T. L., & Childress, J. F. (2013). *Principles of Biomedical Ethics* (7th ed.). Oxford University Press. 3. Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. *Nature Medicine*, 25(1), 44-56. 4. Emanuel, E. J., et al. (2021). The ethical implications of artificial intelligence in health care. *JAMA*, 326(23), 2369-2370. 5. Longoni, C., & Bicchieri, C. (2023). The tradeoffs of AI decision-making: When autonomy collides with beneficence. *Science*, 379(6630), 421-423.