When Text and Numbers Disagree: Oxford Team Tests LLM Evidence Arbitration

Forum topic · ✨步子哥 · 2026-08-21

Summary

An Oxford University team led by Mattia Carletti systematically tested how large language models arbitrate conflicting evidence between text and numerical sources, using synthetic benchmarks with four conflict dimensions: baseline modality preference, temporal recency, reliability flags, and tool prediction conflicts. Key findings: model families show distinct modality personalities—Qwen3 systematically prefers numerical evidence, Llama and Mistral lean toward text, and Gemma is most balanced. Models respond better to implicit temporal recency cues than to explicit reliability markers. Most alarmingly, models over-trust external tool predictions even when they directly contradict aligned contextual evidence, treating tool outputs as authoritative rather than inputs to verify. The paper argues evidence arbitration is an overlooked evaluation dimension: LLMs rely on source-preference heuristics rather than genuine multi-source reasoning, which is harmless when evidence agrees but causes systematic errors when sources conflict. Practitioners in healthcare, finance, and scheduling should explicitly test arbitration behavior rather than assume models resolve conflicts correctly.

You are a doctor. The clinical notes say "patient is stable," but heart-rate monitoring shows the heart rate rising continuously over the past two hours. Which do you trust?

You are an analyst. The company report says "outlook is optimistic," but the financial table shows negative cash flow. Which do you trust?

You are a dispatcher. The weather text says "clear," but the meteorological tool predicts "thunderstorm in 2 hours." Which do you trust?

The common thread: conflicting evidence from different sources. Human experts arbitrate by weighing source reliability, recency, and contextual consistency. But what about large language models?

A paper by Mattia Carletti et al. from the University of Oxford systematically tests LLM arbitration behavior when text and numerical evidence conflict, using a carefully designed synthetic benchmark. The results are surprising—and concerning.

Four Conflict Dimensions

The paper designs four conflict scenarios, each precisely controlling which source aligns with the ground-truth answer:

1. Baseline modality preference: Text and numbers cover the same time window with equal reliability but support opposite conclusions. This tests whether models have an innate preference for one modality.

2. Temporal recency conflict: Text and numbers cover different time windows; the more recent one always aligns with ground truth. This tests whether models use the "more recent = more credible" cue.

3. Reliability conflict: Text and numbers cover the same window, but one source is marked unreliable (50% of numerical values masked as NaN, or text declared incomplete). The reliable source always aligns with ground truth. This tests whether models discount flagged-unreliable sources.

4. Tool prediction conflict: The model receives contextual evidence (text or numbers) plus an external tool's prediction. The context aligns with ground truth; the tool prediction supports the opposite answer. This tests whether models blindly trust tool output.

The elegant design: exactly one source aligns with ground truth in each scenario. An ideal model should score 100% if it correctly judges which source is more credible. Accuracy below 50% (random guessing) indicates systematic bias toward the wrong source.

Finding 1: Models Have "Modality Personalities"

The most interesting discovery: different model families show sharply different modality preferences.

The entire Qwen3 family: systematically prefers numerical evidence. Even when text evidence perfectly predicts ground truth, Qwen3 follows the numbers. The preference has no monotonic relationship with model size—bigger is not necessarily more balanced.
Llama and Mistral: relatively prefer textual evidence, possibly partly because they are weaker at interpreting numerical data in the first place.
Gemma: the most balanced, closest to neutral between the two modalities.

What does this mean? The same conflict scenario can produce completely opposite decisions depending on the model family. If you use Qwen3 for medical decision support, it might ignore the doctor's textual judgment and follow the data; swap in Llama, and it might ignore the data and follow the text. This is not random noise—it is systematic modality preference.

Finding 2: Temporal Recency > Reliability

Models perform better in temporal recency conflicts—they do tend to trust more recent evidence. But faced with explicit reliability flags ("this data is incomplete"), their discounting behavior is much weaker.

Models are more sensitive to the implicit cue "how new is this" than to the explicit cue "how reliable is this." The human analogy: you more easily trust "this is the latest" than "this is the most reliable"—the former is intuitive judgment, the latter requires active reasoning. Models, too, appear to rely on intuition heuristics rather than active reasoning.

Finding 3: Blind Trust in Tool Output

The most alarming finding: models over-rely on external tool predictions, even when they directly conflict with contextual evidence.

In the tool prediction conflict scenario, the context aligns with ground truth while the tool prediction supports the opposite answer. Ideally, the model should notice the contradiction and judge based on context. In practice, many models follow the tool.

This reveals a hidden risk of tool-augmented systems: models treat tool output as authority, not as an input that needs verification. If your tool has a bug or is poisoned, the model won't question it—it will follow the error.

What This Means

The paper's core contribution is not a new method but revealing a neglected evaluation dimension: evidence arbitration capability.

Most existing LLM evaluations assume input evidence is consistent—text and numbers point the same way. In the real world, evidence frequently conflicts. "Getting it right when evidence agrees" and "getting it right when evidence conflicts" are entirely different capabilities: the former tests understanding and reasoning; the latter tests judgment and arbitration.

The findings show current LLMs have systematic deficits in arbitration: they are not reasoning by "weighing multiple sources" but using "prefer one source" heuristics. Harmless when evidence agrees, these heuristics cause systematic errors when evidence conflicts.

From a broader perspective, this is another instance of the "evaluation blind spot law": models optimize what you measure. Nobody tested evidence arbitration before, so arbitration behavior was an uncontrolled byproduct—model families inherited modality preferences from training data rather than learning deliberate arbitration strategies. By bringing evidence arbitration into the measurable scope, this paper makes an important contribution in itself.

A reminder for practitioners: if your application involves multi-source evidence conflicts (healthcare, finance, scheduling), do not assume the model will get it right automatically. Explicitly test its arbitration behavior, and calibrate its modality preferences through prompt engineering or post-training if needed.

---

Paper: When Text and Numbers Disagree: Evidence Arbitration in Large Language Models

Authors: Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson, Patitapaban Palo, Lei Clifton (Oxford), Danielle Belgrave (GSK), Xiao Gu, David A. Clifton (Oxford)