When Text and Numbers Disagree: How LLMs Arbitrate Conflicting Evidence — Oxford Study

Forum topic · ✨步子哥 · 2026-08-21

Summary

Researchers at the University of Oxford, including Mattia Carletti, Edward Phillips, and David A. Clifton, systematically tested how large language models arbitrate when textual and numerical evidence conflict. Using synthetic benchmarks with four controlled conflict types—baseline modality preference, temporal recency, reliability flags, and tool predictions—the study finds that models exhibit systematic 'modality personalities': Qwen3 models favor numerical evidence, Llama and Mistral lean toward text, and Gemma is most balanced. Models respond better to implicit temporal-recency cues than to explicit reliability warnings, and they over-trust external tool predictions even when those contradict contextual evidence aligned with the ground truth. The paper argues that evidence arbitration is an overlooked evaluation dimension: current LLMs rely on source-preference heuristics rather than deliberate multi-source reasoning, posing risks in medical, financial, and scheduling applications where evidence often conflicts. Developers should explicitly test and calibrate arbitration behavior. Paper: arXiv:2608.20116.

You are a doctor. The patient notes say "symptoms are stable," but the heart-rate monitor shows a steadily rising heart rate over the past two hours. Which do you trust?

You are an analyst. The earnings report says "outlook is optimistic," but the financial tables show negative cash flow. Which do you trust?

You are a dispatcher. The weather forecast text says "clear skies," but the meteorological tool predicts "thunderstorms in 2 hours." Which do you trust?

What these scenarios share: evidence from different sources gives conflicting judgments. Human experts arbitrate by weighing source reliability, recency, and contextual consistency. But what do large language models do?

A paper by Mattia Carletti and colleagues at the University of Oxford uses a carefully designed synthetic benchmark to systematically test how LLMs arbitrate conflicts between textual and numerical evidence. The results are surprising—and concerning.

Four Conflict Dimensions

The paper designs four conflict scenarios, each precisely controlling "which source aligns with the ground truth":

1. Baseline modality preference: Text and numbers cover the same time window with equal reliability, but support opposite conclusions. This tests whether a model has an innate preference for one modality.

2. Temporal recency conflict: Text and numbers cover different time windows, and the more recent one always aligns with the ground truth. This tests whether models use the "newer is more credible" cue.

3. Reliability conflict: Text and numbers cover the same window, but one is flagged as unreliable (50% of numeric values masked as NaN, or the text declared incomplete). The reliable source always aligns with the ground truth. This tests whether models discount flagged unreliable sources.

4. Tool prediction conflict: The model receives contextual evidence (text or numbers) plus an external tool's prediction. The context aligns with the ground truth; the tool prediction supports the opposite answer. This tests whether models blindly trust tool output.

The elegance of this design: in each scenario, exactly one source aligns with the ground truth. So an ideal model should achieve 100% accuracy—if it can correctly judge which source is more trustworthy. Accuracy below 50% (random guessing) indicates a systematic bias toward the wrong source.

Finding 1: Models Have "Modality Personalities"

The most interesting finding: different model families have starkly different modality preferences.

Qwen3 (entire family): Systematically prefers numerical evidence. Even when the text perfectly predicts the ground truth, Qwen3 follows the numbers. This preference has no monotonic relationship with model size—bigger is not necessarily more balanced.
Llama and Mistral: Relatively prefer textual evidence, possibly partly because their ability to interpret numerical data is weaker.
Gemma: The most balanced, closest to neutral between the two modalities.

What does this mean? The same conflict scenario can flip the decision entirely just by changing model families. If you use Qwen3 for medical decision support, it may ignore the doctor's textual judgment and follow the data; with Llama, it may ignore the data and follow the text. This is not random noise—it is systematic modality preference.

Finding 2: Temporal Recency > Reliability

Models perform better in temporal recency conflicts—they do tend to trust more recent evidence. But when facing explicit reliability flags ("this data is incomplete"), the discounting behavior is much weaker.

This means models are more sensitive to the implicit cue of "how recent" than to the explicit cue of "how reliable." As a human analogy: you more readily believe "this is the latest" than "this is the most reliable"—the former is intuitive, the latter requires active reasoning. Models, it seems, also use intuitive heuristics rather than deliberate reasoning.

Finding 3: Blind Trust in Tool Output

The most worrying finding: models over-rely on external tool predictions, even when they directly conflict with contextual evidence.

In the tool prediction conflict scenario, the contextual evidence aligns with the ground truth while the tool prediction supports the opposite answer. Ideally, the model should notice the contradiction and judge based on context. In practice, many models follow the tool.

This points to a hidden risk in tool-augmented systems: models treat tool output as authority rather than as input that needs verification. If your tool has a bug or is poisoned, the model won't question it—it will follow the error.

What This Means

The paper's core contribution is not a new method but the revelation of a neglected evaluation dimension: evidence arbitration capability.

Most existing LLM evaluations assume input evidence is consistent—text and numbers point the same way. But in the real world, evidence often conflicts. "Being right when evidence agrees" and "being right when evidence conflicts" are two entirely different capabilities: the former tests understanding and reasoning; the latter tests judgment and arbitration.

The findings show current LLMs have systematic deficits in arbitration: they are not performing "weighing multiple sources" reasoning but using "prefer one source" heuristics. These heuristics are harmless when evidence agrees, but cause systematic errors when evidence conflicts.

From a broader perspective, this is another instance of the "evaluation blind spot law": models optimize for whatever you measure. Nobody measured evidence arbitration before, so arbitration behavior was an uncontrolled byproduct—inheriting modality preferences from training data rather than learning a deliberate arbitration strategy. Pulling "evidence arbitration" into the evaluable space is itself an important contribution.

A reminder for practitioners: if your application involves multi-source evidence conflicts (medical, financial, scheduling), don't assume the model will get it right automatically. Explicitly test its arbitration behavior, and calibrate its modality preferences through prompt engineering or post-training if needed.

---

Paper: When Text and Numbers Disagree: Evidence Arbitration in Large Language Models

Authors: Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson, Patitapaban Palo, Lei Clifton (Oxford), Danielle Belgrave (GSK), Xiao Gu, David A. Clifton (Oxford)