Evidence-Type Competition: Why LLMs Copy from Observational Data Even When Trained on Interventions

Background: Hypothesis H1

A 2026 arXiv paper from Tsinghua University (arXiv:2607.29484) rigorously tests a common intuition in LLM causal reasoning: increasing the proportion of interventional data in pre-training should improve causal inference ability (H1). The authors note this hypothesis had never been tested under strictly controlled conditions, requiring:

• Known causal graph structures
• Computable causal ground truth
• Leakage-free protocols (non-overlapping train/test sets)
• Controllable confounder strength

A synthetic world generator was used to satisfy these requirements, since real-world data cannot.

Simpson-Paradox World Design

The experiments use Simpson-paradox worlds where:

• Observational correlation is negative: Corr_obs(X, Y) < 0
• True causal effect is positive: eff(X→Y) > 0

This creates a causal trap: an observational-only learner gets the direction wrong. Six experiments vary intervention ratio α from 0 (pure observational) to 1.0 (pure interventional).

Key Findings

H1 is Refuted

• Direction accuracy across six α levels: 0.5 / 0.3 / 0.2 / 0.4 / 0.4 / 0.4 — no monotonic improvement
• Simpson slope magnitude grows monotonically: 0.169 → 0.384
• Direction consistently stays negative, copying observational sign

This is the magnitude–direction duality: the model learns effect magnitude from interventions but copies direction from observational context, like a student drawing the right gravity magnitude but flipping its direction by copying the neighbor's answer.

The Switch Lives in Context, Not Weights

Three inference-time intervention experiments localize the failure:

1. Doubling intervention evidence → no effect; direction stays wrong 2. Erasing observational evidence rows → direction immediately corrects; ratio_true jumps from −0.09 to +0.56 3. Activation patching → middle-layer observational evidence rows are the critical site; erasing them restores causal reasoning

Content manipulation shows the effect is content-mediated, not format-mediated: replacing observational values (preserving structure) releases the suppression. Even 2 observational records begin diluting the signal; 4 records fully capture it.

26% Sampling Noise Floor

A methodological contribution: even a perfect least-squares interpolator makes 26% direction errors when evaluated via probes, because probes sample only 4 intervention points each sampled once — a 4-coin-flip estimator with high variance. Evidence averaging across 8 groups lowers the noise floor from 26% to 9%. This is a warning for all probe-based causal evaluations: the evaluator itself may be noisier than the model.

CLadder Audit: Positive-Effect Prior

An external audit on the CLadder benchmark reveals a learned positive-effect prior with two layers:

• Surface layer: removable through in-distribution retraining with symbol randomization
• Deep layer: persists as a default out-of-distribution

This suggests unstable causal-reasoning benchmarks may reflect prior-guessing rather than reasoning.

Implications

1. Capability in weights, switch in context: The model has learned causal ability but observational evidence suppresses it during inference. Two statistical evidence types compete, and the model defaults to the wrong one. 2. Training-ratio is not a universal knob: When root cause is inference-time evidence competition, adjusting training ratios fails — inference-time context intervention is required. 3. Evaluation blind-spot law: Aggregate accuracy and magnitude metrics miss directional failure; unmodeled sampling noise inflates apparent evaluation randomness. 4. Simpson paradox as diagnostic tool: Using worlds where observation and intervention point opposite directions reveals evidence-selection strategies invisible in direction-aligned worlds.

Limitations

1. Synthetic worlds: 25.7M-parameter models with synthetic causal structures may not generalize to real LLMs on real data 2. Scale check: A 0.93B-parameter validation showed rate-level persistence but absolute intervention benefit shrank 4× — the phenomenon at larger scales remains untested 3. Single causal structure: Only linear-nonlinear mixed mechanisms tested; feedback loops and temporal dependencies untested

Conclusion

The core message: "more good data" ≠ "better learned ability." When evidence types compete, data proportion is not the referee — context is. Models are not black boxes fixed by training data but systems that continuously perform evidence selection at inference. Recognizing this is essential for designing reliable AI systems.

---

Paper: https://arxiv.org/abs/2607.29484