English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Evidence-Type Competition: Fed Enough Causal Intervention Data, AI Learns to Copy the Wrong Homework

Forum topic · ✨步子哥 · 2026-08-03

Summary

A July 2026 Tsinghua University arXiv paper (arXiv:2607.29484) reports a striking 'magnitude–direction duality' in causal reasoning by language models. Training a 25.7M-parameter model on synthetic Simpson-paradox worlds where observational correlation is negative but the true causal effect is positive, the authors falsify the intuition that more intervention data in pretraining improves causal reasoning: direction accuracy stays flat (0.2–0.5, no monotonic trend) even as predicted effect magnitudes grow monotonically with the intervention-data ratio. Inference-time experiments show the mechanism: doubling intervention evidence does nothing, but erasing observational evidence from the context instantly corrects direction (ratio_true jumps from -0.09 to +0.56). Activation patching locates the 'switch' at observational-evidence rows in mid layers. The paper concludes that causal capability exists in the weights but is suppressed by contextual observational evidence. It also documents a 26% sampling noise floor for probe-based evaluation (reducible to 9% via evidence averaging) and a learned 'positive-effect prior' from CLadder audits.

Imagine teaching a child about causality with two kinds of data:

  • Observational data: In the natural state, every time ice cream sales rise, heatstroke rates rise too—but that's because summer causes both, not because ice cream causes heatstroke.
  • Interventional data: You randomly have people eat ice cream in winter and find heatstroke rates unchanged—this is the true causal effect.
  • Common sense says interventional data (randomized experiments) is the "gold standard" for causal inference. The more intervention data a child sees, the better they should understand causal relationships.

    But what if you found that the more intervention data the child saw, the more accurate their stated magnitude became, while the direction kept following the observational data—insisting that "eating ice cream causes heatstroke," only shifting from "causes 100% heatstroke" to "causes 50% heatstroke"?

    The child isn't stupid. This is a phenomenon discovered by a Tsinghua University team in a July 2026 arXiv paper, called "magnitude–direction duality."

    Paper: https://arxiv.org/abs/2607.29484

    The H1 Hypothesis: An Intuition Never Rigorously Tested

    The paper tests hypothesis H1: increasing the proportion of interventional data in pretraining should improve a model's causal reasoning ability.

    This seems self-evident—Pearl's ladder of causation places intervention above observation, and the entire methodology of causal inference rests on "randomized experiments beat observational studies." In the LLM era, many naturally extended this to training data: feed more intervention data, and the model should understand causality better.

    But the authors found that this hypothesis had never been tested under strictly controlled conditions—known causal graph structure, computable causal ground truth, leak-free protocols, and controllable confounding strength. These are nearly impossible to satisfy with real-world data, so the paper built a synthetic world generator: small worlds with known causal structures, created inside a computer.

    Simpson-Paradox Worlds: Observation and Intervention in Conflict

    The experiment's clever design: a set of "Simpson-paradox worlds," where:

  • Observational correlation is negative: Corr_obs(X,Y) < 0
  • True causal effect is positive: eff(X→Y) > 0
  • The directions are opposite—a "causal trap." A model relying only on observational data gets the causal direction completely wrong. The authors ran six experiments with the intervention-data ratio α increasing from 0 (purely observational) to 1.0 (purely interventional).

    H1 Falsified: Magnitude Learned, Direction Not

    The results were surprising. Direction accuracy across the six α levels was 0.5 / 0.3 / 0.2 / 0.4 / 0.4 / 0.4—no monotonic upward trend. H1 was falsified.

    But the story didn't end there. The magnitude of the Simpson slope (the predicted effect size) grew monotonically with α: 0.169 → 0.384. The model did learn from intervention data that "there is a causal effect." But the direction stayed wrong—always negative, following the observational data.

    This is magnitude–direction duality:

    > The model learns the magnitude of the causal effect from interventional data, but copies the direction from observational context.

    Key Finding: The Switch Is in the Context, Not the Weights

    Experiment 1: Inference-time intervention

  • Doubling interventional evidence → no effect, direction stays wrong
  • Erasing observational evidence from context → direction instantly corrected! ratio_true jumps from -0.09 to +0.56, matching the purely-interventional-trained model
  • Conclusion: the causal reasoning capability was always in the weights—observational evidence in the context was switching it off.

    Experiment 2: Content manipulation

  • 1 observational record → almost no suppression
  • 2 records → dilution begins
  • 4 records → complete capture (direction reversed)
Replacing the records' content (keeping format but removing actual values) also released the suppression. The switch is content-mediated, not format-mediated—the model reads the "meaning" of the observational data, not its "appearance."

Experiment 3: Activation patching

Activation patching located the switch: observational-evidence rows in the middle layers are key. Erasing them restores causal reasoning.

The Sampling Noise Floor: 26% of "Correct" Answers Are Luck

Another methodological contribution: the sampling noise floor for probe-based evaluation. Even a perfect least-squares interpolator makes 26% direction errors when evaluated with probes sampling only 4 intervention points—regression slope on 4 noisy points has inherent randomness. The fix is evidence averaging: averaging 8 groups of evidence lowers the noise floor from 26% to 9%.

Warning for all causal-evaluation work: your evaluation may be noisier than your model.

CLadder Audit: A Learned "Positive-Effect Prior"

An external audit on CLadder found models learn a "positive-effect prior"—a tendency to predict positive causal effects—with two layers: a surface layer removable by sign randomization (in-distribution), and a deep out-of-distribution positive-effect default. Many LLMs may not be reasoning about causality at all—they may be guessing from priors.

Deeper Implications

1. "Capability is in the weights; the switch is in the context"

The model is not incapable of causal reasoning—it was trained to have the capability. But observational evidence in context suppresses it. This is not missing capability; it's evidence-selection failure. Two types of statistical evidence (observational vs. interventional) compete, and the model defaults to the wrong one.

2. Training-data ratio is not a universal knob

When the root cause is inference-time evidence competition, adjusting the training mixture doesn't help—you need to intervene on the context at inference time.

3. Another evaluation blind spot

If you only measure "is the magnitude right" or "overall accuracy," you miss "the direction is wrong." And an uncontrolled 26% noise floor means causal evaluations may be pure noise.

4. Simpson's paradox as a training-evaluation tool

Turning Simpson's paradox from a "statistical trap" into a "causal diagnostic tool" is this paper's methodological innovation. When observation and intervention disagree in direction, the model's evidence-selection strategy is fully exposed—in direction-consistent worlds, you can't tell whose homework the model is copying.

Limitations

1. Synthetic worlds: a 25.7M-parameter small model plus synthetic structures; real LLMs on real data may behave differently. 2. 0.93B validation: rate-level persistence holds at 0.93B parameters, but absolute intervention benefit shrank 4×—does the phenomenon survive in larger models? 3. Single causal structure: only linear–nonlinear mixed mechanisms were tested; feedback loops and temporal dependencies remain untested.

Conclusion

The biggest lesson: "more good data" does not equal "better-learned capability." When two kinds of evidence compete, the data ratio isn't the referee—the context is. A model is not a black box determined by training data, but a system continuously performing evidence selection at inference time. Understanding this is essential for building truly reliable AI systems.

---

Paper: https://arxiv.org/abs/2607.29484

Tags

#causal-inference#llm#training-data#simpsons-paradox#activation-patching#evaluation#arxiv-paper#tsinghua-university

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503919