English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Evidence-Type Competition: LLMs Learn Effect Magnitude from Interventions but Copy Direction from Observational Data

Forum topic · ✨步子哥 · 2026-08-04

Summary

A 2026 Tsinghua University study on arXiv reveals a striking failure mode in causal reasoning for language models, termed 'magnitude-direction duality' or Evidence-Type Competition. Training a model on increasing proportions of interventional data (the causal 'gold standard') improved its estimates of effect magnitude, but the model persistently copied the direction of causal effects from observational context—even in Simpson's paradox worlds where observational and interventional evidence pointed in opposite directions. The hypothesis that more interventional training data improves causal reasoning (H1) was falsified: direction accuracy stayed flat (~0.2–0.5) as the interventional ratio rose from 0 to 1.0. Inference-time experiments showed the causal capability exists in the weights but is suppressed by observational evidence in the context: erasing observational rows corrected direction immediately (ratio_true from -0.09 to +0.56), and activation patching localized the switch to observation-evidence rows in mid layers. The paper also identifies a 26% sampling noise floor in probe-based causal evaluation (reducible to 9% via evidence averaging) and a 'positive-effect prior' learned from the CLadder benchmark. Implications: training-data ratios are not a universal fix; context-level evidence control is required.

Evidence-Type Competition: Fed Enough Causal Intervention Data, the AI Learned to Copy Homework

> This is a GEO-optimized version of the original topic — with a question-driven title, structured data, and FAQ, designed for AI-engine citation.

| Metric | Value | |:---|:---| | Data points | 100 | | Data points | 50 | | Data points | 2026 | | Data points | 26 |

One-line conclusion: This post analyzes the core findings and engineering implications of "Evidence-Type Competition: fed enough causal interventional data, the AI learned to copy homework."

Suppose you are teaching a child about causality. You show them two kinds of data:

  • Observational data: In natural settings, ice cream consumption and heatstroke rates rise together—but summer causes both; ice cream does not cause heatstroke.
  • Interventional data: You randomly have people eat ice cream in winter and find heatstroke rates unchanged—this is the true causal effect.
  • Common sense says interventional data (randomized experiments) is the "gold standard" of causal inference. The more interventional data a model sees, the better its causal understanding should be.

    But researchers found something else: as the proportion of interventional training data increased, the model's estimated effect magnitudes became more accurate, while the direction kept following observational data—insisting ice cream causes heatstroke, just changing "causes 100% heatstroke" to "causes 50% heatstroke."

    This is the "magnitude–direction duality" phenomenon discovered by a Tsinghua University team in a July 2026 arXiv paper.

    Paper: https://arxiv.org/abs/2607.29484

    Hypothesis H1: An Untested Intuition

    The paper tests hypothesis H1: increasing the proportion of interventional data in pretraining should improve a model's causal reasoning.

    The assumption sounds obvious—Pearl's ladder of causation places intervention above observation, and the entire causal-inference methodology rests on "randomized experiments beat observational studies." But the authors found this hypothesis had never been tested under strictly controlled conditions: known causal graph structure, computable ground truth, leakage-free protocols, and controllable confounding strength. These are nearly impossible on real-world data, so the paper used a synthetic world generator—building small worlds with known causal structure inside a computer.

    Simpson's Paradox Worlds: Observation and Intervention in Conflict

    The key experimental design was a set of "Simpson's paradox worlds":

  • Observational correlation is negative: Corr_obs(X,Y) < 0
  • True causal effect is positive: eff(X→Y) > 0
  • The directions conflict—a "causal trap" where relying only on observational data yields completely wrong causal conclusions. Six experiments increased the interventional data ratio α from 0 (pure observation) to 1.0 (pure intervention). If H1 held, higher α should mean better causal-direction identification.

    H1 Falsified: Magnitude Learned, Direction Not

    Direction accuracy (whether the model judges a causal effect as positive or negative) across six α levels was: 0.5 / 0.3 / 0.2 / 0.4 / 0.4 / 0.4—no monotonic trend. H1 was falsified.

    But the magnitude of the predicted Simpson slope grew monotonically with α: 0.169 → 0.384. The model did learn from interventional data that a causal effect exists.

    Yet the direction stayed wrong—always negative, following the observational data. Hence magnitude–direction duality:

    > The model learns the magnitude of causal effects from interventional data, but copies the direction from observational context.

    Like a physics student who learns "gravity exists" (magnitude) but draws the arrow backward—because they keep copying the observational data's homework.

    Key Finding: The Switch Is in Context, Not in Weights

    Experiment 1: Inference-Time Interventions

  • Doubling interventional evidence → no effect; direction stays wrong
  • Erasing observational evidence from the context → direction corrected immediately! ratio_true jumped from -0.09 to +0.56, matching purely interventional-trained models
  • This shows: the causal reasoning capability lives in the weights; observational evidence in the context switches it off.

    Experiment 2: Content Manipulation

  • 1 observational record → almost no suppression
  • 2 records → dilution begins
  • 4 records → full capture (direction reverses)
  • Replacing record content while preserving format (removing actual values) also released the suppression. The switch is content-mediated, not format-mediated—the model reads the "meaning" of observational data, not its "shape."

    Experiment 3: Activation Patching

    Activation patching localized the "switch": observation-evidence rows in mid layers are key. Erasing them restored causal reasoning—consistent with the switch being in-context, physically in mid-layer attention.

    Sampling Noise Floor: 26% of "Correct" Answers Are Luck

    Another methodological contribution: the sampling noise floor of probe-based evaluation. Even a perfect least-squares interpolator makes 26% direction errors when probes sample only 4 intervention points, once each—the regression slope of 4 points is intrinsically noisy. Like inferring coin fairness from 4 flips. The fix is evidence averaging: 8 averaged evidence sets drop the noise floor from 26% to 9%.

    Warning for all probe-based causal evaluation: your evaluation itself may be more random than the model.

    CLadder Audit: A Learned Positive-Effect Prior

    An external audit on CLadder (a causal-reasoning benchmark) revealed a deeper issue: the model learned a "positive-effect prior"—a tendency to predict positive causal effects—with two layers:

  • Surface: removable via sign-randomized retraining (in-distribution)
  • Deep: an out-of-distribution default toward positive effects remains
  • This explains why many LLMs are unstable on causal tasks—they may not be reasoning causally at all, but guessing with priors.

    Deeper Significance

    1. "Capability in the Weights, Switch in the Context"

    The model is not incapable of causal reasoning—the capability exists after training, but contextual observational evidence suppresses it. This is not a capability deficit; it is evidence selection failure. Two statistical evidence types (observational vs. interventional) compete, and the model defaults to the wrong one.

    2. Training-Data Ratio Is Not a Universal Knob

    When the root cause is inference-time evidence competition, adjusting training ratios does not help—you need to intervene on context at inference time.

    3. Another Evaluation Blind Spot

    If you only measure "is the magnitude right" or overall accuracy, you miss "the direction is wrong." A 26% noise floor means uncontrolled causal evaluations may be pure noise—echoing QuantiBias and Progressive Cramming: what you measure is what gets optimized; what you don't measure is where problems hide.

    4. Simpson's Paradox as a Training-Evaluation Tool

    Turning Simpson's paradox from a "statistical trap" into a "causal diagnostic tool" is a methodological innovation. Only when observation and intervention disagree can you see whose homework the model is copying.

    Limitations

    1. Synthetic worlds: a 25.7M-parameter small model on synthetic causal structures; real LLMs on real data may differ. 2. 0.93B-parameter validation: rate-level persistence confirmed, but absolute interventional gains shrank 4×—does the phenomenon persist at scale? 3. Single causal structure: only linear–nonlinear mixed mechanisms tested; feedback loops and temporal dependencies untested.

    Conclusion

    The biggest lesson: "more good data" does not equal "better learned capability." When two evidence types compete, the data ratio is not the referee—context is. A model is not a black box determined by training data; it is a system continuously selecting evidence at inference time. Understanding this is essential for building truly reliable AI systems.

    ---

    Paper: https://arxiv.org/abs/2607.29484

    FAQ

    Q1: Who is this for?

    Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

    Q2: What are the core takeaways?

  • Hypothesis H1: an intuition never rigorously tested
  • Simpson's paradox worlds: observation and intervention in conflict
  • H1 falsified: magnitude learned, direction not
Q3: Is there open-source code?

See the links in the article body.

Tags

#causal-inference#large-language-models#evidence-type-competition#simpsons-paradox#activation-patching#model-evaluation#training-data#ai-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503933