There is an unwritten iron rule in statistics: correlation does not equal causation. Every statistics student memorizes it, but most people—including many data scientists—still treat correlation as causation in practice.
Judea Pearl found this unacceptable. In 2018 he published *The Book of Why: The New Science of Cause and Effect*. Its core thesis in one sentence: causal inference is not an accessory of statistics; it is an independent science, and by refusing to discuss causation, statistics crippled itself.
Pearl is not speculating. He is the inventor of Bayesian networks, a Turing Award laureate, and the founder of causal inference. The book explains thirty years of his work to the public: a rigorous language that lets mathematics talk about "cause."
1. The Ladder of Causation: Where Do You Think?
Pearl's central framework is the Ladder of Causation, with three rungs.
Rung 1: Association — "If I observe X, what happens to Y?"
- Traditional statistics and most current machine learning live here
- P(Y | X) — probability of Y given observation of X
- Classic question: do customers who buy diapers also buy beer?
- Limitation: only detects co-occurrence, cannot answer "if I intervene on X, what happens to Y?"
- The core of causal inference
- P(Y | do(X)) — probability of Y given active intervention on X
- Classic question: if I give patient drug X, will they recover?
- Key distinction: observation ≠ intervention. Observing that a patient took a drug may reflect underlying severity (a confounder); intervention means random assignment (as in an RCT)
- The highest rung of causal inference
- P(Y_x | X', Y') — given that X happened and Y happened, what would Y have been if X had not happened
- Classic questions: would the defendant still have been arrested if he had not committed the crime? would the patient still have died without the drug?
- Foundational for legal, medical, and moral reasoning
- P(Y|X): probability of Y after observing X. X may be caused by another variable Z that also affects Y (confounding).
- P(Y|do(X)): probability of Y after forcing X to a value (cutting all causal paths into X). This removes confounding.
- Learn statistical associations from massive text (Rung 1)
- Mimic human language patterns
- Answer "if X had not happened, what would Y be" (Rung 3 counterfactuals)
- Make predictions under unseen interventions (Rung 2 intervention)
- Distinguish "observed X" from "did X"
- Alternative to A/B testing: Internet companies love A/B tests, but many scenarios forbid them (e.g., price changes cannot be randomized). Causal inference estimates intervention effects from observational data.
- Ad attribution: Did the ad cause the purchase, or was the user already going to buy? Confounders (user interest) make this extremely hard. Difference-in-differences and synthetic control are standard causal tools.
- Medical evidence: Observational data suggested hormone replacement therapy reduces heart disease risk, but RCTs showed the opposite. The confounder was affluence (wealthier women both used HRT and were healthier). Without causal reasoning, observational data can kill.
- Policy evaluation: Does raising the minimum wage reduce employment? Observational data says yes (low-employment states have low wages), but causal analysis (Card & Krueger's natural experiment) says no. This conclusion reshaped labor economics.
Rung 2: Intervention — "If I do X, what happens to Y?"
Rung 3: Counterfactual — "If X had not happened, what would Y have been?"
> Pearl's key insight: all current machine learning (including LLMs) stays at Rung 1. They detect associations but cannot reason about interventions or counterfactuals. This is why LLMs hallucinate—they have statistical associations, not causal models.
2. The do-Operator: Letting Mathematics Say "Because"
Pearl's mathematical contribution is the do-operator.
In traditional probability theory, P(Y|X) and P(Y|do(X)) are not distinguished—statisticians treat all conditional probability the same. Pearl says no: they are fundamentally different.
The do-operator is implemented graphically: on a causal directed acyclic graph (DAG), do(X) is equivalent to deleting all edges into X, then computing probabilities on the new graph.
> Physically, this corresponds to random assignment of X (an RCT). But often you cannot run an RCT—you cannot randomize people into smoking, unemployment, or war. Pearl's do-operator lets you compute causal effects from observational data, provided you know the structure of the causal graph.
3. Back-Door Criterion: A Shortcut to Causal Effects
One of Pearl's most practical tools is the back-door criterion.
If you want the causal effect of X on Y but a set of confounders Z affects both, what do you do?
The back-door criterion says: find a set Z satisfying two conditions: 1. Z contains no descendants of X (does not block front-door paths X → Y) 2. Z blocks every back-door path from X to Y (every X ← … → Y path)
Then you can adjust on Z to recover the causal effect:
P(Y | do(X)) = Σ_z P(Y | X, Z=z) · P(Z=z)
This is the famous back-door adjustment formula. It tells you: no RCT needed—as long as you can measure all relevant confounders, you can recover causal effects from observational data.
> This is revolutionary in medicine, economics, and sociology. Many questions forbid RCTs (you cannot randomize smoking), but with enough observational data and a causal graph, back-door adjustment gives causal answers.
4. Why Did Statisticians Once Reject Causation?
Pearl sharply criticizes the statistical field. In the early 20th century, entangled in philosophical debates (especially the Pearson–Yule controversy), statisticians abandoned "cause" as metaphysics and restricted themselves to correlation.
> The absurd result: statistics could tell you "smoking correlates with lung cancer" but not "smoking causes lung cancer." The latter requires causal reasoning, which statistics refused to discuss.
Pearl considers this self-imposed limit disastrous. He spent three decades building the mathematical framework showing that causation can be rigorously formalized—no philosophy required, only graph theory plus probability.
5. Implications for AI: What Do LLMs Lack?
Pearl repeatedly emphasizes: current AI (including deep learning and LLMs) lacks causal reasoning.
What LLMs can do:
What LLMs cannot do:
> Pearl's verdict: LLM intelligence is "curve fitting" pushed to the limit—a high-dimensional fit of human language statistics, but no causal model. This is why LLMs hallucinate on reasoning tasks—they are matching associations, not performing causal inference.
Pearl argues next-generation AI needs a causal world model, an internal representation that can answer "what if" and "why." Bigger Transformers will not suffice; architectural breakthroughs are required.
6. Practical Value of Causal Inference
Causal inference is not philosophy; it has huge real-world impact.
7. Pearl's Legacy
Pearl's framework (Structural Causal Models + do-calculus + counterfactuals) is not the only approach—Donald Rubin's potential outcomes framework and James Robins's marginal structural models are powerful competitors. But Pearl's framework has two unique advantages:
1. Graphical representation: Causal diagrams make assumptions visible and auditable. You can point at the graph and say "I assume X affects Y only through Z"—a transparency that pure formulas cannot match. 2. Mathematization of counterfactuals: Structural equation models make counterfactual reasoning computable, which the Rubin framework cannot do.
> Pearl's ultimate vision: causal inference is the bridge that lifts data science from "curve fitting" to "science." Without it, you have data; with it, you have a tool for understanding the world.
After reading this book, you see the world differently. You no longer settle for "X and Y are correlated"; you ask "why? Does X cause Y, or Y cause X, or does a third variable Z cause both?" That question is where scientific thinking begins.
---
Book: *The Book of Why: The New Science of Cause and Effect* (2018) Authors: Judea Pearl & Dana Mackenzie
Free access link mentioned in the original article: https://b23.tv/4vCEQYn