Statistics has an unwritten iron rule: correlation is not causation. Every statistics student memorizes it, yet most practitioners—including many data scientists—still treat correlation as causation in their daily work.
Judea Pearl found this unacceptable. In his 2018 book *The Book of Why: The New Science of Cause and Effect*, his core thesis can be summarized in one sentence: causal inference is not an appendix to statistics—it is an independent science, and by refusing to discuss causation, statistics mutilated itself.
Pearl is not speaking loosely. He invented Bayesian networks, won the Turing Award, and founded the field of causal inference. The book explains thirty years of his academic work: a rigorous mathematical language that lets mathematics talk about "causes."
1. The Ladder of Causation: Which Level Can You Reason At?
Pearl's core framework is the Ladder of Causation, with three rungs:
Level 1: Association — "If I observe X, what happens to Y?"
- The level of traditional statistics and most current machine learning
- P(Y | X) — probability of Y given observation of X
- Typical question: do customers who buy diapers also buy beer?
- Limitation: it only finds co-occurrence patterns; it cannot tell you what happens if you intervene on X
- The core level of causal inference
- P(Y | do(X)) — probability of Y under active intervention on X
- Typical question: if I give a patient drug X, will they recover?
- Key distinction: observation ≠ intervention. Observing a patient take the drug may reflect that they were seriously ill (confounding); intervention is random assignment (like an RCT)
- The highest level of causal inference
- P(Y_x | X', Y') — given that X and Y were observed to happen, what would Y be if X had not?
- Typical questions: would the defendant have been arrested if innocent? Would the patient have died without the drug?
- This is the foundation of legal, medical, and moral reasoning
- P(Y|X): probability of Y after observing X. X may itself be caused by other variables Z that also affect Y (confounding).
- P(Y|do(X)): probability of Y after forcibly setting X's value (severing all causal paths into X). This removes confounding.
- Learn statistical associations from massive text (Level 1)
- Imitate human language patterns
- Answer "what would have happened if X hadn't occurred" (Level 3 counterfactuals)
- Predict in unseen intervention scenarios (Level 2 intervention)
- Distinguish "observing X" from "doing X"
Level 2: Intervention — "What happens to Y if I do X?"
Level 3: Counterfactual — "What would have happened had X not occurred?"
> Pearl's key insight: all current machine learning (including LLMs) sits on Level 1. It can find associations but cannot reason about interventions, let alone counterfactuals. This is why LLMs "hallucinate" — they have no causal model, only statistical associations.
2. The do-Operator: Teaching Mathematics to Say "Because"
Pearl's mathematical contribution is the do-operator.
In traditional probability theory, P(Y|X) and P(Y|do(X)) are not distinguished — statisticians treat "conditional probability" as conditional probability, whether observed or intervened. Pearl says: no, they are fundamentally different.
The do-operator is implemented via graph theory: on a causal DAG, do(X) is equivalent to deleting all edges pointing into X, then computing probabilities on the modified graph.
> The physical meaning: if you can randomize X (a randomized controlled trial), you naturally implement do(X). But often you cannot run an RCT — you cannot randomize smoking, unemployment, or war. Pearl's do-operator lets you compute causal effects from observational data, provided you know the structure of the causal graph.
3. The Back-Door Criterion: A Shortcut to Causal Effects
One of Pearl's most practical tools is the Back-Door Criterion.
Suppose you want the causal effect of X on Y, but a set of confounders Z influences both. The criterion says: if you can find a variable set Z satisfying two conditions: 1. Z contains no descendants of X (it does not block the front-door path X→Y) 2. Z blocks all "back-door paths" from X to Y (all paths X←...→Y)
then adjusting over Z yields the causal effect of X on Y:
P(Y|do(X)) = Σ_z P(Y|X, Z=z) · P(Z=z)
This is the famous back-door adjustment formula. It tells you: no RCT needed — if you can measure all confounders, you can compute causal effects from observational data.
> This was revolutionary for medicine, economics, and sociology. Many questions cannot be studied by RCT (you cannot randomize smoking), but with sufficient observational data and a causal graph, back-door adjustment delivers causal answers.
4. Why Did Statisticians Once Reject Causation?
Pearl sharply criticizes the statistics community. In the early 20th century, entangled in philosophical disputes over causality (notably the Pearson–Yule debate), statistics voluntarily abandoned the concept of "cause," relegating it to metaphysics. From then on, statisticians spoke only of correlation.
> The absurd result: statistics could say "smoking is correlated with lung cancer" but not "smoking causes lung cancer." The latter requires causal reasoning, which statistics refused to discuss.
Pearl considered this self-imposed restriction disastrous. He spent thirty years building a mathematical framework for causation, proving it can be rigorously mathematized — not through philosophical debate, but through combining graph theory and probability.
5. Implications for AI: What Are LLMs Missing?
Pearl repeatedly emphasizes: current AI (including deep learning and LLMs) lacks causal reasoning.
What LLMs can do:
What LLMs cannot do:
Pearl believes next-generation AI needs causal world models — internal representations that can answer "what if" and "why." This is not something a bigger Transformer solves; it requires an architectural breakthrough.
6. Practical Value of Causal Inference
Causal inference is not just philosophy — it has enormous practical value in business and policy:
Alternatives to A/B testing: tech companies love A/B tests, but many scenarios forbid them (e.g., pricing changes can't be randomized). Causal inference estimates intervention effects from observational data.
Ad attribution: a user bought after seeing an ad — was it the ad's credit, or would they have bought anyway? Confounders (user interest) make this hard. Difference-in-differences and synthetic control are standard tools.
Medical evidence: observational data suggested "hormone replacement therapy reduces heart disease risk," but RCTs showed the opposite. The confounder: affluent women were both more likely to use HRT and healthier. Without causal reasoning, observational data can kill.
Policy evaluation: does raising the minimum wage reduce employment? Observational data said yes (low-employment states had low wages), but causal analysis (Card & Krueger's natural experiment) said no — a conclusion that reshaped labor economics.
7. Pearl's Legacy
Pearl's framework (structural causal models + do-calculus + counterfactuals) is not the only approach — Donald Rubin's potential outcomes framework and James Robins' marginal structural models are strong competitors. But Pearl's framework has two unique advantages:
1. Graphical representation: causal graphs make assumptions visual and auditable. You can point at a diagram and say "I assume X affects Y only through Z" — a transparency pure formulas cannot provide.
2. Mathematization of counterfactuals: Pearl made counterfactual reasoning computable via structural equation models — something the Rubin framework cannot do.
> Pearl's ultimate vision: causal inference is the bridge that upgrades "data science" from "curve fitting" to "science." Without it, you only have data; with it, you have tools for understanding the world.
After reading this book, the way you see the world changes. You are no longer satisfied with "X is correlated with Y" — you ask "why? Does X cause Y, does Y cause X, or does a third variable Z cause both?" That questioning is the starting point of scientific thinking.
---
Book: *The Book of Why: The New Science of Cause and Effect* (2018) Authors: Judea Pearl & Dana Mackenzie