> Author: Xiaokai > References: Silver et al., Nature 2016, 2017; DeepMind technical blog
---
1. Introduction: Why Look Back at AlphaGo Ten Years Later?
In March 2016, AlphaGo defeated Lee Sedol. Most people saw only a Go match—machine defeating the crystallization of millennia of human wisdom. But what deserves re-examination today is not the match itself, but the engineering architecture behind AlphaGo, which in an almost prophetic way previewed, ten years in advance, the core paradigms of today's LLM training.
The fundamental problem Silver's team solved is the same one OpenAI, DeepSeek, and Anthropic are solving today:
> How do you make a model with limited expressive capacity (millions of parameters) approximate a complex problem that cannot in principle be exhaustively enumerated?
AlphaGo's answers were written in the 2016 Nature paper, but it was only in 2024–2025 that the LLM community began to realize: these answers never went stale—they were simply waiting for the right era to be rediscovered.
---
2. AlphaGo's Four Engineering Components
To understand AlphaGo's "prophetic" quality, decompose it into its core components:
| Component | Function | Parameter scale | LLM counterpart today | |------|------|----------|-------------------| | SL Policy Network | Imitates human expert moves | 13-layer CNN | SFT (supervised fine-tuning) | | RL Policy Network | Optimizes policy via self-play | Same as SL network | PPO/GRPO (reinforcement learning) | | Value Network | Evaluates win probability of a position | Same architecture, single-output head | Reward Model / Critic | | MCTS | Searches optimal paths at inference | Parameter-free | Test-Time Compute / inference search | | Fast Rollout | Rapidly simulates games to completion | Linear features | Lightweight heuristic / distilled model |
This table alone makes a key point: AlphaGo was not a single neural network, but a composite system of "learning + search + evaluation." Today's LLM training is evolving in the same direction—only the 19×19 board has been replaced by token sequences.
---
3. Core Mapping: From Go to Language Models
3.1 Policy Network → LLM / Generator
AlphaGo's Policy Network takes a board position as input and outputs a probability distribution over all legal moves. In today's language models, this is autoregressive generation: given context, predict the probability distribution of the next token.
But AlphaGo made a distinction that was overlooked then and rediscovered today:
- SL Policy (supervised version): trained on human game records, learning to "play like a human"
- RL Policy (reinforcement learning version): trained via self-play, learning to "win"
- SFT stage: teaches the model to "speak/reason like a human"
- RL stage (RLHF / GRPO): teaches the model to "produce correct answers"
- Reward Model (RLHF): a model trained to evaluate "is this response good"
- Critic (PPO/GRPO): provides baseline estimates in policy gradients
- Process Reward Model (PRM): used by DeepSeek for mathematical reasoning, evaluating the quality of each reasoning step
- At training time: use a neural network to learn a fast but coarse intuition
- At inference time: use search (MCTS / CoT / long thinking) to refine that intuition
- Key insight: a weaker model + good search can beat a strong model + poor search
- DeepSeek-R1's GRPO: the model generates its own reasoning trajectories, a Reward Model evaluates them, policy gradients update weights
- VinePPO: uses MCTS-like search at inference to collect more samples and improve credit assignment
- o1 / o3: inference-time search + training-time distillation
- SFT camp: "High-quality human data is needed first so models can learn basic grammar and reasoning formats"
- Pure RL camp: "AlphaGo Zero proved pure self-play works; LLMs should too"
- Reality: today's LLM training is hybrid—SFT warm start, RL optimization, synthetic data expansion
- AlphaGo's MCTS, when evaluating this move, assigned it too low a probability (the Policy Network's intuition deemed it bad), so it did not allocate enough simulations to discover its deep value
- The Value Network's assessment of the resulting position was also too optimistic (believing Black still held the advantage)
- Only when Lee Sedol actually played it was AlphaGo forced to re-evaluate, but it was too late
- The model has seen enormous data during training and formed a set of "intuitions" (prior probabilities)
- When facing inputs outside the training distribution, intuition can be wrong
- Without inference-time search/verification to correct it, the model directly outputs a wrong answer
- Human data has a ceiling (the skill limit of human players)
- Self-play data quality keeps improving as the model gets stronger (virtuous cycle)
- Once the model exceeds human level, only self-play can produce "superhuman" training data
- Policy provides fast intuition
- Value provides accurate evaluation
- MCTS provides search optimization
- Base Model: provides language ability and knowledge
- Reward Model: provides evaluation capability
- Inference Search (MCTS / CoT / long thinking): provides inference-time optimization
- Tool Use: provides external compute
- Verifiers: provide fact-checking
- Silver, D., et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529, 484-489 (2016).
- Silver, D., et al. "Mastering the game of Go without human knowledge." Nature 550, 354-359 (2017).
- DeepMind Blog: AlphaGo Zero (2017)
- VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment (2024)
- A Survey on Self-Play Methods in Reinforcement Learning (2024)
This distinction has an almost verbatim counterpart in the LLM field:
AlphaGo's ablation experiments already proved: without the SL Policy as a warm start, RL training fails to converge entirely. This explains why today's LLM training still needs an SFT stage—pure RL from scratch is too slow and requires an initialization that has "first learned to be human-like."
3.2 Value Network → Reward Model / Critic
The Value Network is AlphaGo's most underrated contribution. It takes a board position and outputs a scalar: from -1 (certain loss) to +1 (certain win). This scalar comes not from human labels but from statistics of self-play—the network learns to judge "how likely is this position to end in a win" from massive data of games it played against itself.
What does this correspond to in LLM training today?
AlphaGo's Value Network was an Outcome Reward Model (ORM)—evaluating only the final result. Today's PRMs are essentially fine-grained versions of AlphaGo's Value Network: evaluating not just the final position but the quality of every move along the way.
3.3 MCTS → Test-Time Compute / Inference-Time Search
This is AlphaGo's most "prophetic" design.
MCTS (Monte Carlo Tree Search) does one thing at inference: trade computation for accuracy. The Policy Network provides "intuition" (prior probabilities), the Value Network provides "evaluation" (position assessment), and MCTS searches between the two, exploring more possibilities to find moves better than raw intuition.
Silver wrote explicitly in the paper:
> "MCTS can be viewed as a policy improvement operator. The probability distribution π produced by search is stronger than the raw network output p."
In 2024, this sentence was republished in different words:
> "o1's long reasoning spends more compute at inference time to explore better reasoning paths."
From AlphaGo to o1, it is essentially the same idea:
AlphaGo proved this inequality in 2016. Today's LLMs are merely revalidating it in another domain (language).
3.4 Self-Play → RLHF / Self-Play / Synthetic Data
AlphaGo Zero (2017) was Silver's team's most radical experiment: completely abandoning human game records, starting from scratch with self-play.
The training loop: 1. Randomly initialize the network 2. The network plays against itself (MCTS guiding every move) 3. Train on game outcomes (the Policy learns MCTS's search distribution; the Value learns game outcomes) 4. The network gets stronger → MCTS search quality improves → game data quality improves → the network gets stronger still
This virtuous cycle reappeared in 2024 under new names:
AlphaGo Zero's core lesson: when an environment allows unlimited self-play, human data is not necessary—it is only an accelerator. Go satisfies this condition (simple rules, clear win/loss, infinitely restartable). Language models are approaching it too (saturating SFT data, improving synthetic data quality, strengthening self-verification).
---
4. AlphaGo Zero's "Warm-Start Paradox"
There is a key difference between AlphaGo (2016) and AlphaGo Zero (2017):
| Version | Human data | Architecture | Search method | |------|----------|----------|----------| | AlphaGo Lee | Required (KGS game records) | Separate Policy + Value | MCTS + Fast Rollout | | AlphaGo Zero | Not needed | Single two-headed network (ResNet) | Pure MCTS, no rollout |
AlphaGo Zero proved: pure self-play can match or exceed human-data warm starts. But this conclusion has a hidden condition:
> It requires more compute and time.
AlphaGo Zero trained for 72 hours on 4 TPUs to surpass the previous version. Without the "warm start" of human game records, it had to climb out of the performance plateau from completely random moves.
This "warm-start paradox" is being re-argued in the 2024 LLM world:
AlphaGo Zero's history tells us: pure self-play has a higher ceiling but a steeper path. The value of human data is not "injecting knowledge" but providing a good initialization point, letting RL converge to useful policies in less time.
---
5. Lee Sedol's "Divine Move": Humanity's Last Glory
March 12, 2016, game two, move 78.
Lee Sedol played a "wedge" in the lower-right corner, later called the "Divine Move." It broke AlphaGo's established evaluation and caused the program to make visible mistakes in subsequent moves.
From today's perspective, the drama of this move lies not just in its aesthetic value but in what it revealed about the limits of search algorithms:
This is the same mechanism behind today's LLM "hallucination" problem:
Lee Sedol's Divine Move was essentially an adversarial example—a carefully designed input landing in the model's "blind spot." Today's LLM RL training, inference-time search, and self-verification all attempt to shrink that blind spot and reduce the chance of being adversarially caught out.
---
6. Why AlphaGo Hasn't Become Obsolete
AlphaGo's legacy can be summarized as three "paradigm shifts":
1. From "training-time optimization" to "inference-time compute"
Traditional ML assumed: once the model is trained, inference is a forward pass. AlphaGo broke this: inference can be a search process, and compute can be dynamically allocated at inference time.
Today's "long thinking" in o1 and DeepSeek-R1 replicates this paradigm. GPT-4's 1.8T parameters are crystallized training-time compute, while o1's reasoning tokens are dynamically allocated inference-time compute. The combination of both is a complete intelligent system.
2. From "human-data-driven" to "self-evolving loops"
AlphaGo Zero proved self-play can produce data exceeding human quality, because:
Today's LLM approaches—synthetic data, Self-Play, distillation from stronger models—all follow this direction. A significant share of DeepSeek-V3's 14.8T training tokens are synthetic. Not because human data is insufficient, but because the quality of model-generated data keeps improving.
3. From "single network" to "composite system"
AlphaGo was not a Policy Network, not a Value Network, not MCTS. It was the combination of the three, each responsible for what it does best:
Today's LLM systems are heading toward the same composite architecture:
No single model can solve everything. AlphaGo demonstrated this in 2016.
---
7. Conclusion: A Future Delivered Early
When AlphaGo beat Lee Sedol, most people saw only the drama of "Go" and "human vs. machine." What truly mattered was the general intelligence architecture Silver's team validated in that match:
> Learning provides intuition, search provides precision, and self-play provides unlimited data.
This architecture succeeded at Go, then at chess (AlphaZero), StarCraft (AlphaStar), and protein folding (AlphaFold). Now it is being re-implemented in language models—only the parameters have grown from millions to hundreds of billions, and the search space from 19×19 to vocab_size^sequence_length.
AlphaGo hasn't become obsolete. It simply arrived a decade early, quietly waiting for the LLM community to catch up.
After the 2016 match, Lee Sedol said:
> "I lost, but this is not humanity's defeat. It is the victory of something humanity created."
Looking back today, he might add:
> "And this thing is now teaching humanity how to train the intelligence of the future."
---