> Author: Xiaokai > References: Silver et al., Nature 2016, 2017; DeepMind technical blog
---
1. Introduction: Why Look Back at AlphaGo Ten Years Later?
In March 2016, AlphaGo defeated Lee Sedol. Most people saw only a Go match—a machine beating the crystallization of a thousand years of human wisdom. But what truly deserves re-examination today is not the match itself, but the engineering architecture behind AlphaGo, which foreshadowed—almost prophetically—the core paradigms of today's large model training.
The fundamental problem Silver's team solved is the same one OpenAI, DeepSeek, and Anthropic are solving today:
> How do you make a model with limited expressive capacity (millions of parameters) approximate a complex problem that cannot be exhaustively enumerated in theory?
AlphaGo's answers were written in its 2016 Nature paper, but it was only in 2024–2025 that the LLM community gradually realized: these answers never went stale—they were simply waiting for the right era to be rediscovered.
---
2. AlphaGo's Four Engineering Components
To understand AlphaGo's "prophetic" nature, first break it into four core components:
| Component | Function | Parameter Scale | Modern LLM Counterpart | |------|------|----------|-------------------| | SL Policy Network | Imitates human expert moves | 13-layer CNN | SFT (supervised fine-tuning) | | RL Policy Network | Optimizes policy via self-play | Same as SL network | PPO/GRPO (reinforcement learning) | | Value Network | Evaluates win probability of a position | Same architecture, single output head | Reward Model / Critic | | MCTS | Searches for optimal moves at inference | No parameters | Test-Time Compute / inference search | | Fast Rollout | Quickly simulates games to completion | Linear features | Lightweight heuristic / distilled model |
This table itself makes a key point: AlphaGo was not a single neural network, but a composite system of "learning + search + evaluation." Today's LLM training is evolving in the same direction—only the 19×19 board has been replaced by token sequences.
---
3. Core Mapping: From Go to Language Models
3.1 Policy Network → LLM / Generator
AlphaGo's Policy Network takes the current board as input and outputs a probability distribution over all legal moves. In today's language models, this is autoregressive generation: given context, predict the probability distribution of the next token.
But AlphaGo made a distinction that was overlooked at the time and rediscovered today:
- SL Policy (supervised learning version): trained on human games, learning to "play like a human"
- RL Policy (reinforcement learning version): trained via self-play, learning to "win"
- SFT stage: teach the model to "speak/reason like a human"
- RL stage (RLHF / GRPO): teach the model to "produce correct answers"
- Reward Model (RLHF): a model trained to evaluate "is this answer good"
- Critic (PPO/GRPO): provides baseline estimates in policy gradients
- Process Reward Model (PRM): used by DeepSeek in mathematical reasoning to evaluate the quality of each reasoning step
- At training time: a neural network learns a fast but rough intuition
- At inference time: search (MCTS / CoT / long thinking) refines that intuition
- Key insight: a weaker model + sufficiently good search can defeat a stronger model + poor search
- DeepSeek-R1's GRPO: the model generates its own reasoning traces, a Reward Model evaluates them, and policy gradients update the model
- VinePPO: MCTS-like search at inference time to collect more samples and improve credit assignment
- o1 / o3: inference-time search + training-time distillation
- SFT camp: "You must first have high-quality human data for a model to learn basic grammar and reasoning formats"
- Pure RL camp: "AlphaGo Zero proved pure self-play works; LLMs should be able to do it too"
- Reality: Today's LLM training is hybrid—SFT warm start first, then RL optimization, augmented with synthetic data
- When AlphaGo's MCTS evaluated this move, because its prior probability was too low (the Policy Network's intuition judged it a bad move), the search did not allocate enough simulations to discover its deep value
- The Value Network's assessment of the resulting position was also overly optimistic (judging Black still ahead)
- Only when Lee Sedol actually played it was AlphaGo forced to re-evaluate—but by then it was too late
- The model has seen massive data during training, forming a set of "intuitions" (prior probabilities)
- When encountering out-of-distribution inputs, intuition can be wrong
- Without inference-time search/verification mechanisms to correct it, the model outputs wrong answers directly
- Human data has a ceiling (the skill limit of human players)
- Self-play data quality keeps rising as the model improves (virtuous cycle)
- Once a model surpasses human level, only self-play can produce "superhuman" training data
- Policy provides fast intuition
- Value provides accurate evaluation
- MCTS provides search-based optimization
- Base Model: language capability and knowledge
- Reward Model: evaluation capability
- Inference Search (MCTS / CoT / long thinking): inference-time optimization
- Tool Use: external compute
- Verifiers: fact-checking capability
- Silver, D., et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529, 484-489 (2016).
- Silver, D., et al. "Mastering the game of Go without human knowledge." Nature 550, 354-359 (2017).
- DeepMind Blog: AlphaGo Zero (2017)
- VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment (2024)
- A Survey on Self-Play Methods in Reinforcement Learning (2024)
This distinction has an almost word-for-word counterpart in today's LLM field:
AlphaGo's ablation experiments proved long ago: without the SL Policy as a warm start, RL training completely fails to converge. This explains why today's LLM training still requires an SFT stage—pure RL from scratch is too slow; it needs an initialization that has "first learned to be human-like."
3.2 Value Network → Reward Model / Critic
The Value Network is AlphaGo's most underrated contribution. It takes a board position as input and outputs a scalar from -1 (certain loss) to +1 (certain win). This scalar comes not from human labels but from statistics of self-play—the network learns to judge "how likely is this position to end in a win" from massive data of games it played against itself.
What does this correspond to in LLM training?
AlphaGo's Value Network was an Outcome Reward Model (ORM)—evaluating only final results. Today's PRMs refine AlphaGo's Value Network to finer granularity: evaluating not just final positions but the quality of every move.
3.3 MCTS → Test-Time Compute / Inference-Time Search
This is AlphaGo's most "prophetic" design.
MCTS (Monte Carlo Tree Search) did one thing at inference: trade compute for precision. The Policy Network provided "intuition" (prior probabilities), the Value Network provided "evaluation" (position judgment), and MCTS searched between the two, exploring more possibilities to find moves better than intuition alone.
Silver wrote explicitly in the paper:
> "MCTS can be viewed as a policy improvement operator. The probability distribution π produced by search is stronger than the raw network output p."
In 2024, this sentence was re-published in different words:
> "o1's long reasoning spends more compute at inference time to explore better reasoning paths."
From AlphaGo to o1, it is essentially the same idea:
AlphaGo proved this inequality in 2016. Today's LLMs are simply re-validating it in another domain (language).
3.4 Self-Play → RLHF / Self-Play / Synthetic Data
AlphaGo Zero (2017) was Silver's team's most radical experiment: completely abandoning human games, learning from zero via self-play.
The training loop: 1. Randomly initialize the network 2. The network plays against itself (MCTS guides every move) 3. Use game outcomes to train the network (Policy learns MCTS's search distribution; Value learns game outcomes) 4. The network gets stronger → MCTS search quality improves → game data quality improves → the network gets stronger still
This virtuous cycle reappeared in 2024 under a new name:
AlphaGo Zero's core lesson: when an environment permits unlimited self-play, human data is not necessary—it is only an accelerator. Go satisfies this condition (simple rules, clear win/loss, infinitely restartable). Language models are approaching this condition too (SFT data saturation, improving synthetic data quality, growing self-verification capability).
---
4. AlphaGo Zero's "Warm-Start Paradox"
There is a key difference between AlphaGo (2016) and AlphaGo Zero (2017):
| Version | Human Data | Architecture | Search Method | |------|----------|----------|----------| | AlphaGo (Lee) | Required (KGS games) | Separate Policy + Value | MCTS + Fast Rollout | | AlphaGo Zero | Not required | Single dual-headed network (ResNet) | Pure MCTS, no Rollout |
AlphaGo Zero proved: pure self-play can match or exceed human-data warm-starting. But this conclusion carries a hidden condition:
> It requires more compute and more time.
AlphaGo Zero needed 72 hours of training on 4 TPUs to surpass the previous version. Without the "warm start" of human games, it had to begin from completely random moves and spend longer crawling out of the performance plateau.
This "warm-start paradox" was re-debated in the 2024 LLM field:
AlphaGo Zero's history tells us: pure self-play has a higher ceiling, but a steeper path. The value of human data is not "instilling knowledge" but providing a good initialization point, allowing RL to converge to useful policies in less time.
---
5. Lee Sedol's "Divine Move": Humanity's Last Glory
March 12, 2016. Game 2. Move 78.
Lee Sedol played a wedge move in the lower-right corner, later called the "Divine Move." It broke AlphaGo's established evaluation, causing the program to make visible errors over the following moves.
From today's perspective, this move's significance lies not just in its aesthetic value, but in what it revealed about the limits of search algorithms:
This is the same mechanism behind today's LLM "hallucination" problem:
Lee Sedol's Divine Move was essentially an adversarial example—a carefully designed input landing in the model's "blind spot." Today's LLM RL training, inference-time search, and self-verification are all attempts to widen that blind spot and reduce the chance of being adversarially defeated.
---
6. Why AlphaGo Hasn't Become Obsolete
AlphaGo's legacy can be summarized as three "paradigm shifts":
1. From "Training-Time Optimization" to "Inference-Time Compute"
Traditional ML held that once a model is trained, inference is just a forward pass. AlphaGo broke this: inference can be a search process, and compute can be dynamically allocated at inference time.
Today's "long thinking" in o1 and DeepSeek-R1 replicates this paradigm. GPT-4's 1.8T parameters are crystallized training-time compute; o1's reasoning tokens are dynamically allocated inference-time compute. Only their combination forms a complete intelligent system.
2. From "Human-Data-Driven" to "Self-Evolving Loops"
AlphaGo Zero proved that self-play can produce data of quality beyond human capability, because:
Today's LLM efforts in synthetic data, Self-Play, and distillation from stronger models all follow this direction. A substantial fraction of DeepSeek-V3's 14.8T training tokens is synthetic—not because human data is insufficient, but because the quality of model-generated data keeps improving.
3. From "Single Network" to "Composite System"
AlphaGo was not the Policy Network, nor the Value Network, nor MCTS. It was the combination of the three, each handling what it did best:
Today's LLM systems are moving toward the same composite architecture:
No single model can solve everything. AlphaGo demonstrated this in 2016.
---
7. Conclusion: A Future Delivered Early
When AlphaGo beat Lee Sedol, most people saw only the drama of "Go" and "man versus machine." But what truly mattered was the general intelligence architecture Silver's team validated:
> Learning provides intuition, search provides precision, self-play provides unlimited data.
This architecture succeeded in Go, then in chess (AlphaZero), StarCraft (AlphaStar), and protein folding (AlphaFold). Now it is being re-implemented in language models—only the parameters have grown from millions to hundreds of billions, and the search space from 19×19 to vocab_size^sequence_length.
AlphaGo is not obsolete. It simply arrived a decade early and has been quietly waiting for the LLM community to catch up.
After the 2016 match, Lee Sedol said:
> "I lost, but this is not humanity's defeat. It is the victory of something humanity created."
Looking back today, he might add:
> "And that something is now teaching humans how to train the intelligence of the future."
---