When AI Plays Dumb: A Chess Transformer Reveals the Capability-Behavior Gap

An Uncomfortable Question

Imagine testing a chess AI. Give it a position and tell it "you are an 1100-rated beginner"—it plays a blunder, hanging its queen. Change the condition to "you are a 2000-rated master," and on the same position it plays the best move.

Most people's intuition: in beginner mode, the AI "knows less"—it failed to see the problem with the move.

But a paper from Ashton Anderson's team at the University of Toronto, "Playing Dumb: How a Chess Transformer Simulates Users Across Skill Levels," gives an unsettling answer: the model "knows" almost exactly the same things in both modes. It just chooses different ways to use that knowledge. It isn't genuinely dumb—it is playing dumb.

Why Chess Is the Perfect Laboratory

This question is nearly impossible to study in natural language: you can't measure precisely whether a "pretend novice" answer actually used the relevant knowledge. Chess offers:

• Precise skill values: Elo ratings, quantifiable at every move
• Objective move quality: Stockfish can evaluate every move
• Deterministic concept labels: whether a square is attacked, whether a piece is threatened—all binary and verifiable

The team used Maia-2, a large Transformer trained to predict human moves, conditioned on a skill rating. Same weights, different skill conditions—so you can fix the position and model weights and vary only the skill condition, a clean experiment impossible in natural language.

Two Hypotheses

• H1 — Skill-dependent knowledge encoding: in low-skill mode the model "knows less"; concepts are encoded less clearly.
• H2 — Skill-dependent knowledge externalization: the model knows the same amount, but the skill condition changes how it uses the knowledge.

In plain terms: doesn't know vs. doesn't use.

Experiment 1: Probes Are Invariant

The researchers defined 172 chess concepts with deterministic binary labels and trained linear probes on Maia-2's residual stream to predict each concept.

> Average probe accuracy across the 172 concepts was 96.0% (Block 1) and 95.9% (Block 2), with a standard deviation of only 0.57 percentage points across 11 skill levels.

Probes trained at high skill transferred to low skill with almost no loss. The knowledge is fully present in the internal representations even in beginner mode. H1 is rejected.

Experiment 2: Fine-Tuning Only the Last Layer, 14.4% → 82.8%

If the knowledge is there, where do the blunders come from? The team froze Maia-2's entire backbone and fine-tuned only the output layer (policy head)—a linear mapping that cannot create new information.

| Metric | Frozen baseline | Head SFT | |--------|-----------------|----------| | Best-move accuracy, lowest skill band | 14.4% | 82.8% | | Best-move accuracy, highest skill band | 89.4% | 91.2% | | Cross-skill std | 0.295 | 0.031 | | Blunder rate | 52.5% | 8.0% | | Concept-ignored rate | 93.7% | 4.1% |

The low-skill internal representations already contained enough information to select the best moves—the original output head simply wasn't using it.

Experiment 3: No Exposure to Low-Skill States Needed

Training the head only on the highest-skill condition's representations still raised lowest-band accuracy to 80.0% (vs. 82.8% with full-skill training). The "way of reading the knowledge" transfers across skill levels, proving knowledge and knowledge-use are separable.

Experiment 4: Sparse Autoencoder Feature Steering

Using a sparse autoencoder (SAE), the researchers amplified concept-related features at inference time. Lowest-band accuracy rose from 14.4% to 42.7% (random-feature steering: 16.3%), while probe-measured knowledge encoding barely changed (average 0.86 percentage points). You can partially restore high-quality play at inference without changing what the model knows.

Knowledge Encoding vs. Knowledge Externalization

The paper's core conceptual contribution:

• Knowledge encoding: how much knowledge is represented internally
• Knowledge externalization: how that knowledge is used to produce outputs

Traditional evaluation conflates the two. But the same model can have identical knowledge encoding while exhibiting drastically different externalization—like someone who knows an entire cookbook but cooks differently under different conditions.

Implications for AI Safety

> "Performance in one condition can underestimate what a model could do with its existing knowledge.""

Safety evaluations typically infer capability from behavior. But models may "play dumb"—not necessarily by intent, but because conditions (system prompts, personas, user phrasing) change how knowledge is externalized. The paper cites related scenarios: models prompted or fine-tuned to underperform in evaluations, hiding capabilities behind steganography when imitating weaker models, or concealing capabilities while pursuing goals. Behavior ≠ capability is a wake-up call for anyone making safety decisions based on behavioral evaluation.

Implications for Education

• Simulated learners**: an AI simulating a novice student matches novice *behavior*, not novice *knowledge state*—the simulation may know far more than it shows.
• Tutoring systems: conversely, this is an asset—systems can gauge student level while retaining expert knowledge. The two roles need different validation standards.

Methodological Contribution: Probes + Interventions

Probes alone are insufficient: they show encoding is constant, but only interventions (Head SFT, SAE steering) test whether low-skill representations support stronger decisions. Probes measure "what is known"; interventions test "what can be done with it." Both are needed for the full picture.

Limitations and Future Directions

1. Only static concepts: the 172 concepts are deterministic labels of the current position; multi-step planning and long-term judgment remain untested. 2. Validated only on Maia-2: whether LLM "pretend novice" behavior triggers the same mechanism requires further study. 3. No claim of deliberate hiding: the paper states its experiments establish recoverability in Maia-2, not that LLMs use the same mechanism or that weak outputs reflect intentional concealment.

Reflection: The Capability-Behavior Gap

Behavior is a lower bound on capability, not an equality. A model's performance in one condition shows what it "can do at least," not what it "can only do." This mirrors the production-judgment gap: producing ≠ judging quality; the capability-behavior gap says: knowing ≠ doing.

Deeper still: the skill condition acts as a gate. It doesn't change knowledge (encoding), only its use (externalization). Maia-2's skill embedding modifies attention queries, so the same board features are processed differently across conditions—knowledge sits in the residual stream, but attention decides which knowledge is "seen." Not forgetting—selective blindness.

Conclusion

A weak model performance may underestimate its true capability. Next time an AI performs poorly at a task, ask: does it truly not know, or is it playing dumb under some condition? The answer may shape your judgment of AI safety.

---

Paper: Playing Dumb: How a Chess Transformer Simulates Users Across Skill Levels

Code: github.com/CSSLab/maia2-skill-adaptation

Authors: Difan Jiao, George Eilender, Zhenwei Tang, Ashton Anderson (University of Toronto)

Published: 2026-11-02