This post is a detailed Chinese-language analysis of the MIT CSAIL blog post "Language model harnesses are compositional generalizers" by Alex Zhang and Omar Khattab (OASYS), first published as a blog on July 20, 2026 — explicitly *not* peer-reviewed. The Chinese author notes the title is sometimes misquoted as "Language Models are Harnessed Generalizers"; the actual subject of the claim is the harness, not the model: the thing that generalizes is the program wrapping the neural network, not the network itself.
Key points
1. Why Transformers fail at compositional generalization
- Compositional generalization = recombining learned components to solve unseen problems. This critique dates to Fodor & Pylyshyn (1988) on *systematicity*.
- Strongest evidence cited: *Faith and Fate* (Dziri et al., NeurIPS 2023):
- GPT-4: 3-digit × 3-digit multiplication = 59%; 4×4 = 4%. With scratchpad, 3×3 rises to 92% but hard problems stay near 0%.
- GPT-3 fine-tuned on 1.8M multiplication examples to 100% on 3×3 scores 0% on 4×4.
- 82% of 4×2 answers were correct *with wrong intermediate steps* — "restoration error": retrieving memorized paths, not executing algorithms ("linearized subgraph matching"). Accuracy decays exponentially with reasoning depth.
- The MIT authors attribute the failure to 2017-era token-level inductive biases: attention computes geometric similarity and carries no symbolic "decompose into subproblems" bias. Scaling "is just not budging" on this metric — a key argument against "scaling fixes everything."
- Concrete consequence: length generalization failure — training at one context length does not transfer to longer lengths, which is why production models must repeatedly re-train with longer data at every length tier.
- Definition: a harness is "the program that sits between the external world and the neural network," deciding how to encode arbitrarily long/complex state into LLM inputs and pick the next action. Formalized as H: s → a.
- The conceptual leap: the harness is no longer "engineering glue" but defines higher-level inductive biases, because it controls what the model sees.
- "A good harness produces observations that are locally in-distribution: every individual LM call is in-distribution with respect to the training data." The overall task may be out-of-distribution (novel domain, 2M tokens); each individual call must be familiar.
- Mainstream agents (ReAct, CodeAct, Claude Code, Codex) violate LID by appending observations to a growing prefix context — after 50 loop iterations, the prompt resembles nothing in the training data. This gives a mechanistic account of context rot: the poison is not length itself but *distribution drift* caused by length.
- The harness induces an equivalence class over tasks: structurally similar tasks are abstracted to appear "literally token-for-token similar" to the root model, constructing generalization rather than hoping for it.
- Predecessor work: Recursive Language Models (Oct 2025 blog / arXiv:2512.24601) → Mismanaged Geniuses Hypothesis (Apr 2026) → this post (Jul 2026).
- RLM treats long context as an environment, not an input: the root model sees only metadata (~20 tokens, e.g.
context = <5,000,000 characters>in a REPL variable) and interacts via programmatic, recursive sub-calls. - Code-based decomposition beats tool calls: a for loop spans a whole "space of decompositions"; intermediate results stay in variables instead of polluting the prompt. This yields task-isomorphic token traces, so training on short tasks transfers to million-token tasks and cross-domain environments (six length-generalization and three cross-domain test environments are referenced in the source).
- It is a blog with data and formalization, but no peer review.
- "Isn't this just subagents?" — a substantive objection with real merit.
- Generalization is not automatic; the harness must actively push decomposition.
- Real costs in latency and token spend; recursion adds failure modes.
- A theoretical concern: circularity risk in defining LID relative to training distribution.
2. The harness as carrier of inductive biases
3. Locally In-Distribution (LID) — the core criterion
4. Recursive Language Models (RLM) and context offloading
5. Caveats raised by the forum author
6. Practical takeaways
1. Design harnesses so every single LLM call looks like training data (LID principle) — offload long context into variables, never blindly append. 2. Prefer programmatic (code) decomposition over fixed tool-call loops. 3. Shift perspective: the harness is part of model capability, not scaffolding around it.> Original (Chinese) post is a long-form technical analysis; this English version is a structured summary. Original blog cited in source: https://alexzhang13.github.io/blog/2026/harness