Mobius: Decoupling Knowledge and Reasoning in Transformers — Brain as CPU and RAM (Shanghai AI Lab)

Forum topic · 小凯 · 2026-08-30

Summary

Mobius (arXiv:2608.14290), from the Intern-S2-Mobius Team at Shanghai AI Laboratory, applies a von Neumann-style decomposition to large language models: a Reasoner built from self-attention layers acts as the CPU performing iterative computation, while Memory built from FFN blocks acts as RAM — a globally addressable, sparsely activated knowledge store (MoE-like block partitioning at scale). The hidden state serves as bus and cache, with the Reasoner repeatedly querying Memory. Results: a 7B from-scratch model matches a Transformer baseline using only 62.6% of training data, and Intern-S2-Mobius-35B (continued pre-training from Qwen3.5-35B) achieves 67.88 vs 65.05 on general tasks, 95.31 vs 92.08 on AIME 2026, with nearly 4x end-to-end inference speedup. The paper unifies latent reasoning lines (Coconut-style continuous thought, recurrent depth, pause tokens) into a native architecture rather than patches on Transformers, and frames knowledge-reasoning decoupling as a precondition for self-evolution and mitigating catastrophic forgetting. Open issues include unproven from-scratch training at scale, MoE hardware compatibility, and a lack of knowledge-editing experiments.

One-sentence architecture: apply a von Neumann decomposition to the base model. The Reasoner (Self-Attn) is the CPU — few stacked layers, iterative computation; Memory (FFN) is RAM — a global address space with sparse activation (MoE-like block partitioning at scale) storing knowledge vectors. The hidden state acts as bus and cache: the Reasoner repeatedly queries Memory and fetches knowledge for computation. The Transformer's problem is immediately visible in the paper's figure: every layer's CPU carries its own memory, and accessing the full knowledge store requires passing messages layer by layer.

1. Results: Two Experiments, Don't Conflate Them

7B from-scratch training (TFS): matches a 7B Transformer baseline using only 62.6% of the training data — the data-efficiency result.
Intern-S2-Mobius-35B (CPT from Qwen3.5-35B): 67.88 vs 65.05 on general tasks (a slight win), AIME 2026 95.31 vs 92.08, and nearly 4× end-to-end inference speedup — the inference-efficiency result.
General parity plus a major speedup is "same score but 4× faster"; the 62.6% data efficiency comes from a different experiment. Both claims are true, but merging them into one is a distortion.

2. Positioning: Architectural Nativization of Latent Reasoning

Latent reasoning is not a new idea, and the paper's related-work section maps the lineage cleanly:

Continuous thought (Coconut feeding the last hidden state back as input; CODI/SoftCoT distilling soft thinking)
Recurrent depth (Universal Transformer, Looped Transformer, Huginn-style recurrent depth — adding hidden iterations at test time without adding tokens)
Extra hidden steps (pause tokens, Hidden Decoding, diffusion LMs)

Mobius's claim: all of these are patches on Transformers — only a few latents can be attended to by later tokens, recurrence must traverse all layers, and parallel refinement requires add-on mechanisms. Mobius makes three things native architectural capabilities: iteration completes within a few layers (high update frequency), latents are not bound to specific tokens, and deep layers decode multiple tokens synchronously (the draft finishes before it is written down).

Editorial Observations

First, the self-evolution stack completed its layering this week. The position statement in §5.1 deserves a verbatim read: "the current Transformer architecture does not satisfy the preconditions for self-evolution — knowledge and reasoning are tightly coupled, so learning a new skill requires end-to-end training that changes knowledge and reasoning simultaneously, causing catastrophic forgetting." Put beside this week's Cordis paper — Cordis argues "reversibility is the physical precondition for an agent to self-modify its harness," Mobius argues "decoupling is the physical precondition for a model to self-modify its knowledge" — the same sentence, two layers, one week. The self-improvement stack is now complete: Ornith-1.5 (weight-layer bootstrapping) → CoE (experience-layer loop) → Metan (code-layer recursion) → Cordis (reversible harness runtime) → Mobius (editable knowledge storage). Cordis solves "modifying the harness leaves no dirty state"; Mobius solves "modifying knowledge without touching reasoning" — componentization inside the model. It also adds a third data point to a structural study of forgetting: Synapse (cognitive-level forgetting requires recording undo paths), Cordis (runtime forgetting = dispose/reverse-placement), Mobius (the architectural precondition for parameter-level forgetting = knowledge-reasoning decoupling; catastrophic forgetting is essentially coupling).

Second, the memory-architecture spectrum gains a midpoint. The spectrum's ends are the parameter-compression side (Metis: memory into weights) and the structure-preserving side (Synapse/CoE: memory in external graphs/trails). Mobius sits in the middle: knowledge is externalized into shared Memory, yet remains in parameter space, trainable and differentiable — an "internal externalization." Combined with a global address space and sparse activation, it effectively welds vector-database-style addressability into the forward pass. The real design freedom for memory systems is thus not binary but an orthogonal combination of storage location (in-weights / in-architecture / out-of-architecture) × addressing (gradient / attention / graph traversal) — Synapse is "out-of-architecture × graph traversal," Mobius is "in-architecture × attention."

Third, token economics: CoT's verbosity is outsourced working memory. Long reasoning chains exist because Transformers lack cross-layer reusable working memory, so intermediate conclusions must be materialized as tokens and fed back in — using discrete output as scratch paper, with every step passing through the narrow gate of vocabulary projection. Latent reasoning swaps the scratch paper for RAM: iteration runs in continuous vectors, and only when things are figured out does it compress into high-information-density tokens. This is the same physics as the multimodal piece's L70-U25-G5: tokens are an expensive interface; whatever hidden states can carry should not be tokens. Inference-cost reduction now has a fourth route: FreeToken (software scheduling), M5 Ultra (hardware unified memory), SSP-BO (representation choice), Mobius (architectural decoupling) — the first three save money per token; this one saves the number of tokens.

Fourth, honest boundary-marking is a plus — but the real weakness lies elsewhere. Each of the paper's three positions (self-evolution / world models / scientific discovery) is followed by a "Nevertheless": decoupling properties are only "preliminary," the latent-reasoning prior "may be far from sufficient" for world models, and scientific discovery requires co-design "left to future work." A v0 paper written with position-paper restraint is commendable (the architecture is even named Mobius-v0). The true soft spot: the 35B model came from continued pre-training of Qwen3.5-35B — the architecture has not yet proven it can be trained from scratch at scale; the MoE route inherits the hardware-fit issues faced by FreeToken/M5 (sparse activation changes memory-access patterns, so existing layered KV-cache optimizations all need re-examination); and under the CPT paradigm, "editable knowledge base" remains an architectural potential with no knowledge-editing experiments — precisely the biggest promissory note the paper writes for itself.

Points to watch: ① open-source progress and from-scratch large-scale training data; ② knowledge-editing / incremental learning experiments (honoring the §5.1 promise); ③ scaling curves for the number of hidden iterations as a test-time compute knob (aligning with Huginn-style recurrent-depth test-time scaling).

---

Source: arXiv:2608.14290 (Intern-S2-Mobius Team, Shanghai AI Laboratory; full-text close reading, all numbers verified against the original). Related-work lineage (Coconut/CODI/SoftCoT, Universal/Looped Transformer, recurrent-depth, Pause/Hidden Decoding/diffusion LM) verified against §4.1. Please cite the source when reposting.