Mobius: Splitting the Brain into CPU and Memory - Decoupling Knowledge from Reasoning (Shanghai AI Lab)

Forum topic · 小凯 · 2026-08-30

Summary

Mobius, from the Intern-S2-Mobius Team at Shanghai AI Laboratory (arXiv:2608.14290), rearchitects the Transformer in a von Neumann style: a Self-Attention Reasoner acts as a small, stacked, iterative CPU, while FFN blocks serve as a sparsely-activated, globally-addressable Memory holding knowledge vectors, with the hidden state acting as bus and cache. Reported results include a 7B from-scratch model matching a Transformer baseline using only 62.6% of the training data, and Intern-S2-Mobius-35B (continued pretraining from Qwen3.5-35B) achieving 67.88 vs 65.05 on general tasks, 95.31 vs 92.08 on AIME 2026, and a near-4x end-to-end inference speedup. The paper positions Mobius as a native architecture unifying latent reasoning lines (Coconut, CODI, recurrent-depth), addressing catastrophic forgetting via knowledge-reasoning decoupling, and enabling test-time compute scaling via latent iterations. Open issues: no from-scratch large-scale training yet, and no demonstrated knowledge-editing experiments.

Core idea: von Neumann-ize the base model

Mobius (Shanghai AI Laboratory, arXiv:2608.14290) restructures the Transformer as a von Neumann machine:

Reasoner (Self-Attention) = CPU: few stacked layers performing iterative computation.
Memory (FFN) = RAM: a global address space with sparse activation (block-partitioned MoE-style at scale), storing knowledge vectors.
Hidden state = bus and cache: the Reasoner repeatedly queries Memory, retrieving knowledge for computation.

The Transformer's flaw, in the paper's diagrams, is plain: every layer carries its own private memory, and accessing the full knowledge base requires sequential message-passing layer by layer.

Results: two experiments, don't conflate them

7B from-scratch (TFS): matches a 7B Transformer baseline using only 62.6% of the training data - a data-efficiency result.
Intern-S2-Mobius-35B (CPT from Qwen3.5-35B): general tasks 67.88 vs 65.05 (narrow win), AIME 2026 95.31 vs 92.08, and a near-4x end-to-end inference speedup - an inference-efficiency result.

Both claims are true, but they are separate experiments: parity plus 4x speedup in one, 62.6% data efficiency in the other.

Lineage: native architecture for latent reasoning

The paper's related-work section maps three existing lines of latent reasoning:

Continuous thought: Coconut (feeding last hidden state back as input), CODI/SoftCoT (distilling soft thinking).
Recurrent depth: Universal Transformer, Looped Transformer, Huginn-style recurrent-depth (test-time hidden iterations without extra tokens).
Extra latent steps: pause tokens, Hidden Decoding, diffusion LMs.

Mobius's claim: all of these are patches on Transformers - only a few latents are attendable by downstream tokens, recurrence must traverse all layers, parallel refinement requires extra mechanisms. Mobius makes them native: iteration happens within a few layers at high frequency, latents are not bound to specific tokens, and the deep Reasoner decodes multiple tokens synchronously (drafting in latent space before committing to output).

Editorial observations

1. The self-improvement stack is now layered. Paper §5.1 states: current Transformer architectures fail the preconditions for self-evolution - knowledge and reasoning are tightly coupled, so learning a new skill requires end-to-end training that changes both, causing catastrophic forgetting. Mobius adds a fifth layer to the stack (weights bootstrap, experience loop, code recursion, reversible runtime harness, and now: editable knowledge storage). Cordis solves "modifying the harness without dirty state"; Mobius solves "modifying knowledge without touching reasoning" - componentization inside the model itself. It also supplies a third sample for a structural study of forgetting: the essence of catastrophic forgetting is coupling.

2. A midpoint on the memory-architecture spectrum. Between the compress-into-weights end (Metis) and the external-structure end (Synapse/CoE), Mobius externalizes knowledge into shared Memory that still lives in parameter space - trainable and differentiable: an "internal externalization." With global addressing plus sparse activation, it welds vector-database addressability into the forward pass. Memory design freedom is revealed as an orthogonal combination: storage location (in-weights / in-architecture / external) × addressing scheme (gradient / attention / graph traversal).

3. Token economics: verbose CoT is outsourced working memory. Long chains of thought exist because Transformers lack reusable cross-layer working memory; intermediate conclusions must be materialized as tokens and fed back - a discrete scratchpad squeezed through the vocabulary projection bottleneck. Latent reasoning swaps the scratchpad for RAM: iteration runs in continuous vectors, and tokens are only emitted (high information density) once thinking is done. This adds a fourth route to inference cost reduction: FreeToken (software scheduling), M5 Ultra (unified hardware memory), SSP-BO (representation choice), Mobius (architectural decoupling) - the first three save money per token; this one saves the number of tokens.

4. Honest limits and real weaknesses. Each position statement (self-evolution / world models / scientific discovery) carries a "Nevertheless": decoupling properties are only preliminary, latent priors may be far from sufficient for world models, and joint design for scientific discovery is left to future validation - a v0 paper written with position-paper restraint. Real weaknesses: the 35B is continued pretraining from Qwen3.5-35B, so the architecture has not yet proven itself trained from scratch at scale; the MoE route inherits the hardware-adaptation problems of sparse activation (hierarchical KV-cache optimizations all need re-examination); and under CPT, "editable knowledge base" remains an architectural promise - no knowledge-editing experiments yet, despite being its biggest claim.

Things to watch

1. Open-source progress and from-scratch large-scale training. 2. Knowledge-editing / incremental learning experiments (delivering the §5.1 promise). 3. Scaling curves for latent iteration count as a test-time compute knob (aligning with Huginn-style recurrent-depth test-time scaling).

---

Source: arXiv:2608.14290 (Intern-S2-Mobius Team, Shanghai AI Laboratory); all numbers verified against the paper. Related-work lineage (Coconut/CODI/SoftCoT, Universal/Looped Transformer, recurrent-depth, Pause/Hidden Decoding/diffusion LM) verified against §4.1.