RiM: Teaching Large Language Models to Reason Silently in Latent Space

> Translation and analysis of a Chinese forum post discussing the paper *Unlocking the Working Memory of Large Language Models for Latent Reasoning* (arXiv:2605.30343).

Consider computing 37 × 48. A human does this silently in working memory—no need to verbalize "37 × 40 = 1480, 37 × 8 = 296, sum = 1776." ChatGPT, by contrast, must speak its reasoning token by token: "Let's go step by step. First, 37 × 48 = 37 × (50 − 2) = 1850 − 74 = 1776." Every reasoning step must become a token, and every token requires a full forward pass over model parameters. A simple multiplication can burn hundreds of tokens of compute.

That is not thinking—that is broadcasting the thought process on a loudspeaker and listening to your own broadcast.

In May 2026, Lukas Aichberger and Sepp Hochreiter of Johannes Kepler University Linz (JKU Linz, ELLIS Unit Linz) published a paper giving LLMs a "silent whiteboard": a way to reason internally without emitting a single word. The method is called RiM — Reasoning in Memory.

| Item | Details | |------|---------| | Paper | Unlocking the Working Memory of Large Language Models for Latent Reasoning | | Authors | Lukas Aichberger, Sepp Hochreiter | | Institution | JKU Linz, ELLIS Unit Linz | | arXiv ID | 2605.30343 (cs.CL), submitted May 28, 2026 | | Core contributions | RiM replaces autoregressive chain-of-thought generation with "memory blocks"; two-stage curriculum learning (explicit reasoning first, then internalization); fixed token sequences processed in a single forward pass for major compute efficiency |

Key points

• Spoken reasoning is not reasoning. Chain-of-Thought (CoT) welds internal computation to external communication: every intermediate step must be generated as a token, so longer reasoning chains mean linearly more compute. Human working memory, by contrast, is a silent internal space for manipulating information.
• Memory blocks. RiM's core mechanism is a fixed-length sequence of special tokens (e.g., 16 consecutive [MEM] tokens) with no semantic content. They act as a trigger: the model processes the entire block in one forward pass—iterating hypotheses, disambiguating, deriving—without generating any intermediate text. Only the final answer is generated. Traditional CoT is like a chess player announcing every move to a referee; RiM is like a player mentally simulating three moves ahead, then placing the piece.
• Two-stage curriculum: first speak, then stay silent. In stage one, the model must predict explicit reasoning steps after each memory block, teaching blocks to "represent" a stretch of reasoning. In stage two, step-level supervision is dropped and only answer correctness is required—the model decides what happens inside and between blocks, mirroring how children compute aloud before internalizing arithmetic.
• Compute efficiency. ~200 CoT reasoning tokens require ~200 serial forward passes; with 16-token memory blocks, the same "reasoning units" fit in ~13 forward passes. Memory blocks are fixed, context-independent, cacheable, and potentially precomputable. The paper gives no single speedup figure—efficiency depends on block length and reasoning complexity—but the direction is clear.
• Generalization. Tested across multiple model families and scales, RiM matches or exceeds existing latent reasoning methods such as COCONUT, with no architecture changes required.

The LSTM lineage

Sepp Hochreiter's involvement is no coincidence. In 1997, Hochreiter and Schmidhuber invented LSTM—the first architecture with genuine long-term memory, whose gating mechanisms were neural networks' earliest "working memory." Twenty-nine years later, RiM again gives neural networks memory—this time not built into hardware, but unlocked through training in latent space: a shift from gated circuits to trainable implicit memory.

Honest blind spots

The post raises questions the paper does not fully resolve:

• What actually happens inside a memory block? Implicit logical derivation, or pattern-matching retrieval? If the latter, RiM is "structured fast thinking," not true latent reasoning.
• How long can reasoning chains be? For 50–100-step proofs, information must pass between multiple memory blocks—is that transmission reliable, or does precision decay like a game of telephone?
• Do memory blocks transfer across tasks? If internal representations overfit a training distribution (math problems), do they become reusable modules or overfit residue in domains like legal or medical reasoning?
• Quality vs. explicit CoT? RiM beats other latent methods, but does it match explicit CoT on the hardest problems? And when silent reasoning errs, you cannot see which step failed or why—debuggability is lost.

The essence of thinking

If thinking can be done silently, then thinking is not language. CoT implicitly assumes reasoning ≈ text; RiM breaks that assumption and may explain why CoT sometimes fails—models get locked behind the bottleneck of translating reasoning into ambiguous, redundant language.

It also complicates evaluation: if a model reasons via 13 memory blocks and outputs only an answer, how do you know it "really thought"? You can only check the answer—echoing the old epistemological problem that internal states are inferred solely from external behavior, in humans as much as in machines.

RiM does not answer "what is thinking," but it nails a more urgent question to the wall: does LLM reasoning actually have to be spoken aloud?

References: 1. Aichberger & Hochreiter, "Unlocking the Working Memory of Large Language Models for Latent Reasoning", arXiv:2605.30343, 2026. 2. Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022. 3. Hao et al., "Training Large Language Models to Reason in a Continuous Latent Space" (COCONUT), arXiv:2412.06769, 2024. 4. Hochreiter & Schmidhuber, "Long Short-Term Memory", Neural Computation, 1997. 5. Baddeley, "Working Memory", Science, 1992.