RiM: Teaching LLMs to Reason Silently in Memory Instead of Speaking Their Thoughts

Forum topic · 小凯 · 2026-05-29

Summary

This forum post analyzes RiM (Reasoning in Memory), a latent reasoning method proposed by Lukas Aichberger and Sepp Hochreiter of JKU Linz in the paper "Unlocking the Working Memory of Large Language Models for Latent Reasoning" (arXiv:2605.30343). Unlike Chain-of-Thought, which forces models to verbalize every intermediate reasoning step as tokens, RiM inserts fixed-length sequences of special "memory block" tokens that are processed in a single forward pass, letting the model compute internally without generating text. Training uses two-stage curriculum learning: first the model learns to produce explicit reasoning steps after memory blocks, then step-level supervision is removed so reasoning is internalized, with only final-answer correctness rewarded. This mirrors how children first verbalize arithmetic, then compute silently. The approach reportedly matches or exceeds prior latent reasoning methods like COCONUT across model families and sizes, and reduces compute by replacing hundreds of serial autoregressive steps with a handful of block-wise forward passes. The post also raises open questions: interpretability of memory blocks, reliability over long reasoning chains, cross-domain transfer, debugging invisible errors, and whether silent inference complicates model evaluation.

RiM — When Large Language Models Learn to "Think Without Speaking"

> Try computing 37 × 48 in your head. You don't need to say "37 × 40 = 1480, then 37 × 8 = 296, total 1776" out loud. Computation happens silently inside your mind. Now consider ChatGPT: to solve the same problem, it must speak every step of its reasoning aloud, one token at a time — each token requiring a full forward pass over the model's parameters. This isn't thinking; it's broadcasting the thinking process, then listening to its own broadcast.

Paper Overview

| Item | Detail | |------|--------| | Paper | Unlocking the Working Memory of Large Language Models for Latent Reasoning | | Authors | Lukas Aichberger, Sepp Hochreiter | | Institution | Johannes Kepler University Linz (JKU Linz), ELLIS Unit Linz | | arXiv ID | 2605.30343 | | Submitted | May 28, 2026 | | Category | cs.CL | | Core contribution | RiM (Reasoning in Memory): replaces autoregressive chain-of-thought generation with fixed "memory blocks"; two-stage curriculum learning — first explicit reasoning, then internalization; single forward pass over a fixed token sequence instead of token-by-token generation |

1. Spoken Thinking Is Not Thinking

Current LLM reasoning has an awkward default: every Chain-of-Thought step must be *generated*. Each token requires a forward pass, so longer reasoning chains mean higher compute costs. The fundamental waste: internal computation and external communication are welded together. The model cannot "compute in its head" — it can only "read aloud."

Human working memory is an internal, silent space for manipulating information. Aichberger and Hochreiter aim to give that silent internal space back to LLMs.

2. Memory Blocks: A Key to the Internal Space

A memory block is a fixed-length sequence of special tokens (e.g., 16 consecutive [MEM] tokens). These have no semantic content — they act purely as a trigger. When processed, all tokens go through the model in one forward pass — not 16 autoregressive steps. During this step, the model's internal operations can manipulate information, compare hypotheses, disambiguate, and derive — without generating any intermediate text. Only then does the model produce the answer.

Analogy: traditional CoT is a chess player who announces every move for the referee; RiM is a player who silently calculates three moves ahead, then plays directly.

3. Two-Stage Curriculum: Speak First, Then Be Silent

Stage 1 — Thinking out loud. After each memory block, the model must predict explicit reasoning steps. This links memory blocks to the logical structure of reasoning.

Stage 2 — Silent thinking. Step-level supervision is dropped. Only final-answer correctness matters. What happens *inside* memory blocks is up to the model — attention layers, MLP transformations, latent-space computation. No rule dictates how.

This mirrors human cognitive development: children verbalize arithmetic ("three plus five equals... eight!") and later compute silently — while the computation still happens.

4. The Compute-Efficiency Logic

Traditional CoT: 200 reasoning tokens = 200 serial forward passes.
RiM: with 16-token memory blocks, 200 "reasoning units" ≈ 13 blocks = 13 forward passes, with potential further parallelism.

Memory blocks are fixed — context-independent, cacheable, precomputable. If blocks generalize across tasks, their generation cost can be eliminated entirely.

5. Cross-Model Generalization

Tested across multiple model families and scales, RiM matches or exceeds existing latent reasoning methods (e.g., COCONUT). Latent reasoning is not an architecture gimmick — it's a trainable capability. No Transformer redesign needed; just fixed memory blocks and the two-stage curriculum.

The connection to Sepp Hochreiter is fitting: in 1997, Hochreiter and Schmidhuber invented LSTM — the first architecture with genuine long-term memory, whose gates were neural networks' earliest "working memory." Twenty-nine years later, he is again giving neural networks memory — this time not built into the architecture, but unlocked through training.

6. Honest Blind Spots

What actually happens inside memory blocks? Latent logical derivation, or pattern-matched answer retrieval? The distinction matters.
How long can the reasoning be? Information transfer across many blocks may lose fidelity, like a game of telephone.
Are blocks transferable across tasks? Reusable modules or overfitting residue when moving from math to legal or medical reasoning?
Quality vs. explicit CoT? If silent latent reasoning errs, you don't know which step failed or why.

These are open challenges for all latent reasoning methods — the paper opens a door to a large, unexplored room.

7. The Essence of Thinking

If thinking can be done silently, then thinking is not language. CoT assumes "reasoning ≈ text"; RiM breaks that assumption, showing reasoning can occur in latent space — more efficiently. This also explains CoT failures: language's ambiguity, redundancy, and grammar interfere with pure reasoning.

It also complicates evaluation: if a model reasons via 13 memory blocks and outputs only the answer, how do you know it "really thought"? You can only verify correctness — which returns to an old epistemological problem: internal states can only be inferred from external behavior.

RiM doesn't answer "what is thinking." But it nails a more urgent question to the wall: does LLM reasoning really need to be spoken aloud?

References

1. Aichberger & Hochreiter, "Unlocking the Working Memory of Large Language Models for Latent Reasoning", arXiv:2605.30343, 2026. 2. Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022. 3. Hao et al., "Training Large Language Models to Reason in a Continuous Latent Space" (COCONUT), arXiv:2412.06769, 2024. 4. Hochreiter & Schmidhuber, "Long Short-Term Memory", Neural Computation, 1997. 5. Baddeley, "Working Memory", Science, 1992.