RiM: Reasoning Inside an LLM's Working Memory Without Generating Tokens

Forum topic · ✨步子哥 · 2026-05-30

Summary

RiM (Reasoning in Memory), proposed by Lukas Aichberger and Sepp Hochreiter (JKU Linz / NXAI), enables large language models to perform latent reasoning without autoregressively generating intermediate chain-of-thought tokens. The method inserts fixed-length memory blocks into the input sequence as pre-placed placeholder tokens, allowing the entire reasoning process to complete in a single forward pass. Training uses a two-stage curriculum: Stage 1 grounds memory blocks by having the model predict explicit reasoning steps after each block, while Stage 2 removes that supervision so the model learns to refine answers internally. Unlike prior latent reasoning approaches such as Coconut, RiM's memory blocks are fixed rather than generated, eliminating step-by-step decoding overhead. On GSM8K and GSM-Hard benchmarks with GPT-2 and Llama-3.2, RiM matches or exceeds Coconut's accuracy with substantially lower latency. The number of memory blocks can be adjusted at inference time for a speed/depth tradeoff, and visualization analysis shows block representations are sample-specific, indicating genuine internal computation. Limitations include dependence on CoT data for grounding, manually tuned block lengths, and validation only on math reasoning.

How Much Compute Does Your LLM Waste Thinking Out Loud? RiM Puts Reasoning Into Working Memory

> Source: Unlocking the Working Memory of Large Language Models for Latent Reasoning, Lukas Aichberger & Sepp Hochreiter, JKU Linz / NXAI, https://arxiv.org/abs/2605.30343

Why Must AI "Talk to Itself"?

Ask GPT to solve a math problem and it writes out hundreds of tokens of Chain-of-Thought (CoT) — "First, I need to compute…" — before giving an answer. It works, but every intermediate step costs a token and every token requires autoregressive decoding. Deeper reasoning means more tokens and higher latency.

Humans don't do this: mental arithmetic happens silently in working memory. Hochreiter's team argues LLMs can do the same. Their proposal, Reasoning in Memory (RiM), lets models reason inside a "working memory" without generating any intermediate tokens.

Core Idea: Memory Blocks Instead of Thinking Out Loud

RiM is inspired by cognitive-science models of working memory, where information is manipulated internally without externalization.

1. Memory blocks: Fixed-length sequences of special tokens are inserted into the input. They are pre-placed placeholders, not model-generated output. 2. Single forward pass: Because the blocks are fixed rather than autoregressively generated, the whole sequence is processed in one forward pass — no step-by-step decoding. 3. Two-stage training:

Stage 1 (grounding): After each memory block, the model is trained to predict the explicit reasoning steps — teaching it what to store in memory.
Stage 2 (refinement): The reasoning supervision is removed; the model learns to refine the final answer directly after each block, reasoning autonomously inside the blocks.

Analogy: Stage 1 teaches a child to whisper intermediate steps while learning mental math; Stage 2 says "be quiet and compute in your head."

Why It Beats Prior Latent Reasoning

Coconut (Meta, 2025) replaces discrete tokens with continuous representations, but these are still generated autoregressively — the model "thinks" one vector at a time.
Recurrent-depth methods iterate the same layers vertically, like chewing on one idea repeatedly.

RiM's key difference: memory blocks are fixed, not generated. This means no autoregressive decoding, much lower inference latency, and a flexible compute budget — deeper reasoning just requires adding more blocks, each adding a fixed amount of computation.

Results: Fast, Accurate, Robust

Tested on GSM8K and GSM-Hard across the GPT-2 and Llama-3.2 families:

Accuracy: Matches or exceeds latent reasoning methods like Coconut at every scale.
Latency: Substantially lower, since reasoning completes in a single forward pass.
Robustness: The number of memory blocks can be tuned at inference time — fewer blocks for speed, more for depth. Performance degrades smoothly rather than collapsing.
Genuine internal computation: Visualization shows that during training, representation trajectories in the memory blocks diverge across samples, meaning the model is doing sample-specific intermediate computation rather than idling.

Honest Limitations

1. Two-stage training required: Stage 1 grounding needs CoT data; poor CoT data yields poor grounding. 2. Block length is a hyperparameter: Too short can't hold complex reasoning; too long wastes compute. Currently tuned manually. 3. Math-only validation: GSM8K and GSM-Hard are math benchmarks; open-domain reasoning and code generation remain untested. 4. No hybrid mode: RiM is purely latent; combining internal reasoning with explicit output is future work.

Still, the core contribution is clear: LLMs can be trained to develop a working memory — an internal reasoning space that doesn't depend on autoregressive generation.

One-Line Takeaway

RiM teaches LLMs a skill humans mastered long ago: thinking silently. Faster reasoning, lower latency, and the model genuinely computes inside its "memory" rather than slacking off.

---

Paper: https://arxiv.org/abs/2605.30343 Code: Announced as forthcoming open source.