How Much Compute Does Your LLM Waste on Thinking Out Loud? RiM Says: Reason in Working Memory, Silently

> Source: *Unlocking the Working Memory of Large Language Models for Latent Reasoning*, Lukas Aichberger & Sepp Hochreiter, JKU Linz / NXAI, https://arxiv.org/abs/2605.30343

1. Why Must AI "Talk to Itself"?

Ask GPT to solve a math problem and it writes: "First, I need to calculate...", then "Now, substituting the result..." — hundreds of tokens of step-by-step narration before the final answer.

This is Chain-of-Thought (CoT): making LLMs "think out loud." It works, but at a cost — every intermediate step becomes a generated token, and every token requires autoregressive decoding. Deeper reasoning means more tokens and higher latency.

But consider humans: when doing mental arithmetic, you don't verbalize each step. Numbers rotate inside your working memory, intermediate results are stored internally, and the answer simply pops out.

Sepp Hochreiter's team says LLMs can do this too. They propose Reasoning in Memory (RiM) — reasoning inside a "working memory" without generating any intermediate tokens.

2. Core Idea: Memory Blocks Instead of Thinking Out Loud

RiM's design intuition comes from cognitive science models of working memory, where information can be manipulated internally without being externalized.

RiM brings this into Transformers:

1. Memory Block: A fixed-length sequence of special tokens inserted into the input. These tokens are not generated by the model — they are pre-placed placeholders. 2. Single Forward Pass: Because memory blocks are fixed (not autoregressively generated), the whole sequence can be processed in one forward pass. 3. Two-Stage Training:

• Stage 1 (Grounding): After each memory block, the model predicts explicit reasoning steps — teaching it what to store in the block.
• Stage 2 (Refinement): Supervision on reasoning steps is removed; the model refines final answers directly after each memory block, learning to reason autonomously inside it.

Analogy: Stage 1 is like teaching a child to whisper intermediate steps during mental math; Stage 2 is telling them to "be quiet and compute in your head."

3. Why Is This Better Than Prior Latent Reasoning?

Previous latent reasoning approaches include:

• Coconut (Meta, 2025): replaces discrete tokens with continuous representations — but these are still autoregressively generated, just as vectors rather than words.
• Recurrent module methods: repeatedly iterate the same layer activations vertically, like "chewing on the same idea."

RiM's key distinction: memory blocks are fixed, not generated.

This means:

• No autoregressive decoding — the entire reasoning process completes in a single forward pass
• Much lower inference latency — no step-by-step generation overhead
• Flexible compute budget — want deeper reasoning? Add more memory blocks, each adding fixed compute

4. Results: Fast, Accurate, Robust

Evaluated on GSM8K (grade-school math) and GSM-Hard (harder math), across GPT-2 and Llama-3.2 families.

Performance

RiM matches or exceeds Coconut and similar methods in accuracy at various model scales, while avoiding autoregressive generation.

Latency

Because memory blocks are fixed sequences processed in one forward pass, inference latency is substantially lower than generation-based approaches.

Robustness

The number of memory blocks is adjustable at inference time — fewer blocks for faster answers, more for deeper reasoning. Performance degrades smoothly with block count rather than collapsing.

The Model Really "Thinks" in the Blocks

Visualization analysis shows that during training, memory block representation trajectories gradually diverge, with different samples forming distinct representations — evidence the model performs sample-specific intermediate computation rather than idling.

5. Honest Assessment: Limitations and Future Work

RiM is not a silver bullet:

1. Two-stage curriculum required: Stage 1's supervision comes from CoT data; poor CoT quality means poor grounding. 2. Memory block length is a hyperparameter: too short can't hold complex reasoning, too long wastes compute. Currently tuned manually. 3. Validated only on math reasoning: GSM8K and GSM-Hard are both math. Open-domain reasoning and code generation remain untested. 4. No hybrid mode yet: some problems may need "think silently, then write out." RiM is purely latent; mixed approaches are future work.

Still, the core contribution is clear: LLMs can be trained to develop a "working memory" — an internal reasoning space independent of autoregressive generation. Perhaps future LLMs won't have to choose between thinking and speaking.

6. One-Sentence Takeaway

RiM teaches LLMs a skill humans mastered long ago: thinking silently, in your head — faster inference, lower latency, and the model genuinely computes inside its "memory" instead of slacking off.

---

Paper: https://arxiv.org/abs/2605.30343 Code: stated to be open-sourced soon