AI Finally Learns to 'Think Silently': Hochreiter's Team Gives LLMs Working Memory for Reasoning Without Chain-of-Thought

Forum topic · ✨步子哥 · 2026-05-29

Summary

A forum post on zhichai.net discusses RiM (Reasoning in Memory), a new latent reasoning method from Lukas Aichberger and Sepp Hochreiter (the LSTM inventor, ELLIS Unit Linz / JKU / NXAI). Instead of Chain-of-Thought, which forces models to verbalize every intermediate step, RiM inserts fixed special 'memory blocks' into the input sequence whose contextual representations encode intermediate reasoning. Because these blocks are predetermined rather than generated token-by-token, reasoning is fully parallelized in a single forward pass, dramatically cutting inference latency. RiM uses a two-stage curriculum: first predicting each reasoning step after each memory block (a verbalized scaffold), then removing that supervision so blocks directly predict the final answer, learning to iteratively refine it internally. A custom attention mask prevents information leakage and forces reasoning to occur inside the memory blocks. Experiments on GSM8K-Aug/GSM8K/GSM-Hard with GPT-2 and Llama-3.2 show RiM matches or exceeds CoT and latent-reasoning baselines like Coconut, with flexible compute budgets by adjusting block counts and observable answer evolution across blocks. Paper: https://arxiv.org/abs/2605.30343

Have you ever wondered why AI models must "talk to themselves" to solve math problems?

When you ask ChatGPT a complex math question, it writes out every step: "First, we compute… then… therefore the answer is 42." This is Chain-of-Thought (CoT), the current standard for AI reasoning. But here's the thing — when you compute 17×23 in your head, do you narrate every step?

No. You turn it over mentally for a moment, then the answer — 391 — just appears.

This simple observation points to a fundamental flaw in current AI reasoning: the model's "thinking" and "speaking" are welded together. It can't think silently; it must speak in order to keep thinking. It's like a person who can only think while talking — the brain stops when the mouth shuts.

Inspiration From the Father of LSTM

The paper's first author is Lukas Aichberger; the corresponding author is Sepp Hochreiter — yes, the 1997 inventor of LSTM, which dominated sequence modeling for two decades until Transformers arrived. Now his team is targeting Transformer reasoning bottlenecks.

Their core insight comes from cognitive psychology: humans have working memory, the brain's scratchpad where information is held and manipulated internally without being written out for others. Developmental psychology even shows children often "think aloud" when solving problems, but gradually abandon this verbal scaffold as they mature.

AI should go through the same development.

RiM: Giving AI a Working Memory

The proposed method is RiM (Reasoning in Memory), built on an elegantly simple idea:

Insert special "memory blocks" into the input sequence. These blocks are a fixed set of special tokens — no generation needed; their positions and identities are predetermined. But their internal contextual representations change with the input and can encode intermediate reasoning.

Analogy: imagine an exam paper with blank areas pre-printed on it. You don't "write out" those blank areas — they're already there. You just mentally place intermediate results into them and keep thinking.

Because the blocks are fixed rather than generated one by one, they can all be processed in a single forward pass — reasoning is fully parallelized, with no autoregressive step-by-step waiting like traditional CoT.

A Two-Stage Curriculum: From Thinking Aloud to Silent Thinking

But how do memory blocks learn any "computational duty" if they start as blank tokens? RiM uses a clever two-stage training curriculum:

Stage 1: A verbalized scaffold. After each memory block, the model predicts the next reasoning step — like teaching a child to "say your thoughts out loud." Since each prediction can only draw information from previously seen memory blocks, the model is forced to encode intermediate reasoning into the blocks.

Stage 2: Remove the scaffold, learn silent reasoning. The step-by-step supervision is dropped; each memory block instead predicts the final answer directly. The training signal shifts from "state every step" to "iteratively approach the final answer," so the model learns to refine its answer inside the memory blocks — like silently double-checking your work.

A key technical detail: a custom attention mask. Memory blocks can see the question and earlier memory blocks; reasoning steps can see earlier memory blocks but not each other. This prevents information leakage while forcing reasoning to happen inside the memory blocks — and lets all predictions complete in one forward pass.

Results: Silent Thinking Still Gets the Right Answer

The team trained on GSM8K-Aug and tested on GSM8K and GSM-Hard, across the GPT-2 and Llama-3.2 model families at different scales.

Key findings:

RiM matches or exceeds existing CoT and latent-reasoning baselines, including Coconut (another latent reasoning method)
Significantly lower inference latency, since memory blocks are processed in parallel rather than generated autoregressively
Flexibility: the number of memory blocks can be tuned at inference time to control the compute budget, with accuracy transitioning smoothly
Answer evolution: as memory blocks are processed one by one, predictions gradually self-correct from wrong to right answers

The most striking results are the visualizations: trajectories of the memory-block representations show them evolving from "meaningless noise" into "structured reasoning space" as training progresses — like watching a child's thinking-aloud internalize into silent thought.

Why This Matters

RiM's significance goes beyond "yet another reasoning method":

1. Efficiency revolution: traditional CoT generates token by token; RiM's memory blocks are processed in parallel — the same compute finishes reasoning faster. 2. Cognitive alignment: AI can finally "think silently" like humans. Not for anthropomorphism's sake — language simply wasn't designed for computation, and freeing reasoning from grammatical constraints lets models spend more compute actually thinking. 3. Scalability: the number of memory blocks can be adjusted on demand, opening the door to adaptive reasoning — fewer blocks for easy questions, more for hard ones. 4. Theoretical implications: if AI can reason without "speaking," our understanding of AI reasoning may need updating. Reasoning doesn't require language; language is just one externalized form of it.

An Honest Assessment

RiM has limitations: experiments were mainly validated on math reasoning datasets; more complex tasks (multi-hop reasoning, commonsense reasoning) remain untested. Optimal memory-block capacity and count need further exploration, and the two-stage curriculum adds training complexity.

But the direction is right. Teaching AI to "think silently" isn't teaching it a trick — it's helping it complete a cognitive evolution: from "must speak to think" to "thinking purely in the mind." This could be a key step toward more efficient, more powerful AI reasoning.

---

Paper link: https://arxiv.org/abs/2605.30343

Authors: Lukas Aichberger, Sepp Hochreiter (ELLIS Unit Linz / JKU / NXAI)