Reasoning in Memory (RiM): Hochreiter's Team Lets LLMs Think Silently with Working Memory Instead of Chain-of-Thought

Forum topic · ✨步子哥 · 2026-05-29

Summary

Researchers led by Sepp Hochreiter (LSTM inventor, first author Lukas Aichberger) propose RiM (Reasoning in Memory), a method that lets large language models reason without generating explicit chain-of-thought text. RiM inserts fixed memory blocks into the input sequence whose internal contextual representations encode intermediate reasoning, so all blocks are processed in a single parallel forward pass rather than autoregressively. Training uses a two-stage curriculum: first the model learns to verbalize reasoning steps after each block (scaffolding), then supervision shifts to predicting only the final answer, forcing internal computation. A custom attention mask prevents information leakage between reasoning steps. On GSM8K and GSM-Hard using GPT-2 and Llama-3.2 models, RiM matches or exceeds CoT and latent reasoning baselines like Coconut, while significantly reducing inference latency. The number of memory blocks can be tuned at inference to control compute budget, and visualizations show block representations evolving into structured reasoning trajectories. Limitations include validation mainly on math benchmarks and added training complexity.

Why must AI "talk to itself" to solve math problems? Chain-of-Thought (CoT) forces models to verbalize every intermediate step, but humans compute 17×23 silently in their heads. A new paper argues AI's "thinking" and "speaking" are unnecessarily coupled—and shows how to decouple them.

The team

First author Lukas Aichberger, corresponding author Sepp Hochreiter—the inventor of LSTM (ELLIS Unit Linz / JKU / NXAI). Their core insight comes from cognitive psychology: humans have working memory, a mental scratchpad. Child development research shows children first "think aloud," then internalize language into silent processing. RiM applies the same developmental arc to AI.

How RiM works

RiM (Reasoning in Memory) inserts special memory blocks—a fixed sequence of tokens that are not generated but whose contextual representations encode intermediate reasoning—into the input sequence.

Because the blocks are fixed rather than generated one by one, they can be fully processed in a single parallel forward pass, unlike autoregressive CoT.

Two-stage training curriculum:

1. Scaffolding stage: After each memory block, the model predicts the next reasoning step. Each prediction can only access earlier blocks, forcing intermediate reasoning into them—like teaching a child to verbalize thoughts. 2. Silent reasoning stage: Stepwise supervision is removed; each block directly predicts the final answer, so the model learns to iteratively refine the answer inside the blocks.

A custom attention mask lets memory blocks see the question and prior blocks, while reasoning steps see prior blocks but not each other—enabling parallel prediction with no information leakage.

Results

Trained on GSM8K-Aug, evaluated on GSM8K and GSM-Hard across GPT-2 and Llama-3.2 families:

RiM matches or exceeds CoT and latent-reasoning baselines, including Coconut
Significantly lower inference latency due to parallel processing
Flexible compute budget: the number of memory blocks can be adjusted at inference, with smooth accuracy transitions
Answer evolution: predictions gradually correct themselves as blocks are processed, moving from wrong to right answers
Visualizations show block representations evolving from noise into structured reasoning trajectories—the model literally internalizes its thinking

Why it matters

1. Efficiency: parallel memory blocks replace token-by-token CoT generation. 2. Cognitive alignment: language isn't designed for computation; freeing reasoning from grammar lets models spend compute on actual thinking. 3. Scalability: block count enables adaptive inference—fewer blocks for easy problems, more for hard ones. 4. Theory: reasoning may not require language at all; language is just one externalized form of it.

Honest limitations

Validation is mainly on math benchmarks; multi-hop and commonsense reasoning remain untested. Optimal memory block capacity/count and the added training complexity of the two-stage curriculum need further study.

Paper: https://arxiv.org/abs/2605.30343

Authors: Lukas Aichberger, Sepp Hochreiter (ELLIS Unit Linz / JKU / NXAI)