Reasoning in Memory: Decoupling Computation from Communication in LLMs

Introduction

Chain-of-thought prompting has become the default lever for improving the reasoning capabilities of large language models. The recipe is familiar: prompt the model to generate intermediate steps, scale test-time compute by producing more tokens, and watch accuracy improve on math and logic benchmarks. The empirical results are hard to argue with. But there is a conceptual problem buried in this approach that deserves more attention than it usually receives.

When a model "thinks out loud," it conflates two distinct activities: internal computation and external communication. Every intermediate result must be serialized into grammatical natural language before subsequent reasoning can condition on it. Language, however, was optimized for communication between agents, not for the efficient manipulation of internal state. A meaningful fraction of the compute budget therefore goes toward producing fluent, well-formed text rather than toward the actual reasoning.

The paper "Unlocking the Working Memory of Large Language Models for Latent Reasoning" by Lukas Aichberger and Sepp Hochreiter (ELLIS Unit Linz / LIT AI Lab, JKU) proposes a clean alternative. Their method, Reasoning in Memory (RiM), replaces autoregressively generated reasoning steps with fixed sequences of special tokens called memory blocks, which can be processed in a single forward pass. The framing draws on the cognitive psychology notion of working memory, an internal workspace for holding and manipulating task-relevant information without externalizing it.

The Core Idea: Fixed Memory Blocks as a Latent Workspace

The central insight is subtle but important. Existing latent reasoning methods, such as Coconut (Hao et al.) and related continuous-thought approaches, already attempt to escape the natural-language bottleneck by feeding continuous representations back into the model instead of discrete tokens. But these methods preserve the step-by-step generation paradigm. They still "think out loud," just in a continuous space rather than a discrete one. Each latent thought must be produced before the next can condition on it, which means computation remains coupled to autoregressive generation.

RiM breaks this coupling. Instead of generating intermediate representations, the model inserts fixed sequences of special tokens (memory blocks) into the context. The token identities and positions are fixed in advance; they are not produced by the model. What remains task-dependent is the contextual representation of those tokens after they pass through the transformer's attention layers. In other words, the memory block tokens are constant, but the hidden states they evoke encode the intermediate reasoning.

Because the tokens are fixed rather than generated, the entire reasoning trace can be processed in a single forward pass. This is the key efficiency win. There is no autoregressive loop where each step waits on the previous one. The method is parallelizable in a way that token-by-token generation simply is not.

The Two-Stage Curriculum and the Attention Mask

The obvious objection is that fixed special tokens have no inherent computational role. Inserting eight copies of a <m> token into the context does nothing useful unless the model has been trained to use them. RiM addresses this with a carefully structured two-stage curriculum.

Stage 1 (grounding). The model is trained to predict the next explicit reasoning step after each memory block. Crucially, a custom attention mask ensures that the readout after each block can recover its target only from the memory blocks seen so far. The model cannot peek at the answer through the actual reasoning-step text. This forces the latent workspace to encode task-relevant intermediate information. In effect, Stage 1 teaches the memory blocks to "mean something" by anchoring them to human-written reasoning steps.

Stage 2 (refinement). Once the memory blocks have a computational role, the step-level supervision is discarded. Now the model is trained to predict the final answer after each memory block, progressively refining its prediction across blocks. The supervision shifts from imitating written reasoning to iteratively improving the answer in latent space.

The attention mask design (Figure 2 in the paper) is what makes this efficient. Memory blocks attend to the question and to previous memory blocks. Target reasoning steps attend to previous memory blocks and optionally the question, but not to other reasoning steps. This prevents information leakage and allows all readouts to be trained in a single forward pass. The result is dense supervision, every block gets a training signal, combined with compute-efficient inference.

This dense-supervision-plus-single-pass combination is the engineering heart of the contribution. Many latent reasoning methods suffer from sparse or unstable training signals because the latent thoughts are not directly supervised. RiM sidesteps this by grounding the blocks in explicit steps first, then weaning the model off that crutch.

Empirical Results and What They Show

The authors evaluate across two model families and scales: GPT-2 and Llama-3.2. Training uses GSM8K-Aug, and evaluation covers GSM8K (in-distribution) and GSM-Hard (out-of-distribution). The headline finding is that RiM matches or exceeds existing CoT and latent reasoning baselines while avoiding autoregressive generation of thoughts entirely.

This is a measured claim, and I appreciate that the authors do not oversell it. RiM does not claim to dramatically beat chain-of-thought on accuracy. It claims parity or modest improvement, achieved at lower inference cost and with a fundamentally different computational structure. For a method that processes reasoning in a single forward pass, parity with autoregressive CoT is itself a strong result, because it suggests the externalized intermediate text was not strictly necessary for the computation.

The choice of GSM8K and GSM-Hard is reasonable for an initial study but also limiting. These are arithmetic word problems with relatively short reasoning chains and well-structured intermediate steps. Whether memory blocks can hold and manipulate the longer, more branching reasoning required for, say, competition mathematics or multi-hop retrieval remains an open question.

My Take: The Interpretability Tension

The technical contribution is solid, but the implication I find most consequential is one the paper does not foreground. Much of the current alignment and safety agenda rests on an assumption that reasoning is legible because it is verbalized. The hope is that if a model writes out its chain of thought, we can monitor that chain for deception, unsafe planning, or flawed logic. This "CoT monitoring" strategy has gained real traction as a near-term oversight mechanism.

RiM, and latent reasoning more broadly, challenges this assumption directly. If the actual reasoning happens silently inside memory-block hidden states, then the externalized text, where it exists at all, is no longer a faithful trace of the computation. A model trained to reason in latent space could produce a fluent post-hoc rationale that has little causal relationship to how it actually arrived at the answer. We already have evidence that chain-of-thought is often unfaithful even in standard models. Methods that explicitly move computation out of the token stream make this worse by design.

There is a genuine tension here between efficiency and oversight. Latent reasoning is more compute-efficient precisely because it stops paying the "grammar tax" of serializing thoughts into language. But that grammar tax was also, inadvertently, an interpretability subsidy. We were getting a partial window into the model's process for free. Remove the subsidy and the window closes.

I do not think this means latent reasoning is unsafe and should be avoided. The honest reading is that transparency-via-CoT was always a fragile guarantee built on a coincidence of the architecture. RiM makes the fragility explicit. The field will need interpretability tools that operate on hidden states directly, probing memory-block representations rather than reading generated tokens, if latent reasoning becomes mainstream.

Limitations and Open Questions

Several questions remain unresolved:

• Scaling behavior. The experiments cover GPT-2 and Llama-3.2 scales. Whether the working-memory capacity grows usefully with model size, and how the number of memory blocks should scale with problem difficulty, is unclear.
• Reasoning depth. Single-pass processing is attractive, but truly hard problems may require iterative refinement that a fixed block count cannot provide. How does RiM handle problems where the required reasoning depth exceeds the allocated blocks?
• Generalization. GSM-Hard tests some out-of-distribution robustness, but the broader question of transfer to non-arithmetic domains is untested.
• Interpretability of memory blocks. Can we read out what a memory block encodes? If Stage 1 grounds them in explicit steps, perhaps probing methods could recover approximate reasoning, offering a partial path back to oversight.

Conclusion

Reasoning in Memory is a clean instantiation of a principled idea: that internal computation should not be hostage to the syntax of external communication. By replacing generated reasoning steps with fixed memory blocks processed in a single forward pass, RiM achieves competitive accuracy at lower inference cost while decoupling reasoning from autoregressive generation. The two-stage curriculum and the leakage-preventing attention mask are the mechanisms that make this practical.

The deeper significance, in my view, is that RiM forces a conversation the field has been deferring. As reasoning moves into latent space, our interpretability and alignment strategies, many of which quietly assume verbalized chains, will need to be rebuilt around the hidden states where the thinking actually happens. The efficiency gains are real, but so is the cost to transparency. The interesting work ahead lies in recovering the latter without sacrificing the former.