Thinking Without Words: When Models Invent Their Own Compressed Language

Reasoning models have settled into a comfortable but expensive habit. To solve a hard math problem or trace a multi-hop question, a large language model produces a long natural language chain-of-thought, often hundreds of tokens of explicit step-by-step deliberation, before committing to an answer. This works. It also burns latency, inflates inference cost, and bloats the trace lengths that reinforcement learning has to grind through. The field has been searching for a way to keep the reasoning gains while shedding the verbal overhead.

The paper "Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought" by Keshav Ramji, Tahira Naseem, and Ramón Fernandez Astudillo at IBM Research offers a clean answer to a deceptively simple question: can a short sequence of discrete, reserved tokens replace a long verbal rationale and still preserve the performance of chain-of-thought? Their result is that it can, and the path there reveals something interesting about how models internalize reasoning.

The Core Idea: A Reserved Abstract Vocabulary

Most prior efficiency work has leaned in one of two directions. The first compresses or internalizes verbal CoT into continuous representations, as in Coconut (Hao et al., 2025), which reasons in a continuous concept space rather than emitting tokens. The second adds filler or pause tokens (Goyal et al., 2024; Pfau et al., 2024) that expand the model's effective per-position computation without carrying readable meaning. Continuous methods tend to be efficient but underperform verbalized CoT; filler tokens help but are not a full reasoning substitute.

Abstract-CoT takes a different bet. Instead of a continuous latent space, the model emits a bounded sequence of discrete tokens drawn from a reserved abstract vocabulary, wrapped in <beginabstract> ... <endabstract> delimiters, in place of the natural language rationale inside <think> ... </think> tags. These tokens are not human-readable. They are distinguishable filler tokens that the model learns to deploy as a compressed latent scratchpad.

The headline numbers are strong: up to 11.6x fewer reasoning tokens while matching or outperforming verbalized CoT on MATH-500 (mathematical reasoning), AlpacaEval (instruction-following), and HotpotQA (multi-hop reasoning). The method generalizes across model families, which matters because brittle, model-specific tricks rarely survive contact with the broader field.

Solving the Cold-Start Problem

The interesting engineering challenge here is that the abstract tokens do not exist in the pretrained vocabulary. Their embeddings are randomly initialized and carry no meaning. You cannot simply prompt a model to "reason in abstract tokens" and expect anything coherent. This is a genuine cold-start problem, and the bulk of the paper's contribution is a training recipe to bootstrap these embeddings into something useful.

The recipe has two stages.

Stage one is a policy iteration warm-up loop that alternates between two operations:

• Bottlenecked supervised fine-tuning. The model generates a verbal CoT, but the final response is masked so that it can only attend to the abstract tokens, not the verbal rationale. This forces the abstract token representations to absorb the information from the verbal CoT, acting as an information bottleneck. The verbal trace is the teacher; the abstract sequence is the compressed channel through which all relevant information must pass.
• Self-distillation. The verbal CoTs are discarded, and the model is trained to generate abstract sequences directly from the prompt alone, using constrained decoding restricted to the codebook of abstract tokens.

These two steps alternate, iteratively refining both the embeddings and the model's ability to produce useful abstract sequences on-policy. The information bottleneck framing is the key conceptual move. By denying the answer-generating pass access to the verbal reasoning, the only way the model can recover performance is to route the necessary computation through the abstract tokens.

Stage two applies reinforcement learning, specifically GRPO, with a generative reward model and constrained decoding to the abstract vocabulary. This warm-started RL induces exploration over abstract token sequences and refines the generation policy beyond what supervised distillation alone achieves. The warm-start is essential; RL from a cold initialization over a meaningless vocabulary would have no signal to climb.

The Emergent Power Law

The finding that most caught my attention is empirical rather than methodological. As training progresses, the distribution over the abstract vocabulary develops a power law, the same heavy-tailed frequency distribution we observe in natural language (Zipf's law). A small number of abstract tokens are used frequently, while a long tail is used rarely, and this structure evolves across the training phases.

This is more than an aesthetic curiosity. A power law over a learned, initially meaningless vocabulary is a signature of compositional structure and concept reuse. The model is not assigning tokens at random; it is allocating a few high-frequency tokens to common reasoning operations and reserving rarer tokens for specialized roles. In other words, the model is inventing a language, with the statistical fingerprints of one, without any pressure to be human-readable.

This connects to the DeepSeek-R1-Zero observation that strong reasoning performance can be separable from human readability. Abstract-CoT pushes that observation to its logical conclusion: if readability is optional, why pay for the verbosity of natural language at all?

My Take: Discreteness Was Never the Problem

The prevailing assumption in latent reasoning has been that efficiency requires moving off the discrete token grid and into continuous concept space. The implicit reasoning is that discrete tokens are a lossy, low-bandwidth channel, so to compress reasoning you need the richer geometry of continuous vectors.

I think this paper quietly undermines that assumption. The bottleneck was never discreteness; it was verbosity. Natural language CoT is inefficient not because tokens are discrete, but because natural language is a redundant, human-oriented encoding optimized for communication, not for computation. A reserved discrete vocabulary, freed from the constraints of being readable, can pack reasoning into far fewer symbols while retaining the practical advantages of the discrete autoregressive machinery: standard decoding, standard RL pipelines, KV caching, and constrained sampling over a fixed codebook.

There is a deeper inference worth drawing. The information bottleneck training, where the response cannot see the verbal CoT, is essentially a rate-distortion problem. The model is being asked to find the minimum-description-length encoding of its own reasoning that still supports a correct answer. The emergent power law is exactly what you would expect from an efficient code under reuse pressure. The model is doing source coding on its own thoughts.

Limitations and Open Questions

Several caveats deserve honesty. First, the abstract tokens are non-interpretable by construction, which trades efficiency for opacity. If verbal CoT can already be unfaithful, as Lanham et al. (2023) and Turpin et al. (2023) argue, then abstract CoT removes even the illusion of a readable trace to audit. For safety-critical deployment, this is a real cost.

Second, the benchmarks, while diverse, are still bounded. MATH-500, AlpacaEval, and HotpotQA cover meaningful territory, but the hardest frontier reasoning tasks involve very long deliberation chains. Whether a bounded-length abstract sequence can compress thousands of tokens of reasoning without saturating is an open empirical question.

Third, the codebook size and the maximum abstract sequence length are hyperparameters that likely interact. A vocabulary too small constrains expressivity; too large reintroduces the cold-start difficulty across more embeddings.

Finally, the multi-stage recipe is intricate. Policy iteration warm-up plus warm-started GRPO is not a trivial pipeline, and reproducing the delicate balance between bottlenecked SFT and self-distillation may prove sensitive in practice.

Closing Thoughts

What I find most compelling about Abstract-CoT is that it reframes latent reasoning as a learned compression problem rather than a geometric one. The model is not handed a new representational space; it is given a blank vocabulary and the incentive to build its own efficient code, and it does, complete with the statistical regularities of natural language.

The forward-looking question is whether these abstract languages transfer. If a model can learn a compressed reasoning code, can that code be shared across tasks, distilled into smaller models, or even partially decoded back into human-readable explanations on demand? An abstract scratchpad that is efficient at inference yet expandable to a verbal trace when auditing is required would resolve the interpretability tension. That hybrid seems like the natural next step.