Neural Garbage Collection: When AI Learns to Manage Its Own Memory

The current generation of reasoning models has demonstrated that extended chains of thought enable remarkable problem solving capabilities, yet this capability comes with a steep computational price. As models think longer, their key-value caches grow linearly with sequence length, quickly exhausting GPU memory and limiting inference throughput. Traditional approaches to this problem rely on human engineering, whether through heuristic eviction policies like StreamingLLM or compression methods trained on reconstruction losses. A new paper from Stanford, "Neural Garbage Collection: Learning to Forget while Learning to Reason," proposes something far more radical. Michael Y. Li and colleagues ask a simple but profound question: if a model can learn to reason, why cannot it learn to forget?

The End-to-End Philosophy of Learned Memory Management

The central thesis of Neural Garbage Collection (NGC) rests on what Rich Sutton termed the "bitter lesson" of artificial intelligence: that general methods leveraging computation ultimately win against handcrafted knowledge. Current KV cache management strategies represent the latter approach. Methods like H2O, SnapKV, and KeyDiff rely on attention weight analysis or reconstruction objectives designed by researchers who assume they know which information the model will need later. NGC abandons this assumption entirely.

Instead, the authors treat cache eviction as a discrete action space analogous to token generation. During chain-of-thought reasoning, the model periodically enters an "eviction round." It computes a softmax distribution over its current KV cache entries, samples which entries to evict using Gumbel top-k sampling, and continues reasoning conditioned only on the remaining cache. Because both token generation and cache eviction are discrete actions sampled from the language model, the authors can apply policy gradient methods to optimize both simultaneously from a single learning signal: the binary outcome of task correctness.

This methodological choice is technically subtle but conceptually significant. Prior work on differentiable cache management required continuous relaxations or separate supervised training stages with auxiliary objectives. By contrast, NGC uses reinforcement learning to backpropagate through both sampled tokens and sampled evictions without proxy objectives. The model learns what to forget based entirely on whether its final answer is correct. If evicting a particular cache entry leads to reasoning errors, the policy gradient penalizes that decision. If pruning transient scratchwork preserves accuracy while reducing memory, the model reinforces that behavior.

Empirical Results and Comparative Analysis

The experimental results demonstrate that learned eviction substantially outperforms engineered heuristics. On the AIME 2025 mathematics benchmark, NGC achieves 21.4% pass@32 accuracy at 4.6x compression, compared to 10.7% for SnapKV and 0% for both StreamingLLM and KNorm. On the Countdown task, NGC reaches 49.6% pass@1 at 2.4x compression, while the strongest baseline, KeyDiff, manages only 21.2%. These are not marginal improvements; they represent the difference between functional and non-functional reasoning under severe memory constraints.

The authors introduce an additional technique they call "budget-aware interoception," where the target eviction rate is included in the prompt during training. This gives the model explicit awareness of its resource constraints before it begins reasoning. The results show that models trained with this interoceptive signal generalize better to different cache budgets at test time, suggesting that the model learns to modulate its reasoning strategy based on available working memory. This mirrors human cognitive flexibility, where we adapt our problem-solving approaches based on how much information we can hold in mind simultaneously.

Critically, the compression ratios reported, 2x to 4.6x, represent peak cache size reduction. Since the model performs evictions periodically throughout the reasoning process, the actual memory savings enable reasoning over much longer horizons than would be possible with static caches. The model effectively learns to distinguish between transient intermediate calculations, which can be safely discarded, and essential premises or partial results that must be preserved for downstream reasoning steps.

Original Insights: The Unification of Capability and Efficiency

What makes this work philosophically significant extends beyond the technical implementation. NGC represents a unification of capability optimization and efficiency optimization into a single objective. Traditionally, we train models to be capable, then compress them to be efficient. We accept a tradeoff where quantization, pruning, or distillation degrade performance in exchange for deployment viability. The Stanford team's approach suggests this separation is artificial and suboptimal.

The model learns to compress its own reasoning trace because compression serves the goal of correct reasoning. This creates a feedback loop where efficiency emerges as a byproduct of capability. The model becomes more efficient not because we force it to be small, but because selective forgetting enables longer reasoning chains, which improve accuracy on complex problems. This aligns with David Silver and Rich Sutton's recent arguments about the "era of experience," where agents improve through self-generated interaction rather than human-curated data.

However, several limitations and open questions remain. The current evaluation focuses on mathematical reasoning tasks with verifiable answers. It remains unclear whether NGC would generalize to open-ended creative writing, multi-turn dialogue, or tasks requiring factual recall over long contexts. The eviction mechanism itself adds computational overhead; while memory bandwidth decreases, the additional forward passes required for eviction rounds consume FLOPs. The paper does not quantify this overhead, leaving open the question of whether NGC improves wall-clock time or merely memory efficiency.

Furthermore, the reliance on policy gradients for discrete eviction decisions introduces training instability risks. The variance in gradient estimates from sparse, binary rewards could make scaling to more complex tasks challenging. The authors note they use outcome-based reward only, without process supervision, which means the model receives no intermediate feedback about which specific eviction caused a reasoning failure. In long reasoning chains, credit assignment over dozens of eviction rounds may become intractable.

Conclusion: Toward Self-Managing Intelligence

Neural Garbage Collection points toward a future where models manage their own computational resources with the same sophistication they apply to problem solving. This represents a shift from viewing efficiency as a systems engineering constraint to treating it as an emergent property of intelligence itself. The implications extend beyond KV caches to broader questions about attention mechanisms, mixture-of-experts routing, and even robotic control, where agents might learn to allocate physical or computational resources dynamically.

The work also raises provocative questions about model interpretability. When a model learns to evict its own cache entries based on internal states we do not fully understand, we gain less visibility into its reasoning process than we would with heuristic methods that rely on attention weight analysis. We trade explainability for performance, a bargain consistent with the bitter lesson but one that demands careful consideration as these systems grow more autonomous.

As reasoning models scale to think for hours or days rather than seconds, self-managed memory will become essential rather than optional. The Stanford team's demonstration that models can learn to forget, not through human specification but through the pressure of task performance, suggests we are entering a phase where AI systems optimize aspects of their own implementation that we previously considered fixed architectural constraints. Whether this leads to more robust, human-like reasoning or to opaque systems that discard information in ways we cannot predict remains an open empirical question, but the approach itself marks a decisive break from the hand-engineered efficiency techniques of the past.