KV-Fold and the Surprising Stability of KV-Cache Recurrence

When we think about extending the context length of a transformer, the instinct is usually architectural. We reach for new positional encodings, sparse attention patterns, learned memory tokens, or a full replacement of attention with state-space dynamics. The paper KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference by Nadali, Cooper, Trivedi, and Velasquez takes a different stance. It asks whether a frozen, pretrained transformer already supports a stable form of recurrence if we simply reinterpret how its key-value cache is used. The answer, somewhat unexpectedly, is yes.

This is a result worth sitting with, because it changes the question from "what new mechanism do we need" to "what latent capability already exists in models we have trained but never used this way."

The Core Idea: A Left Fold Over Chunks

The KV cache in a decoder-only transformer is usually treated as a serving optimization. Once a token is processed, its layer-wise key and value projections are stored so that later tokens can attend to them without recomputation. The authors point out that this cache is not just a speed trick. It is a structured record of the model's prior computation, and in principle it can act as a recurrent state.

KV-Fold operationalizes this. A long input sequence is divided into chunks. When the model processes chunk $t$, it attends to the accumulated KV cache from all earlier chunks as a prefix, produces new keys and values for the current chunk, appends them, and carries the enlarged cache to chunk $t+1$. The same one-step update applies repeatedly. The functional programming analogy is exact: this is foldl, where the KV cache is the accumulator and each chunk is folded into it.

What makes this notable is what does not happen:

• No retraining or fine-tuning. The model weights are frozen.
• No special memory tokens or architectural modifications.
• No compression, eviction, or pruning of the past.

The shift is purely operational. Instead of one monolithic full-context forward pass, you run a chain of tractable forward passes that thread state through the cache. The lineage here matters: the authors build on a KV-cache concatenation primitive introduced for latent multi-agent communication, where one transformer pass attends to another pass's cache as a prefix. KV-Fold repurposes that primitive as a chunk-to-chunk recurrence within a single model.

Why Bounded Drift Is the Real Finding

The naive expectation is that this should not work well. If each chunk boundary introduces some independent error relative to a true full-attention computation, then over many chunks that error should compound. A chain of depth 511 should look badly degraded compared to one full pass. This is the standard intuition behind why people distrust approximate or streaming methods over long horizons.

The empirical result contradicts that intuition. The authors measure per-step drift, the deviation from full attention introduced at each transition, and observe a short transient followed by a flat plateau. Drift rises in the first few chunk transitions and then stabilizes, remaining essentially unchanged across deep chains. This is the central technical observation of the paper, and it is more interesting than the benchmark numbers.

The robustness checks make the case stronger:

• The plateau is insensitive to a 10,000x change in numerical precision. If the drift were accumulated floating-point error, precision should matter enormously. It does not.
• The plateau is robust across chunk sizes.
• The plateau is consistent across transformer families.

The combination of these three facts is what rules out the boring explanation. This is not numerical error piling up. Instead, KV-Fold moves the model into a slightly shifted but stable attention regime. The system finds a fixed point, or something close to one, rather than diverging.

On the task side, the consequences are concrete. Because the KV state is never compressed, earlier tokens remain available for content-based addressing. On a needle-in-a-haystack benchmark spanning contexts of 16K, 32K, 64K, 96K, and 128K tokens, KV-Fold recovers 100% of inserted facts across 152 trials, including chain depth 511 on Llama-3.1-8B at the full 128K native window, all within the memory budget of a single 40GB GPU. For comparison, a single full-attention forward at 128K is infeasible on that hardware; the attention-scores matrix alone is roughly 1TB.

Where This Sits in the Landscape

It helps to locate KV-Fold among the families of long-context methods, because its trade-off profile is distinctive.

Streaming methods such as attention sinks plus a sliding window keep inference memory bounded by discarding old tokens. They scale beautifully in memory but cannot recover information that has been evicted. They trade fidelity for a constant footprint.

KV-cache compression methods prune, evict, or quantize parts of the cache. These are lossy by construction. They preserve more than streaming but still gamble on which past information will matter later.

Recurrent memory and learned compression approaches compress the past into a fixed-size state, often requiring training, and inherit the lossiness of any bottleneck.

KV-Fold occupies a different corner. It is lossless with respect to the cache, because nothing is discarded, which is precisely why exact recall is preserved. The cost is that the cache grows linearly with sequence length, raising memory and per-step latency relative to streaming. So this is not a method that gives you unbounded context for free. It gives you exact recall over long but bounded sequences, processed as a chain of manageable passes rather than one infeasible pass.

The honest framing in the paper is that this is a point on a trade-off surface, not a domination of all prior work. For workloads where exact recall is non-negotiable, recovering an identifier from a long log or a factual detail from early in a document, the lossless property is the entire point.

My Take: State Richness as the Enabler of Stable Recurrence

The deeper principle here, which the authors gesture at and which I think deserves emphasis, is about what makes a fixed function become a stable recurrence. A transformer was not trained to be applied iteratively to its own cache. Yet it behaves stably when used that way. The reason, I would argue, is that the KV cache is a rich enough state representation to act as its own carrier. It is not a compressed summary that must be re-expanded; it is the literal substrate the model would have produced anyway.

This reframes the lesson of recurrent architectures. Classical RNNs were brittle over long horizons in part because their hidden state was a fixed-dimensional bottleneck, forcing lossy compression at every step and inviting divergence or vanishing dynamics. KV-Fold sidesteps that by letting the state grow. Bounded drift, not zero drift, is what makes long chains tractable. A system can absorb a small, stable per-step perturbation indefinitely; it cannot absorb an unbounded one.

This suggests a general design heuristic: if you want a frozen model to function as a stable recurrence, keep the state representation faithful to the model's own internal format and let it carry enough information to avoid lossy re-expression at each step. Compression is exactly what destabilizes the loop.

Limitations and Open Questions

The linear memory growth is the obvious ceiling. KV-Fold makes infeasible passes feasible, but it does not make the cache constant-size, so genuinely unbounded streams remain out of reach without some eviction or compression that would reintroduce loss.

Several questions remain open. Why does the drift plateau exist at all, and can we characterize the shifted attention regime analytically rather than empirically? Does the 100% retrieval result hold under harder tasks than single-needle retrieval, such as multi-hop reasoning or aggregation across many distant facts where attention must integrate rather than locate? And does the stable regime degrade for reasoning-heavy generation, where small representational shifts could compound into different outputs even if retrieval is preserved?

The most provocative implication is that we may be underusing the models we already have. If frozen transformers already support stable KV-cache recurrence, the boundary between "single forward pass" and "recurrent system" is softer than the architecture suggests. That is a useful thing to keep in mind the next time the instinct is to reach for a new mechanism.