When Search Agents Stop Searching: The Hidden Cost of Memory-Backed Verification

Introduction

The deployment of large language models as autonomous search agents has produced a comforting narrative: feed a model the ability to browse the web, and it transforms from a static text generator into an investigator that gathers evidence, cross-references sources, and discovers facts it could not produce on its own. Benchmark scores on suites like BrowseComp seem to confirm this story, with frontier models posting steadily improving numbers.

A recent paper, LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?, challenges that narrative directly. The authors argue that much of what we interpret as search capability is actually something far less impressive: agents retrieving the web to confirm answers they already had encoded in their parameters. They name this failure mode Intrinsic Knowledge Dependence (IKD), and they construct a benchmark designed specifically to expose it. The findings deserve careful attention, because they call into question how we measure progress in agentic retrieval.

The Diagnostic Case for Intrinsic Knowledge Dependence

The paper's strongest contribution is methodological. Rather than simply proposing a new benchmark, the authors build three diagnostic experiments that progressively strip away the role of retrieved evidence, isolating whether tool use is actually driving answers.

First, the closed-book test. If a search benchmark genuinely requires search, agents should fail badly when all tools are removed. They do not. On BrowseComp, closed-book pass@4 reaches up to 44.5% (MiniMax M2.5), and every evaluated model scores non-trivially. A benchmark that nearly half the time can be solved from memory alone is not, in any clean sense, measuring search.

Second, the evidence-blocking test. This one is more subtle and more revealing. Here the search interface stays fully available, but all answer-supporting documents are removed from what the agent can retrieve. If agents were using search for genuine discovery, removing the answer-bearing evidence should simply collapse them toward their closed-book performance. Instead, every model performs worse than its closed-book baseline. MiniMax M2.5 falls from 44.5% to 8.0%, and Kimi-K2.6 from 25.5% to 2.3%. In other words, the presence of a broken search loop actively degrades performance below simply answering from memory.

Third, the trajectory analysis. This explains the mechanism. More than half of agents' search queries are seeded by entities that first appear in the model's own reasoning rather than in any retrieved document. After failed searches, agents frequently only rephrase the previous query rather than building a new hypothesis. And even when useful evidence is retrieved, they often fail to incorporate it.

Taken together, these three diagnostics paint a coherent picture. The agent's internal knowledge is the anchor. Search confirms it. When the fact lies inside the model's parametric boundary, the loop converges quickly. When it lies outside, the loop has nothing to latch onto, drifts into repetitive rephrasing, and the agent ends up worse off than if it had never searched at all.

Why Static Benchmarks Decay

The deeper claim, and the one I find most important, is that this is not merely data contamination. Contamination is the usual scapegoat: the test set leaked into training, the model memorized the answer. The authors are careful to distinguish IKD from this. Even questions that never appeared in training can be answerable through broad parametric world knowledge. A model with rich coverage of obscure but historically stable facts can reconstruct answers without ever retrieving them.

This reframes the problem as structural rather than accidental. As models grow more knowledgeable, the effective difficulty of any fixed benchmark erodes silently. Questions that once required genuine search become solvable from memory, and the benchmark quietly shifts from measuring discovery to measuring recall. The score keeps going up, but what it measures has changed underneath us.

This is the insight captured in the heuristic worth restating: a benchmark measures capability only if its test set lies outside the model's parametric knowledge. Otherwise it measures memory. For most NLP tasks this distinction has always existed, but for search agents it is acute, because the entire value proposition of a search agent is its ability to find things it does not already know.

The LiveBrowseComp Construction

The proposed fix is temporal grounding. LiveBrowseComp contains 335 human-authored questions, each depending on facts published within the 90 days preceding benchmark construction, drawn from six continuously updated sources: GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS. The choice of sources is deliberate and worth noting. These are structured, frequently updated, publicly verifiable feeds covering geopolitical events, films, video games, security vulnerabilities, sports, and seismic activity. They generate a steady stream of new factual ground truth.

Two design decisions stand out. First, the questions are filtered to exclude globally salient events, retaining obscure but verifiable facts. This matters because a globally salient recent event could still be in a model's knowledge if the horizon is recent, and broadly reported events get summarized everywhere. Long-tail recency is what neutralizes intrinsic knowledge. Second, each question is independently validated by human verifiers using only web search, confirming both solvability and uniqueness, with archived snapshots preserved for reproducibility.

The results are stark. Every evaluated model falls below 2% closed-book accuracy, confirming that the temporal and long-tail constraints largely eliminate the memory backstop. Once that backstop is gone, search-augmented scores drop by roughly 25 to 40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance.

The most persuasive control in the paper is the human comparison. Human searchers require comparable effort on LiveBrowseComp and BrowseComp. This rules out the obvious objection that the new questions are simply harder. They are not harder for a human with a search engine; they are harder for agents specifically because the memory-backed shortcut no longer exists.

My Take

The control involving human searchers is what elevates this work from a critique into a genuine measurement instrument. It cleanly separates two things that static benchmarks conflate: the intrinsic difficulty of a question and the availability of a parametric shortcut. By holding human effort roughly constant across the two benchmarks, the authors demonstrate that the performance collapse is attributable to the removal of the shortcut, not to harder tasks.

There is an uncomfortable implication for the broader retrieval-augmented generation literature. Many RAG systems are evaluated on datasets where the model could plausibly answer from parametric knowledge alone. If we cannot distinguish retrieval-driven answers from memory-driven ones, we cannot tell whether our retrieval pipeline is contributing anything. The evidence-blocking diagnostic is a tool any practitioner could adopt: keep the retriever running, remove the answer-bearing passages, and measure whether the system degrades gracefully or collapses below its closed-book floor.

I would also flag a limitation the authors themselves imply. A 90-day temporal benchmark is inherently perishable. Within months, those facts drift toward salience and toward the next training cut, and the benchmark must be rebuilt. LiveBrowseComp is therefore less a fixed dataset than a protocol for continuous regeneration. That is the right design, but it imposes maintenance cost and raises questions about cross-version comparability. Comparing a model evaluated on the January snapshot against one evaluated on the April snapshot requires care, since the question distributions across the six sources may shift.

A further open question concerns whether IKD is purely a knowledge-boundary problem or also a behavioral one. The trajectory analysis suggests agents fail to build new hypotheses from evidence and merely rephrase failed queries. This points to a deficiency in search policy that is independent of knowledge coverage. Even a perfectly informed retriever would not help an agent that cannot reformulate its strategy after a dead end. Disentangling the knowledge boundary from the search-policy boundary seems like the natural next study.

Conclusion

LiveBrowseComp delivers a clear message: the impressive scores of search agents on static benchmarks substantially reflect memory-backed verification rather than evidence-driven discovery. The diagnostics are simple and reproducible, the failure mode is named and mechanistically explained, and the proposed benchmark isolates exactly what the old ones obscured. The forward path is temporal grounding combined with long-tail filtering, treating evaluation as a continuously regenerated protocol rather than a frozen artifact. The harder open question is whether agents can be trained to keep searching productively once the parametric anchor disappears, or whether the search loop, deprived of memory, simply has nowhere to go.