When Better Retrieval Fails: The Component-Performance Gap in Policy-Focused RAG Systems

Introduction

The deployment of large language models in high-stakes domains has intensified interest in retrieval-augmented generation (RAG) as a mechanism for grounding outputs in authoritative sources. For AI governance, a field characterized by dense regulatory text, cross jurisdictional complexity, and rapid evolution, RAG appears particularly promising. The AI Governance and Regulatory Archive (AGORA) provides a curated corpus of 947 policy documents spanning global jurisdictions, offering a testbed for automated policy analysis. Intuitively, improving the retrieval component of a RAG system, ensuring it fetches the most relevant policy documents, should yield better answers. However, recent work challenges this assumption in ways that carry significant implications for system designers.

In "Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA," Mathur et al. demonstrate that domain-specific retriever optimization does not reliably translate into improved end to end question answering performance. Their analysis of a ColBERT-based system fine-tuned for the AGORA corpus reveals a troubling pattern: stronger retrieval capabilities can, under certain conditions, produce more confident hallucinations rather than accurate responses. This finding disrupts the prevailing narrative that component-level improvements necessarily compound into system-level reliability, particularly in specialized domains like AI policy.

Technical Architecture and Domain Challenges

The researchers construct a sophisticated RAG pipeline specifically adapted to the complexities of regulatory text. At the retrieval layer, they employ ColBERT, a late interaction architecture that enables fine grained token level matching between queries and documents. Unlike bi-encoders that compress texts into single vectors, ColBERT preserves token representations, allowing it to capture nuanced legal terminology and specific statutory references. The team fine-tunes this retriever using contrastive learning with synthetically generated queries derived from the AGORA documents, supplemented by manually labeled examples. This approach targets the domain shift between general web text and specialized policy language.

For generation, the system utilizes a language model aligned to human preferences through Direct Preference Optimization (DPO). Unlike reinforcement learning from human feedback (RLHF) which requires explicit reward modeling, DPO directly optimizes the policy to satisfy pairwise preferences, making it computationally efficient for domain adaptation. The researchers collect preference data specifically from policy-focused question-answer tasks, ensuring the generator learns to produce responses valued by domain experts.

This dual optimization, retrieval and generation, addresses the unique challenges of AI governance corpora. Policy documents employ dense legal language with nested definitions, temporal qualifications, and cross references between jurisdictions. The terminology evolves rapidly as regulatory frameworks mature, creating a moving target for automated systems. The AGORA corpus, while comprehensive, remains incomplete; regulatory developments outpace curation, and certain jurisdictions or policy areas may lack coverage.

The Retrieval-Performance Paradox

The central empirical contribution of the paper lies in demonstrating the disconnect between retrieval metrics and end to end system performance. The researchers find that while contrastive fine-tuning significantly improves standard retrieval metrics, such as recall and mean reciprocal rank, these gains do not consistently produce better answers to policy questions. In some experimental conditions, improved retrieval actually degrades output quality.

The mechanism behind this counterintuitive result involves what the authors term confident hallucinations when relevant documents are absent from the corpus. When the retriever functions poorly, it returns obviously irrelevant documents, which signals to the generator that the knowledge base lacks sufficient information. This uncertainty often prompts more cautious responses or acknowledgments of ignorance. However, when the retriever is highly optimized, it returns documents that appear relevant, even when the corpus contains no truly pertinent information. The generator, conditioned to trust retrieved context and aligned to produce confident, helpful responses through DPO, synthesizes these tangentially related documents into plausible but factually incorrect answers.

This phenomenon reveals a fundamental tension in RAG system design. The generator learns to treat retrieved documents as ground truth, but retrieval quality metrics measure relevance, not sufficiency. A document can be the most relevant available while still containing insufficient information to answer a specific query. In policy domains, where questions often require synthesis across multiple sources or precise citation of specific provisions, this distinction matters enormously. The system produces fluent, authoritative sounding responses that misrepresent regulatory requirements or