When Should You Trust an LLM Judge? Per-Instance Reliability Through the Lens of Conformal Prediction and Transitivity

Large language models deployed as automated judges have become the invisible infrastructure of modern NLP evaluation. From ranking summarization systems to scoring chatbot responses, LLM-as-judge frameworks promise scalability at orders of magnitude lower cost than human annotation. Yet this efficiency comes with a critical epistemic blind spot: aggregate metrics such as system-level Kendall’s tau or Pearson correlation tell us that judges are accurate on average, while remaining silent about whether we can trust any specific judgment. In their recent preprint, Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations, Manan Gupta and Dhruv Kumar address this gap with a rigorous two-pronged diagnostic framework that reveals how reliability varies not just across models, but across individual documents and evaluation criteria.

The Aggregation Problem: Why Average Accuracy Misleads

The central tension in LLM evaluation lies in the mismatch between aggregate statistics and per-instance decisions. A judge that correlates with human preferences at tau = 0.8 across five hundred summaries might nevertheless fail catastrophically on the fifty instances that determine whether a production summarization system gets deployed. Gupta and Kumar demonstrate that this is not merely theoretical: across four judges (GPT-4o-mini, LLaMA-3.1-70B, Qwen-2.5-72B, and Mistral-Small) evaluating SummEval summaries, aggregate transitivity violation rates appear reassuringly low at 0.8% to 4.1%. Yet when disaggregated by input document, 33% to 67% of documents exhibit at least one directed 3-cycle (A preferred to B, B to C, yet C to A), with per-document violation rates spiking to 30.4% for Mistral-Small-3.1.

This discrepancy between aggregate and per-instance behavior illustrates a fundamental statistical phenomenon: averaging can render sporadic but severe inconsistencies invisible. The authors formalize this through tournament theory, modeling pairwise LLM comparisons as directed graphs where edges represent preferences. A transitivity violation constitutes a directed 3-cycle within this tournament. Crucially, they test whether these violations represent systematic directional bias that could be repaired through Minimum Feedback Arc Set (MFAS) ranking repair, a technique from preference aggregation that removes the minimum set of edges to eliminate cycles. The finding is decisive: MFAS repair does not improve agreement with human rankings, establishing that transitivity violations constitute sparse noise rather than coherent, correctable bias. This distinction matters for practitioners because it implies that confidence calibration through ranking aggregation is unlikely to resolve the underlying unreliability.

Conformal Prediction: From Point Estimates to Reliability Sets

While transitivity analysis diagnoses inconsistency in pairwise comparisons, the second diagnostic addresses direct Likert scoring through split conformal prediction. Rather than treating a score of 3.5 as a precise point estimate of quality, conformal methods construct prediction sets (intervals of scores) that contain the true human judgment with finite-sample, distribution-free coverage guarantees of at least (1-alpha). The width of these sets serves as a per-instance uncertainty quantifier: narrow sets indicate high confidence, while wide sets signal ambiguity.

The empirical results reveal a striking pattern. Pooling 1,918 observations across all judges and criteria yields a Spearman correlation of rs = +0.576 (p < 10^-100) between prediction set width and actual judge-human disagreement. More remarkably, different judges assign wide sets to the same documents, with cross-judge correlations of r = 0.32 to 0.38 for criteria such as fluency, consistency, and relevance. This agreement demonstrates that set width captures document-level difficulty rather than judge-specific noise or model epistemic uncertainty. The signal generalizes across model architectures and training regimes, suggesting it reflects intrinsic properties of the evaluation instances, such as ambiguity in the source text or near-equivalence of candidate summaries.

The granularity of this approach enables actionable comparisons across evaluation dimensions. On a 1-5 Likert scale, relevance achieves an average prediction set size of approximately 3.0, indicating relatively reliable judgment. Coherence follows at approximately 3.9, while fluency and consistency average approximately 4.9, approaching the maximum possible uncertainty. This hierarchy suggests that certain evaluation criteria are inherently more amenable to automated assessment than others, regardless of which frontier model performs the evaluation.

Original Insights: From Diagnostics to Deployment Gates

The convergence of these two independent diagnostics offers more than methodological validation; it provides a blueprint for responsible deployment of LLM evaluation systems. The finding that criterion matters more than judge implies a resource allocation strategy: practitioners should reserve human oversight for criteria exhibiting high prediction set widths (fluency and consistency) while potentially automating decisions on criteria with narrow sets (relevance). This criterion-aware routing represents a departure from current practice, which typically treats all dimensions of the SummEval rubric as equally automatable.

The cross-judge agreement on conformal width suggests a particularly promising application: uncertainty-based routing gates. If wide prediction sets consistently indicate document difficulty across heterogeneous judges, then a deployment system could use conformal width as a trigger for human escalation without requiring model-specific calibration. Documents with narrow sets proceed through automated evaluation; documents with wide sets route to human annotators. The correlation of r = 0.32 to 0.38 between judges, while moderate, indicates that this gate would generalize across judge swaps without recalibration, turning a diagnostic tool into a quality-control layer that respects the theoretical coverage guarantees of conformal prediction.

However, several limitations warrant consideration. The study focuses on summarization evaluation via SummEval, and the document difficulty effects may not transfer to domains with different ambiguity structures, such as mathematical reasoning or code generation. Additionally, while MFAS repair proves ineffective for transitivity violations, this does not preclude other aggregation methods from mitigating pairwise inconsistencies. The conformal approach requires a calibration set of human judgments, which may be expensive to obtain for niche domains, though likely less costly than full human evaluation of all instances.

Conclusion

Gupta and Kumar’s work shifts the discourse on LLM judges from aggregate accuracy to per-instance reliability, a transition necessary for high-stakes deployment of automated evaluation. By combining transitivity analysis with conformal prediction sets, they provide both a warning and a path forward: warning that low aggregate violation rates mask widespread per-document inconsistency, and path forward through theoretically grounded uncertainty quantification that generalizes across judges. The hierarchy of criteria reliability (relevance > coherence > fluency/consistency) offers immediate practical guidance, while the conformal width signal opens possibilities for dynamic human-AI evaluation pipelines.

Open questions remain regarding the source of document-level difficulty. Do wide prediction sets correlate with specific linguistic features, such as lexical ambiguity or syntactic complexity? Can fine-tuning or prompt engineering reduce set widths for unreliable criteria, or are these limitations fundamental to the construct definitions themselves? As the field moves toward automated evaluation at scale, answers to these questions will determine whether we can build evaluation systems that know when they do not know, a capability essential for trustworthy deployment.