The Tacit Knowledge Gap: Why LLM Judges Fail to Capture Developer Preferences in Code Evaluation

As large language models become ubiquitous in software engineering workflows, the question of how to evaluate their outputs has shifted from academic curiosity to practical necessity. The dominant approach, LLM-as-a-judge, promises scalable evaluation without the prohibitive cost of human annotation. Yet this scalability may come at the expense of validity when judges fail to capture the nuanced, context-dependent criteria that developers actually use. In their paper Comparing Developer and LLM Biases in Code Evaluation, Mittal et al. present compelling evidence that current LLM judges systematically misalign with human preferences across realistic coding scenarios, revealing not merely a performance gap but a fundamental disconnect in how machines and humans conceptualize code quality.

The TRACE Framework and Empirical Reality

To investigate this misalignment, the authors introduce TRACE (Tool for Rubric Analysis in Code Evaluation), a framework designed to evaluate LLM judges in interactive, ambiguous settings rather than static benchmarks. Unlike prior work that relies on polished GitHub repositories or competitive programming tasks, TRACE examines three distinct interaction modalities where developers actually work: chat-based programming assistance, IDE autocompletion, and instructed code editing. These contexts capture the partial information and underspecified intent characteristic of real software development, where requirements emerge through interaction rather than being fully defined a priori.

The methodology is rigorous. Across 13 distinct models, including general-purpose LLMs, specialized judge models, and reward models, the researchers measure alignment with human preferences by extracting interpretable rubric items that explain points of disagreement. The results are sobering. Even the strongest LLM judges underperform human annotators by 12 to 23 percentage points, with no single model achieving consistent dominance across modalities. More surprisingly, fine-tuned judge models fail to reliably outperform general-purpose alternatives, suggesting that the problem transcends simple data scarcity or architectural specialization.

The analysis identifies 35 significant sources of misalignment between human and machine evaluation. These map onto 16 recurring evaluation themes, 11 of which correspond closely to established software engineering criteria such as syntactic correctness, formatting, and robustness. Crucially, judges remain significantly misaligned with human preferences on 6 of these 11 canonical dimensions. For instance, in chat-based programming, models demonstrate a pronounced bias toward verbose code explanations while developers prefer concise, context-aware solutions. In IDE autocompletion, judges overweight functional correctness while underweighting readability within live codebases. During code editing tasks, judges prioritize defensive error handling, whereas developers often prefer precise, commented modifications that prioritize clarity over exhaustive robustness.

Tacit Knowledge and the Expert Blind Spot

These findings illuminate a deeper issue than mere metric optimization. The consistent divergence between LLM and human judgments, particularly the preference for defensive programming patterns over human-centered readability, points to what cognitive scientists call the expert blind spot. Developers possess tacit knowledge, criteria so deeply internalized through practice that they become nearly impossible to articulate explicitly, let alone encode in training data.

Consider the specific example from the paper involving font loading logic. The LLM judge preferred a solution with nested try-except blocks covering multiple file paths, representing exhaustive error handling. The human developer preferred a shorter version with clearer comments, accepting a simpler fallback mechanism. The judge evaluated this as lower quality on robustness criteria, while the human valued cognitive load and maintainability. This is not a technical error on the part of the LLM; it is a value judgment rooted in different ontologies of what constitutes "good" code. The model optimizes for edge case coverage and syntactic safety, while the human optimizes for collaborative maintainability and contextual appropriateness.

The fact that fine-tuned judges do not resolve these biases suggests we face a tacit knowledge problem rather than a data volume problem. Current training paradigms can capture explicit code quality heuristics, but they struggle to model the situated, pragmatic reasoning that developers employ when evaluating code within specific workflow constraints. The 12 to 23 percent performance gap represents not missing labels in a dataset, but missing comprehension of the social and practical dimensions of software engineering.

Implications for AI-Assisted Development

This research forces us to confront a uncomfortable possibility: the aspiration of fully automated, scalable evaluation of code quality may be fundamentally limited when applied to real-world development contexts. If LLM judges cannot reliably discern why developers prefer specific solutions over functionally equivalent alternatives, their deployment as evaluation metrics risks encoding a particular, machine-centric bias into the development pipeline.

I would argue that we are witnessing a clash between two distinct epistemologies of code. The LLM judges operate from a specification-based epistemology, evaluating code against formal properties like correctness, completeness, and robustness. Developers, however, operate from a practice-based epistemology, evaluating code against criteria like appropriateness, clarity, and alignment with evolving intent. These are not merely different weights on the same rubric; they are qualitatively different dimensions of evaluation.

The danger lies in automation bias. If engineering teams adopt LLM judges as cost-effective proxies for human evaluation, they may inadvertently optimize for the wrong qualities. Codebases could become increasingly defensive and verbose, optimized for machine approval rather than human comprehension. The finding that judges systematically undervalue comments and clarity in favor of error handling suggests a future where AI-assisted development produces brittle, over engineered solutions that satisfy formal metrics while frustrating the developers who must maintain them.

Moving Forward: Hybrid Evaluation and Contextual Rubrics

The path forward requires abandoning the fantasy of replacing human judgment with automated metrics and instead designing hybrid evaluation protocols that acknowledge the complementary strengths of human and machine evaluation. LLM judges excel at detecting syntactic errors, type inconsistencies, and obvious logical flaws; humans excel at evaluating contextual fit, maintainability, and alignment with unstated requirements. Rather than forcing judges to approximate human preference globally, we should deploy them for verifiable correctness checks while reserving human judgment for the qualitative, tacit dimensions where machines consistently fail.

Future research must also investigate whether dynamic, contextual rubrics can bridge this gap. Rather than static criteria applied uniformly, evaluation frameworks might adapt rubric weights based on project phase, team conventions, or specific task constraints. If the misalignment stems from judges' inability to modulate their criteria based on situational context, then personalized or fine-grained contextual prompting may offer partial remedies, though the persistence of the expert blind spot suggests inherent limits to how completely tacit knowledge can be externalized.

The work of Mittal et al. serves as a necessary corrective to the uncritical adoption of LLM-as-a-judge paradigms in software engineering. Their identification of 35 specific bias sources provides a roadmap for improvement, but the underlying challenge remains. Until we can encode not just what developers do, but why they make the contextual choices they make, the 12 to 23 percent gap will persist not as a bug to be patched, but as a feature distinguishing human expertise from algorithmic approximation. The question is no longer whether we can make LLM judges agree with developers, but whether we should trust them when they disagree.