The Specificity Paradox in LLM Code Generation

When we prompt a large language model to write code, we face a deceptively simple question: how much should we tell it? Intuitively, more detail should help. Give the model the function signature, the expected behavior, a few examples, the edge cases, and surely the output improves. Yet anyone who has spent serious time pair-programming with an LLM has noticed something stranger. There is a point past which additional instruction stops helping and starts hurting. The paper "Testing LLMs on Code Generation with Varying Levels of Prompt Specificity" provides a structured look at exactly this phenomenon, and the results are worth examining carefully because they generalize well beyond code.

What the Study Measured

The authors evaluated four models that represented the frontier at the time of the study: Bard, ChatGPT-3.5, ChatGPT-4, and Claude-2. Their benchmark consisted of 104 Python coding problems, and crucially, each problem was presented under four distinct prompt formulations that varied in their degree of specificity and the presence of explicit test cases.

This four-way design is the methodological heart of the work. Rather than asking the binary question "can the model solve this problem," the study asks the conditional question "how does the framing of the problem change the solution." The team did not stop at correctness. They measured three dimensions of output quality:

• Accuracy: whether the generated function produced correct results across the test suite
• Time efficiency: the runtime characteristics of the generated code
• Space efficiency: the memory footprint of the solution

Measuring time and space efficiency alongside raw correctness is the detail that elevates this study above a simple pass-or-fail benchmark. A function that returns the right answer in exponential time is correct in only the most academic sense. Production code lives and dies on its complexity profile, and by tracking these dimensions the authors capture something closer to what practitioners actually care about.

The Core Finding: A Non-Monotonic Relationship

The headline result is that performance varies significantly both across models and across prompt types, and that the relationship between specificity and quality is not monotonic. Adding detail helps, until it does not.

This deserves unpacking, because the mechanism is more interesting than the observation. Consider what an LLM is actually doing during code generation. It is performing a conditional search over plausible continuations, weighting them by patterns learned during training. When you provide a sparse prompt that states only the goal, the model retains a wide search space and can draw on the full distribution of solution patterns it has internalized. Many of those patterns encode efficient, idiomatic approaches that a competent engineer would recognize.

When you over-specify, you progressively collapse that distribution. Each additional constraint, each explicit test case framed as an implementation hint rather than a behavioral contract, prunes branches of the search tree. If your specification happens to nudge the model toward your own mental model of the solution, and your mental model is suboptimal, you have effectively traded the model's broad competence for your narrower one. The model becomes compliant rather than creative.

This is the distinction I find most useful from the study, even though the paper frames it more conservatively. There is a difference between specifying the contract and specifying the solution. A contract describes what correct behavior looks like: the input domain, the output guarantees, the invariants that must hold. A solution describes how to get there. Test cases that assert behavior are contract specification. Pseudocode embedded in the prompt is solution specification. The first expands what counts as success without constraining the path. The second narrows the path directly.

Test Cases as a Double-Edged Tool

The inclusion of explicit test cases in some prompt variants is where the study connects to test-driven development, and this is worth dwelling on. Test cases serve two functions simultaneously, and the study implicitly surfaces the tension between them.

On one hand, test cases anchor the model. They tell it precisely what success means, reducing ambiguity about edge conditions and expected output formats. The paper's results suggest this anchoring genuinely lifts accuracy, which matches the broader literature on few-shot prompting and the well-documented gains from in-context examples.

On the other hand, test cases can leak implementation bias. If the examples you choose implicitly assume a particular data structure or algorithm, the model reads that signal and conforms. The accuracy gain is real, but it can come bundled with an efficiency loss that a coarser test suite would have avoided. This is precisely why tracking time and space efficiency mattered. A study that measured only correctness would have concluded that more test cases are strictly better. The multidimensional measurement reveals the hidden cost.

Connecting to the Broader Landscape

This work sits alongside benchmark efforts like HumanEval and MBPP, which established the standard pass@k methodology for evaluating code generation. Where those benchmarks fix the prompt format and vary the model, this study fixes the problem set and varies the prompt, which is a complementary and underexplored axis. The field has poured enormous effort into making models better while treating the prompt as a relatively static input. Studies like this one remind us that the prompt is a tunable variable with its own optimization surface, and that the optimum is problem dependent rather than universal.

The model-to-model variation the authors report is also significant. ChatGPT-4 and Claude-2 occupying different points on the accuracy and efficiency tradeoff implies that prompting strategy is not transferable across models in any simple way. A prompt tuned for one model's inductive biases may underperform on another. This has practical consequences for anyone building tooling on top of multiple LLM backends.

My Take

The deeper principle here, which I argued in brief on social media, is that over-specification trades exploration for compliance, and this generalizes far past code. Whenever you constrain an LLM tightly, you are substituting your priors for its learned distribution. Sometimes that is exactly right, when you know something the model does not. But often you are simply imposing a suboptimal solution under the comfortable illusion of control.

The practical heuristic I draw from this study is to specify the invariants and leave the search open. State what must be true of any acceptable answer. Provide behavioral test cases that assert outcomes rather than methods. Then resist the urge to dictate the implementation. Let the model explore the space of solutions it knows better than you do for most routine problems.

Limitations and Open Questions

The study has honest limitations worth noting. A benchmark of 104 problems is sufficient to detect trends but small enough that fine-grained per-category conclusions should be treated cautiously. The models tested are now superseded by later generations, and it is an open empirical question whether the specificity paradox persists, weakens, or sharpens as models grow more capable. A more capable model might be more robust to over-specification, or it might be more sensitive to it precisely because it had a better solution to abandon.

Several questions remain open. Where exactly does the inflection point sit, and does it move with problem difficulty? Can we automate the detection of solution-leaking prompts versus contract-defining ones? And as agentic coding systems take over more of the development loop, who writes the prompts, and do these heuristics survive when the prompt author is itself a model? The answer to the simple question of how much to tell the model turns out to be: enough to define success, and no more.