From Feelings to Metrics: Formalizing the Structure of Informal LLM Evaluation

The Benchmark Gap and the Rise of Vibe-Testing

The evaluation of large language models has long been dominated by standardized benchmarks. Metrics such as accuracy on coding challenges, performance on academic exams, or scores on reasoning datasets provide convenient summary statistics for comparing models. Yet a persistent frustration plagues both researchers and practitioners. These aggregated scores often fail to predict whether a model will actually be useful in real-world workflows. A system that excels on HumanEval may still produce code that experienced developers find verbose, brittle, or misaligned with production conventions. Similarly, high scores on reasoning benchmarks do not guarantee that explanations will be clear, trustworthy, or stylistically appropriate for a given user.

This divergence between benchmark performance and practical utility has given rise to an informal practice known as "vibe-testing." Rather than relying on published leaderboards, users evaluate models through personal experimentation. They craft prompts drawn from their own tasks, compare outputs side by side, and judge quality based on subjective criteria such as clarity, style, workflow fit, or trustworthiness. These evaluations appear in blog posts, forum threads, and social media discussions, forming a distributed, unstructured layer of assessment that operates parallel to formal benchmarks.

In their paper, "From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs," Itzhak et al. take a systematic look at this phenomenon. They argue that vibe-testing is not merely noise or anecdotal evidence. Instead, it follows identifiable patterns that can be formalized and analyzed. By bridging the gap between subjective user experience and systematic evaluation, they propose a framework that could reshape how we assess model utility.

Empirical Foundations: How Users Actually Evaluate

To understand vibe-testing as a practice, the researchers assembled two complementary empirical resources. First, they conducted a survey asking participants about their evaluation habits and the qualities they seek in model outputs. Second, they collected and annotated an "in-the-wild" corpus of model comparisons from tech blogs, forums, and video reviews.

The survey results reveal a striking mismatch between what benchmarks measure and what users care about. When asked what benchmarks fail to capture, 61.4% of respondents cited workflow fit, while 40.9% mentioned style, and 34.1% highlighted the handling of ambiguity. Trust and stability also ranked high, with 27.3% of participants noting these as missing from standard evaluations. These are not easily quantifiable properties. They are context-dependent, user-specific, and inherently subjective.

The corpus analysis reinforces these findings by showing how these concerns translate into practice. Users frequently test models on tasks drawn directly from their own work, with 73.5% of survey respondents reporting that they use workflow-based testing. Common strategies include side-by-side comparisons (65.3%), probing stylistic preferences (49.0%), stress-testing ambiguity handling (44.9%), and checking stability across repeated runs (38.8%). Crucially, 38.8% of users report having recurring "test prompts," suggesting that vibe-testing, while informal, is not arbitrary. Users develop consistent, personalized heuristics for evaluation.

Formalizing the Two-Part Process

Building on these observations, Itzhak et al. formalize vibe-testing as a structured two-part process involving distinct input and output dimensions. Input dimensions define what users test and how they construct their prompts. These include the type of task, the amount of context provided, and the specific constraints or stylistic preferences embedded in the query. For instance, a developer testing a coding assistant might provide not just a function signature, but detailed context about a legacy codebase, specific library versions, and stylistic constraints inherited from existing code.

Output dimensions capture how users judge the resulting responses. These criteria include clarity, adherence to constraints, production readiness, and alignment with the user’s cognitive style or workflow. Unlike standard benchmarks that apply uniform metrics across all instances, vibe-testing applies user-aware criteria that vary by individual and context.

This formalization is significant because it moves beyond simply acknowledging that "users are different." It provides a framework for systematically varying both the test conditions and the evaluation criteria. By parameterizing these dimensions, researchers can begin to study how specific perturbations in task framing or judgment criteria affect model rankings.

Experimental Validation: When Preferences Flip

To demonstrate the practical impact of this formalization, the authors implemented a proof-of-concept evaluation pipeline. Given a brief description of a user, the system first rewrites standardized benchmark prompts to reflect that user’s likely context and preferences. It then compares model outputs not with generic accuracy metrics, but with judgments tailored to user-relevant output dimensions.

The experiments focus on coding benchmarks, a domain where standardized evaluation is well-established but often criticized for failing to capture software engineering reality. The results are instructive. When prompts are rewritten to reflect personalized contexts and judgments are made using user-aware criteria, the preferred model often changes relative to standard benchmark rankings. In several head-to-head comparisons, the "winner" flips entirely compared to the original benchmark formulation. Interestingly, non-personalized rewrites, which modify the prompt but do not tailor it to specific user profiles, largely preserve the original ordering. This suggests that the personalization component, not just prompt variation, drives the divergence in preferences.

These findings provide empirical support for the intuition behind vibe-testing. The "vibe" of a model, its fit for a specific user in a specific context, can override aggregate performance metrics. A model that scores lower on HumanEval might still be preferred by a developer for its concise style, its ability to work with ambiguous requirements, or its compatibility with existing code patterns. The flip in preferences is not a measurement error. It is a signal that benchmarks are missing.

Implications and Limitations

The work of Itzhak et al. points toward a necessary evolution in how we think about model evaluation. For too long, the field has pursued the illusion of a single, objective ranking. The reality is that utility is inherently contextual. A model optimized for one workflow may be suboptimal for another. By formalizing vibe-testing, we can begin to build evaluation systems that respect this heterogeneity rather than averaging it away.

However, significant challenges remain. The current proof of concept relies on automated persona generation and judgment, which may not fully capture the nuance of human subjective experience. Scaling personalized evaluation to cover the full diversity of user contexts requires robust methods for eliciting user preferences without burdening individuals with extensive surveys. There is also a tension between personalization and reproducibility. If every evaluation is tailored to a specific user, how do we compare models across studies? The authors suggest that their dimensional framework provides a common vocabulary, but standardizing subjective criteria remains an open problem.

Furthermore, the coding domain, while practical, may not generalize to all applications. Creative writing, legal analysis, and medical advice each involve different types of ambiguity, stylistic constraints, and risk profiles. The input and output dimensions that matter for code generation may differ substantially from those relevant to other domains.

Conclusion: Toward User-Centered Evaluation

The formalization of vibe-testing represents more than a methodological innovation. It signals a shift in perspective, from viewing users as passive recipients of model capabilities to recognizing them as active evaluators with legitimate, structured criteria that benchmarks often overlook. As models become increasingly integrated into specialized professional workflows, the need for evaluation methods that capture this specificity will only grow.

Future work might explore adaptive evaluation systems that learn a user’s preferences over time, or community-driven platforms where vibe-testing patterns are aggregated to identify clusters of user types. We might also investigate how model training can incorporate these personalized dimensions directly, optimizing not just for aggregate accuracy but for the diverse criteria that constitute real-world utility.

The divergence between benchmark scores and user experience is not a problem to be solved by better benchmarks alone. It is a fundamental feature of deploying general systems in specific contexts. By taking "feelings" seriously and translating them into metrics, Itzhak et al. have provided a framework for closing this gap. The question now is whether the field will embrace the complexity of user-centered evaluation, or continue to optimize for metrics that miss what matters most.