Beyond Functional Correctness: The Case for Multi-Criteria Code Reward Models

Introduction

The current landscape of language model post-training for code generation operates under a strikingly narrow conception of software quality. Dominant reinforcement learning from human feedback (RLHF) pipelines and test-time scaling methods rely almost exclusively on execution feedback, treating code merely as a computational artifact to be validated through input-output matching. This paradigm, while effective for benchmarking functional correctness in sanitized environments, fundamentally misaligns with how software is actually evaluated in production settings. The paper Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring by Paul et al. presents a compelling alternative: reward models (RMs) capable of evaluating code across multiple qualitative dimensions and programming languages, independent of execution constraints.

The limitations of execution-based rewards extend beyond mere inconvenience. They enforce a brittle requirement that code be self-contained and immediately executable, excluding the vast majority of real-world software that depends on external libraries, module systems, and complex build configurations. Languages such as Rust, Go, and Swift, which rely on module-level compilation units, are effectively sidelined from post-training pipelines that depend on runtime verification. Furthermore, execution feedback reduces software quality to a binary signal (pass/fail), ignoring the multifaceted nature of professional code review that encompasses efficiency, security, maintainability, and adherence to idiomatic conventions.

The Execution Bottleneck and Its Consequences

Current code generation post-training predominantly employs test-case execution as its reward signal, a strategy that optimizes for functional correctness alone. While this approach offers the advantage of objective ground truth for self-contained problems, it introduces several structural deficiencies that constrain model development. First, procuring high-quality test cases requires either expensive human annotation or synthetic generation through circular consensus mechanisms, which research has shown produce brittle or erroneous test suites. These synthetic tests often fail to capture subtle interface contract violations or edge cases that characterize robust software engineering.

More critically, execution-based rewards are inherently limited to code that can run in isolated sandbox environments. This excludes repository-level code with complex dependency graphs, compiled languages requiring build systems, and partial code snippets that comprise the majority of professional software development. The paper notes that this restriction creates a distributional mismatch; models trained exclusively on executable Python snippets learn to generate code that differs systematically from the "natural code distribution in the wild" that includes imports, external API calls, and multi-file contexts.

The noise inherent in proxy rewards presents additional challenges. Attempts to expand execution feedback to include auxiliary signals such as wall time or memory consumption require tightly controlled emulated environments to prevent reward hacking. Without such controls, models exploit superficial correlations; for instance, optimizing for compilation speed rather than syntactic correctness, or reducing memory allocation through logic errors that happen to use less RAM. These failures illustrate why execution signals, while necessary, are insufficient for comprehensive code quality assessment.

The Themis Framework: Benchmarks, Data, and Models

To address these gaps, the authors introduce a comprehensive ecosystem comprising Themis-CodeRewardBench, Themis-CodePreference, and Themis-RM. The benchmark evaluates code RMs across five preference dimensions (correctness, efficiency, security, style, and readability) and eight programming languages (Python, Java, JavaScript, C++, Go, Rust, Swift, and PHP), providing the first systematic evaluation of multi-criteria code reward modeling beyond functional correctness.

The training corpus, Themis-CodePreference, represents the largest open-source collection of code preferences to date, containing more than 350,000 preference pairs. This dataset enables the training of Themis-RM, a suite of models ranging from 600M to 32B parameters. The scaling experiments reveal positive trends across model sizes, with larger models demonstrating superior discrimination between code snippets differing in non-functional attributes. Notably, the authors observe strong cross-lingual transfer when training on diverse multilingual preferences; a model trained on polyglot data outperforms monolingual baselines even when evaluated on individual languages, suggesting that underlying software engineering principles generalize across syntactic boundaries.

The technical methodology employs standard Bradley-Terry preference learning, but with a crucial distinction: the preference pairs are constructed to reflect multi-criteria judgments rather than binary correctness. For instance, a preference pair might contrast two functionally correct solutions where one exhibits quadratic time complexity while the other operates in linear time, or where one contains SQL injection vulnerabilities while the other sanitizes inputs properly. This multi-criteria training proves essential for reliable reward modeling; models trained exclusively on functional correctness data fail to generalize to security or efficiency evaluation, whereas multi-criteria training improves performance across all dimensions, including correctness itself.

Critical Analysis: Software Engineering Reality and Economic Constraints

The significance of Themis extends beyond technical benchmarking into the economic and practical realities of software development. Execution-based reward generation requires substantial computational infrastructure: sandboxed environments, containerized execution, timeout handling, and resource monitoring. In contrast, RM-based scoring, once trained, requires only forward passes through a neural network. For test-time scaling applications such as Best-of-N sampling or Monte Carlo Tree Search, where thousands of candidate solutions must be scored, the latency and compute savings of RM evaluation versus execution become substantial. This economic efficiency could democratize access to sophisticated code generation post-training for organizations lacking the infrastructure to safely execute arbitrary generated code.

However, the transition to RM-based evaluation introduces new challenges. Unlike execution feedback, which provides objective ground truth (modulo test quality), RM signals are inherently subjective and potentially subject to reward hacking. The paper acknowledges that current RMs show "limited proficiency" on non-functional criteria compared to functional correctness, suggesting that the 350k preference pairs, while large, may still underrepresent the nuanced judgments required for expert-level code review. Furthermore, the largest Themis-RM variant at 32B parameters, while effective, requires significant memory and inference compute, potentially limiting deployment in resource-constrained environments.

The multi-criteria approach also raises fundamental questions about criteria weighting. Professional code review contexts differ drastically; a startup prototyping a feature prioritizes speed of implementation differently than a bank reviewing payment processing code. The current framework treats criteria as separate evaluation dimensions, but practical deployment requires dynamic weighting mechanisms that adapt to user-specified priorities. The paper establishes that models can distinguish between criteria, but leaves open the question of how to optimally combine these signals for specific deployment contexts.

Conclusion and Future Directions

The Themis framework represents a necessary evolution in code generation post-training, moving from narrow execution-based optimization toward comprehensive software quality assessment. By demonstrating that multilingual, multi-criteria reward models exhibit positive scaling characteristics and cross-lingual transfer, Paul et al. establish the feasibility of training RMs that evaluate code as software engineers do, rather than merely as test runners.

Looking forward, the field must address several open questions. First, the integration of execution and model-based feedback remains unresolved; hybrid approaches that use RMs to filter candidate solutions before expensive execution, or that combine execution signals with RM predictions, may offer the reliability of runtime verification with the flexibility of learned evaluation. Second, repository-level evaluation requires attention, as current benchmarks focus on function-level or snippet-level code, whereas real software quality emerges from architectural decisions spanning multiple files and modules. Finally, the calibration of RMs for low-resource programming languages requires continued investigation, as performance disparities between high-resource languages like Python and niche languages persist despite cross-lingual transfer benefits.

The transition from single-metric to multi-criteria code evaluation mirrors the evolution of software engineering itself, from naive performance optimization toward holistic system design. As code generation models increasingly operate as collaborative agents in software development workflows, reward models capable of assessing security, efficiency, and maintainability will prove essential for generating not just functional code, but code worthy of production systems.