The Limits of Distribution Sharpening: Why Task Rewards Drive Real Capability Gains in Language Models

The integration of reinforcement learning into large language model post-training has fundamentally altered the trajectory of artificial intelligence development. Where once next-token prediction dominated, we now see sophisticated systems capable of extended reasoning, tool use, and complex planning emerging from reward-based optimization. Yet beneath these empirical successes lies a persistent theoretical ambiguity. Does reinforcement learning genuinely instill novel capabilities within base models, or does it merely act as a sophisticated filter, amplifying existing behaviors through increased confidence? The preprint "Beyond Distribution Sharpening: The Importance of Task Rewards" by Mittal, Gagnon, and Lajoie provides a rigorous examination of this question, offering both mathematical analysis and empirical evidence that clarifies the distinct roles of distribution sharpening and task-reward optimization.

Disentangling Mechanisms Within a Unified Framework

To understand the debate, one must first grasp the mathematical structure of modern RL fine-tuning. The standard approach employs KL-regularized reinforcement learning, which optimizes a composite objective balancing reward maximization against divergence from a reference model. Formally, this objective takes the shape:

arg max_θ E_{x∼D}[E_{ŷ∼π_θ(·|x)}[r(ŷ,x)] − β D_KL(π_θ(·|x) || π_θ_ref(·|x))]

This framework admits a closed-form optimal solution: π_θ*(·|x) ∝ π_θ_ref(·|x) exp(β^{-1}r(·,x)). The elegance of this formulation lies in its flexibility. By manipulating the reward signal r and the KL regularization parameters, one can instantiate qualitatively different learning paradigms within identical training infrastructure.

Mittal et al. exploit this property to conduct a controlled comparison between two distinct approaches. Task-reward optimization employs the actual task-specific reward, such as binary correctness on mathematical problems. Distribution sharpening, by contrast, effectively sets the reward to the log-likelihood under the reference model itself, causing the optimization to concentrate probability mass on outputs the base model already considers likely. The paper examines four specific configurations: pure task-reward optimization, tilted sampling (which combines base likelihood with task rewards), pure distribution sharpening, and tempered sampling (inference-time sharpening without training).

This methodological isolation is crucial. Previous debates about sharpening versus capability acquisition often suffered from confounding variables; different papers employed different training procedures, inference methods, or evaluation protocols. By constraining both approaches within the same KL-regularized framework, Mittal et al. ensure that observed differences stem from the fundamental mathematical properties of the objectives rather than implementation artifacts.

The Instability of Sharpening and the Necessity of External Signal

The theoretical analysis presented in the paper reveals a fundamental limitation inherent to distribution sharpening. When one optimizes purely to concentrate mass on the modes of the base distribution, the resulting optima possess unfavorable mathematical properties. The optimization landscape becomes unstable by construction; small perturbations can lead to qualitatively different solutions as the policy attempts to place all probability mass on a narrow subset of already-likely trajectories.

This instability manifests empirically in the paper's experiments using Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, and Qwen3-4B-Instruct-2507 evaluated on mathematical reasoning datasets. Models trained via pure distribution sharpening demonstrate limited performance gains compared to their base versions. More tellingly, these gains plateau rapidly and fail to generalize to more difficult problem instances where the base model initially performs poorly. The sharpening approach essentially amplifies the base model's existing confidence structure, meaning it cannot recover correct solutions that the base model assigns low initial probability.

Conversely, task-reward optimization demonstrates robust improvements across difficulty levels. When the reward signal carries information about task success external to the model's initial beliefs, the optimization can discover high-reward trajectories that the base model would not have generated spontaneously. This distinction becomes most pronounced on challenging problems; where sharpening fails to escape the gravitational pull of the base distribution's modes, task-reward optimization navigates toward correct solutions regardless of their initial likelihood under π_base.

The paper's findings resonate with recent observations that certain post-training improvements can indeed be recovered through inference-time procedures like beam search or temperature scaling. However, Mittal et al. demonstrate that such recovery is bounded. Inference-time sharpening cannot transcend the fundamental limitations of the pre-trained distribution, whereas task-reward optimization provides a pathway to genuine capability expansion.

Toward a Taxonomy of Reward Signals

The implications of this work extend beyond the specific comparison of sharpening versus task rewards. The paper suggests, though does not fully develop, a broader taxonomy of reward signal types that warrants deeper investigation. Not all task rewards are created equal; some may indeed function primarily as sharpening mechanisms in disguise, while others unlock behaviors that require substantial deviation from the base distribution.

Consider the distinction between verifier rewards and process rewards. A sparse binary reward indicating final answer correctness provides a strong external signal that inherently differs from the base model's generative distribution. By contrast, a reward based on stylistic conformity or format adherence might closely track the base model's existing preferences, effectively operating as a form of sharpening despite being framed as task optimization. The field currently lacks precise language to distinguish rewards that compress existing distributions from those that expand the reachable behavior space.

Furthermore, the instability of pure sharpening suggests that we should view the KL regularization term not merely as a training stabilizer, but as a crucial inductive bias that determines the nature of learning. The coefficient β controls whether optimization primarily follows the gradient of the reward (enabling new behaviors) or remains anchored to the reference distribution (sharpening existing patterns). Future work might investigate intermediate regimes where partial sharpening enables more stable discovery of high-reward regions without collapsing to pure mode-seeking.

Limitations and Open Questions

The analysis provided by Mittal et al. is not without constraints. The experiments focus specifically on mathematical reasoning tasks, where correctness verification is unambiguous and external to the model's internal uncertainty. It remains unclear whether the same dynamics hold for more subjective tasks, such as creative writing or conversational alignment, where the "task reward" itself may be entangled with distributional properties of the training data.

Additionally, the paper assumes access to a well-specified reward function. In practice, reward hacking and misspecification remain significant challenges. One could imagine scenarios where optimizing an imperfect task reward leads to behavior that appears distinct from sharpening yet remains systematically undesirable. The relationship between reward quality and the sharpening versus capability acquisition dichotomy requires further exploration.

Conclusion

The distinction between distribution sharpening and task-reward-based learning is not merely academic; it determines the scalability and ultimate limits of post-training methods. As Mittal et al. demonstrate, sharpening alone offers diminishing returns, particularly as we push toward problems where current models show weakness. The mathematical unfavorability of sharpening optima, combined with their empirical instability, suggests that sustained progress in artificial intelligence will require careful engineering of reward signals that carry genuine information about task success.

Looking forward, the field must develop more nuanced theoretical tools to classify reward signals according to their capacity to expand versus compress the behavior distribution. We need training paradigms that explicitly account for the sharpening dynamics inherent in all RL fine-tuning, perhaps through adaptive regularization schemes that modulate the KL constraint based on task difficulty. The question is no longer whether RL fine-tuning works, but rather which reward structures enable the continued expansion of capabilities beyond the frontier of pre-trained knowledge.