The Tsallis Loss Continuum: Rethinking Commitment in Reasoning Model Training

Introduction

The post-training adaptation of large language models to reasoning tasks remains fraught with mechanical instability. When we attempt to teach models to generate latent computational trajectories, such as chains of thought or search traces, through reinforcement learning from verifiable rewards (RLVR), the process often collapses at cold start. If the initial model assigns a low success probability (p_0) to correct answers, standard policy gradient methods require prohibitively many updates to escape the basin of random performance. This stagnation has historically been addressed through instruction engineering or Rao-Blackwellized reward shaping, yet these remedies treat symptoms rather than the underlying gradient dynamics.

In their paper "How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum," Chu-Cheng Lin and Eugene Ie present a rigorous theoretical framework that recharacterizes this problem as a question of gradient amplification. By deriving a one-parameter family of losses (J_Q) based on the Tsallis (q)-logarithm, they demonstrate that RLVR and standard supervised fine-tuning represent opposing poles of a continuous spectrum. The critical insight lies not in the gradient direction, which remains constant across this family, but in a per-instance scalar amplification factor (P^{-q}\theta) that reweights examples based on the model’s current success probability (P\theta). This mechanism, which the authors term commitment, provides a tunable knob for navigating the tension between cold start escape velocity and noise robustness.

The Geometry of Cold Start and Gradient Amplification

The theoretical core of Lin and Ie’s work rests on a simple yet profound gradient identity. For any (q \in [0, 1]), the gradient of the Tsallis loss (\ell_q = -\log_q(P_\theta)) decomposes as:

$$\nabla_\theta \ell_q = P^{-q}\theta \nabla\theta \ell_0 = P^{1-q}\theta \nabla\theta \ell_1$$

where (\ell_0) corresponds to the RLVR objective (the exploitation pole) and (\ell_1) corresponds to the log marginal likelihood over latent trajectories (the density estimation pole). The factor (P^{-q}\theta) acts as an instance-specific learning rate amplifier. For low (q), the gradient on high-probability (familiar) examples is attenuated only slightly, while low-probability (unfamiliar) examples receive minimal updates. For high (q), the amplification grows polynomially as (P\theta) shrinks, forcing the model to attend aggressively to its failure modes.

This amplification directly determines cold start complexity. Under gradient flow assumptions with bounded score functions, Lin and Ie prove that the exploitation pole ((q=0)) requires (\Omega(1/p_0)) time to escape cold start, while the density estimation pole ((q=1)) escapes in (\Theta(\log(1/p_0))) time. This exponential separation in convergence rate explains why RLVR stalls on tasks where the base model initially succeeds rarely. The intermediate region (q \in (0,1)) offers a continuum of trade-offs: higher commitment resolves ambiguity faster but risks memorizing label noise, since the unbounded log loss at (q=1) fits the empirical distribution exactly, including annotation errors. Lower commitment provides noise robustness through bounded loss and escort tempering, but prolongs the initial stagnation.

Estimators and Empirical Trade-offs

Because the marginal success probability (P_\theta) is intractable for autoregressive models, practical optimization requires Monte Carlo estimation. The gradient identity admits two distinct factorizations, yielding complementary estimators. Gradient-Amplified RL (GARL) samples trajectories from the prior (p_\theta(z|x^)) and scales the standard RL gradient by the empirical success rate raised to (-q). Posterior-Attenuated Fine-Tuning (PAFT) instead approximates sampling from the posterior (p_\theta(z|x^, y^*)) using importance resampling, then applies standard supervised fine-tuning scaled by (P^{1-q}_\theta).

Both estimators share the same asymptotic bias (O(q/MP^{q+1}_\theta)) for (M) samples, but differ critically in variance and semantic coherence. GARL conditionally Rao-Blackwellizes PAFT, achieving lower variance by utilizing all (M) sampled rationales, including those that lead to incorrect answers. However, this comes at the cost of mixing semantically incoherent gradients; the gradient update includes contributions from trajectories that contradict the supervision signal. PAFT excludes these contradictory paths through posterior resampling, yielding semantically pure gradients at the expense of higher variance and computational overhead.

Empirical evaluation on FinQA, HotPotQA, and MuSiQue reveals distinct regimes for each estimator. In cold start scenarios where standard GRPO (Group Relative Policy Optimization) fails entirely, GARL with (q=0.75) successfully escapes stagnation, validating the theoretical prediction that elevated commitment accelerates initial learning. In warm start regimes, the picture fragments by task complexity. On FinQA, where training remains stable, GARL with low (q) dominates. However, on the multi-hop reasoning tasks HotPotQA and MuSiQue, GARL destabilizes during training, exhibiting the volatility that often plagues high-variance policy gradient methods. Here, PAFT at (q=0.75) provides stable convergence, achieving 47.9 maj@16 on HotPotQA, an improvement of +14.4 points over GRPO. This bifurcation suggests that the choice between amplification strategies depends not merely on the temperature of the cold start, but on the underlying stability of the reasoning task’s gradient landscape.

Original Insights: Commitment as Curriculum

Beyond the technical machinery, the concept of commitment offers a reinterpretation of the relationship between reinforcement learning and supervised learning. Rather than viewing these as distinct paradigms, the Tsallis continuum reveals them as points on a spectrum of trust in the supervision signal. Low commitment ((q \rightarrow 0)) represents conservative optimization, updating parameters only when the model already assigns reasonable probability to the correct answer, effectively filtering noise through the model’s own inductive biases. High commitment ((q \rightarrow 1)) represents credulous optimization, treating every supervision signal as worthy of aggressive pursuit regardless of current model beliefs.

This suggests an unstudied extension: dynamic commitment scheduling. Just as learning rate schedules modulate optimization speed globally, q-schedules could modulate per-instance attention throughout training. One might initialize with high (q) to escape cold start rapidly, then anneal toward low (q) to refine reasoning without overfitting to spurious correlations. Such an approach would mimic curriculum learning, but with the curriculum determined automatically by the model’s evolving success probabilities rather than hand-engineered difficulty metrics.

Furthermore, the framework illuminates why recent reasoning models trained via distillation or synthetic data generation often outperform their RLVR counterparts. These methods implicitly operate at high (q), generating diverse reasoning paths and fitting the marginal likelihood of correct answers. The Tsallis formulation provides a principled mechanism to interpolate between such density estimation and pure RL, potentially explaining why mixtures of SFT and RL phases prove empirically superior to either in isolation.

Limitations and Forward-Looking Questions

The analysis rests on several assumptions that constrain its immediate applicability. The theoretical convergence results assume bounded score functions and exact-match rewards (R(\hat{y}, y^) = \mathbb{I}(\hat{y} = y^)), excluding the general reward functions common in RLHF or RLAIF pipelines. The estimators introduce bias that scales inversely with both the number of samples (M) and the success probability (P_\theta), suggesting that very hard tasks with (P_\theta \approx 0) remain problematic even under the Tsallis framework. Additionally, the empirical evaluation focuses on question-answering datasets with verifiable discrete answers; how these dynamics extend to open-ended generation or mathematical proof verification remains unclear.

Several questions emerge for future work. Can we extend the Tsallis framework to general reward functions beyond exact-match supervision, perhaps through importance weighting on reward trajectories? How does the variance of PAFT scale with the entropy of the posterior over reasoning paths, and can we develop variance reduction techniques specific to high (q) regimes? Finally, does dynamic scheduling of (q) throughout training provide the robustness benefits of both poles while avoiding their respective pathologies of stagnation and overfitting?

Conclusion

Lin and Ie’s work provides a much-needed theoretical foundation for understanding why reasoning models stall at cold start and how to accelerate their escape. By identifying gradient amplification as the critical mechanism missing from standard RLVR, and by grounding this mechanism in the Tsallis (q)-logarithm, they offer practitioners not a single new algorithm, but a flexible continuum for navigating the exploration-exploitation tradeoff during training. The empirical superiority of GARL and PAFT in distinct regimes suggests that future reasoning systems may benefit from hybrid training protocols that modulate commitment based on task difficulty and model calibration. As we move toward training on increasingly complex reasoning tasks where (p_0) grows ever smaller, controlling how fast a model commits to supervision may prove as important as the supervision itself.