Sample Routing as a Hybrid Strategy for Reinforcement Learning in Language Models

Post-training of large language models through Reinforcement Learning with Verifiable Rewards has emerged as the dominant paradigm for enhancing reasoning capabilities in mathematical, scientific, and agentic domains. While Group Relative Policy Optimization offers a stable baseline through relative advantage estimation, its coarse-grained credit assignment limits sample efficiency when addressing specific reasoning errors. Conversely, Self-Distillation Policy Optimization provides fine-grained, token-level supervision but suffers from catastrophic collapse during prolonged training. In Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing, Li et al. propose Sample-Routed Policy Optimization, a framework that dynamically allocates optimization strategies based on sample correctness. Their work demonstrates that hybridizing these approaches through intelligent routing not only prevents the instability modes of pure self-distillation but also surpasses the performance ceiling of GRPO alone, achieving 77.4% average accuracy on Qwen3-8B across five benchmarks.

The Credit Assignment Problem and Self-Distillation Pathologies

GRPO operates by estimating advantages through outcome reward normalization across groups of rollouts, applying a uniform scalar advantage to every token within a trajectory. For successful rollouts, this approach works adequately because intermediate steps generally support the correct outcome. However, for failed trajectories, the uniform penalty fails to localize specific deviations, effectively treating a minor arithmetic error identically to a completely incoherent reasoning chain. This sparsity in credit assignment motivates the search for denser supervision signals.

SDPO addresses this by leveraging logit-level supervision from a self-teacher conditioned on privileged context, typically a correct sibling rollout from the same group. This provides token-level gradients that precisely target reasoning deviations, enabling rapid early convergence in complex domains. However, Li et al. identify two intrinsic flaws that cause SDPO to collapse during prolonged training. First, applying self-distillation to already-correct samples introduces optimization ambiguity. When a successful rollout is forced to match a different successful sibling, the model must arbitrarily discriminate between reward-equivalent reasoning paths, creating conflicting gradient signals. The authors empirically validate this by demonstrating that restricting SDPO updates to failed samples preserves most benefits, while applying SDPO exclusively to correct samples accelerates performance degradation and destabilizes training.

Second, the self-teacher's signal quality degrades temporally. As training progresses and the gap between teacher and student narrows, the distillation signal becomes less informative. Concurrently, the teacher's token-level entropy rises, indicating growing uncertainty in its predictions. This entropy increase transforms the distillation target from a confident guide into a noisy, unreliable signal, explaining the late-stage collapse observed in pure SDPO implementations.

The SRPO Architecture and Mechanistic Design

Sample-Routed Policy Optimization resolves these tensions through a routing mechanism that partitions samples based on outcome correctness. Correct samples route to a GRPO branch, leveraging Monte Carlo advantage estimation for stable, reward-aligned updates that avoid arbitrary discrimination between valid solution paths. Failed samples route to an SDPO branch, receiving targeted logit-level corrections without the ambiguity of cross-matching successful trajectories.

Crucially, SRPO incorporates an entropy-aware dynamic weighting mechanism within the SDPO branch. This component monitors the self-teacher's prediction entropy and downweights high-entropy, unreliable distillation targets while emphasizing confident corrections. This creates an automatic curriculum where early training relies heavily on precise SDPO corrections for frequent errors, then gradually transitions toward GRPO dominance as rollouts become correct and teacher entropy rises.

Evaluated across five benchmarks following the protocol established by Hübotter et al., SRPO achieves consistent improvements across model scales. On Qwen3-8B, SRPO reaches 77.4% average accuracy, representing gains of 3.4 percentage points over GRPO and 6.3 over SDPO. On Qwen3-4B, the framework achieves 74.2% average accuracy, improving by 4.5 and 7.5 percentage points respectively. Additionally, SRPO reduces per-step compute cost by up to 17.2% while maintaining moderate response lengths, avoiding both the verbosity inflation common in GRPO and