Internalizing Recovery: Why Agentic AI Must Learn From Its Mistakes, Not Just Its Successes

Current post-training methods for large language models treat autonomous agency as an optimization problem over final outcomes. When we deploy models as agents in interactive environments, whether coding with compiler feedback or navigating web interfaces, the dominant paradigm remains outcome based reinforcement learning with verifiable rewards. This approach, exemplified by methods such as GRPO, assigns scalar success signals to complete trajectories and uses policy gradients to upweight winning behaviors. Yet this methodology contains a fundamental misalignment with the nature of agentic competence. In their paper Internalizing Agency from Reflective Experience, Ge et al. identify a critical failure mode they term distribution sharpening, wherein models become increasingly proficient at reproducing already successful behaviors while failing to expand their capacity to solve novel problems. The authors propose LEAFE, a framework that shifts the training objective from outcome optimization to the internalization of recovery procedures. Their work suggests we have been conflating exploitation of existing capabilities with the acquisition of genuine agency.

The Distribution Sharpening Trap in Outcome Driven Training

The central observation driving this research is that scalar reward signals provide insufficient guidance for long horizon decision making. In complex interactive domains such as competitive programming or embodied navigation, agents must make sequences of interdependent decisions where early errors propagate and compound. When training uses only terminal rewards, the learning signal reduces rich environmental feedback, including compiler errors, invalid action notifications, and state transition observations, into a single binary or scalar value.

This compression creates what the authors identify as distribution sharpening. The policy gradient updates concentrate probability mass on trajectories that happen to succeed, which typically represent behaviors already present in the base model's distribution. While this improves Pass@1 metrics, it yields diminishing or even negative returns on Pass@k evaluations that measure the breadth of solvable problems. The model effectively memorizes paths to solutions it can already reach, rather than learning to navigate the recovery procedures necessary to escape local optima and error states. Ge et al. demonstrate this empirically across CodeContests, WebShop, ALFWorld, ScienceWorld, and Sokoban, showing that GRPO baselines exhibit minimal improvement over base models when measured at high k values such as Pass@128.

The technical limitation here is not merely a sampling issue but an information theoretic one. Successful trajectories contain information about what works, but failed trajectories contain information about what fails and potentially how to fix it. By discarding the latter as noise, outcome driven methods waste the majority of their interaction data. In long horizon settings where successful trajectories are rare, this creates a