SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

The reinforcement learning paradigm in robotics has long been constrained by the reward engineering bottleneck. While foundation models promise to abstract away task specific programming, their deployment as reward signals has been fraught with practical failures. The paper SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning presents a targeted architectural solution to this problem, introducing a video native reasoning model that generates dense, temporally grounded reward signals without task specific tuning.

Current vision language models, despite their impressive performance on static benchmarks, exhibit systematic vulnerabilities when deployed as evaluators in closed loop control. As the authors demonstrate, models such as GPT-5 and Gemini-3-Pro fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than achieve genuine task success. This phenomenon, often termed reward hacking, emerges because general purpose VLMs lack explicit spatiotemporal reasoning capabilities necessary to track continuous task progress from video sequences.

Video Native Reasoning for Dense Progress Estimation

SOLE-R1 addresses these limitations through an architecture explicitly designed for video language reasoning over robot trajectories. Rather than treating reward modeling as a binary classification problem, SOLE-R1 performs per timestep spatiotemporal chain of thought reasoning, generating dense estimates of task progress relative to natural language goals. This approach fundamentally differs from prior work that either evaluates final states or provides sparse success indicators.

The training methodology reflects the dual requirements of reasoning quality and reward calibration. The authors developed a large scale video trajectory and reasoning synthesis pipeline, generating over one million chain of thought reasoning examples from more than 40,000 real world and simulated videos. This dataset provides temporally grounded reasoning traces aligned with continuous progress supervision, enabling the model to understand not just spatial relationships within single frames, but temporal dynamics across sequences.

Complementing this trajectory data, the training mixture incorporates foundational spatial reasoning tasks and multi frame temporal reasoning datasets. This layered approach ensures the model develops video native reasoning capabilities that explicitly integrate spatial and temporal structure. The hybrid training framework couples supervised fine tuning for high quality chain of thought generation with reinforcement learning from verifiable rewards to refine progress prediction accuracy.

Zero Shot Generalization and Robustness

The experimental validation of SOLE-R1 demonstrates remarkable zero shot generalization capabilities. Across four distinct simulation environments and a real robot setting, SOLE-R1 enabled online reinforcement learning from random initialization on 24 previously unseen manipulation tasks. This stands in stark contrast to strong baseline vision language models including GPT-5 and Gemini-3-Pro, which succeeded on fewer than 10 tasks under identical conditions.

The tasks span diverse families including pick and place operations, articulated object manipulation, and interactions with buttons, levers, and knobs. Crucially, the experiments required no ground truth rewards, success indicators, demonstrations, or task specific tuning. The model generalized to unseen scene layouts, camera viewpoints, and robot embodiments, suggesting that the spatiotemporal reasoning capabilities transfer across visual and mechanical domains.

Beyond raw success rates, SOLE-R1 exhibits markedly greater robustness to reward hacking compared to general purpose VLMs. This robustness stems from the explicit chain of thought reasoning requirement; by forcing the model to articulate spatial and temporal observations before predicting progress, the architecture becomes less susceptible to spurious correlations that exploit visual shortcuts. The dense, per timestep reward structure further