Perceval and the Granularity of Perception: Refining Vision-Language Models Through Process Rewards

The current generation of vision-language models demonstrates remarkable capabilities across multimodal mathematics, chart analysis, and visual question answering. Yet beneath these achievements lies a persistent fragility: when these models engage in complex, multi-step reasoning, they frequently hallucinate objects, misattribute spatial relationships, or drift from the visual evidence mid-chain. While reinforcement learning with verifiable rewards has emerged as the dominant post-training paradigm for improving reasoning, its reliance on outcome-level supervision creates a fundamental bottleneck. The paper "Improving Vision-language Models with Perception-centric Process Reward Models" introduces Perceval, a process reward model designed to provide token-level supervision that directly targets perceptual errors within reasoning chains. This work represents a significant refinement of how we train VLMs, shifting from coarse sequence-level rewards to fine-grained, claim-by-claim verification against visual evidence.

The Architecture of Perceptual Supervision

At the core of Perceval lies a reconceptualization of what constitutes a reward in reinforcement learning for vision tasks. Traditional RLVR implementations, such as those built upon Group Relative Policy Optimization, assign a single scalar reward to an entire response sequence. This approach proves inadequate for visual reasoning because the final answer correctness reveals little about where the model failed. A correct conclusion might mask hallucinated intermediate steps, while an incorrect final answer could stem from either perceptual mis grounding or subsequent logical errors. The resulting credit assignment problem forces the policy to learn through sparse, noisy signals.

Perceval addresses this by operating as a perception-centric process reward model capable of token-level error grounding. Rather than evaluating outcomes, the model extracts image-related claims from generated rationales, such as assertions about object attributes or spatial relations, and compares each claim individually against the visual evidence. This requires a carefully curated training regime using perception-intensive data drawn from goal-directed visual search and referring-expression grounding tasks. The authors employ a strong LLM to generate structured annotations that explicitly mark hallucinatory spans and provide visual counter-evidence, enabling the PRM to learn not merely binary correctness but explanatory error detection.

The technical sophistication here extends beyond simple verification. By decomposing generated text into verifiable perceptual claims, Perceval effectively creates an intermediate representation layer between raw pixels and abstract reasoning. This architecture acknowledges that visual reasoning errors often manifest as specific instances of image-text misalignment rather than general logical failures. The model learns to return structured feedback identifying precisely which tokens or spans constitute perceptual hallucinations, providing the granular supervision necessary for targeted correction.

Fine-Grained Credit Assignment in Training

The integration of Perceval into the RL training pipeline reveals the paper's most compelling technical contribution: a token-level advantage reallocation framework that modifies standard GRPO objectives. In conventional GRPO, the advantage for a response is computed relative to the group mean reward and applied uniformly across the sequence. Perceval enables a more surgical approach. By identifying specific hallucinated spans within otherwise reasonable chains of thought, the method can apply targeted penalties to perceptually erroneous tokens while preserving gradients for correct reasoning steps.

This distinction between sequence-level and token-level advantages addresses the fundamental sparsity problem plaguing current RLVR methods. When a VLM generates a ten-step reasoning chain where step three contains a hallucinated object description, traditional approaches must either reward or penalize the entire sequence based on the final answer. Perceval's architecture allows the policy to receive fine-grained feedback: the tokens corresponding to the hallucination receive negative advantage estimates, while subsequent correct logical deductions maintain appropriate weighting. This precision accelerates learning by eliminating the noise that arises from conflating perceptual errors with reasoning errors.

The empirical results demonstrate that this approach significantly enhances perceptual capabilities on perception-centric tasks. More surprisingly, the authors observe substantial generalization to complex reasoning tasks even without applying PRM supervision during training for those specific tasks. This suggests that improving foundational perceptual grounding produces cascading benefits for higher-level reasoning, indicating that perceptual fidelity may constitute a binding constraint on current VLM capabilities.

Test-Time Scaling Through Iterative Refinement

Beyond training modifications, Perceval enables a novel inference strategy that trades computational overhead for accuracy through iterative refinement. The authors propose a Truncation-Regeneration loop where the PRM analyzes generated responses, identifies erroneous perceptual claims, truncates the response at the point of hallucination, and either prompts regeneration from that point or induces explicit reflection on the error. This process can repeat multiple times, creating a form of test-time scaling distinct from brute force approaches like majority voting.

This inference mechanism leverages the structured error outputs of Perceval to perform targeted correction rather than wholesale resampling. While majority voting requires generating multiple complete responses and selecting the most consistent, the Truncation-Regeneration strategy focuses compute on fixing specific failure modes. The paper demonstrates consistent performance gains over alternative scaling strategies, suggesting that intelligent error detection may prove more efficient than statistical averaging for visual reasoning tasks. However, this approach introduces latency concerns and creates a dependency on the PRM's error detection accuracy; a false positive from Perceval could lead to the unnecessary discarding of correct reasoning steps.

Critical Analysis and Broader Implications

The generalization effect observed in this work carries profound implications for our understanding of vision-language model limitations. The finding that improved perceptual supervision enhances complex reasoning without explicit reasoning training suggests that current VLMs may be more constrained by their ability to faithfully ground language in visual reality than by their capacity for logical inference. This aligns with emerging evidence that many reasoning failures in multimodal systems originate from early-stage perceptual hallucinations that propagate through subsequent steps.

However, several limitations warrant consideration. The effectiveness of Perceval depends entirely on the quality and coverage of its training annotations. The use of a strong LLM to generate supervision signals for the PRM creates a potential capability ceiling; the PRM cannot detect errors that exceed the supervising model's own perceptual abilities. Additionally, the iterative inference strategy, while effective, introduces computational costs that may prove prohibitive for real-time applications. There remains an open question regarding the scalability of the annotation process required to train Perceval across diverse visual domains beyond the perception-intensive settings described.

This work situates itself within a broader trend toward process supervision in large models, extending concepts previously explored in text-based mathematical reasoning to the multimodal domain. The perception-centric focus offers a concrete instantiation of process rewards that leverages the unique verifiability of visual claims against image evidence. Unlike abstract reasoning steps that may require future context to evaluate, perceptual claims offer immediate, locally verifiable grounding points. This characteristic makes visual reasoning an particularly promising domain for PRM development, potentially offering a pathway toward more interpretable and correctable AI systems.

Conclusion

Perceval demonstrates that the path toward robust visual reasoning requires moving beyond outcome-level optimization to embrace the granularity of process supervision. By targeting perceptual errors at the token level and enabling iterative refinement during inference, this approach addresses the credit assignment problems that have limited traditional RLVR methods. The observed generalization from perception to reasoning suggests that we may need to reconsider the hierarchy of capabilities in multimodal models; perhaps sophisticated reasoning requires first mastering the seemingly basic task of not hallucinating what one sees.

Looking forward, several questions emerge. Can similar process reward models be developed for other modalities, such as video or audio, where temporal dynamics complicate claim verification? How might we reduce the annotation burden required to train high-quality PRMs, possibly through synthetic data generation or self-supervised approaches? Most importantly, as we develop increasingly sophisticated error detection mechanisms, we must consider the interplay between correction and exploration. An overly aggressive PRM might constrain model creativity by penalizing perceptually unusual but potentially valid interpretations. Balancing factual grounding with the flexibility to reason about counterfactual or ambiguous visual scenarios remains the next challenge for perception-centric reinforcement learning.