Trusting the Critic: How Task-Specific Reward Models Are Reshaping RL for Image Generation

Introduction: The Fragile Foundation of Visual RL

The integration of Reinforcement Learning (RL) into text-to-image generation and image editing pipelines represents one of the most significant methodological shifts in contemporary computer vision. By treating diffusion and autoregressive models as policies to be optimized, researchers have opened pathways to fine-tune generative systems toward nuanced human preferences. Yet this approach introduces a fundamental dependency. The entire optimization trajectory hinges upon the quality of the reward signal; if the critic hallucinates or assigns noisy scores, the policy learns to satisfy illusory objectives rather than genuine human intent.

In "Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation," Zhao et al. identify this bottleneck and propose FIRM, a comprehensive framework for training faithful, task-specific reward models. Their work reveals that general-purpose Multimodal Large Language Models (MLLMs), despite impressive zero-shot capabilities, suffer from severe limitations when deployed as critics for fine-grained visual tasks. These models exhibit object neglect, poor spatial reasoning, and hallucinations that propagate noise through the RL pipeline, fundamentally misguiding the optimization process.

Architectural Innovations in Reward Modeling

The FIRM framework distinguishes itself through tailored data curation strategies that acknowledge the distinct fidelity requirements of image editing versus text-to-image generation. For image editing, the authors observe a counter-intuitive phenomenon. While MLLMs struggle to directly assess whether an edited image perfectly follows instructions while preserving consistency with the source, they demonstrate remarkable competence at identifying visual differences between paired images. Exploiting this asymmetry, the researchers implement a "difference-first" pipeline. An MLLM first generates captions describing exact visual differences between source and edited images. These textual differences then feed into a scoring module that deduces both execution scores (did the edit follow the instruction?) and consistency scores (how much did the image change?).

For text-to-image generation, the team employs a checklist-based prompting strategy. A language model extracts key verification points from user prompts, appending these as explicit checking instructions to the MLLM scorer. This procedural approach mitigates hallucination by constraining evaluation to verifiable attributes rather than allowing open-ended aesthetic judgment.

Using these pipelines, the authors construct two large-scale datasets: FIRM-Edit-370K and FIRM-Gen-293K. They subsequently train specialized 8B parameter reward models, FIRM-Edit-8B and FIRM-Gen-8B, initialized from Qwen3-VL-8B-Instruct. The resulting models demonstrate superior alignment with human judgment on FIRM-Bench, a newly introduced human-annotated benchmark specifically designed for evaluating editing and generation critics.

Balancing Competing Objectives through Strategic Reward Formulation

A persistent challenge in RL for generative models involves balancing competing objectives without triggering optimization collapse. Naively maximizing multiple rewards often leads to reward hacking, where models exploit loopholes in the reward function to achieve high scores while producing visually degraded