Beyond "Looks Nice": Structuring Video Aesthetics into Measurable Rewards

Video generation has reached a point where photorealism is no longer the bottleneck. Models like Wan2.2, Seedance, and HunyuanVideo can produce clips that pass casual inspection. The harder problem, the one that separates a generated clip from a piece of cinema, is aesthetics: color grading, lighting design, shot composition, and the subtle plausibility that makes a scene feel intentional rather than synthesized. The paper AesRM: Improving Video Aesthetics with Expert-Level Feedback tackles this directly, and it does so by refusing the lazy shortcut that has plagued aesthetics research for years: reducing the entire problem to a single scalar "this looks nice" score.

This is worth examining in detail, both for what it proposes and for what it implies about where reward modeling for generative video is heading.

Decomposing Aesthetics Into Something You Can Measure

The central contribution is a hierarchical rubric that splits video aesthetics into three core dimensions:

• Visual Aesthetics (VA): the artistic qualities, including six criteria for color and lighting such as color quality and light source handling.
• Visual Fidelity (VF): the technical quality of the rendered frames.
• Visual Plausibility (VP): whether the content is physically and semantically coherent over time.

These three dimensions expand into 15 fine-grained criteria, with shot composition being one representative example. This matters because aesthetics is not a monolith. A clip can have gorgeous color but broken motion, or perfect physical plausibility while being compositionally flat. Collapsing these into one number throws away exactly the information you would need to fix a model.

From this rubric, the authors built two artifacts that I suspect will outlast the specific models trained on them:

1. A 19K preference-pair training set generated from roughly 20K synthesized prompts across multiple text-to-video models, with aesthetics experts performing pairwise best/worst labeling along VA, VF, and VP.
2. AesVideo-Bench, an evaluation benchmark of about 2,500 video pairs with expert annotations across the same dimensions.

The expert annotation is the expensive part and the part that gives this work credibility. Prior efforts often relied on aggregate aesthetic scores from datasets like the LAION-Aesthetics predictor lineage, which were trained on noisy, crowd-sourced or model-predicted labels. Grounding preferences in expert judgment guided by an explicit rubric is a meaningful step up in label quality, even if it limits scale.

The Reward Model Architecture and Training Recipe

The authors deliberately avoid the common pattern of bolting a regression head onto a vision-language model. Instead, they cast aesthetic reward modeling as native next-token prediction, keeping the task aligned with the VLM's original generative objective. The reasoning is sound: a regression head discards the model's pretrained reasoning capacity, while a generative formulation lets the model reason its way to a judgment. This is the same insight that has driven generative reward models in the text domain, and applying it to video aesthetics is a natural extension.

The training proceeds in three progressive stages:

1. Atomic Aesthetic Capability Learning: fine-tuning on atomic datasets so the model reliably recognizes fundamental concepts, for example correctly identifying a centered composition. This is essentially building the perceptual primitives before asking for judgments.
2. Cold-Start: instruction-tuning on the preference data to teach pairwise comparison with structured reasoning.
3. GRPO: Group Relative Policy Optimization with verified rewards to sharpen accuracy.

The family splits into two members. AesRM-Base directly outputs pairwise comparisons across VA, VF, and VP, giving efficient reward signals for post-training. AesRM-CoT additionally produces chain-of-thought reasoning aligned to all 15 criteria, trading some efficiency for interpretability.

The most technically interesting wrinkle is how they handle CoT quality. The authors observed that low-quality CoT amplifies hallucinations, for instance the model declaring a perfectly normal video to have severe artifacts. Their fix is a self-consistency-based CoT synthesis method: rather than distilling from a single sampled reasoning trajectory from the teacher (here Gemini 2.5 Pro plus expert priors), they ensemble multiple trajectories and select using a consistency metric. They then carry this further into GRPO by adding a process reward based on the similarity between teacher-generated and student-generated CoTs.

This is a thoughtful piece of engineering. Distilling reasoning is notoriously fragile because the teacher's mistakes get baked in as ground truth. Using consistency across trajectories as a filter is a cheap and principled way to suppress the teacher's hallucinations before they pollute the student. The process reward then ensures the student is rewarded not just for the right verdict but for the right reasoning path, which should reduce reward hacking where a model reaches a correct preference through spurious logic.

Results and What They Tell Us

The reported outcomes are twofold. On the evaluation side, AesRM outperforms baseline reward models across multiple aesthetics benchmarks and shows lower position bias, meaning it is less likely to prefer whichever video happens to be presented first. Position bias is a chronic problem in pairwise LLM and VLM judges, so a measurable reduction here is a meaningful robustness gain rather than a cosmetic one.

On the application side, the authors align Wan2.2-TI2V-5B with AesRM and report clear aesthetic gains over existing aesthetic reward models, with generated videos exhibiting better color, more sophisticated lighting, and richer detail. This closes the loop: the reward model is not just a benchmark instrument but a usable training signal that improves a real generator.

My Take: The Static Reward Is the Real Limitation

The decomposition into VA, VF, and VP with per-criterion reasoning is the right move, and it escapes the "looks nice" trap cleanly. But I think the most important limitation is one the framework itself sets up but does not yet solve: the reward is still effectively static.

When you optimize a generator against a fixed aesthetic reward, you are optimizing toward an aesthetic mean. That mean is precisely where everything begins to look the same. Film noir wants crushed shadows, hard key lighting, and off-center composition. A nature documentary wants even exposure, natural color, and stable framing. A music video might want deliberately "broken" composition for effect. A single reward, even a well-decomposed 15-criterion one, collapses these into a global preference that rewards a generic prettiness.

The structure AesRM builds is actually the ideal substrate for fixing this. Because the reward is already decomposed into criteria, you could introduce per-criterion weighting that conditions on a style prompt. Imagine the GRPO reward computed as a weighted sum where the weights are produced by a small head that reads a target style descriptor. "Film noir" would up-weight the lighting and composition criteria associated with that idiom and down-weight, say, color vibrancy. "Documentary realism" would do the opposite, and might even let plausibility dominate. This would let you steer generation toward a target cinematic language rather than toward an aesthetic average.

Two practical concerns follow. First, the expert annotations would need to be conditioned on intent; a video that is "good film noir" is not the same as a video that is "good" in the abstract, and the current pairwise protocol seems intent-agnostic. Second, validating genre-conditional rewards is harder because you lose a single universal ground truth. But these are tractable, and the payoff is avoiding the homogenization that aesthetic optimization tends to produce.

A second limitation worth naming honestly is scale and coverage. 19K expert-labeled pairs is high quality but small by modern standards, and the prompts and source models define the distribution the reward model trusts. As generators improve, the failure modes shift, and a reward model trained on today's artifacts may misjudge tomorrow's outputs. Continual relabeling will be necessary.

Conclusion

AesRM makes a clean and defensible argument: video aesthetics deserve structured, interpretable, expert-grounded rewards rather than opaque scalars. The hierarchical rubric, the generative reward formulation, and the consistency-filtered CoT distillation are each sensible choices, and the demonstrated improvement on Wan2.2 shows the pipeline produces a usable signal.

The open question I find most compelling is whether the field will treat aesthetics as a single objective to be maximized or as a space of styles to be navigated. AesRM gives us the decomposition needed for the latter. The next step is making the weights over those criteria conditional, so that we can ask a model for a specific cinematic language instead of settling for the comfortable, forgettable middle.