PhyCo: Embedding Controllable Physical Laws into Video Diffusion Models

Modern video diffusion models have achieved remarkable fidelity in visual appearance, synthesizing photorealistic textures, coherent lighting, and fluid camera motions. Yet these models remain fundamentally detached from physical reality. Objects hover inexplicably, collisions occur without momentum transfer, and materials behave according to visual stereotypes rather than underlying mechanical properties. In their recent work, PhyCo: Learning Controllable Physical Priors for Generative Motion, Narayanan et al. present a framework that addresses this disconnect by embedding continuous, interpretable physical controls directly into generative video models. Their approach is notable not merely for improving physical plausibility, but for achieving controllable variation in friction, restitution, deformation, and applied force without requiring geometric reconstruction or runtime physics simulation.

From Explicit Simulators to Learned Physical Priors

The central technical innovation of PhyCo lies in its training paradigm, which decouples physics supervision from inference requirements. Prior approaches to physically consistent video generation typically fall into two categories, each with significant limitations. Methods such as PhysDreamer and PhysGen rely on explicit physics engines (Material Point Method or rigid body dynamics) and require 3D geometry reconstruction during inference. While these produce accurate motion, the computational overhead and need for predefined materials make them impractical for scalable generation. Conversely, implicit approaches like ForcePrompting or PhysCtrl embed physical cues through language prompting or trajectory constraints, but offer only coarse, categorical control rather than continuous property variation.

PhyCo bridges this gap through a three stage pipeline. First, the authors construct a large scale dataset of over 100,000 photorealistic simulation videos using Kubric with PyBullet for physics simulation and Blender for rendering. Unlike existing benchmarks such as CLEVRER or Physion, which provide limited property annotations or single object dynamics, this dataset systematically varies friction, restitution, deformation, and applied force across diverse scenarios with multiple interacting objects. This systematic variation is crucial; by disentangling visual appearance from physical parameters during training, the model learns representations that generalize beyond specific textures or geometries.

The second component involves physics supervised fine tuning of the Cosmos-Predict2 diffusion backbone using a ControlNet architecture. Rather than conditioning on global text descriptions or sparse trajectories, PhyCo injects pixel aligned physical property maps as control signals. This spatial conditioning allows the model to localize physical properties to specific regions; for instance, assigning high friction to one surface while allowing low friction elsewhere in the same scene. The ControlNet architecture ensures that these physical constraints propagate through the denoising process while preserving the visual prior learned from large scale video data.

The Vision Language Model as a Differentiable Physics Critic

Perhaps the most intriguing aspect of PhyCo is its third component: VLM guided reward optimization. The authors fine tune a vision language model to evaluate generated videos through targeted physics queries, effectively training a learned critic that understands mechanical plausibility. This critic provides differentiable rewards that fine tune the diffusion model toward more physically consistent outputs.

This mechanism addresses a fundamental challenge in physics aware generation. Traditional supervised learning on simulation data risks overfitting to the specific visual artifacts of the training simulator, limiting transfer to real world scenarios. By incorporating a VLM that evaluates semantic consistency with physical laws, PhyCo introduces a layer of abstraction that aligns generated motion with human intuition about physics, rather than mere pixel level similarity to training videos. The differentiable nature of this feedback suggests the use of score distillation or similar techniques to propagate gradients from the VLM's judgments back to the video generator's parameters.

The results on the Physics IQ benchmark demonstrate significant improvements over strong baselines in physical realism metrics. Human preference studies further confirm that observers perceive clearer control over physical attributes in PhyCo's outputs compared to standard video diffusion models or even physics informed baselines. Particularly impressive is the model's compositional generalization; it can combine multiple physical attributes (such as force plus friction) in stylized scenes that diverge significantly from the training distribution, suggesting that the learned representations capture disentangled physical concepts rather than memorized motion patterns.

Limitations and the Path to General World Models

Despite these advances, PhyCo invites several critical questions about the nature of learned physical understanding. First, while the framework offers continuous control over individual properties, real world materials rarely exhibit independent variation of these parameters. Rubber typically displays both high friction and high restitution; ice exhibits low friction and minimal deformation. The extent to which PhyCo can model these natural correlations, or whether it produces physically implausible combinations (such as high friction with perfectly elastic collisions), remains an open question regarding the entanglement of its learned priors.

Second, the reliance on simulation data, while necessary for obtaining ground truth physical annotations, introduces a domain gap that the VLM critic attempts to bridge. However, vision language models carry their own biases regarding physical scenes, having been trained on internet scale data that includes cartoons, animations, and physically impossible imagery. When the VLM evaluates a generated video, it may favor motion that appears intuitively correct but violates precise physical laws, or conversely, penalize unusual but physically valid behaviors. This creates a tension between simulator accuracy and human perceptual judgments that the framework does not fully resolve.

Furthermore, the current implementation focuses on rigid and deformable body dynamics within controlled scenarios. Extending this approach to fluids, granular materials, or chaotic systems presents substantial challenges, as these domains require continuous field representations rather than discrete object properties, and their simulation is significantly more computationally expensive.

Conclusion

PhyCo represents a meaningful advance in the quest for physically consistent generative video models. By demonstrating that continuous physical control can be achieved through pure generative modeling without runtime simulation, the work suggests a scalable path toward world models that understand how objects behave, not merely how they appear. The combination of pixel aligned conditioning and VLM guided optimization offers a template for incorporating structured physical knowledge into foundation models.

Looking forward, several research directions emerge from this work. Can similar frameworks learn physics entirely from real world video, eliminating the need for synthetic training data? How might these models handle extreme physical regimes, such as zero gravity environments or materials with negative viscosity? As generative models increasingly serve as simulators for robotics and scientific applications, the ability to embed controllable, interpretable physical laws will prove essential. PhyCo establishes that this integration is possible, laying groundwork for the next generation of video generation systems that respect the constraints of physical reality.