Manufacturing 4D Supervision: How Reshoot-Anything Sidesteps the Multi-View Data Problem

Camera control is one of the most persistent unsolved problems in video generation. We can synthesize photorealistic frames, follow text prompts, and condition on poses, yet asking a model to take an existing dynamic video and reshoot it from a different camera trajectory remains hard. The reason is not architectural. It is data. To teach a model what a scene looks like from a new angle, you need paired footage of the same action recorded simultaneously from multiple cameras. For static scenes this is merely tedious. For dynamic, non-rigid scenes, a hand catching a ball, a face turning, cloth folding, such paired data barely exists at scale.

The paper Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting from Morphic Inc. proposes a way around this constraint that I find genuinely instructive, less for the specific results and more for what it reveals about how self-supervision is maturing as a discipline.

The Core Trick: Cropping as Synthetic Parallax

The central observation is deceptively simple. If you cannot collect two real camera views of the same action, you can manufacture two pseudo-views from a single monocular video. The authors extract two distinct smooth random-walk crop trajectories from one input clip. One trajectory becomes the source view, the other becomes the target view. Because the two crops follow different paths across the frame over time, the target view will frequently demand image content that is occluded, cut off, or simply absent in the corresponding source frame.

This misalignment is the entire point. It functions as what the authors call an information erasure mechanism. If the source and target crops were aligned, the model could solve the reconstruction task by copying pixels directly. By forcing spatial misalignment and artificial occlusion, the task becomes impossible to solve via trivial copying. To fill a dis-occluded region in the target, the model must locate the missing texture somewhere else in the source video, typically in a different temporal context, then route and re-project that content into the target frame.

The training triplet is completed by a third element: the anchor. The anchor is generated by forward-warping the first frame of the source video using a dense 2D tracking field. This is the clever bridge between training and inference. At inference time, the model receives a 4D point cloud derived from the source video, re-projected under a novel camera trajectory. That re-projection is noisy and distorted, full of holes and reconstruction artifacts. By synthesizing the training anchor through forward-warping, the authors deliberately reproduce those same distortions during training. The model therefore learns from the start to treat the anchor as approximate geometric guidance rather than ground truth, and to correct it using the source video.

Why the Bottleneck Produces 4D Priors

This is the part worth dwelling on. The model never observes true 4D geometry. It is never given depth maps as targets, never trained with a 4D reconstruction loss, never shown a real multi-view pair. And yet, to minimize its reconstruction objective, it is compelled to build an implicit understanding of how objects occupy space and persist through time.

The logic is a chain of forced consequences:

• The target crop needs content not present in the matching source frame.
• That content exists elsewhere in the source video, at different times and apparent viewpoints.
• To retrieve it correctly, the model must reason about which earlier or later frame contains the dis-occluded surface and how to re-project it.
• Doing this consistently across an entire video requires an internal representation of scene structure that is spatially and temporally coherent, in other words, a 4D prior.

The architecture itself is intentionally minimal. Rather than building bespoke 4D modules, the authors adapt the self-attention layers of a pre-trained video diffusion transformer. This is a reasonable bet: self-attention is already the mechanism by which transformers route information across spatial and temporal positions. The task design simply repurposes that routing capacity toward cross-view texture retrieval. They pair this with targeted augmentations meant to harden the model against the kind of distortions the inference-time anchor will exhibit.

The contrast with prior work clarifies the contribution. The paper identifies two dominant families and their failure modes:

• Synthetic-data methods train on rendered scenes and suffer the well-known synthetic-to-real gap. The paper shows artifacts on unseen real interactions, such as a tennis ball being struck, where the rendered training distribution offers no analog.
• Anchor-only methods lean directly on 3D/4D reconstruction from pretrained models. When that reconstruction is poor, the errors propagate straight into the output. TrajectoryCrafter and related approaches mitigate this by feeding the source video as an auxiliary input, but remain bottlenecked by the cost and quality of their 4D pipelines.

Reshoot-Anything avoids both traps by getting its supervision signal from structure that already exists inside ordinary monocular video. The data pipeline is, by the authors' framing, scalable and domain-agnostic, which means it can in principle feed on internet-scale video without any reconstruction or rendering step at training time.

My Take: Supervision by Design, Not by Label

The deeper lesson here, which I gestured at in my earlier note, is that self-supervision is increasingly an exercise in bottleneck design rather than label collection. The history of the field has steadily moved in this direction: masked prediction in language and vision, contrastive views in image representation learning, next-frame prediction in video. In each case the trick is to remove information the model would prefer to copy, leaving it no choice but to learn the underlying structure.

Reshoot-Anything is a particularly pure instance because the bottleneck is spatial and geometric rather than statistical. The misaligned crop is not random masking; it is a constraint that maps directly onto the physical phenomenon the model needs to internalize, namely occlusion and parallax. The task is constructed so that the only generalizable solution is the desired one. That is good engineering of an inductive pressure.

I would push on a few inferences the paper invites. First, the quality ceiling of this approach is implicitly bounded by the diversity of camera-like motion that crop trajectories can simulate. A 2D crop walk approximates translation and a narrow band of apparent viewpoint change, but it cannot reproduce genuine large-baseline rotation around an object. The model learns 4D priors, but it learns them from a thin slice of the true viewpoint distribution. I would expect the method to be strongest for modest camera moves and to degrade as the requested novel trajectory diverges far from anything a crop can simulate.

Second, the forward-warping anchor is doing heavy lifting as a domain adapter between training and inference. Its faithfulness to real point-cloud distortions determines how well training transfers. If inference-time 4D reconstruction produces a class of artifacts not represented by the 2D-tracker-based warps, the model could be surprised. The augmentation strategy is presumably meant to broaden this coverage, but it is an empirical bet rather than a guarantee.

Third, there is a subtle dependency on dense 2D tracking quality. Tracking failures on fast or thin motion would corrupt the anchor and, more importantly, could bias what the model believes about scene structure. The method trades the cost of 4D reconstruction at training time for a reliance on robust 2D correspondence, which is cheaper but not free.

Conclusion

Reshoot-Anything reframes dynamic video reshooting from a data-collection problem into a task-design problem. By sampling two crop trajectories from a single clip and synthesizing a distortion-matched anchor, it manufactures the multi-view supervision that does not exist in the wild, and it does so in a way that forces a pre-trained diffusion transformer to build implicit 4D priors. The reported state-of-the-art temporal consistency and camera control are evidence that the bottleneck is well chosen.

The open questions are about reach. How far from crop-simulable motion can these priors extrapolate? How tightly is performance coupled to tracker and warp fidelity? And more broadly, what other geometric capabilities, depth estimation, relighting, scene editing, might fall out of similarly engineered erasure tasks? If the pattern holds, the most valuable contributions in this space may increasingly come not from new losses or larger labeled sets, but from finding the precise piece of information to hide so that the model has no choice but to learn what we wanted all along.