Rethinking Supervision in Novel View Synthesis: How Monocular Depth Enables Internet-Scale Training

Introduction

The human visual system possesses a remarkable capacity for spatial reasoning. Presented with a single photograph of a cathedral, we effortlessly extrapolate the geometry of flying buttresses, infer occluded facades, and imagine alternative vantage points. For computer vision, replicating this capacity, known as monocular novel view synthesis, has remained stubbornly dependent on multi-view supervision. Training pipelines have historically required carefully curated datasets of posed image pairs, such as RealEstate10K or DL3DV, fundamentally constraining model development to narrow visual domains and limited data scales.

A new paper, "One View Is Enough! Monocular Training for In-the-Wild Novel View Generation" by Ramanana Rahary et al., challenges this foundational assumption. The authors demonstrate that monocular depth estimation has matured sufficiently to serve as a geometric scaffold, enabling training on arbitrary single images without multi-view correspondence. Their model, OVIE, leverages pseudo-novel views generated via off-the-shelf depth estimators, training on 30 million uncurated internet images while achieving inference speeds exceeding 100 frames per second. This work represents a significant departure from established training paradigms, with implications that extend beyond view synthesis into the broader economics of 3D vision supervision.

The Pseudo-View Paradigm: Decoupling Geometry from Supervision

The technical innovation underlying OVIE lies in its reconfiguration of the training target. Rather than requiring paired photographs from different camera positions, the method constructs synthetic training pairs from solitary images. At training time, OVIE lifts a source image into a 3D point cloud using a pretrained monocular depth estimator, applies a sampled camera transformation, and reprojects the result to obtain a partial target view. This pseudo-novel view serves as supervision, with valid pixels restricted to unoccluded regions visible from both perspectives.

This approach necessitates careful handling of disocclusions, those regions visible in the target view but occluded in the source. The authors introduce a masked training formulation that restricts geometric, perceptual, and textural losses to valid regions. Specifically, they employ a masked L1 reconstruction loss, mask the inputs to a perceptual loss network (LPIPS) to ensure feature comparisons occur only on available pixels, and supplement this with a PatchGAN adversarial term to encourage realistic texture synthesis in the disoccluded regions. This masking strategy is critical; without it, the model would learn to hallucinate or average over the unreliable regions of the pseudo-target, degrading performance.

The use of a metric depth estimator, specifically one trained to predict metric rather than relative scale, introduces an underappreciated advantage: metric scale awareness. Traditional structure-from-motion pipelines used to generate training data for view synthesis produce scale-ambiguous reconstructions. By contrast, OVIE's pseudo-pair translations are expressed in metric units, allowing the model to learn physically meaningful camera movements without requiring scale calibration at test time. This property is particularly valuable for robotics applications where metrically consistent view synthesis could aid in path planning and spatial reasoning.

Efficiency Through Geometry-Free Design

Beyond the training paradigm, OVIE distinguishes itself through architectural minimalism. At inference, the model is entirely geometry-free, requiring no depth estimation, no 3D representation, and no per-scene optimization. The architecture accepts a source image and target camera pose as input, producing a synthesized image directly in pixel space through a feed-forward convolutional network. This stands in stark contrast to contemporary approaches that rely on Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting as intermediate representations, which necessitate volumetric rendering or point rasterization during inference.

This design choice yields dramatic efficiency gains. The authors report inference speeds of over 100 FPS, rendering OVIE more than 600 times faster than the second-best baseline in their comparisons. This throughput enables real-time interactive navigation from a single image, a capability that remains computationally prohibitive for methods requiring explicit 3D representations. The trade-off, however, lies in the model's capacity to handle complex view-dependent effects. By bypassing explicit volumetric rendering, OVIE may struggle with phenomena such as specular reflections or translucent surfaces that benefit from ray-traced rendering integrals. The paper acknowledges this implicitly through its focus on largely Lambertian scenes, leaving photorealistic handling of challenging materials as an open problem.

Generalization and the Data Scaling Hypothesis

Perhaps the most compelling results concern OVIE's generalization capabilities. Trained exclusively on in-the-wild images from ImageNet-21K, Places, Open Images, and OpenStreetView5M, the model achieves competitive or superior performance on RealEstate10K and DL3DV benchmarks, despite never having seen posed multi-view data during training. On RealEstate10K, OVIE rivals state-of-the-art methods trained in-domain. On DL3DV, which represents an out-of-domain setting for all compared methods, OVIE surpasses all baselines.

This performance pattern supports a data scaling hypothesis: for monocular view synthesis, the quantity of training data may matter more than precise geometric correspondence. The authors note that data scale appears to outweigh diversity, though broader coverage yields marginal gains. This finding suggests that the field's historical reliance on small, perfectly calibrated multi-view datasets may have been a local optimum, sacrificing the benefits of internet-scale data curation for geometric precision that can be approximated through monocular depth estimation.

Your Take: The Depth Estimator as a Bottleneck and a Bridge

The OVIE framework establishes a fascinating dependency relationship between view synthesis and depth estimation. By using a frozen depth estimator as a teacher, the authors effectively transfer the geometric biases and capabilities of that model into the view synthesis network. This creates a potential compounding error problem: if the depth estimator fails on a particular category of scenes, OVIE receives corrupted supervision. The paper evaluates on relatively benign domains where modern depth estimators excel, but one wonders how the model would perform on domains where depth estimation remains challenging, such as scenes with extreme lighting, mirror reflections, or aerial photography with limited texture.

Furthermore, OVIE's reliance on the static scene assumption, while standard in the field, remains its most significant limitation for real-world deployment. The method cannot handle dynamic elements in training images, meaning the 30 million images must be filtered or accepted with noise from moving objects. This constraint highlights a curious asymmetry: the method achieves internet scale in terms of data quantity, but not in terms of data realism, as it must discard the dynamic, messy reality that constitutes much of visual experience.

The metric scale awareness, however, represents a genuine advance that deserves broader recognition. Most monocular view synthesis methods operate in a normalized coordinate system, requiring post-hoc scale alignment for practical applications. OVIE's implicit metric understanding could facilitate direct integration with SLAM systems or robotic simulators without additional calibration steps, potentially making it more useful for embodied AI applications than higher-fidelity but scale-ambiguous alternatives.

Conclusion

"One View Is Enough" challenges the data bottleneck that has constrained monocular view synthesis to narrow domains. By demonstrating that monocular depth estimation can substitute for multi-view correspondence during training, Ramanana Rahary et al. open a pathway toward training on billions of internet images rather than millions of posed pairs. The resulting model's combination of speed, metric awareness, and out-of-domain robustness suggests a practical near-term future for single-image 3D navigation.

Several questions remain open. How will this approach integrate with diffusion-based generative models, which currently dominate image synthesis but struggle with 3D consistency? Can the framework be extended to handle dynamic scenes through temporal consistency losses or 4D representations? And as depth estimation continues to improve, will the error propagation from pseudo-views diminish sufficiently to enable training on video without static scene assumptions? The field appears to be moving toward a convergence of geometric reasoning and generative modeling, and OVIE provides a compelling template for how geometric scaffolding can enable scalable supervision without sacrificing inference efficiency.