Controlled Forgetting: How Sensor2Sensor Solves the Paired Data Problem in Autonomous Driving

Autonomous driving systems live and die by their data. The unglamorous truth of the field is that validating a self-driving stack against the full distribution of real-world events requires staggering volumes of sensor data, and the most valuable slices of that distribution are precisely the ones you almost never see: the erratic merge, the pedestrian who steps off a curb without looking, the debris on the highway at dusk. These long-tail events are rare by definition, and collecting them at fleet scale is prohibitively expensive. A recent paper from Waymo and collaborators, Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving, tackles this scarcity from an unusual angle. Rather than synthesizing new scenarios from scratch, it proposes to convert the enormous existing pool of in-the-wild monocular video into the structured, multi-modal sensor logs that an autonomous vehicle (AV) stack expects.

What makes the paper worth careful attention is not the headline capability of turning dashcam footage into LiDAR and multi-camera output. It is the training trick that makes the whole thing possible. I want to spend most of this article on that trick, because I think it generalizes well beyond driving.

The embodiment gap and why unpaired translation fails

The core problem is structural. In-the-wild data, whether scraped from internet driving videos, partner dashcam fleets, or phone recordings, is grounded in physical reality. The events happened. The dynamics are plausible by construction. And because mundane footage is less likely to be recorded and shared, this data is naturally skewed toward the long tail. That is exactly what you want.

The catch is what the authors call the embodiment gap. A dashcam gives you a single, uncalibrated, monocular video stream. A target AV platform expects eight calibrated cameras providing 360-degree coverage plus LiDAR point clouds. The geometry is misaligned, the modalities are missing, and there is no calibration metadata. Framed naively, this is an unpaired domain translation problem, and a brutally hard one. Classical unpaired methods such as CycleGAN-style approaches lack the geometric priors and the modal capacity to hallucinate a temporally consistent, multi-sensor suite from a single video. The domain gap is simply too wide.

The two established alternatives both have problems. De novo scenario synthesis with generative world models can invent novel events, but it suffers from a plausibility gap (non-physical dynamics) and a realism gap (sensor fidelity too low for validation). Leveraging in-the-wild data avoids the plausibility problem entirely, but runs straight into the embodiment gap. Sensor2Sensor tries to keep the plausibility of real data while re-rendering it into the precise format of a target embodiment.

Controlled forgetting: inverting the data problem

Here is the part I find genuinely clever. To train a model that maps cheap monocular video to rich multi-modal logs, you need paired examples: the same physical moment captured both as a dashcam stream and as a full AV sensor suite. That data does not exist in nature. You cannot mount a dashcam at the exact viewpoint and timing of an AV's sensor rig and capture the identical scene from both perspectives at scale.

So the authors invert the problem. They start with real AV logs, which already contain rich 3D information and 360-degree coverage. From each log they reconstruct a 4D scene representation using dynamic 3D Gaussian Splatting (3DGS). Because the reconstruction is a full spatial-temporal model of the scene, they can then render novel views from arbitrary camera poses, with intrinsics and extrinsics sampled from the real-world distribution of dashcam parameters. The output is a synthetic but realistic dashcam-style video that corresponds exactly to a known, real AV log.

This yields the paired corpus that nature refuses to provide: (synthetic dashcam, real AV log). The model then learns the reverse mapping, from the degraded monocular view back to the full sensor suite, using a conditional diffusion architecture that produces multi-sensor (eight cameras) and multi-modal (camera and LiDAR) output conditioned on the input video.

The principle, as I summarized it earlier, is controlled forgetting. You take a rich data source, deliberately throw away most of its information to mimic a cheap data source, and then train a model to recover what you discarded. The supervision signal is free because you control exactly what was forgotten. This is a recurring and underappreciated pattern in machine learning, and it is the conceptual heart of the paper. Masked autoencoders, denoising objectives, and self-supervised view synthesis all share the same DNA: manufacture the input by corrupting the target.

What distinguishes Sensor2Sensor is that the "forgetting" operation is geometrically principled. The 4DGS reconstruction is not a random crop or noise injection. It is a physically grounded re-rendering through a different virtual camera, which means the synthetic dashcam shares the statistical character of a real one (perspective, field of view, mounting position) while remaining tied to ground-truth 3D geometry. The diffusion model therefore learns a mapping with real geometric structure baked into both ends of the pair.

Strengths, and the seams where I would push

The diffusion choice is well motivated. Diffusion models have proven effective for geometrically aware cross-domain transfer, and their generative capacity is suited to the inherently ambiguous task of hallucinating occluded regions and an entire LiDAR modality from a single view. The evaluation strategy combines a bespoke, manually collected ground-truth dataset for quantitative fidelity with broad qualitative demonstrations on internet and dashcam footage, which is the honest way to validate a system whose real test data is by definition unpaired.

That said, there are seams worth examining.

• The synthetic-to-real gap in the training data. The model is trained on synthetic dashcam views rendered from 3DGS reconstructions, but deployed on real dashcams. Any systematic artifacts in Gaussian Splatting rendering, such as floaters, blurred dynamic objects, or specular failures, become part of the model's learned notion of "dashcam." If real footage differs in its degradation profile, the conversion quality could suffer in ways the synthetic evaluation will not reveal.
• LiDAR is fully hallucinated. There is no LiDAR signal in a monocular video. The generated point clouds are an inference from learned priors and monocular depth cues. For perception training this may be acceptable, but for safety validation, where the entire point is to test against hard cases, a hallucinated LiDAR return on a genuinely novel object is a serious epistemic concern. The model can only generate geometry consistent with what it has seen before.
• The long tail is exactly where priors are weakest. The pitch is that this method unlocks long-tail data. But a diffusion model trained on AV logs has its strongest priors in common scenarios and its weakest in rare ones. The conversion is most reliable precisely where it is least needed, and most uncertain precisely where it matters most. Calibrating and communicating that uncertainty is, in my view, the central open problem for using this in validation.

Why this matters beyond driving

The broader claim I want to make is that the controlled-forgetting recipe should generalize to any cross-embodiment sensor gap where one rich modality can be degraded into a cheaper one. Consider robotics: a fleet of well-instrumented robots with depth cameras, force sensors, and proprioception could generate synthetic monocular-only views, training a model that upgrades cheap commodity-camera robots into the full sensor representation. Consider medical imaging: a high-field MRI degraded to mimic a low-field scanner provides paired data for super-resolution that respects real anatomy. The pattern is the same in each case. Where you have one embodiment rich enough to reconstruct a scene and re-render it through a poorer sensor, you can manufacture supervision for the reverse mapping.

The constraint, and it is a real one, is that the reverse mapping can only recover information that is statistically predictable from the degraded input. Forgetting is reversible only on average. The recovered modality is a plausible reconstruction, not a measurement.

Closing thoughts

Sensor2Sensor is a strong example of solving a data problem by reframing it. Instead of chasing impossible paired collection or accepting the plausibility gap of pure synthesis, it converts a scarce-pairing problem into an abundant-supervision problem through geometrically grounded degradation. The open questions I would most want answered are how the synthetic-to-real dashcam gap affects deployment, and how the system communicates uncertainty on genuinely novel long-tail content, since that is where its priors are thinnest and its stakes are highest. If those can be characterized rigorously, the controlled-forgetting paradigm could become a standard tool for bridging sensor embodiments well outside autonomous driving.