Generating Audio Without a Tokenizer: What WavFlow Tells Us About Compression

For the last several years, the recipe for generating audio with neural networks has been remarkably consistent. Take a raw waveform, compress it into a latent representation using a pretrained variational autoencoder or neural codec, train a diffusion or flow-matching model in that compact space, and then decode back to a waveform. This pipeline has produced impressive systems for text-to-audio, music generation, and Foley-style video-to-audio synthesis. It has also become so standard that few practitioners pause to ask whether the compression step is actually necessary, or merely convenient.

A new paper from Meta AI and Northeastern University, WavFlow: Audio Generation in Waveform Space, takes that question seriously. The authors build a generative model that operates directly on raw waveforms, with no intermediate tokenizer or VAE, and report results that match or exceed established latent-based methods on standard benchmarks. The claim is worth examining carefully, because if true, it implies that one of the most entrenched architectural assumptions in audio generation is optional.

Why Latent Compression Became the Default

There are good reasons the field gravitated toward latent-space modeling, and the WavFlow authors are clear about them.

First, raw waveforms are extremely high-dimensional. Audio sampled at 44.1 kHz produces tens of thousands of samples per second. A ten-second clip is hundreds of thousands of values, far longer than the token sequences transformers comfortably handle. Compressing audio into a latent grid reduces sequence length dramatically, which is essential for tractable attention computation.

Second, waveform amplitudes have an awkward statistical structure. The signal has a high dynamic range but spends most of its time near zero. This produces a poor signal-to-noise ratio for the generative objective: a model trained with standard flow matching against a unit-variance Gaussian prior sees a target distribution whose energy is badly mismatched with the noise it is learning to remove. Latent spaces, by construction, have cleaner statistics that are easier to optimize against.

Third, paired multimodal data is scarce. VGGSound, the workhorse video-to-audio dataset, contains roughly 200,000 samples, about 500 hours. Latent models inherit useful inductive biases from their pretrained tokenizers, so they can learn cross-modal alignment from limited paired data. A model learning acoustic structure from scratch in raw space has no such head start.

These three obstacles are real, and they explain why direct waveform generation has largely been abandoned outside of vocoders. WavFlow's contribution is a set of targeted fixes for each.

How WavFlow Makes Raw-Space Generation Work

The architecture is deliberately plain: a multimodal diffusion transformer (MMDiT) conditioned on visual and text encoders, trained with conditional flow matching. The interesting engineering lives in three components.

Waveform patchify. Rather than treating the waveform as a single long 1D sequence, WavFlow reshapes it into a 2D token grid. This is the audio analog of how vision transformers patchify images. Folding the temporal axis into a grid shortens the effective sequence length seen by attention while preserving local structure within each patch. It is a simple trick, but it directly addresses the dimensionality problem without introducing a learned compression bottleneck.

Amplitude lifting. This is the part that I find most conceptually important. The authors apply RMS normalization and amplitude scaling to lift the near-zero, high-dynamic-range signal into a range where the flow-matching objective is well conditioned. The point is to align the scale of the data distribution with the unit-variance Gaussian prior. Without this, the optimization landscape is dominated by the mismatch between a signal concentrated near zero and noise spread across a much wider range. Amplitude lifting is the quiet hero of the whole approach; it is what makes the rest tractable.

x-prediction under flow matching. Instead of predicting the velocity field or noise, WavFlow predicts the clean signal directly. The authors argue this is a more stable target for raw audio. This connects to a broader thread in the diffusion literature about parameterization choices, where the right prediction target depends heavily on the noise schedule and the data statistics. For a signal as unusual as a raw waveform, direct x-prediction appears to give the model a cleaner gradient signal.

Finally, the data pipeline addresses scarcity. The authors built an automated curation system to filter large-scale media for audio quality and event diversity, producing approximately 5 million video-text-audio triplets. This is more than an order of magnitude beyond VGGSound, and it is plausibly the necessary precondition for any of the architectural tricks to pay off. Raw-space models have less inductive bias, so they need more supervision to compensate.

The Results, and What They Actually Show

On VGGSound for video-to-audio, WavFlow reports FD PaSST of 55.82 at 44.1 kHz and 59.98 at 16 kHz, IS PANNs of 17.40, and a DeSync of 0.44. On AudioCaps for text-to-audio, it reports FD PANNs of 10.63 and IS PANNs of 12.62. The headline is that these numbers are competitive with, and in some cases better than, strong latent-based systems such as MMAudio.

A few observations on interpreting these metrics. Frechet Distance scores measure distributional similarity to real audio in a feature space, while Inception Score proxies for diversity and recognizability. DeSync measures temporal misalignment between generated audio and video, which is the hardest part of Foley generation. A DeSync of 0.44 is competitive but not dominant; this is consistent with the intuition that synchronization is bottlenecked by the visual conditioning and the model's temporal modeling, not by the audio representation per se.

The text-to-audio results are notable because the authors obtain them by simply zeroing out the visual condition. This suggests the model learned a reasonably general audio prior rather than a narrowly video-coupled one.

My Take

The framing in the accompanying summary, that the visual cortex works on raw signal rather than a fixed compression codec, is a useful intuition pump, though I would not push the biological analogy too far. The deeper and more defensible lesson is the one the paper states plainly: intermediate compression is a convenience, not a law.

What WavFlow really demonstrates is that the difficulty of raw-space generation was never fundamental; it was a conditioning and data problem disguised as a representation problem. Once you fix the amplitude statistics and supply enough data, a standard transformer with flow matching handles the rest. This mirrors a recurring pattern in deep learning, where capabilities once thought to require specialized inductive biases turn out to require mostly scale and careful normalization.

I would temper the enthusiasm with a cost observation that the paper underplays. Eliminating the VAE simplifies the pipeline conceptually, but operating on longer sequences and relying on 5 million curated samples shifts the cost elsewhere, into compute and data engineering. The simplicity is architectural, not necessarily computational. Whether raw-space generation is more efficient than latent generation at matched quality remains an open empirical question, and the paper does not give a clean compute comparison.

There is also a subtle dependency worth flagging: the result rests heavily on a proprietary 5M-sample dataset. Reproducing WavFlow's quality without that curated data may be difficult, which makes it harder to disentangle the contribution of the architecture from the contribution of the data.

Closing Thoughts

WavFlow is a clean refutation of an assumption that hardened into dogma. It does not prove that latent compression is bad; latent spaces still offer real efficiency advantages and may remain preferable in compute-constrained settings. What it proves is that compression is not a prerequisite for fidelity, and that the historical difficulties of raw-space modeling were addressable with modest, well-targeted interventions.

The open questions I find most interesting going forward: How does raw-space generation scale to longer durations, where sequence length again becomes punishing? Can amplitude lifting and patchify transfer to music, where harmonic structure is more demanding than environmental Foley? And does removing the tokenizer change the failure modes, producing artifacts that latent decoders would have smoothed away? Those answers will determine whether tokenizer-free audio generation becomes a genuine alternative or a clever proof of concept.