When the Safety Feature Becomes the Attack Surface: Watermarking Autoregressive Image Generators

Introduction

Watermarking has become one of the default proposals for managing the externalities of generative models. If we cannot stop synthetic images from flooding the internet, the reasoning goes, we can at least mark them at the source, embedding a signal during generation that downstream detectors can later recover. For autoregressive (AR) image generators, this promise is doubled. In-generation watermarks are supposed to support both provenance (attributing an image to a model) and dataset hygiene (filtering synthetic content out of training corpora to prevent model collapse).

The paper "On the Robustness of Watermarking for Autoregressive Image Generation" by Müller et al. takes a hard look at whether these schemes actually deliver on that promise. The short answer is no. The authors show that with access to a single watermarked reference image, and crucially without access to model weights or the secret watermarking parameters, an adversary can both remove the watermark and forge it onto unrelated images. The forgery result, which the authors call Watermark Mimicry, inverts the entire protective logic of the system. I want to walk through how these attacks work, why the threat model was misframed from the start, and what this tells us about the broader project of signal-based provenance.

How AR Watermarking Works, and Where It Breaks

Modern AR image generators do not predict pixels directly. They operate in a latent space, representing a high-resolution image as a grid of discrete tokens drawn from a learned codebook, then decoding those tokens back to pixels. Generation proceeds token by token, with each token sampled from a categorical distribution produced by a softmax over network logits, conditioned on previously generated tokens. Models like VAR, HMAR, and Infinity follow variants of this recipe.

In-generation watermarking exploits exactly this sampling step. Rather than altering the image after the fact, the watermark biases the token selection process, slightly shifting the probabilities of which tokens get chosen so that the resulting token sequence carries a hidden statistical signature. Because the signal is baked into the generation trajectory rather than stamped on top of pixels, these watermarks tend to be more robust than post-hoc methods against the usual battery of perturbations: JPEG compression, additive noise, color jitter, and even diffusion-based regeneration attacks.

This robustness is real, and it is the basis for the field's confidence. The problem is that robustness against generic transformations says nothing about robustness against attacks that target the watermarking mechanism itself. The paper introduces three new attacks that do exactly this:

• Vector-quantized regeneration removal. Since AR generators quantize images into codebook tokens, an adversary can re-encode a watermarked image through a VQ pipeline, effectively re-tokenizing it and disrupting the carefully tuned token statistics that encode the watermark. This is a more surgical version of the regeneration attack, exploiting the same discretization the generator relies on.
• Adversarial optimization based attack. Here the attacker directly optimizes a perturbation to push the image across the detector's decision boundary, either away from the watermarked region (removal) or into it (forgery), while constraining perceptual change.
• Frequency injection attack. This targets the spectral signature of the watermark, injecting or removing energy in specific frequency bands to manipulate detection.

The combination matters. Removal attacks defeat provenance. Forgery attacks defeat dataset hygiene. The paper demonstrates both with minimal adversary capabilities.

Watermark Mimicry: The Quiet Danger

The forgery result deserves the most attention, because it is the one that turns a safety feature into a weapon. The relevant target is BitMark, a radioactive watermarking scheme for bitwise autoregressive generation. The word "radioactive" here refers to the idea that a watermark survives being used as training data: if a watermarked image is consumed by a future model, the watermark contaminates that model in a detectable way. The intended use is defensive. A service provider scans candidate training images, discards those carrying the generator's watermark, and thereby avoids feeding its own synthetic outputs back into training, which is the recipe for model collapse.

Watermark Mimicry inverts this. The paper shows that an adversary, with a single watermarked reference, can transfer that watermark onto arbitrary authentic images. The detector then flags those genuine photos as AI-generated and excludes them from training. The paper's own illustration uses a photograph of a poison frog, an apt choice given that the strategy is essentially mimicry in the biological sense. The image is real, but it wears the costume of a synthetic one.

Consider what this enables in practice. A photographer, an artist, or an organization that does not want its imagery harvested for model training no longer needs to argue about copyright or robots.txt. They can simply stamp the generator's watermark onto their own work and let the model builder's own filter do the excluding for them. What was designed as a shield against model collapse becomes a vector for data poisoning by exclusion, where the poison is invisible and the system removes the data voluntarily.

There is a second consequence the authors connect to earlier work on diffusion watermark forgery: forgery can discredit evidence. If anyone can forge a generator's watermark onto an arbitrary image, then the presence of the watermark stops being proof of anything. A real photograph carrying a forged mark can be dismissed as "obviously AI-generated," and a malicious synthetic image can be passed off as authentic by removing the mark. Provenance claims become contestable in both directions.

My Take: The Threat Model Was Backwards

The deeper issue here is not any single attack but the structure of the assumption underneath the whole approach. In-generation watermarking implicitly treats the embedded signal as a secret whose security rests on the adversary not knowing the watermarking parameters. But the security actually rests on something much weaker: the cost of observing and reproducing the signal's effect on an image.

The paper's most damaging finding is the single-reference requirement. One watermarked image is enough. This is the watermarking equivalent of a one-time-pad reused once. If a single sample reveals enough structure to both subtract and replicate the signal, then the secret parameters are irrelevant. The detector defines a decision boundary in image space, and any adversary who can probe that boundary can navigate it. The authors confirm that you cannot patch this by adjusting decision boundaries, because tightening the threshold to resist forgery makes legitimate detection fail, and loosening it to catch real watermarks makes forgery easier. This is the standard tension between false positives and false negatives, but here both error modes are independently attacker-controllable.

I would push the inference one step further. Any watermark designed to be robust to perturbation is, almost by construction, easy to forge. Robustness means the signal survives many transformations, which means the signal occupies a large, stable region of feature space. A large stable region is exactly what makes a target easy to hit with an optimization attack. The very property that makes these watermarks attractive for provenance is the property that makes mimicry tractable. You cannot maximize robustness and forgery resistance simultaneously without a genuine secret, and a single observable reference erases the secret.

Limitations and Open Questions

To be fair to the watermarking schemes, the paper does not claim every attack succeeds in every configuration, and image quality after removal is described as having "limited impact" rather than zero impact. There remains a perceptual budget, and a defender who could enforce strict quality requirements on training data might filter some cruder attacks. The evaluation also focuses on specific AR architectures and the BitMark family; whether future schemes with cryptographically bound, image-dependent signals could resist single-reference forgery is genuinely open.

That last point is where I would direct future work. The failure here is the use of a fixed signal that is the same across generated images. A watermark that is cryptographically tied to the content of each individual image, such that the mark on image A reveals nothing about the valid mark on image B, would defeat single-reference transfer. The question is whether such a scheme can retain the robustness that made in-generation watermarking appealing in the first place. My suspicion is that it cannot, at least not fully, and that the field will keep rediscovering the same lesson: detection built on subtle embedded signals is only as strong as the cheapest available spoof.

For now, anyone proposing to use AR watermarks for dataset filtering should treat the filter itself as an attack surface, not a defense.