Closing the Perception Gap: How Synthetic VQA from a Single Keyword Improves VLMs

Vision-language models have become competent at a surprising range of tasks: reading text in images, answering complex questions, grounding interface elements, even solving multimodal math problems. Yet they continue to stumble on tasks that a young child handles effortlessly. Which way is the giraffe facing? Which of these two objects is closer to the camera? Is the cup to the left or right of the plate? These are not reasoning failures in the conventional sense. They are perception failures, and they have proven stubbornly resistant to scale.

A new paper, VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images (Zhou et al., Princeton University and NYU), makes a focused argument about why this gap persists and what to do about it. The claim is simple and, I think, mostly correct: the bottleneck is not model capacity but supervision coverage. Natural image corpora, however vast, do not systematically sample the combinatorial space of viewpoints, depth orderings, and spatial relations that robust low-level perception requires. The proposed fix is to generate that coverage synthetically, starting from nothing more than a task name.

The Pipeline: From Keyword to Verified Triples

The core contribution is a closed-loop generation system that takes a single task keyword, such as Depth Order, and produces verifier-filtered image-question-answer triples with no reference images and no human annotation. The pipeline composes three components:

• An LLM generates question-answer pairs and detailed text-to-image (T2I) prompts conditioned on the target task. The system maintains an adaptive concept pool spanning objects, attributes, scenes, and styles (for example, "a red apple on a wooden table in oil painting style"), which gives the generator combinatorial reach over visual variations that are rare in real datasets.
• A modern T2I model synthesizes images conditioned on those prompts.
• A frontier VLM acts as a judge, verifying that the generated image actually supports the answer-determining visual statement. Misaligned samples are filtered out, with a fallback mechanism for handling failures.

The elegance here is that the entire supervision signal is bootstrapped from the cross-modal capabilities already present in T2I and judge models. The LLM knows what a depth-ordering question looks like; the T2I model can render a scene with a clear foreground and background; the judge can check whether the rendering matches the intended answer. No human ever labels an image. This is the visual analogue of the synthetic-SFT recipes that reshaped instruction-following in language models, where models like Phi were trained substantially on generated data (Gunasekar et al., 2023; Abdin et al., 2024).

Using this pipeline, the authors build VisionFoundry-10K: 10,000 triples across 10 perception tasks, 1,000 per task, covering spatial understanding, relative depth, viewpoint variation, color, object state, and related low-level skills. The task selection is deliberately motivated by the failure modes documented in diagnostic benchmarks such as MMVP and CV-Bench (Tong et al., 2024a,b) and RealWorldQA (xAI, 2024).

The Results and What They Actually Show

Finetuning on VisionFoundry-10K produces consistent gains across three open-source models of different scales and architectures: Qwen2.5-VL-3B-Instruct, Llama-3.2-11B-Vision-Instruct, and MiMo-VL-7B-SFT. The headline numbers are +7% on MMVP and +10% on CV-Bench-3D. Three aspects of the evaluation are worth highlighting because they distinguish this from a typical "we finetuned and the number went up" result.

First, broader capabilities are largely preserved. The authors report benchmark-wise results across multimodal reasoning, OCR, and GUI grounding. Changes are benchmark-dependent: substantial gains on some, slight fluctuations on others. This matters because targeted finetuning often causes catastrophic interference, trading the target skill against everything else. The reported absence of severe regression suggests the synthetic supervision is adding a genuinely complementary signal rather than overwriting existing competence.

Second, scaling behaves predictably. For a representative task, performance improves as synthetic data size increases. A clean data-size trend is exactly what you want to see if the method is to become infrastructure rather than a one-off trick. It implies the gains are not an artifact of a lucky data mixture but a function of coverage that can be dialed up.

Third, and most telling, an equal-sized synthetic-natural mixture outperforms pure natural subsets. This is the cleanest evidence for the paper's central thesis. If natural data already contained the perceptual signal and synthetic data were merely a noisier substitute, you would expect the natural subset to win or tie at equal size. Instead the synthetic samples provide something the real data lacks. That something is targeted, balanced coverage of the perceptual variations that natural corpora sample only incidentally.

My Take: Coverage, Not Capacity

The framing that I find most defensible is the one in the accompanying summary: the bottleneck was never model capacity, it was supervision coverage. This deserves elaboration because it reframes a problem the field has often misdiagnosed.

When a 7B or 11B VLM cannot tell which of two objects is closer, the instinctive explanation is that the visual encoder is too weak or the model is too small. But MMVP-style benchmarks were specifically designed to decouple perception from language priors, and the persistence of these failures across model scales argues against a pure capacity story. A model that can perform GUI grounding and read dense text clearly has rich visual features available. What it lacks is a learned mapping from those features to explicit perceptual judgments, because the training distribution rarely demanded that mapping. Depth ordering is implicit in almost every natural image but explicitly labeled in almost none.

This connects VisionFoundry to a broader principle that I expect to generalize. Synthetic data is most valuable precisely where the supervision is sparse in nature but cheap to specify. Spatial relations, viewpoints, and depth are easy to render with a T2I model and easy to verify with a judge, but tedious and inconsistent for humans to annotate. The same logic explains why synthetic SFT worked so well for instruction-following: the target format was easy to generate and verify, even though clean examples were scarce in raw web text.

There is also a subtle methodological point about using a frontier VLM as the verifier. This introduces a teacher-student dynamic. The student model can only learn perceptual distinctions the judge can reliably check. That implies an asymmetry worth watching: if the judge itself has a perceptual blind spot, the synthetic curriculum will inherit it, and no amount of scaling fixes a systematically mislabeled axis. The verification step is the load-bearing component of the whole pipeline, and its reliability ceiling is the pipeline's reliability ceiling.

Limitations and Open Questions

Several caveats temper my enthusiasm. The 10-task, 1k-per-task setting is explicitly described as an initial validation, and the gains, while consistent, are modest in absolute terms. T2I models have their own failure modes, particularly with counting, fine spatial precision, and unusual viewpoints, which are exactly the hard cases. A pipeline that can only generate the perceptual configurations its T2I model renders cleanly may struggle to cover the long tail where models fail most. There is also the question of whether synthetic-image artifacts induce a distribution shift that helps benchmarks but transfers imperfectly to messy real photographs.

The deeper open question is compositional generalization. The adaptive concept pool gives combinatorial reach over individual axes, but does training on enumerated combinations teach the underlying perceptual primitive, or does it teach a lookup table over seen combinations? The favorable scaling trend is encouraging but does not settle this.

Looking Ahead

If the coverage thesis holds, the natural endpoint is what I would call a synthetic perception curriculum: specify a skill taxonomy, auto-generate millions of verified triples, and fold them into pretraining rather than late-stage finetuning. VisionFoundry is a small, controlled demonstration that this direction works, that the limiting reagent is targeted supervision, and that we can manufacture it on demand. The interesting work now is in scaling the verifier's reliability, expanding the task taxonomy to the genuinely hard cases T2I models render poorly, and demonstrating that the learned skills transfer to real-world distributions rather than to synthetic-friendly benchmarks alone. Those are tractable engineering problems, which is the most optimistic thing one can say about a research direction.