Compressing 3D Scenes for Vision-Language Models: A Closer Look at Proxy3D

Spatial intelligence remains one of the more stubborn gaps in multimodal large language models. A model can describe a photograph with impressive fluency, yet ask it to reason about the relative positions of objects in a room it has just watched on video, and the answers degrade quickly. The recent paper Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment by Jiang, Sun, Gudovskiy, and colleagues from Tsinghua, Panasonic AI Lab, and UC Berkeley takes a direct approach to one specific bottleneck: how vision information is represented and serialized before it reaches the language model. Their answer is to stop treating 3D scenes as long sequences of pixel-aligned tokens and instead compress them into a compact set of semantically meaningful proxies.

This is a representation problem dressed up as an efficiency problem, and I think that framing is exactly right.

The Representation Bottleneck in 3D VLMs

To understand why Proxy3D matters, it helps to separate the two dominant families of 3D vision-language models that the paper contrasts.

The first family is correspondence-based. Methods like 3DRS, Video-3D LLM, SR-3D, and Ross3D develop a sense of 3D space implicitly. They feed the model multiple video frames and rely on cross-frame feature matching or 3D positional encodings to let the language model infer spatial structure. The appeal is that they stay close to the conventional 2D VLM pipeline. The problem, as the authors point out citing Yang et al., is that these models tend to form a collection of local world models anchored to the ego perspective rather than a single coherent global model of the scene. They also use training data inefficiently and carry high computational cost because the vision sequence is long.

The second family is representation-based. These methods explicitly model the 3D scene using point clouds, depth maps, 3D Gaussian splats, or scene graphs. They inject genuine geometric priors, but they struggle with serialization. Converting an explicit 3D structure into a token sequence that a language model can ingest, without ballooning the sequence length or relying on a heavy neural serializer, is the open challenge the paper identifies directly: how to construct a sequence of tokens with accurate spatial information while keeping it small.

Proxy3D's central observation is deceptively simple. The encoded vision modality has a sparse semantic distribution. A room contains a finite number of meaningful entities, and most of the raw pixel-aligned features are redundant. If the semantic content is sparse, then latent-space clustering should be able to compress it without much loss.

How Proxy3D Works

The pipeline, given only video frames as input, proceeds in three conceptual stages.

• Foundation encoders. The method runs the frames through pretrained semantic and geometric encoders. The semantic encoder captures what objects are, while the geometric encoder captures where they sit in space. This dual-encoder design is the mechanism by which the model gets both appearance and 3D structure without reconstructing a full explicit scene.

• Semantic-aware clustering into proxies. Rather than serializing every pixel-aligned feature, Proxy3D clusters features in latent space to produce a compact set of 3D proxy tokens. Each proxy aggregates a semantically coherent region of the scene. This is the key move: it replaces a learned neural serializer (as in some prior work) with a clustering operation that compresses the sequence while preserving spatial grounding through a 3D spatial positional encoding.

• Multi-stage alignment with the LLM. The proxies are aligned to the language model through a progressive, iterative training pipeline. The authors curate the SpaceSpan dataset, a 318K-sample collection built on top of eight or more existing sources (ScanQA, SQA3D, Scan2Cap, ScanRefer, Multi3DRefer, MMScan, and others) and reformatted into a unified structure rich in spatial reasoning queries. The dataset includes quantitative QA (for example, asking for the direct distance in meters between two objects), captioning, and visual grounding tasks, along with object-object spatial relationship pairs that are typically underrepresented in 3D training data.

The reported outcome is that Proxy3D achieves competitive or state-of-the-art performance on 3D QA, visual grounding, and general spatial intelligence benchmarks while using shorter vision sequences. That last point is the crux of the contribution. If you can match or beat longer-sequence baselines with fewer tokens, you have demonstrated that the limiting factor was representation quality, not raw token budget.

Why This Matters Beyond the Benchmarks

The practical implication is straightforward and worth stating plainly. Vision sequence length drives both latency and memory in transformer-based VLMs because attention scales quadratically with sequence length. For any embodied application, a robot navigating a kitchen or AR glasses reasoning about a room in real time, the cost of long pixel-aligned token sequences is prohibitive. Proxy3D's compression is not merely an academic nicety; it is the kind of change that moves a model from offline analysis toward on-device or near-real-time deployment.

But I want to push on the deeper claim, because I think it is more interesting than the efficiency story. The result suggests that spatial intelligence scales with representation quality rather than token count. This echoes a pattern we have seen elsewhere in the field. In 2D vision, the move from dense patch grids to fewer, semantically organized tokens (as in some perceiver-style architectures and learned token reduction methods) often preserved accuracy. The fact that the same holds in 3D, where one might assume spatial precision demands dense coverage, is a meaningful empirical signal. It implies that the language model does not need a high-resolution scan of every surface; it needs a small set of well-grounded object-level anchors and reliable relative positions between them.

This connects naturally to the failure mode that Yang et al. identified. If correspondence-based models fail to form a unified global scene model, it may be precisely because their representation never commits to a stable global structure. By clustering into proxies positioned in a shared 3D coordinate frame, Proxy3D is, in effect, forcing a single global representation rather than letting the model stitch together ego-centric fragments. The architecture nudges the model toward the kind of allocentric, scene-level reasoning that humans use.

Limitations and Open Questions

There are honest caveats. The clustering step introduces a fixed bottleneck, and the quality of proxies depends heavily on the pretrained encoders chosen. If the semantic encoder misses fine-grained distinctions, those distinctions are gone before the language model ever sees them. Quantitative QA that demands precise distances may be especially sensitive to how aggressively the scene is compressed. The paper's strong numbers on distance-style questions, if they hold up, would be reassuring evidence that the geometric encoder and spatial positional encoding compensate for this.

A second open question concerns failure modes in cluttered or repetitive scenes. Semantic clustering assumes the scene is sparse in distinct semantics. A warehouse full of identical boxes or a dense office of similar chairs may break that assumption, collapsing distinct objects into a single proxy. How gracefully the method degrades in such cases is not obvious from the framing.

Finally, the heavy reliance on curated datasets built from ScanNet derivatives raises the usual generalization question. ScanNet-based benchmarks share statistical regularities, and strong performance there does not guarantee transfer to unconstrained, in-the-wild video.

Closing Thoughts

Proxy3D fits into a broader and welcome trend: treating tokenization and representation design as first-class problems rather than implementation details. The most interesting takeaway is conceptual. We have been compensating for weak 3D representations by throwing more tokens at the language model, and this work suggests that strategy was masking the real issue. A compact, semantically clustered, spatially grounded representation may serve spatial reasoning better than an exhaustive pixel dump.

The forward-looking question I find most compelling is whether proxy representations can become dynamic, updating as an agent moves and gathers new views, rather than computed once over a fixed video. A robot does not perceive a room in a single batch; it accumulates and revises. If proxy tokens can be incrementally maintained, Proxy3D's compression could become the foundation for streaming spatial memory in embodied agents. That, more than any single benchmark number, is where this line of work could prove most consequential.