Grounded World Models: Reimagining Visuomotor Planning Through Language Alignment

Current Vision Language Action models face a fundamental tension. They inherit rich semantic knowledge from pretrained Vision Language Models, yet this knowledge often degrades during fine-tuning for robotic control. The paper "Grounded World Model for Semantically Generalizable Planning" proposes an elegant solution that sidesteps this tension entirely. Rather than fine-tuning a VLM to generate actions, the authors leverage a frozen multimodal embedding space to ground world model predictions in natural language.

The core insight involves transforming Model Predictive Control from an image matching problem into a semantic retrieval task. Traditional latent world models like DINO-WM and JEPA-WM require goal images to compute planning costs, limiting their applicability to scenarios where demonstrations are available. GWM instead operates in the shared embedding space of Qwen3-VL-Embedding, allowing cosine similarity between predicted futures and text instructions to guide action selection.

Decoupling Dynamics from Semantics

The architectural innovation of GWM lies in its strict separation between world dynamics and semantic understanding. While conventional VLAs fine-tune pretrained models like LLaVA or Qwen on robot trajectories, risking catastrophic forgetting of visual concepts, GWM treats the foundation model as a fixed metric space.

Specifically, GWM learns a state transition function that maps current states and actions into the latent space of Qwen3-VL-Embedding. This foundation model encodes images, text, and video into a unified space where semantic relationships are preserved through cosine similarity. During inference, the system proposes candidate action trajectories, predicts their visual outcomes using the learned dynamics model, and scores each prediction against the language instruction.

This approach offers three technical advantages:

• The rendering-based action encoder requires no training and generalizes across robot embodiments. The authors demonstrate zero-shot transfer to the xArm6 robot despite differences in kinematics, action space, and visual appearance.
• By avoiding gradient updates to the foundation model, GWM preserves the world knowledge encoded in Qwen3-VL-Embedding.
• The MPC framework naturally handles multimodal trajectory distributions, selecting actions based on their predicted semantic alignment rather than mode-seeking behavior common in direct policy learning.

The WISER Benchmark and the Generalization Gap

To quantify semantic generalization, the authors introduce WISER, a benchmark comprising 288 test tasks across 24 categories of world knowledge including numbers, food, animals, and landmarks. The benchmark design is particularly clever; test tasks use unseen visual signals and referring expressions, yet require only motions demonstrated during training. This isolates semantic generalization from motor skill acquisition.

The results reveal a stark generalization gap in traditional VLAs. While these models achieve 90% success on training tasks, their performance drops to 22% on test tasks. Some architectures fail completely on novel combinations of objects and instructions despite perfect training performance. This suggests that fine-tuning for action generation