Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models

Vision-Language-Action (VLA) models typically build action generation on top of pretrained vision-language backbones. While this allows for rapid progress, it creates a structural mismatch: the inherited multimodal pathway must support both language understanding and action control. The authors investigate whether action supervision, when applied directly to these inherited representations, inadvertently disrupts the semantic and grounding structures necessary for language-side processing.

The authors introduce Action QFormer, a query-based interface inserted between the pretrained backbone and the policy head. Instead of allowing action-loss gradients to flow directly into the inherited multimodal pathway, Action QFormer uses learnable queries conditioned on instructions to selectively extract and reorganize visual information. This design creates an intermediate adaptation path, allowing the model to optimize action-facing representations without requiring broad, potentially destructive, updates to the upstream backbone.

In zero-shot sim-to-real navigation, Action QFormer significantly outperforms the direct-fusion baseline, increasing closed-loop task success from 18.8% to 56.3% and improving action-generation correctness from 22.5% to 75.5%. Mechanistic analysis reveals that the direct-fusion approach often causes broad upstream rewriting of the backbone, which degrades object grounding and instruction-following capabilities. By contrast, Action QFormer localizes the action-supervised adaptation to the query interface, preserving the stability of the inherited multimodal representations while still enabling effective control.

This work shifts the focus in VLA research from merely scaling backbones to understanding the interaction between action supervision and inherited representations. It demonstrates that the design of the action-facing interface is critical for maintaining the integrity of pretrained models, suggesting that future VLA architectures should prioritize structured, query-based interfaces to control how action-loss signals reshape multimodal knowledge.