Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

Vision-Language Models (VLMs) are increasingly used as the backbone for diffusion-based image editing, where they process text instructions to guide the generation of modified images. However, these models often struggle to accurately localize edits in complex, multi-entity scenes. This paper identifies a critical discrepancy: while a standalone VLM can accurately localize objects when allowed to generate text autoregressively, the same model fails to maintain this accuracy when restricted to a single forward pass—the standard operating mode for current editing pipelines. The authors hypothesize that this failure occurs because the spatial information required for precise localization is not being effectively captured by the final-layer representations typically used for conditioning.

To investigate where this spatial knowledge resides, the authors introduce the Analysis-by-Proxy framework. Instead of relying on the VLM's final output, they train a lightweight, interpretable proxy model (a Q-Former) to predict target bounding boxes directly from the VLM's intermediate hidden states. This approach allows the researchers to probe the model's internal representations without requiring autoregressive text generation. By training this proxy, they reveal that localization signals are not uniformly distributed; rather, they are concentrated in specific intermediate layers and are highly sparse across the token sequence, focusing on semantically significant nouns and adjectives.

Building on these findings, the authors demonstrate that the localization failure is not an inherent limitation of the VLM, but a failure of the extraction strategy. By using their proxy to identify the optimal, input-dependent layer for extracting spatial signals, they can generate a bounding box that acts as an explicit spatial marker. When this marker is overlaid onto the source image and fed into the diffusion model, the editing pipeline achieves significantly higher localization accuracy. This method reconciles the gap between the VLM's internal spatial reasoning and the final image output, providing a more principled approach to designing conditioning architectures.