Qize Yu, Lianrui Fan, Bowen Ping, Xini Ding, Zetian Song, Junbo Niu, Kaixuan Wang, Tianxing Chen, Yue Chen, Minghua He, Yuran Wang, Jie Huang, Haojun Zhang, Min Chen, Hao Li, Wenxuan Song, Ruihai Wu, Xianming Liu, Shilong Liu, Shuchang Zhou, Ping Luo, Shiyu Huang
9 min
Abstract
Autoregressive (AR) grounding models serialize spatial predictions, introducing sequential latency and imposing a causal order on output tokens. We view grounding as visual evidence extraction: objects, locations, and spatial relations are jointly constrained by the image and query, yet their dependencies do not imply an intrinsic left-to-right generation order. This distinction makes bidirectional diffusion a natural fit, allowing spatial hypotheses to emerge in parallel and be jointly refined through iterative denoising. We introduce GroundAnything, a 4B-parameter grounding foundation model that reconciles fast parallel decoding with precise localization through blockwise denoising. Training combines grounding pretraining from public datasets and dedicated data engines, direct AR-to-diffusion conversion with joint AR and diffusion objectives, supervised fine-tuning, and GRPO-based reinforcement post-training. Across 30 grounding benchmarks, our autoregressive variant, GroundAnything-VLM, establishes a new overall state of the art among similarly sized models at 72.42%, remaining competitive with GPT-6 Astra (71.35%). With entropy-guided decoding, GroundAnything also surpasses the prior state of the art at this scale, averaging 61.75% versus 53.32% for the fast MTP-based LocateAnything model. We further explore decoding strategies, showing that an optional self-speculative mode achieves a $4.51\times$ speedup over the AR counterpart with a 0.74 percentage-point drop in COCO F1mIoU. Infrastructure experiments show that progressive inference optimizations translate parallel decoding into practical speedups. These support efficient visual grounding in latency-sensitive real-world systems.
Alex: If those positions can all interact, does the model have to reconsider the entire response every time?
Sam: No. It works in blocks. Within a block, attention is bidirectional, so positions can use evidence from both directions. Completed blocks become a fixed prefix for later blocks, and the model reuses their cached computations.
Alex: So the proposal removes a forced order locally, without discarding every kind of order. A trajectory would still need its time sequence.
Sam: The paper explicitly preserves semantic ordering, including trajectory time order. Its concrete example asks for minions, humans, and cars. The response gives boxes for the minions and human, while the absent car category receives a None payload.
Alex: Those outputs combine categories, coordinates, and absence. How do they train one model to handle that mixture?
Sam: They use a shared vocabulary with dedicated coordinate tokens, rather than writing coordinates as ordinary text. Training combines public datasets with annotations from a data engine that fuses multiple teachers' labels and validates them. The engine checks label validity separately from query coverage: an empty result alone doesn't establish absence.
Alex: That handles the supervision. How do they switch a sequential language model into one that fills gaps?
Sam: They retain the pretrained backbone and grounding vocabulary, then add a mask token. Training combines ordinary next-token prediction with recovering masked responses. Both modes share weights, followed by supervised fine-tuning and reinforcement post-training.
Alex: Shared weights explain why a sequential checker is available later. Before that, how broad is the evaluation?
Sam: They evaluate across thirty benchmarks covering detection, referring expressions, pointing, spatial questions, interfaces, text recognition, document layout, and exemplar-based queries. Their own results are averaged over repeated runs with multiple random seeds. Some baseline scores are externally reported, and the paper flags interface and protocol exceptions.
Alex: An overall score can hide different outcomes. What does the headline comparison actually say?
Sam: Their autoregressive version, GroundAnything-VLM, averages about seventy-two percent across the suite. Pure diffusion GroundAnything averages about sixty-two percent, compared with about fifty-three percent for fast LocateAnything. The authors claim a new overall best at comparable model scale for the autoregressive version, and a diffusion result above the prior overall autoregressive best.
Alex: So diffusion beats that earlier overall reference, but not the authors' own sequential model. That seems more useful than saying parallel generation simply wins.
Sam: The task differences matter. Diffusion improves over their autoregressive model on some referring and interface tasks. But text recognition and document layout show larger losses, and tiny-object localization remains challenging. Overall breadth doesn't mean equal precision everywhere.
Alex: Now separate the speed claims. If I hear the headline acceleration, which model and task am I actually hearing about?
Sam: The headline is an optional self-speculative mode on COCO, an object-detection benchmark. Diffusion drafts ahead, and the same model checks the draft autoregressively. It reaches a four-point-five-one-fold speedup over GroundAnything-VLM, with a drop of zero-point-seven-four percentage points in the grounding score.
Alex: What does that grounding score penalize? Fast boxes aren't useful if they're missing objects or loosely placed.
Sam: The score combines an F-one measure, which balances misses and false positives, with intersection-over-union, which measures box overlap. Also, that speed headline is not the mode used in the main benchmark comparisons. Those use pure diffusion, without autoregressive verification.
Alex: Without a checker, how does pure diffusion decide which predictions are ready to keep?
Sam: Its default strategy uses entropy, meaning uncertainty in the distribution over possible tokens. It commits low-uncertainty predictions and leaves others masked for further recovery. Committed tokens stay fixed; if nothing qualifies, it commits the least uncertain position to ensure progress.
Alex: Then parallelism is selective, not simply filling every blank at once. What happens when they push it harder?
Sam: Relaxing the uncertainty threshold increases speed, but quality falls sharply beyond the authors' favored setting on COCO. Larger blocks aren't uniformly better either. And a schedule that reduces network calls can still be slower if each call does much more work.
Alex: The earlier multi-token approach can draft for a checker too. Did they compare drafting methods without changing the checker underneath?
Sam: They matched visual inputs, adaptation data, tokenization, and prefixes, and froze the same causal verifier. Diffusion drafts gained more accepted tokens and lower time per accepted token at the reported operating point. Limited refinement helped, but further refinement cost more than it repaid. Bidirectional multi-token prediction also improved, so the attention benefit wasn't exclusive to diffusion.
Alex: That supports a specific computational advantage, not an unlimited one. What should we still hesitate to conclude?
Sam: These are benchmark and inference experiments, not demonstrated real-world robotic control. The authors explicitly describe the model as research-oriented, not ready for safety-critical deployment. They also measure infrastructure improvements separately from decoding improvements, so those speedups shouldn't be casually multiplied.
Alex: Who should read the full document, and where would you start if time were short?
Sam: Grounding researchers should start with Main Results and examine the task-dependent accuracy gaps. Inference researchers should read Efficient Parallel Inference and Speed Evaluation, especially the matched diffusion-versus-multi-token drafting comparison. Keep pure diffusion separate from the hybrid headline result. For everyone else, remember: spatial evidence need not be inferred in the order it is written.
Alex: Thanks for listening.