Leilei Wang, Longfei Liu, Xi Shen, Xuanlong Yu, Ying Tiffany He, Fei Richard Yu, Yingyi Chen
7 min
Abstract
Real-time open-vocabulary object detection (OVOD) is essential for practical deployment in dynamic environments, where models must recognize a large and evolving set of categories under strict latency constraints. Current real-time OVOD methods are predominantly built upon YOLO-style models. In contrast, real-time DETR-based methods still lag behind in terms of inference latency, model lightweightness, and overall performance. In this work, we present OV-DEIM, an end-to-end DETR-style open-vocabulary detector built upon the recent DEIMv2 framework with integrated vision-language modeling for efficient open-vocabulary inference. We further introduce a simple query supplement strategy that improves Fixed AP without compromising inference speed. Beyond architectural improvements, we introduce GridSynthetic, a simple yet effective data augmentation strategy that composes multiple training samples into structured image grids. By exposing the model to richer object co-occurrence patterns and spatial layouts within a single forward pass, GridSynthetic mitigates the negative impact of noisy localization signals on the classification loss and improves semantic discrimination, particularly for rare categories. Extensive experiments demonstrate that OV-DEIM achieves state-of-the-art performance on open-vocabulary detection benchmarks, delivering superior efficiency and notable improvements on challenging rare categories. Code and pretrained models are available at https://github.com/wleilei/OV-DEIM.
Sam: The model starts by pulling visual details from different scales of the image—like zooming in and out to catch both big and small objects—and fuses them into a flat map of possible spots. From that map, it selects a small, fixed number of starting points called object queries, which act like educated guesses on where objects could be, avoiding the overload of checking every pixel. To make it text-aware, it compares each query's visual traits to the text description's features using a simple similarity measure, picking the top matches to refine further.
Alex: So those queries get ranked by how well their looks match the words—like sorting flashcards by description before studying?
Sam: Yes. A lightweight part called the vision-text alignment head computes those matches: it takes the query's image features and the text's embedding, measures their closeness with cosine similarity, and uses that as the classification score. During training, this drives a contrastive loss where the right text-object pair scores highest against others, aligning vision and language without fixed categories.
Alex: Got it—the encoder mixes scales, queries get text-filtered, and alignment scores say 'yes, that's the pallet jack.' And to keep training stable?
Sam: They add a denoising loss—where noisy position hints are given and cleaned up—and an auxiliary loss for extra supervision on intermediate steps. These help the decoder converge faster without extra runtime cost. Overall, this setup lets DETR-style detection scale cleanly to open-vocabulary, vocabulary-independent inference.
Alex: And no post-filtering mess like YOLO... that efficiency adds up in real-time scenarios. What exactly in the DETR-style setup gives it better discrimination for those infrequent objects?
Sam: DETR-style detectors use a transformer architecture, which works like a smart attention system—much like how language models connect words by focusing on relevant parts. Here, it lets the model look across the whole image and predict a direct, clean set of objects all at once, without generating tons of overlapping guesses that need cleanup. This skips the non-maximum suppression step YOLO needs, cutting delay while sharpening focus on what objects truly look like, which helps rare ones stand out semantically against common clutter.
Alex: So the direct prediction keeps things clean and lets semantics shine through... makes sense for sparse data like pallet jacks. That edge on rare categories is notable. But what makes GridSynthetic particularly effective for those rare categories—beyond just the clean layout?
Sam: In standard training, the loss function weighs how well the predicted box matches the real object location—using a quality score that drops sharply if the match is off. When that score is low or middling, it either skips teaching the model about the object's look entirely or forces it to blend in background junk as part of the category, muddying what "pallet jack" really means visually. GridSynthetic fixes this by placing objects perfectly on the grid, so the location score hits the ideal case—letting the model drill pure category matching without distractions.
Alex: So poor box fits starve or confuse the learning on what matters... and the grid enforces perfect fits to teach clean visuals. LVIS sounds like a tough real-world test. How does it stack up there specifically?
Sam: The paper tests on LVIS, a benchmark with 1203 categories heavy on infrequent ones, using average precision on rares as the key measure. On the rare split, OV-DEIM beats YOLO mid-size models notably while running over five times faster end-to-end. This comes from the clean predictions skipping post-filter cleanup, paired with grid-sharpened semantics.
Alex: Huh. That dense prediction allowance reveals the edge in cluttered warehouses. But in real-world use, like that warehouse robot, are there scenes where the fixed number of predictions might fall short?
Sam: Yes, the decoder works with a fixed set of around 300 object guesses per image, which handles most cases efficiently. In extremely crowded scenes with far more objects—like a packed shelf with hundreds of tiny items—it could miss some beyond that limit, as it doesn't generate unlimited predictions like denser YOLO scans. The paper notes this as a trade-off for speed.
Alex: So the fixed slots keep it fast but cap dense recall. Practically, beyond warehouses, where else does this deploy well?
Sam: In AR glasses, for instance, it could name thousands of everyday objects instantly from text prompts, enabling open-world vision without retraining—like spotting a rare tool or plant on sight. The clean, NMS-free predictions make it suitable for edge devices needing low latency. As a baseline, it sets a notable standard for future real-time OVOD refinements.
Alex: That's a grounded step forward—balancing speed, rares, and scalability without overreaching.
Sam: Precisely. The paper's contribution lies in making DETR-style viable for real-time open-vocabulary detection, especially on infrequent categories. That's the core takeaway from OV-DEIM.
Alex: Well put, Sam. Thanks for joining us on ResearchPod.