NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision Language Models

As vision-language models (VLMs) like YOLO-World and Grounding DINO become the standard for object detection, their ability to handle objects outside their training distribution (OOD) remains a critical safety concern. Standard VLM detectors are optimized for in-distribution (ID) accuracy and often treat background regions uniformly, failing to distinguish between true background and unseen OOD objects. This paper investigates how to adapt these models to reliably detect OOD objects without sacrificing ID performance.

The authors propose Negative Label Guided Attention and Scoring (NegAS), which introduces two primary components:

1. Negative Label Guided Attention (NegA): The researchers use an LLM to generate a set of negative labels—categories that are visually similar to ID classes but semantically distinct. These labels are used to create an auxiliary attention branch that forces the model to focus on background regions that might contain OOD objects.
2. Negative Label Guided Scoring (NegS): Because VLM detectors typically use sigmoid-based outputs rather than the softmax outputs common in traditional OOD classification, the authors developed a custom scoring function. This function uses both ID and negative labels to produce a robust OOD score, effectively suppressing overconfident predictions on unseen data.

During training, the model uses these negative labels to refine its decision boundaries. Crucially, the NegA branch is discarded at inference time, meaning the performance gains come with no additional computational overhead during deployment.

This work is the first to systematically address OOD detection within the VLM-based object detection paradigm. By bridging the gap between dense detectors (like YOLO-World) and query-based transformers (like Grounding DINO), the authors demonstrate that their approach is highly generalizable. The significant reduction in false positive rates (e.g., 25.5% on OpenImages) suggests that NegAS is a practical, low-cost solution for improving the reliability of AI systems in safety-critical environments like autonomous driving.