Confidence Scores in Open-Vocabulary Detection Are a Biased Mixture of Scale and Semantics

Open-vocabulary object detectors, such as GroundingDINO and OWL-ViT, rely on vision-language foundation models like CLIP to identify objects based on text queries. These systems output a scalar confidence score representing the similarity between a visual region and a text embedding. This paper investigates whether these scores are reliable indicators of localization probability, or if they are systematically distorted by factors unrelated to the actual detection quality.

Using controlled experiments on the COCO and LVIS datasets, the authors analyze the confidence scores of three prominent foundation-model-based detectors. They decompose the confidence score into an additive model, identifying two primary sources of bias: scale bias (the tendency for large objects to receive higher scores) and semantic bias (the tendency for generic queries to receive lower scores than specific ones). The authors provide a theoretical derivation showing that these biases arise from the spatial pooling of features and the geometry of CLIP’s contrastive pretraining objective.

• Scale Bias: There is a strong, statistically significant correlation between object area and confidence scores. Large objects are systematically favored, while small objects are penalized, regardless of the detector's actual localization accuracy.
• Semantic Bias: For a fixed visual region, generic queries (e.g., "an object") consistently yield lower confidence scores than specific queries (e.g., "a dog"), as specific queries occupy more concentrated regions in the embedding space.
• Structural Inevitability: These biases are not mere implementation errors but are fundamental to how image-level models are adapted for region-level tasks. Standard thresholding techniques fail to recover the localization signal for small objects.
• Correction: The authors introduce a parameter-free, test-time temperature scaling method that improves small-object Recall@10 by 19.6% without requiring model retraining. However, this correction involves a trade-off, as it slightly reduces overall precision by surfacing more false positives.