HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning

Modern Vision-Language Models (VLMs) have improved image geo-localization, but they often suffer from 'landmark bias'—a tendency to over-rely on visually salient landmarks (like the Eiffel Tower) while ignoring broader, more reliable geographic context. This leads to confident but incorrect predictions when landmarks are ambiguous or misleading. The authors seek to quantify this bias and develop a framework that forces models to aggregate diverse geographic cues rather than relying on single-cue shortcuts.

The authors first formalize landmark bias by introducing two quantitative metrics: Bias Intensity (BI), which measures how much a landmark anchors a model's prediction, and Bias Harmfulness (BH), which measures how much that landmark shifts the model away from the ground truth. Using these metrics, they curate a new benchmark, LandmarkBias-3K, containing images where landmark cues are deceptive.

To mitigate this, they introduce HoloGeo, a framework trained on a new dataset, BF-30k. This dataset provides structured 'reasoning chains' that force the model to decompose an image into multiple independent geographic elements (e.g., architecture, vegetation, climate) before aggregating them into a final decision. HoloGeo uses multi-dimensional rewards during training to explicitly penalize reliance on isolated cues and encourage balanced attention across the entire image.

Experiments show that existing state-of-the-art VLMs perform poorly on the LandmarkBias-3K benchmark, often achieving less than 20% accuracy. HoloGeo significantly outperforms these models, reaching 27.27% accuracy on the bias-heavy benchmark while maintaining strong performance on standard datasets like IM2GPS3K and YFCC4k. Saliency map visualizations confirm that HoloGeo successfully distributes its attention across broader geographic context, whereas baseline models remain fixated on isolated, misleading landmarks.