Sagar Lekhak, Prasanna Reddy Pulakurthi, Lalit Joshi, Ramesh Bhatta, Emmett J. Ientilucci
5 min
Abstract
RGB imagery offers a practical, low-cost option for Unmanned Aerial/Ground Vehicle (UAV/UGV) survey support in surface-landmine detection, but object detectors remain underexplored in this safety-critical domain. Limited cross-architecture benchmarking and insufficient out-of-distribution (OOD) analysis obscure whether detectors generalize across deployment conditions. This challenge is amplified by the scarcity of public RGB landmine datasets, making SULAND a key benchmark for PFM-1 and PMA-2 detection. However, inspection reveals missing/false annotations, localization errors, inconsistent visibility criteria, visual artifacts, temporal labeling inconsistencies, and an inverted OOD class-ID convention in SULAND. We present SULAND_v2, a refined RGB surface-landmine dataset and benchmark. Preserving original images and splits, we manually revise annotations to ensure completeness, precise localization, label validity, and class consistency. SULAND_v2 contains 33,771 images and 12,433 bounding boxes. We benchmark 35 detector configurations across nine families. Annotation refinement improves YOLOv8 in-distribution (IID) test mAP@50 by 14.6-19.6 percentage points, while fixing the OOD class-ID convention increases mean YOLOv8 OOD mAP@50 by ~25 percentage points. On SULAND_v2, YOLOv12-Small achieves the highest IID mAP@50 (0.908), while RF-DETR-Large yields the strongest OOD performance (0.799 mAP@50, 0.675 recall). Our results demonstrate that high IID accuracy does not guarantee operational readiness. SULAND_v2 provides a reliable benchmark for evaluating domain-shift robustness in RGB-based mine-action survey support.
Alex: That's the right way to think about it. And to check whether this was a fluke, they tested the cleaned data across 35 different model configurations—different AI architectures, different setups. The improvement held consistently across all of them.
Sam: Did some models handle the bad data better than others?
Alex: Not in any meaningful way. What that tells you is that the labeling errors were acting as a ceiling on performance. No matter how well-designed the model was, it couldn't fully compensate for being trained on incorrect information.
Sam: That raises another question. How do we know the model is actually learning what a mine looks like, rather than just memorizing the specific backgrounds or lighting conditions in the training photos?
Alex: That's exactly the right concern. The researchers address it using what's called "out-of-distribution" evaluation. You train the model on data from one environment, then test it on footage from a completely different environment it has never seen before. If the model has genuinely learned to recognize a mine, it should still perform reasonably well. If it was just memorizing the scenery, it falls apart.
Sam: It's like studying for an exam using one textbook, then being tested on questions from a completely different one. It forces you to show whether you actually understand the concept.
Alex: Precisely. And what the researchers found is that even with clean data, a meaningful gap remains between lab performance and real-world results. Fixing the labels is a necessary first step—it gives you an honest baseline—but it doesn't eliminate the underlying challenge of adapting to new terrains, new lighting, new soil types.
Sam: So the paper isn't claiming to have solved the problem. It's saying: before we can make progress, we need to stop measuring ourselves against a broken ruler.
Alex: That's a fair summary. A model trained on sunny, grassy fields in one region will struggle when it's flown over rocky, shadowed terrain somewhere else. And if the training data was inconsistent to begin with, the model has no reliable foundation to generalize from. What this research establishes is that data quality isn't a secondary concern—for safety-critical tasks like this, it's foundational. You cannot meaningfully compare models, or claim real-world progress, if the benchmark itself is unreliable.
Sam: And the stakes here are about as high as they get. These aren't academic benchmarks—they're tools that could one day help clear land that's genuinely dangerous for people to walk on.
Alex: Which is precisely why the researchers argue this kind of careful, unglamorous work—auditing tens of thousands of images, correcting small errors, establishing honest evaluation protocols—matters as much as any advance in the models themselves. Thanks for listening to ResearchPod.