An Integrated Hardware-Software Design for Low-Data Spatial Defect Detection in Robotic Visual Inspection with Hybrid Optoelectronic Neural Networks

In industrial robotic visual inspection, high-definition imaging often creates massive data bottlenecks, while traditional deep learning models require extensive, labor-intensive manual annotations to localize defects. This paper introduces a hardware-software co-design that replaces traditional imaging with a non-imaging, low-data paradigm. By integrating physical optical feature extraction with pre-trained vision-language models, the system achieves accurate defect detection and localization while significantly reducing data redundancy and annotation requirements.

The system utilizes a Digital Micromirror Device (DMD) as a physical optical convolutional layer, which performs feature extraction directly in the photonic domain. This "sensor-in-the-loop" strategy employs block-based compressed sensing to convert spatial information into low-dimensional temporal signals, bypassing the need for computationally expensive image reconstruction. To eliminate the need for manual shape labeling, the researchers use natural language descriptions of defect types to guide the network, aligning its internal feature space with generalizable features from the CLIP model. This allows the network to generate attention maps that highlight defect shapes automatically.

The integrated architecture demonstrates that hardware-level compression can maintain detection accuracy parity with traditional imaging while drastically lowering resource consumption. By leveraging cross-modal alignment, the system successfully identifies and localizes defects using only categorical labels or, in some cases, no labels at all. The authors also introduced the Localization Accuracy for Attention (LAA) metric to quantify the precision of these attention-based heatmaps, providing a more suitable evaluation tool for morphological defect detection than traditional bounding-box metrics like Intersection over Union (IoU).