Anomalous Frame Detection by Grouping Frame Similarities between Two Videos Computed by Vision-Language Model to Extract Expert Workers’ Unique Actions

In critical infrastructure maintenance, expert workers often possess tacit knowledge—actions they perform automatically or intuitively that are not documented in standard operating manuals. Traditional interview-based methods fail to capture these unconscious behaviors. The authors seek a way to automatically identify these unique expert actions by comparing video recordings of expert performance with videos of workers following a standard manual.

The researchers propose a framework that detects "anomalous frames"—segments where an expert's actions deviate from the manual. The process involves two main steps:

1. Similarity Map Generation: The system generates captions for every frame in both the manual-based video and the expert video. To ensure robustness against noise like lighting or camera angle, these captions are refined using a pre-trained vision-language model (VLM) to ensure they accurately represent the work content. These captions are then converted into vectors to compute a similarity map between the two videos.
2. Anomalous Frame Detection: The system identifies regions in the similarity map where the similarity between the two videos is low. These low-similarity regions indicate segments where an action exists in one video but not the other, effectively isolating the expert's unique "know-how" actions.

In a simulated maintenance experiment involving a distribution board, the proposed method successfully identified 11 types of expert actions that were not described in the official manual. The system achieved a 66.9% extraction rate, representing a 50-percentage-point improvement over conventional video-comparison techniques. This demonstrates that leveraging pre-trained VLMs can effectively codify tacit expert knowledge without requiring extensive training on domain-specific datasets.