Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

How can vision-language models (VLMs) be effectively adapted to understand complex, fine-grained assembly actions in human-robot collaborative environments while ensuring the deterministic, interpretable outputs required for robotics?

The authors propose two primary technical innovations. First, Compositional Context Fine-Tuning (CCFT) decomposes assembly actions into distinct semantic elements—specifically verbs, objects, and tools—and uses templated visual question-answering (VQA) pairs to train the model. This structure forces the VLM to produce near-deterministic, interpretable outputs rather than open-ended, potentially hallucinatory text.

Second, to manage multi-task learning efficiently, they introduce Layer-Partitioned Alternating Training (LP-AT). This method assigns disjoint groups of model layers to specific action elements and applies task-specific low-rank adapters (LoRA). By alternating weight updates across these adapters, the model reduces cross-task interference and allows for granular hyperparameter optimization for each subtask. The authors also reformulated existing assembly video datasets (HA-ViD and IKEA-ASM) into structured VQA formats to support this training approach.

Experimental results on the newly created HA-ViD-VQA and IKEA-ASM-VQA datasets demonstrate that the CCFT and LP-AT framework consistently outperforms traditional action recognition baselines (such as TSM, UniFormerV2, and VideoMAE V2). The approach provides not only higher accuracy in holistic action recognition but also granular, interpretable predictions for individual action components, which is critical for downstream robotic assistance and skill acquisition.