Agentic Collaborative Cognition for Zero-Shot 3D Understanding

This paper addresses the limitations of existing zero-shot 3D understanding methods, which typically rely on static, pre-captured video keyframes. These methods often suffer from finite observation perspectives and an inability to maintain consistent spatial reasoning across different views. To overcome this, the authors propose a collaborative multi-agent framework consisting of two specialized agents: a Planning Agent and a Perception Agent.

The Planning Agent is responsible for high-level viewpoint selection. It analyzes a shared 'cognitive map' to identify query-relevant objects and determines which viewpoints are necessary to resolve ambiguities. If existing video frames are insufficient, it supplements them with rendered views. The Perception Agent then processes these views, documenting object-level attributes and updating the cognitive map with consistent instance identifiers. This creates a closed-loop, iterative process where the agents refine their understanding of the scene until the query requirements are met.

By decoupling viewpoint planning from perceptual verification, the framework achieves significant performance gains over previous zero-shot and many fully supervised methods. The authors report state-of-the-art results across six benchmarks, including an 11.1% improvement in Acc@0.5 on ScanRefer, a 14.6 BLEU-1 increase on 3D-assisted dialog tasks, and a 2.1 EM improvement on SQA3D. These results demonstrate that active, agent-driven viewpoint selection is superior to passive keyframe analysis for complex 3D reasoning.

This research provides a scalable way to perform 3D scene understanding without the need for massive, manually annotated 3D-text datasets. By enabling agents to 'look' at a scene from the most informative angles, the framework bridges the gap between static 2D image understanding and the dynamic requirements of 3D environments, which is essential for applications in robotics and augmented reality.