R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video

Long-horizon egocentric videos are difficult to query because they lack structured, persistent object-level information. Existing methods often rely on flat captions or assume high-quality 3D geometry (like point clouds or depth maps) that is unavailable in casual, free-motion wearable video. The authors ask: can we build a queryable, object-centric memory for long videos using only monocular RGB input?

The authors introduce R4DSG, a relative 4D scene graph memory. Instead of attempting to build a globally consistent 3D map, the system identifies stable objects (anchors like tables or fridges) and dynamic objects (items like bags or phones). It tracks these objects over time and records their state changes relative to these anchors. The pipeline uses foundation models for promptable video segmentation and relative 3D lifting to generate frame-level graphs, which are then compressed into episodic memory entries containing time, place, object identity, and anchor-relative transitions.

By organizing video data into these relative 4D scene graphs, the system creates a compact, retrieval-ready memory. When tested on a 255-question subset of the EgoLifeQA benchmark, R4DSG outperformed text-based retrieval methods (EgoRAG-Text) by 6.7 points overall. It showed a significant 12.5-point gain on 'when' questions, demonstrating that this structured, anchor-relative representation is more effective at capturing the temporal dynamics of object movement than traditional caption-based approaches.

This work provides a practical path for wearable AI assistants to maintain a persistent, queryable memory of a user's environment. By avoiding the need for expensive 3D reconstruction and focusing on relative spatial relationships, R4DSG makes it possible for AI agents to answer complex questions about object history—such as where an item was moved or why it was relocated—using only standard camera footage.