GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes

Robots operating in human environments must understand not only the current state of the world but also the history of human activities within it. Existing 4D scene graphs effectively track objects and places over time but lack the semantic structure to represent goal-directed human behavior. This paper addresses the challenge of building a memory that links persistent spatial entities to the activities that involve them, allowing for retrospective reasoning about human-object interactions.

The authors introduce GESTO (Grounded Event and Spatio-Temporal memOry), a framework that couples a persistent 4D scene graph with a two-level activity hierarchy. The hierarchy consists of atomic human-object interactions (e.g., "rinses cup") and higher-level, goal-driven events (e.g., "prepares coffee").

GESTO operates in two main stages:

1. Construction: The system processes an RGB-D stream to extract atomic interactions and ground them to persistent objects in the scene graph using geometric and semantic cues.
2. Refinement: Once interactions are grouped into events, the system performs a context-aware refinement step. This allows the model to resolve ambiguous object associations by leveraging the broader activity context, ensuring that the spatial and activity hierarchies are mutually consistent.

Finally, a relation-aware tool-calling agent queries this memory to answer natural language questions, providing both the answer and the supporting spatio-temporal evidence.

GESTO demonstrates strong performance on standard benchmarks for spatio-temporal reasoning, achieving scores of 0.71, 0.75, and 0.70 across text, binary, and time-based categories. Notably, the system approaches the performance of methods provided with ground-truth event and object data, despite constructing its memory fully autonomously. Ablation studies confirm that the hierarchical event structure and the context-aware grounding refinement provide complementary benefits, significantly improving the agent's ability to reason about complex, multi-step human activities.