The GEST-Engine: From Event Graphs to Synthetic Video

The GEST-Engine addresses the challenge of generating high-quality, fully annotated training data for video understanding models. While implicit neural generators (like latent dynamics models) struggle with object permanence and temporal consistency, the authors propose an explicit world model. By using a formal Graph of Events in Space and Time (GEST) as an input specification, the engine ensures that every generated frame is grounded in a semantic plan. The system leverages the open-source Multi Theft Auto (MTA) framework to programmatically control the environment of Grand Theft Auto: San Andreas, allowing for the deterministic orchestration of multi-actor narratives.

The system processes input GESTs through a four-stage pipeline:

1. Graph Parsing and Validation: Selects valid episodes and handles multi-location transitions.
2. Entity and Action Grounding: Binds abstract entities to concrete 3D objects and actor skins using a chain-ID system.
3. Temporal Orchestration: Uses Allen’s interval algebra and Floyd-Warshall transitive closure to resolve complex multi-actor timing constraints.
4. Execution and Capture: Dispatches actions to the game engine and collects multi-modal artifacts (RGB, depth, segmentation, skeletal pose, and spatial graphs) via a synchronized EventBus.

This work provides a solution to the "annotation bottleneck" in computer vision. Because the engine knows the state of the world at every frame by construction, it produces dense, frame-aligned labels without the need for manual labeling. This makes the output highly valuable for training and evaluating models that require strict temporal and spatial reasoning, such as those tasked with understanding multi-actor social interactions or long-term narrative progression. The architecture is designed to be modular, with an adapter pattern that aims to eventually decouple the orchestration logic from the specific game engine, potentially allowing for future portability to more modern rendering platforms.