GS-Agent: Creating 4D Physical Worlds With Generative Simulation

GS-Agent addresses the challenge of generating dynamic, physically grounded 4D environments without the manual labor typically required in computer graphics. Instead of relying on large-scale video diffusion models—which often struggle with physical consistency and precise control—the authors propose an agentic system that mimics a human production pipeline. The framework consists of three specialized agents: a Manager Agent for high-level planning and task delegation, an Entity Agent for asset curation and material tuning, and a Render Agent for camera and lighting control. These agents interact with a physics engine by generating and executing code, allowing for iterative refinement based on multimodal feedback.

By integrating a physics engine directly into the generation loop, GS-Agent ensures that object interactions—such as liquids splashing, deformable objects squishing, or rigid bodies colliding—adhere to physical laws. The system supports complex scene configurations, including:

• Entity Management: Automated retrieval or generation of 3D assets, followed by material parameter tuning (e.g., Young's modulus for deformation) and spatial placement.
• Cinematic Control: The Render Agent translates natural language directives into specific camera trajectories and lighting setups, which are executed as code to ensure temporal coherence.
• Iterative Refinement: The agents use visual and physical feedback to detect collisions, misalignments, or simulation instabilities, allowing them to autonomously correct errors until the output matches the user's intent.

Traditional text-to-video models often produce visually impressive but physically nonsensical results. GS-Agent provides a paradigm shift by treating world creation as a software engineering task rather than a pure pixel-prediction problem. This approach enables the generation of high-fidelity, controllable 4D data, which is critical for training embodied AI agents, improving simulation-to-reality transfer, and lowering the barrier for creative content production in gaming and film.