FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

As multimodal applications (e.g., voice agents, interactive video) grow in complexity, developers face a difficult trade-off: they must either write inefficient single-GPU code or manually engineer complex, hardware-specific deployments to achieve acceptable latency and throughput. The authors ask whether a coding agent can be guided to automatically synthesize these optimized, multi-GPU deployments from simple, unoptimized reference implementations.

FlashRT introduces a "chain-of-program" paradigm that moves away from naive, single-step code generation. Instead, it forces the agent to follow a structured, multi-pass workflow:

1. Intermediate Representation (IR): The agent first converts the reference implementation into a hierarchical graph IR that explicitly captures data dependencies and persistent-state scopes.
2. Static Analysis: The agent analyzes this IR to identify candidate transformations, such as pipeline parallelism, streaming, or co-location.
3. Measurement-Gated Loop: The agent iteratively implements, verifies, and benchmarks these candidates using an application-specific test harness. This loop allows the agent to "self-correct" and explore different hardware budgets (e.g., latency vs. throughput) without human intervention.

FlashRT demonstrates significant performance gains across diverse multimodal pipelines, including video world models and multimodal LLMs. On NVIDIA B200 GPUs, it achieved up to 70x latency reduction and 2.8x throughput improvement. On AMD MI355X hardware, it showed even greater scalability, achieving a 3.6x throughput improvement and reducing response latency for Qwen3-Omni by 65% compared to expert-written vLLM-Omni implementations. These results suggest that agent-driven optimization can effectively bridge the gap between high-level code and low-level system performance, particularly on hardware platforms where expert-level optimization is less mature.