Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

Research Context and Motivation

Mainstream LLM serving systems like vLLM and SGLang are engineered for high-throughput, multi-tenant environments. They rely on paged or radix-based KV caches, which treat the KV cache as a positionally addressable fragment of the execution state. While highly effective for maximizing aggregate throughput, these systems struggle in the 'physical-AI' regime—interactive, single-stream applications like robotics, voice assistants, or coding agents. These applications require extreme responsiveness, frequent branching, and the ability to reset or re-enter states under tight latency budgets, where the overhead of cold-starting or recomputing prefixes is prohibitive.

The FlashRT Approach

FlashRT addresses this by shifting the unit of reuse from token-addressed KV fragments to graph-bound execution-state boundaries. The system uses a white-box, latency-first runtime that captures the entire forward pass as a graph plan over contiguous, static buffers. By eliminating block-table indirection, the runtime ensures that the entire live state—including recurrent states, convolution states, and KV caches—is contained within a closed, named set of buffers. This design allows the system to treat the entire execution boundary as a single, first-class object that can be snapshotted, restored, forked, or rolled back in sub-millisecond time.

Performance and Correctness

On an RTX 5090, FlashRT demonstrates that capturing the full execution state is essential for correctness; an ablation study shows that restoring only the positional KV cache leads to divergence in hybrid models, whereas full capsule restoration remains token-identical. The system provides significant speedups over cold prefill, with performance gains increasing as the prefix length grows (e.g., 27x speedup at 16k tokens). Because the capsule mechanism is built on a static, contiguous buffer contract, it provides a consistent, low-latency execution floor that is maintained across session reuse and branching, making it uniquely suited for interactive, embodied AI.