LynnReal AI
8 min
Abstract
Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
Alex: Before acceleration, what information does the model receive? “Hand-controlled” could mean anything from a gesture label to a complete movement sequence.
Sam: It receives an initial image, an activity description, a hand-motion sequence, and camera poses. The image establishes appearance; the other inputs specify activity, articulation, and viewpoint. Training uses about a hundred thousand interaction clips spanning real and synthetic environments, with those conditions aligned over time.
Alex: The alignment matters here. If the hand movement and target video disagree, extra hand supervision could reinforce the wrong thing.
Sam: The report describes substantial annotation and filtering. It detects and tracks hands, fills some detection gaps, estimates hand geometry, and rejects clips with insufficient hand coverage. A separate verification pass retains purposeful hand–object interactions, excluding empty gestures and static display.
Alex: That builds a detailed generator. How do they make it respond quickly without throwing away the control they've just taught it?
Sam: Through distillation: training an accelerated model using guidance from another model. Their streaming generator produces each chunk with three denoising transitions, progressively turning noise into video. But learning initially uses clean reference history, while deployment must use the model's own generated history.
Alex: So it learns with a reliable past, then works with an imperfect one. That seems like a route for small errors to compound.
Sam: The authors address that mismatch by refining the accelerated model on its own generated sequences. A hand-conditioned video teacher guides those sequences. They also propose an image teacher to improve appearance, but deliberately prevent that teacher from changing the hand regions.
Alex: Why exclude hands from an appearance teacher? That's where convincing appearance seems most valuable.
Sam: Because that image teacher doesn't receive the hand conditions. Its preferred hand might conflict with the requested movement. The video guidance covers the whole scene; the image guidance contributes only outside the hand regions.
Alex: So this isn't simply adding more supervision. It's deciding which source of supervision is entitled to change which pixels.
Sam: In their diagnostic example, video guidance evaluated at higher noise produces stronger corrections around fingers. Image guidance changes sink and countertop boundaries. That motivates separating their responsibilities, though those probes aren't a performance test by themselves.
Alex: Does the headline streaming result test that complete combination? Otherwise listeners could attribute the improvement to a component that wasn't actually enabled.
Sam: It does not. The evaluated Fast model uses video-score distillation with image guidance disabled. That's a crucial boundary: the benchmark supports the reported Fast configuration, not a measured benefit from adding the proposed image branch.
Alex: Let's look at that benchmark. How do they distinguish obeying the supplied movement from simply generating plausible-looking hands?
Sam: HandBench contains two hundred video sequences drawn from diverse interaction sources. Its primary control measure reconstructs hand geometry independently from generated frames, then compares projected hand silhouettes with the reference. That overlap measure is called MANO Dice; higher means closer agreement.
Alex: Silhouette agreement sounds useful, but narrower than proving successful manipulation. Does the evaluation also test appearance and generalization?
Sam: It includes video quality, perceptual similarity, scene consistency, and camera-following measures. Some sources are held out from world-model training for testing unfamiliar data. Still, projected hand agreement isn't itself a direct measure of whether an object interaction obeys physical constraints.
Alex: What result best captures the improvement over prior hand-controlled generation?
Sam: On the main benchmark, Base reaches about ninety-three on the hand-overlap score, compared with about ninety-one for adapted bidirectional Hand2World. Base also reduces Fréchet Video Distance by roughly thirty-eight percent against that model. That's a comparison of video distributions, where lower is better.
Alex: That supports both control and appearance, rather than trading one entirely for the other. What survives when they switch to streaming?
Sam: Fast retains nearly the same hand-overlap score, although its video-distribution quality is worse than Base's. On a single H200 graphics processor, optimized streaming reaches about eighteen frames per second. Those measurements use lower resolution than the main quality comparisons.
Alex: Frame rate alone doesn't tell me how quickly a new hand action appears. What is the delay?
Sam: The reported mean chunk latency is about seven hundred forty milliseconds, including online hand encoding and decoding. Their compact decoder helps by converting compressed video into frames more cheaply. It preserves the generator's existing representation while narrowing expensive internal computations.
Alex: So the evidence is promising, but I shouldn't treat every speed entry as a matched race. The report uses different resolutions and timing units across methods.
Sam: Nor should we confuse continuous output with persistent memory. The streaming model keeps a bounded recent history and resets local temporal coordinates. The authors explicitly say this bounds attention state; it doesn't provide long-term memory, which may constrain consistency over extended interactions.
Alex: Who should read the full report, and where should they start if their time is limited?
Sam: Researchers building hand-controlled video systems should start with “Hand-Aware Full Sequence Learning” and the ablation study. Those connect the local-supervision idea to measured improvements. For acceleration, read “Region-Aware Score Distillation,” remembering the disabled image branch, then check the runtime protocols and limitations.
Alex: And what should everyone else carry away from the result?
Sam: A generated world can look plausible while ignoring your hands. Faster interaction needs training that protects the details carrying control, not just training that improves the average frame.
Alex: Keep the hands in the evaluation.