Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
Sam: A generated world needs to follow your fingers, not just your viewpoint. LynnReal-Hand generates videos from supplied hand movements, preserving much of that control when accelerated for streaming.
Alex: Following fingers sounds more demanding than moving a camera. Does the report show better control, or just convincing pictures of hands?
Sam: It reports improvements in both, using a benchmark that measures hand alignment separately from visual quality. This is a technical report from LynnReal AI, called “LynnReal-Hand: Unlocking World Interaction with Hand Control.”
Alex: So the audience isn't only people trying to make better-looking video. It's researchers who want generated environments to respond to manipulation.
Sam: That's the target: reaching, grasping, releasing, and using both hands together. We'll unpack how the authors protect those details, what their benchmark establishes, and why streaming indefinitely isn't the same as remembering indefinitely.
Alex: Place this against earlier work. Are hands a new control interface here, or is the advance in making an existing interface work better?
Sam: The interface already exists. The report names HandWorld, Hand2World, Generated Reality, and others as earlier efforts connecting hand motion to video generation. Its contribution is a training and inference pipeline designed to preserve hand control while making generation fast enough for streaming.
Alex: What makes hands especially vulnerable? A model that produces an entire scene should, in principle, also produce the hands.
Sam: Hands and contact regions occupy little of the image. A training objective averaged across the whole frame can give them too little weight. The authors observe weak loss and gradient contributions there, alongside distorted hands or hands collapsing into noise.
Alex: So the model can improve its overall score while neglecting the region carrying the user's instruction. How do they change that incentive?
Sam: They explicitly supervise the hand regions. They compare the model's compressed video representation with the target, concentrating extra attention on hand locations. They also decode those regions into images and compare their perceptual appearance and learned visual features.
Alex: Those sound like several ways of asking whether the hand looks right. What does each comparison add?
Sam: The compressed-space comparison penalizes local reconstruction errors directly. The decoded comparisons constrain what those internal representations actually produce: perceptual structure and hand appearance. In the component tests, combining them gives stronger hand alignment than training with the whole-frame objective alone.
Alex: Before acceleration, what information does the model receive? “Hand-controlled” could mean anything from a gesture label to a complete movement sequence.
Sam: It receives an initial image, an activity description, a hand-motion sequence, and camera poses. The image establishes appearance; the other inputs specify activity, articulation, and viewpoint. Training uses about a hundred thousand interaction clips spanning real and synthetic environments, with those conditions aligned over time.
Alex: The alignment matters here. If the hand movement and target video disagree, extra hand supervision could reinforce the wrong thing.
Sam: The report describes substantial annotation and filtering. It detects and tracks hands, fills some detection gaps, estimates hand geometry, and rejects clips with insufficient hand coverage. A separate verification pass retains purposeful hand–object interactions, excluding empty gestures and static display.
Alex: That builds a detailed generator. How do they make it respond quickly without throwing away the control they've just taught it?
Sam: Through distillation: training an accelerated model using guidance from another model. Their streaming generator produces each chunk with three denoising transitions, progressively turning noise into video. But learning initially uses clean reference history, while deployment must use the model's own generated history.
Alex: So it learns with a reliable past, then works with an imperfect one. That seems like a route for small errors to compound.
Sam: The authors address that mismatch by refining the accelerated model on its own generated sequences. A hand-conditioned video teacher guides those sequences. They also propose an image teacher to improve appearance, but deliberately prevent that teacher from changing the hand regions.
Alex: Why exclude hands from an appearance teacher? That's where convincing appearance seems most valuable.
Sam: Because that image teacher doesn't receive the hand conditions. Its preferred hand might conflict with the requested movement. The video guidance covers the whole scene; the image guidance contributes only outside the hand regions.
Alex: So this isn't simply adding more supervision. It's deciding which source of supervision is entitled to change which pixels.
Sam: In their diagnostic example, video guidance evaluated at higher noise produces stronger corrections around fingers. Image guidance changes sink and countertop boundaries. That motivates separating their responsibilities, though those probes aren't a performance test by themselves.
Alex: Does the headline streaming result test that complete combination? Otherwise listeners could attribute the improvement to a component that wasn't actually enabled.
Sam: It does not. The evaluated Fast model uses video-score distillation with image guidance disabled. That's a crucial boundary: the benchmark supports the reported Fast configuration, not a measured benefit from adding the proposed image branch.
Alex: Let's look at that benchmark. How do they distinguish obeying the supplied movement from simply generating plausible-looking hands?
Sam: HandBench contains two hundred video sequences drawn from diverse interaction sources. Its primary control measure reconstructs hand geometry independently from generated frames, then compares projected hand silhouettes with the reference. That overlap measure is called MANO Dice; higher means closer agreement.
Alex: Silhouette agreement sounds useful, but narrower than proving successful manipulation. Does the evaluation also test appearance and generalization?
Sam: It includes video quality, perceptual similarity, scene consistency, and camera-following measures. Some sources are held out from world-model training for testing unfamiliar data. Still, projected hand agreement isn't itself a direct measure of whether an object interaction obeys physical constraints.
Alex: What result best captures the improvement over prior hand-controlled generation?
Sam: On the main benchmark, Base reaches about ninety-three on the hand-overlap score, compared with about ninety-one for adapted bidirectional Hand2World. Base also reduces Fréchet Video Distance by roughly thirty-eight percent against that model. That's a comparison of video distributions, where lower is better.
Alex: That supports both control and appearance, rather than trading one entirely for the other. What survives when they switch to streaming?
Sam: Fast retains nearly the same hand-overlap score, although its video-distribution quality is worse than Base's. On a single H200 graphics processor, optimized streaming reaches about eighteen frames per second. Those measurements use lower resolution than the main quality comparisons.
Alex: Frame rate alone doesn't tell me how quickly a new hand action appears. What is the delay?
Sam: The reported mean chunk latency is about seven hundred forty milliseconds, including online hand encoding and decoding. Their compact decoder helps by converting compressed video into frames more cheaply. It preserves the generator's existing representation while narrowing expensive internal computations.
Alex: So the evidence is promising, but I shouldn't treat every speed entry as a matched race. The report uses different resolutions and timing units across methods.
Sam: Nor should we confuse continuous output with persistent memory. The streaming model keeps a bounded recent history and resets local temporal coordinates. The authors explicitly say this bounds attention state; it doesn't provide long-term memory, which may constrain consistency over extended interactions.
Alex: Who should read the full report, and where should they start if their time is limited?
Sam: Researchers building hand-controlled video systems should start with “Hand-Aware Full Sequence Learning” and the ablation study. Those connect the local-supervision idea to measured improvements. For acceleration, read “Region-Aware Score Distillation,” remembering the disabled image branch, then check the runtime protocols and limitations.
Alex: And what should everyone else carry away from the result?
Sam: A generated world can look plausible while ignoring your hands. Faster interaction needs training that protects the details carrying control, not just training that improves the average frame.
Alex: Keep the hands in the evaluation.