Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a system called ARDY—short for Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation.
Sam: That's quite a title. But the core problem it's solving is pretty relatable, right? Making digital characters move like real people, in a way that reacts instantly to a player's input?
Alex: Exactly. Here's the tension: current high-quality animation is often calculated in advance—slow, careful, but completely unable to adapt on the fly. Real-time methods are fast, but they tend to produce stiff, imprecise movement that struggles to follow complex instructions or specific paths. So you're choosing between quality and responsiveness.
Sam: So the goal is to bridge that gap—getting the quality of those slow, careful methods while keeping the speed you'd need for a video game or a robot that has to move right now.
Alex: Right. And to do that, ARDY uses what they call a "hybrid" approach. Think of it like a puppeteer. The root of the character—where they are on stage and which direction they're facing—is the handle you hold. The body, with all its fluid limb movements, is the internal mechanism. These two things are controlled separately, because they have very different jobs.
Sam: So instead of trying to calculate every single joint and the character's position all at once, they split the problem in two?
Alex: Precisely. The position and direction—what they call the "root"—are handled with very direct, explicit instructions. This lets the system force the character to hit specific waypoints or follow a precise path accurately. Then the rest of the body—the arms swinging, the weight shifting—is handled differently.
Sam: How so?
Alex: They compress the body's motion into what's called a "latent embedding." Imagine you could summarise an entire dance routine as a short code, rather than describing every muscle movement individually. That compressed code is much easier and faster for a computer to work with, which is what makes real-time generation possible.
Sam: Okay, but how does the system actually generate the motion? Is it just predicting the next frame from the last one?
Alex: It uses something called an autoregressive diffusion model—so let's unpack both of those words. "Autoregressive" just means it works step by step: each new moment in the animation is built on the previous one. "Diffusion" describes the generation process itself. The system starts with pure random noise and gradually refines it—like a sculptor chipping away at a block of stone until a clear shape emerges. It repeats that refinement process at every step of the animation.
Sam: So it's not guessing randomly. It's doing a careful, iterative refinement at each moment in time.
Alex: That's right. And it does this in two stages. First, it predicts where the root should go. Then it uses that prediction to guide the generation of the body motion. Because the body always knows where the root is headed, the two stay perfectly in sync—no sliding feet, no jittering limbs.
Sam: That's the key, isn't it? If you tried to generate both at once, they might drift apart. But by doing root first, body second, you guarantee they're always telling the same story.
Alex: Exactly. And all of this happens within about 33 milliseconds—fast enough to feel instantaneous to a player.
Sam: That works for a short burst of movement. But what about longer sequences? If a character needs to walk across a large space over ten seconds, how does the model stay on track without losing the goal?
Alex: This is where the "sliding window" comes in. Think of it like reading a long scroll. You can only see one section at a time, but you never lose track of where the story is going because the goal is always visible at the edge of the window. As the character moves, future waypoints drift into view, and the model continuously updates its plan.
Sam: So it's not trying to memorise the entire ten-second journey at the start. It's constantly re-reading the destination and adjusting.
Alex: Exactly. And that also means it can handle changes mid-motion. If you move a waypoint while the character is already halfway there, the system detects the new input and recalculates the path immediately.
Sam: But wouldn't that cause a stutter? If the computer has to stop and recalculate, the character should freeze for a split second.
Alex: They solve that with a "replan buffer." It's a small stock of pre-generated animation frames kept ready to play at all times—like the buffer in a video streaming app that loads a few seconds ahead so you never see the loading screen. While those buffered frames are playing, the model quietly calculates the new path in the background. The character keeps moving smoothly while the system is thinking. The seam between the old path and the new one is invisible.
Sam: And I'd imagine there's a trade-off in how large you make that buffer?
Alex: There is. A larger buffer gives the model more time to think, which allows for higher-quality, more complex motion. But it also means the character takes slightly longer to respond to rapid changes. Developers can tune that balance depending on what their application needs—it's a dial between responsiveness and motion quality, not a fixed setting.
Sam: What strikes me is that the cleverness here isn't just in the mathematics. It's in how they manage the flow of time and information—splitting the body from the root, buffering against lag, using a sliding window to stay on task.
Alex: That's a fair summary. Each of those design choices addresses a specific failure mode in earlier systems. Taken together, they represent a meaningful step toward digital characters that feel genuinely present and responsive—not just animated, but reactive.
Sam: I'll admit, next time I'm playing a game and a character smoothly sidesteps an obstacle mid-sprint, I'll probably think about the scaffolding holding that moment together.
Alex: That's exactly the right way to think about it. There's a lot of careful engineering behind what looks effortless. Thanks for listening to ResearchPod.