Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang, Liwei Wang
7 min
Talking-video character replacement requires the simultaneous transfer of both visual appearance and vocal timbre while faithfully preserving the source video's motion, background, scene dynamics, and linguistic content. Existing approaches typically decouple this challenge by using separately optimized models for video modification and voice conversion. Because these modules operate independently, no joint optimization objective enforces audio-visual consistency, making it difficult to synchronize lip movements with converted speech or correct cross-modal errors. Furthermore, most diffusion-based video generation techniques require access to the entire video clip before processing, restricting their use in real-time, interactive, or streaming workflows.
This paper introduces UniSwap, the first framework capable of streaming joint audio-visual identity replacement within a unified architecture. Given a source talking video alongside a reference image and a reference voice clip, UniSwap generates synchronized video and audio frames autoregressively. By processing both modalities inside a single diffusion transformer, the model leverages cross-modal interactions to coordinate generated speech and facial motion.
A primary bottleneck in training joint audio-video identity replacement models is the lack of large-scale, paired training data where two different individuals perform identical motions and speech with precise temporal alignment. To overcome this limitation, UniSwap treats every real talking video as its own self-reconstruction target.
In this pipeline, a real video clip serves as the ground-truth target for both appearance and audio. The visual identity is removed by estimating whole-body 2D poses, segmenting the person, and compositing a rendered pose sequence onto the background plate. This produces an identity-swapped visual source that retains the original motion and background without original facial features. Concurrently, the original audio is converted using an off-the-shelf voice conversion model to alter speaker timbre while keeping linguistic content and timing intact. A portrait frame and a random audio segment from the real clip serve as the reference image and reference audio.
UniSwap adapts a pretrained audio-video diffusion backbone through three distinct training stages to achieve low-latency streaming generation:
In-context Pretraining: The source, reference, and noisy target latents are concatenated into a unified sequence. Using condition positional encoding offsets and joint flow-matching loss, the model learns full-sequence cross-modal attention for identity replacement.
Conditional Streaming Adaptation: A Decoupled Streaming Conditioning Mask converts the bidirectional model into a block-causal generator. This restricts attention receptive fields to valid historical blocks, enabling key-value (KV) cached autoregressive inference.
Efficient Self-forcing DMD: Distribution Matching Distillation combined with self-forcing rollout exposes the student model to its own historical outputs, mitigating exposure bias. This stage reduces the required denoising steps per block from 30 down to 3.
To make this distillation tractable, Efficient Multi-LoRA Switching allows the teacher, generator, and critic roles to share a single frozen backbone via lightweight adapters instead of maintaining three full model copies. Finally, Feature-RoPE Decomposition separates unrotated keys from rotary coordinates, ensuring stable, long-form hour-scale generation without positional drift.
Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.
Alex: So it's trained to work on the full sequence first. Then how does it shift to streaming?
Sam: They apply what they call a streaming conditioning mask, which converts that bidirectional model—one that can look at the whole video at once—into a block-by-block generator. Instead of needing to see the entire future video, the model only looks at what's happening right now and what it has already produced and cached.
Alex: Like reading a book one chapter at a time, where you remember what came before but can't peek at the ending.
Sam: That's a good way to put it. And to make it fast enough for live use, they apply a distillation technique—essentially a compression step—that reduces the number of refinement passes from thirty down to just three per block. That's what allows it to run at a usable speed.
Alex: How does the model keep audio and video locked together during long sessions, though? I'd imagine that gets harder the longer the video runs.
Sam: That's a real problem. Standard models use a coordinate system to track where they are in a sequence—think of it like timestamps. But if you let those timestamps tick continuously for hours, the internal math starts to drift or break. To fix this, they developed a technique that separates the stored features from the position markers. The features from past blocks stay cached and stable, while the position markers reset to a bounded range for each new block.
Alex: So it keeps a stable anchor for what it's already processed, while sliding the position clock forward only for the active block.
Sam: Exactly. They call this Feature-RoPE Decomposition—RoPE stands for Rotary Position Embedding, which is a specific mathematical way of encoding position information. Combined with a memory buffer that drops old history while keeping fixed reference points, the position math never overflows. That keeps temporal alignment stable across hour-long generation without reprocessing the entire past video from scratch.
Alex: And the training itself happens in stages, right? How does that work?
Sam: Three stages. In the first, the model learns how the different inputs—reference images, source footage, and the noisy target—relate to each other. It processes them all together in one shared attention context, so it understands how a face, a voice, and a body pose connect before it ever tries to generate anything.
Alex: And the second stage is where streaming comes in?
Sam: Yes. Here they introduce what are called low-rank adaptation modules—think of these as lightweight add-ons, like interchangeable lenses on a camera. Instead of retraining the entire massive base model, which would be computationally impractical, they freeze it and only train these small adapter layers. That lets the model switch efficiently between different tasks like streaming generation and noise refinement.
Alex: So the adapters redirect the frozen base model without having to rebuild it from scratch.
Sam: Precisely. Then the third stage applies the distillation step we mentioned—a critic network compares the model's outputs to the target distribution and pushes the generator to match it in just three steps instead of thirty.
Alex: What does the overall picture look like when you pull all of this together?
Sam: The paper demonstrates that handling face and voice replacement as a single joint task avoids the synchronization drift that plagues traditional multi-step pipelines. By integrating the process into one architecture, the model maintains stable identity preservation and lip synchronization even over minute-long sequences.
Alex: But it's not a complete sweep across every category?
Sam: That's correct. While UniSwap leads in synchronization and long-form stability, its image quality scores remain slightly behind specialized models built solely for offline editing. There's a clear trade-off between the consistency of joint streaming and the raw visual fidelity of single-purpose tools. And its current throughput means it supports streaming generation, but doesn't yet reach the frame rate required for fully smooth real-time playback.
Alex: So the work points toward something like real-time video conferencing with live avatar and voice swapping, but it's not quite there yet.
Sam: That's a fair summary. The framework establishes that joint audio-visual identity replacement can work within a unified streaming architecture. Future iterations could extend it toward multi-character interactive streaming—but those remain open problems for now.
Alex: Thanks for walking us through the details today, Sam. And thank you for listening to ResearchPod.