Xinyang Li, Zeran Xu, Qijun Gan, Chenwei Zhang, Meiguang Jin, Junfeng Ma
8 min
Abstract
AI-generated videos often depict people within detailed scenes while leaving their small facial regions poorly resolved. Visual super-resolution can sharpen these regions without resolving whether the recovered lips and teeth agree with the spoken audio. We present an audio-guided facial video restoration framework that addresses this ambiguity within a reference-guided video flow model. Source--noise mixing initializes reconstruction from a partially preserved observation, and structured reference injection separates appearance anchors from continuous motion context. Training combines restoration of degraded images and videos with correction of degraded dubbed sources under target audio. Appearance and synchronization supervision allow the model to recover facial detail and support visual dubbing when the audio is replaced. On talking-head restoration benchmarks, our method achieves the best paired perceptual similarity and synchronization scores among the compared restoration methods. Component comparisons examine audio conditioning, the reconstruction formulation, and reference injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage. injection. Native AIGC examples further illustrate the recovery of lip contours, dental structure, and surrounding facial texture in scene-level footage.
Alex: That makes copying the input mouth an unsuccessful strategy during training. The source still supplies the scene and person, but speech tells the model which articulation it needs to reconstruct.
Sam: The training also includes a synchronization objective, using a frozen network to judge agreement between mouth movements and speech. Appearance losses constrain the reconstructed frames. A temporal loss constrains changes between adjacent frames, so the task isn’t just producing individually convincing still images.
Alex: There’s another possible conflict: a clean reference photograph has its own mouth shape. How does the model use that image without making every frame inherit its expression?
Sam: It gives appearance and motion separate conditioning slots. Clean reference frames are encoded independently to anchor identity and texture. Recent frames are encoded together to supply motion context. The fixed appearance reference stays available as reconstruction proceeds, rather than surviving only through previously generated frames.
Alex: That distinction seems especially relevant for longer clips, where generated history could carry its own mistakes forward. Do the component tests show a benefit beyond the overall system comparison?
Sam: They do support it within the tested setup. Replacing the persistent appearance anchor with direct conditioning on recent generated frames worsens target similarity and synchronization. The authors show overly smooth texture and artificial dot patterns in examples, consistent with errors accumulating through that history.
Alex: And what about the audio itself? If most pixels are outside the mouth, a conventional image score could miss whether speech guidance helps.
Sam: That happens in the ablation. The visual-only version has slightly better pixel fidelity, but much worse synchronization. In a selected example, it reconstructs a narrower mouth opening than the target. The audio-guided version recovers the open-mouth configuration, while overall perceptual error stays nearly unchanged.
Alex: Let’s put scale behind those examples. What was the main restoration test, and what did the strongest result actually measure?
Sam: The benchmark contains 150 videos drawn from public talking-head datasets and an internal dataset. Each method gets the same synthetic degradation, including downsampling and resizing. Results are averaged equally across videos, with metrics computed on complete prepared frames rather than only the mouth.
Alex: So these are controlled restoration tests, not a benchmark made entirely from native generated footage. How large is the perceptual advantage?
Sam: On a learned perceptual-error measure, where lower is better, the proposed model scores about point zero eight. InstantRestore, the strongest baseline on that measure, scores about point one four. This compares each reconstruction against its clean target, rather than judging sharpness alone.
Alex: That supports recovered appearance, but the central claim also concerns speech. Does the model improve synchronization against the restoration baselines?
Sam: Its synchronization-confidence score is about seven point four. SparkVSR, the strongest restoration baseline on the synchronization measures, scores about six point five. Those are network-based scores, not listener judgments, and SparkVSR still leads the pixel and structural fidelity measures.
Alex: Before treating this as a production result, I want to know what reference they supplied. A clean image of the target person could make a substantial difference.
Sam: That is the most important qualification. In the quantitative tests, the proposed model and reference-capable baselines receive the undegraded target’s first frame. The enhanced version uses an edited copy of that same clean frame. That is a stronger reference than a damaged generated video necessarily provides.
Alex: And editing the reference changes the target appearance. Does a better-looking reference improve fidelity, or just push the output toward something visually appealing?
Sam: It shifts the balance. The enhanced reference improves the reported no-reference quality scores, which judge appearance without comparing against a clean target. But paired fidelity gets substantially worse. The unenhanced model’s perceptual reconstruction advantage therefore needs to be kept separate from the enhanced model’s appearance-quality advantage.
Alex: What happens when the speech changes altogether? That seems a stronger test of whether audio can override the original mouth.
Sam: They test replacement audio on 30 clips, comparing their enhanced-reference model against restoration followed by separate dubbing systems. Their model leads all the tested cascades on synchronization and no-reference quality. But there is no matching target-mouth video for the new speech, so paired reconstruction accuracy cannot be evaluated.
Alex: The motivating problem was generated footage, though. How much direct evidence comes from that setting?
Sam: Four native generated clips, without added synthetic degradation or clean targets. Those examples show clearer lip contours, tooth separation, and surrounding texture. References come from edited input frames there. I’d treat them as useful illustrations, not a broad quantitative validation of generated-video repair.
Alex: So who should read the full report, and where should they start?
Sam: Researchers in restoration, talking-face editing, or dubbing should start with Facial Restoration under Original Audio, especially the reference protocol. Then read the Ablation Study to separate the contributions of speech, source–noise mixing, and persistent appearance guidance. For production use, inspect the native generated-video examples without confusing them with benchmark evidence.
Alex: And what should everyone else remember when they see a restored face that looks convincing?
Sam: Speech can constrain which sharp mouth is plausible; it cannot prove that reconstructed detail is true. The authors explicitly call that detail a generative estimate, and urge permission and disclosure for audio-driven editing.
Alex: Keep that distinction in mind when the picture gets sharper.