Tencent Propose AniPortrait: An Audio-Driven Synthesis of Photorealistic Portrait Animation

AniPortrait is a framework that generates high-quality portrait animations driven by audio and a reference image. It overcomes challenges in producing visually captivating animations by utilizing transformer-based models and a robust diffusion model. The framework comprises two stages: extracting facial mesh and head pose from audio in the first stage, and transforming facial landmark sequences into a photorealistic animated portrait using a motion module in the second stage.