Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation
Authors & Institutions
Haijie Yang
Nanjing University of Science and Technology, China
Jindi Bao
Nanjing University of Science and Technology, China
Yixuan Dong
Tsientang Institute for Advanced Study, China
Hongliang Zhang
Nanjing University of Science and Technology, China
Jian Bi
Nanjing University of Science and Technology, China
Hao Tang
Peking University, China
Zhenyu Zhang
Nanjing University, China
Jianjun Qian
Nanjing University of Science and Technology, China
Jian Yang
Nanjing University of Science and Technology, China
What Problem It Solves
The paper separates reusable emotional motion, cross-identity retargeting, and appearance rendering so that a single reference portrait can be animated in real time from audio and an emotion label. It specifically removes repeated reference-feature computation without retraining a cache for each target identity.
Key Result
On an RTX 4090 the full system runs at 32 FPS. In self-reconstruction it reaches 31.24 PSNR, 0.018 LPIPS, 0.883 SSIM, and a 6.314 lip-sync score, leading most quality and motion metrics despite EmoTag's higher raw frame rate. In cross-identity driving it records 29.64 FID, 0.745 identity similarity, 75.4% emotion accuracy, and 6.096 lip sync. Participants preferred it for emotional expressiveness in 70% of comparisons, lip synchronization in 60%, and visual realism in 65%; extended clips showed no obvious identity or sync drift.
Abstract
Audio-driven portrait animation has advanced rapidly with diffusion-based generative models, yet real-time one-shot generation with expressive emotion control remains challenging. Existing methods often suffer from insufficient emotion-aware motion priors and expensive appearance computation during multi-step denoising. To address these issues, we propose Proxy Avatar Meets Low-Rank Caching, a cascaded framework for real-time one-shot emotion-controllable portrait animation. Instead of directly generating the target portrait from audio, our method uses a Gaussian-based emotion proxy avatar as a reusable motion generator, which is trained once on a single identity to produce expressive driving videos from audio and emotion labels. Since the proxy avatar only provides motion rather than target appearance or geometry, a large-scale one-shot retargeting model further extracts identity-independent motion from the proxy performance and adapts it to arbitrary target portraits. To improve inference efficiency, we introduce zero-shot appearance reuse with low-rank caching, which caches reference appearance features at the initial denoising step and models subsequent feature variations using lightweight low-rank adapters. Extensive experiments demonstrate that our method achieves stronger emotional expressiveness, better identity-preserving animation, and substantially reduced inference cost, enabling real-time one-shot portrait animation.
Research Starting Point
One-shot talking portraits need to preserve an unseen person's appearance, follow speech, express a requested emotion, and run interactively on available hardware. Diffusion systems repeatedly recompute nearly static appearance features during denoising, while emotion controls often entangle identity and motion or require a reference video for every style. The result is either expressive but slow generation or fast animation with weak emotional range.
Method
A Gaussian-based proxy avatar is trained once on one performer and serves only as an audio-and-emotion-conditioned motion generator. A large-scale one-shot retargeting network extracts identity-independent motion from that proxy video and applies it to arbitrary target portraits. During few-step diffusion, zero-shot appearance reuse caches reference features at the first step, while lightweight low-rank adapters model only their later variation. This cascaded division of labor is evaluated in self-reconstruction, cross-identity one-shot driving, long-video stability, ablations, and a blinded study with twenty participants.
Paper Summary
The reusable proxy-plus-retargeting design makes real-time emotional avatars practical without per-user diffusion training. The reported 32 FPS on a consumer RTX 4090 is promising, but deployment sizing should include the one-time proxy training and the quality/cost trade-off of the large retargeting model, not just steady-state frame rate.