MimicTalk: Mimicking a personalized and expressive 3D talking face in few minutes
Zhenhui Ye, Tianyun Zhong, Yi Ren, Ziyue Jiang, Jiawei Huang, Rongjie Huang, Jinglin Liu, Chen Zhang, Zehan Wang, Xize Chen, Xiang Yin, Zhou Zhao

NeurIPS 2024

Project Page

• MimicTalk aims at training a high-quality personalized digital avatar in several minutes.
• By ICS-A2M Model, we can in-context mimicking the talking style of the target ID.
• By fine-tuning Real3D-Portrait (Our previous large-scale talking face model), we can achieve fast adaptation in several minutes.

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun MA, Zhou Zhao

ICLR 2024 Spotlight

Project Page

• Real3D-Portrait is the first one-shot NeRF-based talking face system with realistic head, torso, and background segments.
• It facilitates both audio / video-driven one-shot talking face generation.

Geneface++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Jiangwei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, Zhou Zhao

Under Review

Project Page)

• GeneFace++ is a modern talking face system that aims to achieve the goal of generalized lip synchronization, good video quality, and high system efficiency.
• It greatly improves the stability and efficiency of NeRF-based methods.

Geneface: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, Zhou Zhao

ICLR 2023 Poster

Project Page

• GeneFace is a NeRF-based talking face system that generalizes well to various OOD audios.
• It first utilizes a generative model to model the audio-to-motion mapping.

CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training
Zhenhui Ye, Rongjie Huang, Yi Ren, Ziyue Jiang, Jinglin Liu, Jinzheng He, Zhou Zhao

ACL 2023 Poster

Project Page

• CLAPSpeech is the first cross-modal contrastive learning method that focus on extracting prosody-related text representation for text-to-speech (TTS).
• It provides a convenient plug-in text encoder applicable for all TTS models to improve prosody.

SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech
Zhenhui Ye, Zhou Zhao, Yi Ren, Fei Wu

IJCAI 2022 Poster

Project Page

• SyntaSpeech is the first syntax-aware non-autoregressive TTS acoustic model.
• We design a syntatic graph encoder to extract syntax-related prosody from text.

Soft-DRGN: Multi-UAV Navigation for Partially Observable Communication Coverage by Graph Reinforcement Learning
Zhenhui Ye, Ke Wang, Yining Chen, Xiaohong Jiang, Guanghua Song.

IEEE transactions on Mobile Computing 2022

Project Page

• We propose Soft-DRGN to learn robust stochastic policies for large-scale multi-agent cooperation.
• We propose to utilize graph attention network to learn the inter-agent communication.

Improving Sample Efficiency in Multi-Agent Actor-Critic Methods

Zhenhui Ye, Yining Chen, Xiaohong Jiang, Guanghua Song,

Applied Intelligence 2022

• We propose Experience Augmentation (EA) to improve the sample efficiency for homogeneous MARL tasks.
• We propose a sample-efficient training pipeline called PEDMA.