I am a fourth-year CS Ph.D. student at Zhejiang University, fortunately advised by Prof. Zhou Zhao (赵洲). I won the National Scholarship in 2021.

My current research interests include 3D Talking Face Generation and Text-to-Speech (TTS). I have also deeply investigated Deep Reinforcement Learning (DRL) and Multi-Agent Systems (MAS). I have published 15+ papers in high-impact conference/journals, including ICLR, IJCAI, ACL, IEEE TMC, etc.

📝 Publications

🦸 Digital Avatar

ICLR 2024 Spotlight
sym

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun MA, Zhou Zhao

ICLR 2024 Spotlight

Project Page img

  • Real3D-Portrait is the first one-shot NeRF-based talking face system with realistic head, torso, and background segments.
  • It facilitates both audio / video-driven one-shot talking face generation.
Arxiv
sym

Geneface++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Jiangwei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, Zhou Zhao

Under Review

Project Page) img

  • GeneFace++ is a modern talking face system that aims to achieve the goal of generalized lip synchronization, good video quality, and high system efficiency.
  • It greatly improves the stability and efficiency of NeRF-based methods.
ICLR 2023
sym

Geneface: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, Zhou Zhao

ICLR 2023 Poster

Project Page img

  • GeneFace is a NeRF-based talking face system that generalizes well to various OOD audios.
  • It first utilizes a generative model to model the audio-to-motion mapping.

🎙 Speech Synthesis

ACL 2023
sym

CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training
Zhenhui Ye, Rongjie Huang, Yi Ren, Ziyue Jiang, Jinglin Liu, Jinzheng He, Zhou Zhao

ACL 2023 Poster

Project Page img

  • CLAPSpeech is the first cross-modal contrastive learning method that focus on extracting prosody-related text representation for text-to-speech (TTS).
  • It provides a convenient plug-in text encoder applicable for all TTS models to improve prosody.
IJCAI 2022
sym

SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech
Zhenhui Ye, Zhou Zhao, Yi Ren, Fei Wu

IJCAI 2022 Poster

Project Page img

  • SyntaSpeech is the first syntax-aware non-autoregressive TTS acoustic model.
  • We design a syntatic graph encoder to extract syntax-related prosody from text.

📚 Deep Reinforcement Learning

IEEE TMC 2022
sym

Soft-DRGN: Multi-UAV Navigation for Partially Observable Communication Coverage by Graph Reinforcement Learning
Zhenhui Ye, Ke Wang, Yining Chen, Xiaohong Jiang, Guanghua Song.

IEEE transactions on Mobile Computing 2022

Project Page img

  • We propose Soft-DRGN to learn robust stochastic policies for large-scale multi-agent cooperation.
  • We propose to utilize graph attention network to learn the inter-agent communication.
Applied Intelligence 2022
sym

Improving Sample Efficiency in Multi-Agent Actor-Critic Methods

Zhenhui Ye, Yining Chen, Xiaohong Jiang, Guanghua Song,

Applied Intelligence 2022

  • We propose Experience Augmentation (EA) to improve the sample efficiency for homogeneous MARL tasks.
  • We propose a sample-efficient training pipeline called PEDMA.

🎖 Honors and Awards

  • 2022.12 Runner-up in China Graduate AI Innovation Competition (2/1217)
  • 2022.10 Tecent Scholarship (as Ph.D Student) (top 1%)
  • 2021.10 National Scholarship (as Master Student) (Top 1%)
  • 2020.6 Outstanding Graduate of Zhejiang University (as Undergraduate Student) (Top 5%)

📖 Educations

  • 2021.9 - 2025.6 (now) Ph.D student, College of Computer Science and Technology, Zhejiang University, Hangzhou.
  • 2020.06 - 2021.9, Master student, School of Aerospace and Astronautics, Zhejiang University, Hangzhou.
  • 2016.09 - 2020.06, Undergraduate, School of Aerospace and Astronautics, Zhejiang Univeristy, Hangzhou.

Academic Services

  • Conference Reviewer: ICLR 2023, EMNLP 2023, NeurIPS 2023, ACL 2024, ICLR 2024, CVPR 2024