Learning 3D-aware Image Synthesis
with Unknown Pose Distribution

Zifan Shi*1, Yujun Shen*2, Yinghao Xu3, Sida Peng4, Yiyi Liao4, Sheng Guo2, Qifeng Chen1, Dit-Yan Yeung1
1HKUST    2Ant Group    3CUHK    4Zhejiang University
(* indicates equal contribution)
CVPR 2023
Paper Supp Demo Code

Images and geometry synthesized by PoF3D under random views,
without any pose prior.

Abstract

Existing methods for 3D-aware image synthesis largely depend on the 3D pose distribution pre-estimated on the training set. An inaccurate estimation may mislead the model into learning faulty geometry. This work proposes PoF3D that frees generative radiance fields from the requirements of 3D pose priors. We first equip the generator with an efficient pose learner, which is able to infer a pose from a latent code, to approximate the underlying true pose distribution automatically. We then assign the discriminator a task to learn pose distribution under the supervision of the generator and to differentiate real and synthesized images with the predicted pose as the condition.

The pose-free generator and the pose-aware discriminator are jointly trained in an adversarial manner. Extensive results on a couple of datasets confirm that the performance of our approach, regarding both image quality and geometry quality, is on par with state of the art. To our best knowledge, PoF3D demonstrates the feasibility of learning high-quality 3D-aware image synthesis without using 3D pose priors for the first time.

Results

Syntheses on FFHQ

Syntheses on Cats

Syntheses on ShapeNet Cars

Demo Video

BibTeX

@inproceedings{shi2023pof3d,
    title   = {Learning 3D-aware Image Synthesis with Unknown Pose Distribution},
    author  = {Shi, Zifan and Shen, Yujun, and Xu, Yinghao and Peng, Sida and Liao, Yiyi and Guo, Sheng and Chen, Qifeng and Dit-Yan Yeung},
    journal = {CVPR},
    year    = {2023}
}

The website template was adapted from Nerfies. We thank the authors for sharing the templates.