Improving 3D-aware Image Synthesis with A
Geometry-aware Discriminator

NeurIPS 2022

Zifan Shi1     Yinghao Xu2     Yujun Shen3     Deli Zhao3     Qifeng Chen1     Dit-Yan Yeung1
1HKUST      2CUHK      3Ant Group
[Paper]   [Code]   [Demo]
Figure: (a) Existing 3D-aware GANs where only the generator is made 3D-aware with the help of NeRF. (b) GeoD where the discriminator supervises the generator with the extracted geometry.
Overview
This work aims at improving 3D-aware image synthesis from the discriminator's perspective. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency.
Results
Qualitative comparison with pi-GAN as the base model.
3D reconstruction and novel view synthesis on real data with GAN inversion.
Demo

We include a demo video, which shows the continuous 3D control achieved by our GeoD.
BibTeX
  @inproceedings{shi2022improving,
    title   = {Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator},
    author  = {Shi, Zifan and Xu, Yinghao and Shen, Yujun and Zhao, Deli and Chen, Qifeng and Yeung, Dit-Yan},
    booktitle = {NeurIPS},
    year    = {2022}
  }
Related Work
Katja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas Geiger. GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.
Comment: Proposes the generative radiance fields for 3D-aware image synthesis.
Michael Niemeyer, Andreas Geiger. GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.
Comment: Proposes the compositional generative neural feature fields for scene synthesis.
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein. pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. CVPR, 2021.
Comment: Proposes the periodic implicit generative neural feature fields for 3d-aware image synthesis.
Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt. StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis. ICLR, 2022.
Comment: Proposes to integrate the neural radiance field into a style-based generator.
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou. 3D-aware Image Synthesis via Learning Structural and Textural Representations. CVPR, 2022.
Comment: Proposes to explicitly learn a structural representation and a texture representation.