Gaussian Shell Maps for Efficient 3D Human Generation

Rameen Abdal^*1, Wang Yifan^*1, Zifan Shi^*2, Yinghao Xu¹, Ryan Po¹, Zhengfei Kuang¹, Qifeng Chen² Dit-Yan Yeung², Gordon Wetzstein¹

CVPR 2024

¹Stanford University, ²HKUST

* Equal Contribution

Paper arXiv Video Code

Gaussian Shell Maps: Gaussian Shell Maps is an efficient framework for 3D human generation connecting 3D Gaussians with CNN-based generators. 3D Gaussians are anchored to “shells” derived from the SMPL template (only two shells are visualized for clarity), and the appearance is modeled in texture space. Trained only on 2D images, we show that our method can generate diverse articulable humans in real-time with state-of-the-art quality directly in high resolution without the need for upsampling and hence avoiding aliasing artifacts.

Method Overview : We propose an expressive yet highly efficient representation, Gaussian Shell Map (GSM), for 3D human generation. Combining the idea of 3D Gaussians and Shell Maps, we sample 3D Gaussians on “shells”, which are mesh layers offsetted from the SMPL template, forming a shell volume to model complex and diverse geometry and appearance; the Gaussian parameters are learned in the texture space, allowing us to leverage existing CNN-based generative architecture. Articulation is straightforward by interpolating the deformation of the shell. The generation is supervised by single-view 2D images using several discriminator critics, including part-specific face, hands, and feet discriminators.

Overview

Poses

Semantic Editing

Interpolations

Animations

Walking sequence

Jumping sequence

Swing sequence

Novel Views

Ours

AG3D

BibTeX

@misc{abdal2023gaussian,
      title={Gaussian Shell Maps for Efficient 3D Human Generation}, 
      author={Rameen Abdal and Wang Yifan and Zifan Shi and Yinghao Xu and Ryan Po and Zhengfei Kuang and Qifeng Chen and Dit-Yan Yeung and Gordon Wetzstein},
      year={2023},
      eprint={2311.17857},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}