On September 21 2022 at 10.00, Pengpeng Hu will defend his PhD entitled “Deep learning-based 3D human body shape reconstruction from point clouds”.
Everybody is invited to attend the presentation online via this link.
3D reconstruction of the human body shape is a fundamental problem in computer vision, which is valuable for various human-centric applications such as computer animation, virtual reality, and clothing design, to name a few. 3D scanning is a popular technology for acquiring the geometry of a subject based on which a 3D body reconstruction can be produced. Although countless body scanners were developed to meet different industrial requirements and a lot of advanced algorithms were proposed for optimizing the reconstructed body models, many problems are still not properly solved. These problems, however, are difficult to address using conventional methods.
Recent years have witnessed the rapid development of artificial intelligence, especially deep learning. Encouraged by the significant success of deep learning in image processing, an increasing number of researchers attempted to extend deep learning to deal with 3D data. Following this trend, we proposed deep learning-based solutions to several challenges existing in modern 3D body scanning and reconstruction.
In this thesis, we focus on four challenges of 3D body scanning, namely, (1) estimation of body shape under clothing, (2) body reconstruction from impaired point clouds, (3) registration of non-overlapping point clouds, and (4) animatable body reconstruction using a single depth camera. The first challenge arises from the fact that existing 3D scanning solutions require the subjects to get scanned with minimal clothing as the scanning device can only record the outmost surface of objects. This scanning procedure is inconvenient to most people and is also an infringement of the right to privacy. The second challenge comes from the observation that impaired point clouds are common in practice but they lack a systematic study. Moreover, the problems of misalignment and problematic posture are neglected in existing solutions. The third challenge is a classical problem: partial point cloud registration. We found that existing methods mainly rely on the assumption that the source and the target point clouds have sufficient overlap and none of them could handle non-overlapping registration. The last challenge is addressed as many applications demand dynamic human body models. Traditional methods require expensive professional devices to produce such models.
We have addressed these four challenges by leveraging the deep learning paradigm. Our first contribution is to propose the first deep learning-based method in the literature for estimating the body shape under clothing from a single 3D dressed body scan. To facilitate the proposed model, a novel dataset consisting of large-scale dressed body scans and corresponding ground-truth body shapes is proposed. Our second contribution is a novel deep learning approach for jointly reconstructing an accurate body mesh and normalizing the posture of the human body model from a low-quality body point cloud in arbitrary postures. It proposes to directly reconstruct high-fidelity body shapes from impaired point clouds instead of attempting to point cloud repairments. Our third contribution is the first deep learning-based method in the literature to align non-overlapping partial point clouds. Using this method, an omnidirectional body can be obtained from only two non-overlapping body scans. The last contribution in this thesis is to propose a novel deep learning-based method to reconstruct an animatable body shape from only two depth images and at the same time allow for large pose variations between the camera shots.
Extensive experiments based on different datasets have demonstrated that the proposed methods outperform the reference methods from the literature. Our work has resulted in numerous high-quality scientific publications and has demonstrated impact at both academic and industrial levels.
