Semantic object reconstruction via casual handheld scanning

Ruizhen Hu, Cheng Wen, Oliver Van Kaick, Luanmin Chen, Di Lin, Daniel Cohen-Or, Hui Huang

Research output: Contribution to journalArticlepeer-review

Abstract

We introduce a learning-based method to reconstruct objects acquired in a casual handheld scanning setting with a depth camera. Our method is based on two core components. First, a deep network that provides a semantic segmentation and labeling of the frames of an input RGBD sequence. Second, an alignment and reconstruction method that employs the semantic labeling to reconstruct the acquired object from the frames. We demonstrate that the use of a semantic labeling improves the reconstructions of the objects, when compared to methods that use only the depth information of the frames. Moreover, since training a deep network requires a large amount of labeled data, a key contribution of our work is an active self-learning framework to simplify the creation of the training data. Speciically, we iteratively predict the labeling of frames with the neural network, reconstruct the object from the labeled frames, and evaluate the conidence of the labeling, to incrementally train the neural network while requiring only a small amount of user-provided annotations. We show that this method enables the creation of data for training a neural network with high accuracy, while requiring only little manual efort.

Original languageEnglish
Article number219
JournalACM Transactions on Graphics
Volume37
Issue number6
DOIs
StatePublished - Nov 2018
Externally publishedYes

Keywords

  • 3D scanning
  • Active learning
  • Semantic reconstruction
  • object registration

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Semantic object reconstruction via casual handheld scanning'. Together they form a unique fingerprint.

Cite this