Learning to Reconstruct Shape and Spatially-Varying Reflectance from
a Single Image
Siggraph Asia 2018
Abstract:
Reconstructing shape and reflectance properties from images is a highly
under-constrained problem, and has previously been addressed by using
specialized hardware to capture calibrated data or by assuming known (or
highly constrained) shape or reflectance. In contrast, we demonstrate that
we can recover non-Lambertian, spatially-varying BRDFs and complex geometry
belonging to any arbitrary shape class, from a single RGB image
captured under a combination of unknown environment illumination and
flash lighting. We achieve this by training a deep neural network to regress
shape and reflectance from the image. Our network is able to address this
problem because of three novel contributions: first, we build a large-scale
dataset of procedurally generated shapes and real-world complex SVBRDFs
that approximate real world appearance well. Second, single image inverse
rendering requires reasoning at multiple scales, and we propose a cascade
network structure that allows this in a tractable manner. Finally, we incorporate
an in-network rendering layer that aids the reconstruction task by
handling global illumination effects that are important for real-world scenes.
Together, these contributions allow us to tackle the entire inverse rendering
problem in a holistic manner and produce state-of-the-art results on both
synthetic and real data.
Download
Bibtex
@inproceedings{li2018learning,
title={Learning to reconstruct shape and spatially-varying reflectance from a single image},
author={Li, Zhengqin and Xu, Zexiang and Ramamoorthi, Ravi and Sunkavalli, Kalyan and Chandraker, Manmohan},
booktitle={SIGGRAPH Asia 2018 Technical Papers},
pages={269},
year={2018},
organization={ACM}
}