Deep Polarization Imaging for 3D Shape and SVBRDF Acquisition
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021 (Oral)
Valentin Deschaintre Yiming Lin Abhijeet Ghosh
Imperial College London, RGI Group
Patents Pending
Fig. 1: Examples of acquired 3D objects, including shape (normals and depth) and spatially varying reflectance, using frontal flash illumination.
Abstract:
We present a novel method for efficient acquisition of shape and spatially varying reflectance of 3D objects using polarization cues. Unlike previous works that have exploited polarization to estimate material or object appearance under certain constraints (known shape or multiview acquisition), we lift such restrictions by coupling polarization imaging with deep learning to achieve high quality estimate of 3D object shape (surface normals and depth) and SVBRDF using single-view polarization imaging under frontal flash illumination. In addition to acquired polarization images, we provide our deep network with strong novel cues related to shape and reflectance, in the form of a normalized Stokes map and an estimate of diffuse color. We additionally describe modifications to network architecture and training loss which provide further qualitative improvements. We demonstrate our approach to achieve superior results compared to recent works employing deep learning in conjunction with flash illumination.
Publication: Deep polarization imaging for 3D shape and SVBRDF acquisition. Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. Proc. of CVPR, June 2021. (Patents Pending)
Paper:
Supplemental material: Zip file (100 Mo)
Code: We make the code available strictly for academic research purposes only, please e-mail Valentin Deschaintre and Abhijeet Ghosh to request access.
Data: You can find here a sample of the dataset: Zip file (249.8 MB)
Please e-mail Valentin Deschaintre and Abhijeet Ghosh if you would like access to the entire dataset for academic research purposes.
Presentation video:
Bibtex:
@InProceedings{Deschaintre21,
author = {Deschaintre, Valentin and Lin, Yiming and Ghosh, Abhijeet},
title = {Deep polarization imaging for 3D shape and SVBRDF acquisition},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021}
}
Acknowledgements:
This work was supported by an EPSRC Early Career Fellowship (EP/N006259/1) and a hardware donation from Nvidia. We particularly thank Dr Bernhard Kainz for access to GPUs which made the CVPR submission possible!
