Precomputed Radiance Transfer (PRT) remains an attractive solution for real-time rendering of complex light transport effects such as glossy global illumination. After precomputation, we can relight the scene with new environment maps while changing viewpoint in real-time. However, practical PRT methods are usually limited to low-frequency spherical harmonic lighting. All-frequency techniques using wavelets are promising but have so far had little practical impact.
In this paper, we demonstrate a hybrid neural-wavelet PRT solution to high-frequency indirect illumination, including glossy reflection, for relighting with changing view. Specifically, we seek to represent the light transport function in the Haar wavelet basis. For global illumination, we learn the wavelet transport using a small multi-layer perceptron (MLP) applied to a feature field as a function of spatial location and wavelet index, with reflected direction and material parameters being other MLP inputs. We optimize the feature field (compactly represented by a tensor decomposition) and MLP parameters from multiple images of the scene under different lighting and viewing conditions.
We demonstrate real-time (512 x 512 at 24 FPS, 800 x 600 at 13 FPS) precomputed rendering of challenging scenes involving view-dependent reflections and even caustics.
Our method is able to render scenes under arbitrary view. For the Four Animals scene, note how the mirror reflections slide across the horse, and note how the reflective caustic part of the reflection within the ring remains view-independent. Also, note how for the diffuse Armadillo scene all the interreflections remain view-independent.
Our method is able to handle changing lighting conditions. For the Kitchen scene, note the reflections on the backboard of the stove. For the Armadillo scene, note how the diffuse interreflections remain unchanged.
Our method is able to generalize complex illumination conditions under novel view and lighting conditions better than previous work. Note the rotation of the caustic within the ring, as well as the glossy interreflections of the animals on the ground.
Our method is able to generalize to any novel lighting condition, including those far from training. By predicting in an orthonormal basis (Haar wavelets), we can recover the correct shading much more accurately than if the lighting were predicted directly via neural network.
@article{raghavanxiao2023nert,
author = {Raghavan, Nithin and Xiao, Yan and Lin, Kai-En and Sun, Tiancheng and Bi, Sai and Xu, Zexiang and Li, Tzu-Mao and Ramamoorthi, Ravi},
title = {Neural Free-Viewpoint Relighting for Glossy Indirect Illumination},
journal = {Computer Graphics Forum (Proc. EGSR 2023)},
year = {2023},
}|
Template taken from Nerfies (source code here) |