Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

Vision Transformer for NeRF-Based View Synthesis
from a Single Input Image

arxiv preprint

Kai-En Lin1 Lin Yen-Chen2 Wei-Sheng Lai3
Tsung-Yi Lin4* Yi-Chang Shih3 Ravi Ramamoorthi1
1University of California, San Diego 2Massachusetts Institute of Technology
3Google 4NVIDIA
*Work done while at Google


Abstract:

Although neural radiance fields (NeRF) have shown impressive advances for novel view synthesis, most methods typically require multiple input images of the same scene with accurate camera poses. In this work, we seek to substantially reduce the inputs to a single unposed image. Existing approaches condition on local image features to reconstruct a 3D object, but often render blurry predictions at viewpoints that are far away from the source view. To address this issue, we propose to leverage both the global and local features to form an expressive 3D representation. The global features are learned from a vision transformer, while the local features are extracted from a 2D convolutional network. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. This novel 3D representation allows the network to reconstruct unseen regions without enforcing constraints like symmetry or canonical coordinate systems. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model. Quantitative and qualitative evaluations demonstrate that the proposed method achieves state-of-the-art performance and renders richer details than existing approaches.


Results

Cars Dataset (128 x 128, 24 FPS)

shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif



Chairs Dataset (128 x 128, 24 FPS)

shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif shapenet results gif

Download

Paper
Code (coming soon)
More results

Bibtex

@inproceedings {lin2022visionnerf,
    booktitle = {arxiv},
    title = {Vision Transformer for NeRF-Based View Synthesis from a Single Input Image},
    author = {Lin, Kai-En and Yen-Chen, Lin and Lai, Wei-Sheng and Lin, Tsung-Yi and 
Shih, Yi-Chang and Ramamoorthi, Ravi}, year = {2022}, }