Welcome to our light-field website!

This is the webpage for light-field related researches in Prof. Ravi Ramamoorthi's lab, which is affiliated with both UC San Diego and UC Berkeley.
It includes all light-field papers (e.g. depth estimation) published in recent top conferences/journals.
If you compare to certain algorithms and/or use the datasets, please also cite the appropriate papers.

***For decoding the Lytro raw input, we recommend using Lytro's official software.
Donald's decoder is also very useful and does not require any registration.***

2017


lfv_sig

Light Field Video Capture Using a Learning-Based Hybrid Imaging System
Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei Efros, Ravi Ramamoorthi
ACM Transactions on Graphics (SIGGRAPH), 2017

paper | lo-res pdf | abstract | bibtex | project page

Capturing light fields requires a huge bandwidth to record the data: a modern light field camera can only take three images per second. Temporal interpolation at such extreme scale is infeasible as too much information will be entirely missing between adjacent frames. Instead, we develop a hybrid imaging system, adding another standard video camera to capture the temporal information. Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps. We adopt a learning-based approach, which can be decomposed into two steps: spatio-temporal flow estimation and appearance estimation. The flow estimation propagates the angular information from the light field sequence to the 2D video, so we can warp input images to the target view. The appearance estimation then combines these warped images to output the final pixels. The whole process is trained end-to-end using convolutional neural networks.

@article{wang2017light,
   author  = {Ting-Chun Wang and Jun-Yan Zhu and Nima Khademi Kalantari 
              and Alexei A. Efros and Ravi Ramamoorthi},
   title   = {Light Field Video Capture Using a Learning-Based Hybrid 
              Imaging System},
   journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH)},
   volume  = {36},
   number  = {4},
   year    = {2017},
}
      
brdf_pami

SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras
Ting-Chun Wang, Manmohan Chandraker, Alexei Efros, Ravi Ramamoorthi
Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017

paper | abstract | bibtex

In this paper, we derive a spatially-varying (SV)BRDF-invariant theory for recovering 3D shape and reflectance from light-field cameras. Our key theoretical insight is a novel analysis of diffuse plus single-lobe SVBRDFs under a light-field setup. We show that, although direct shape recovery is not possible, an equation relating depths and normals can still be derived. Using this equation, we then propose using a polynomial (quadratic) shape prior to resolve the shape ambiguity. Once shape is estimated, we also recover the reflectance. We present extensive synthetic data on the entire MERL BRDF dataset, as well as a number of real examples to validate the theory, where we simultaneously recover shape and BRDFs from a single image taken with a Lytro Illum camera.

@article{wang2017svbrdf,
   title={{SVBRDF}-Invariant Shape and Reflectance 
   Estimation from Light-Field Cameras},
   author={Wang, Ting-Chun and Chandraker, Manmohan
   and Efros, Alexei and Ramamoorthi, Ravi},
   journal={IEEE Transactions on Pattern 
   Analysis and Machine Intelligence (TPAMI)},
   year={2017},
}
      
lfmb_cvpr

Light Field Blind Motion Deblurring
Pratul P. Srinivasan, Ren Ng, Ravi Ramamoorthi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

paper | abstract | bibtex

We study the problem of deblurring light fields of general 3D scenes captured under 3D camera motion and present both theoretical and practical contributions. By analyzing the motion-blurred light field in the primal and Fourier domains, we develop intuition into the effects of camera motion on the light field, show the advantages of capturing a 4D light field instead of a conventional 2D image for motion deblurring, and derive simple methods of motion deblurring in certain cases. We then present an algorithm to blindly deblur light fields of general scenes without any estimation of scene geometry, and demonstrate that we can recover both the sharp light field and the 3D camera motion path of real and synthetically-blurred light fields.


      
lfd_cvpr

Robust Energy Minimization for BRDF-Invariant Shape from Light Fields
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Manmohan Chandraker
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

paper | abstract | bibtex | supplementary | code

Highly effective optimization frameworks have been developed for traditional multiview stereo relying on Lambertian photoconsistency. However, they do not account for complex material properties. On the other hand, recent works have explored PDE invariants for shape recovery with complex BRDFs, but they have not been incorporated into robust numerical optimization frameworks. We present a variational energy minimization framework for robust recovery of shape in multiview stereo with complex, unknown BRDFs. While our formulation is general, we demonstrate its efficacy on shape recovery using a single light field image, where the microlens array may be considered as a realization of a purely translational multiview stereo setup. Our formulation automatically balances contributions from texture gradients, traditional Lambertian photoconsistency, an appropriate BRDF-invariant PDE and a smoothness prior. Unlike prior works, our energy function inherently handles spatially-varying BRDFs and albedos. Extensive experiments with synthetic and real data show that our optimization framework consistently achieves errors lower than Lambertian baselines and further, is more robust than prior BRDF-invariant reconstruction methods.

      

2016


view_siga

Learning-Based View Synthesis for Light Field Cameras
Nima Khademi Kalantari, Ting-Chun Wang, Ravi Ramamoorthi
ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), 2016

paper | abstract | bibtex | project page

With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes. We believe our method could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.

@article{LearningViewSynthesis,
   author  = {Nima Khademi Kalantari and 
   Ting-Chun Wang and Ravi Ramamoorthi},
   title   = {Learning-Based View Synthesis 
   for Light Field Cameras},
   journal = {ACM Transactions on Graphics 
   (Proceedings of SIGGRAPH Asia 2016)},
   volume  = {35},
   number  = {6},
   year    = {2016},
}
      
lfmr_eccv

A 4D Light-Field Dataset and CNN Architectures for Material Recognition
Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei Efros, Ravi Ramamoorthi
European Conference on Computer Vision (ECCV), 2016

paper | abstract | HTML comparison | bibtex | dataset (2D thumbnail)
full dataset (15.9G)

We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field. Our dataset contains 12 material categories, each with 100 images taken with a Lytro Illum, from which we extract about 30,000 patches in total. Since recognition networks have not been trained on 4D images before, we propose and compare several novel CNN architectures to train on light-field images. In our experiments, the best performing CNN architecture achieves a 7% boost compared with 2D image classification (70% to 77%).

@inproceedings{wang2016dataset,
   title={A {4D} light-field dataset and {CNN} 
   architectures for material recognition},
   author={Wang, Ting-Chun and Zhu, Jun-Yan 
   and Hiroaki, Ebi and Chandraker, Manmohan 
   and Efros, Alexei and Ramamoorthi, Ravi},
   booktitle={Proceedings of European Conference on 
   Computer Vision (ECCV)},
   year={2016}
}
      
shading_pami

Shape Estimation from Shading, Defocus, and Correspondence Using Light-Field Angular Coherence
Michael Tao, Pratul Srinivasan, Sunil Hadap, Szymon Rusinkiewicz, Jitendra Malik, Ravi Ramamoorthi
Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2016

paper | abstract | bibtex

Light-field cameras are quickly becoming commodity items, with consumer and industrial applications. They capture many nearby views simultaneously using a single image with a micro-lens array, thereby providing a wealth of cues for depth recovery: defocus, correspondence, and shading. In particular, apart from conventional image shading, one can refocus images after acquisition, and shift one’s viewpoint within the sub-apertures of the main lens, effectively obtaining multiple views. We present a principled algorithm for dense depth estimation that combines defocus and correspondence metrics. We then extend our analysis to the additional cue of shading, using it to refine fine details in the shape. By exploiting an all-in-focus image, in which pixels are expected to exhibit angular coherence, we define an optimization framework that integrates photo consistency, depth consistency, and shading consistency. We show that combining all three sources of information: defocus, correspondence, and shading, outperforms state-of-the-art light-field depth estimation algorithms in multiple scenarios.

@article{tao2016shape,
   title={Shape Estimation from Shading, Defocus, and 
   Correspondence Using Light-Field Angular Coherence},
   author={Tao, Michael and Srinivasan, Pratul 
   and Hadap, Sunil and Rusinkiewicz, Szymon 
   and Malik, Jitendra and Ramamoorthi, Ravi},
   journal={IEEE Transactions on Pattern 
   Analysis and Machine Intelligence (TPAMI)},
   year={2016},
}
      
brdf_cvpr

SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras
Ting-Chun Wang, Manmohan Chandraker, Alexei Efros, Ravi Ramamoorthi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
(oral presentation)

paper | abstract | supplementary | HTML comparison | bibtex

In this paper, we derive a spatially-varying (SV)BRDF-invariant theory for recovering 3D shape and reflectance from light-field cameras. Our key theoretical insight is a novel analysis of diffuse plus single-lobe SVBRDFs under a light-field setup. We show that, although direct shape recovery is not possible, an equation relating depths and normals can still be derived. Using this equation, we then propose using a polynomial (quadratic) shape prior to resolve the shape ambiguity. Once shape is estimated, we also recover the reflectance. We present extensive synthetic data on the entire MERL BRDF dataset, as well as a number of real examples to validate the theory, where we simultaneously recover shape and BRDFs from a single image taken with a Lytro Illum camera.

@inproceedings{wang2016svbrdf,
   title={SVBRDF-invariant shape and reflectance 
   estimation from light-field cameras},
   author={Wang, Ting-Chun and Chandraker, Manmohan 
   and Efros, Alexei and Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE Conference on 
   Computer Vision and Pattern Recognition (CVPR)},
   year={2016}
}
      
stereo_cvpr

Depth from Semi-Calibrated Stereo and Defocus
Ting-Chun Wang, Manohar Srikanth, Ravi Ramamoorthi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

paper | abstract | HTML comparison | bibtex

In this work, we propose a multi-camera system where we combine a main high-quality camera with two low-res auxiliary cameras. The auxiliary cameras are well calibrated and act as a passive depth sensor by generating disparity maps. The main camera has an interchangeable lens and can produce good quality images at high resolution. Our goal is, given the low-res depth map from the auxiliary cameras, generate a depth map from the viewpoint of the main camera. The advantage of our system, compared to other systems such as light-field cameras or RGBD sensors, is the ability to generate a high-resolution color image with a complete depth map, without sacrificing resolution and with minimal auxiliary hardware.

@inproceedings{wang2016semi,
   title={Depth from semi-calibrated stereo and defocus},
   author={Wang, Ting-Chun and Srikanth, Manohar
   and Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE Conference on 
   Computer Vision and Pattern Recognition (CVPR)},
   year={2016}
}
      
occlusion_pami

Depth Estimation with Occlusion Modeling Using Light-field Cameras
Ting-Chun Wang, Alexei Efros, Ravi Ramamoorthi
Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2016

paper | abstract | bibtex

In this paper, an occlusion-aware depth estimation algorithm is developed; the method also enables identification of occlusion edges, which may be useful in other applications. It can be shown that although photo-consistency is not preserved for pixels at occlusions, it still holds in approximately half the viewpoints. Moreover, the line separating the two view regions (occluded object vs. occluder) has the same orientation as that of the occlusion edge in the spatial domain. By ensuring photo-consistency in only the occluded view region, depth estimation can be improved.

@article{wang2016depth,
   title={Depth estimation with occlusion modeling 
   using light-field cameras},
   author={Wang, Ting-Chun and Efros, Alexei and 
   Ramamoorthi, Ravi},
   journal={IEEE Transactions on Pattern 
   Analysis and Machine Intelligence (TPAMI)},
   year={2016},
}
      

2015


occlusion

Occlusion-aware depth estimation using light-field cameras
Ting-Chun Wang, Alexei Efros, Ravi Ramamoorthi
International Conference on Computer Vision (ICCV), 2015

paper | abstract | bibtex | supp
code | dataset (3.3GB)

In this paper, we develop a depth estimation algorithm for light field cameras that treats occlusion explicitly; the method also enables identification of occlusion edges, which may be useful in other applications. We show that, although pixels at occlusions do not preserve photo-consistency in general, they are still consistent in approximately half the viewpoints.

@inproceedings{wang2015occlusion,
   title={Occlusion-aware depth estimation using 
   light-field cameras},
   author={Wang, Ting-Chun and Efros, Alexei and 
   Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE International 
   Conference on Computer Vision (ICCV)},
   year={2015}
}
      
flow

Oriented Light-Field Windows for Scene Flow
Pratul Srinivasan, Michael Tao, Ren Ng, Ravi Ramamoorthi
International Conference on Computer Vision (ICCV), 2015

paper | abstract | bibtex | code (152MB)

For Lambertian surfaces focused to the correct depth, the 2D distribution of angular rays from a pixel remains consistent. We build on this idea to develop an oriented 4D light-field window that accounts for shearing(depth), translation (matching), and windowing. Our main application is to scene flow, a generalization of optical flow to the 3D vector field describing the motion of each point in the scene.

@inproceedings{srinivasan2015oriented,
   title={Oriented Light-Field Windows for Scene Flow},
   author={Srinivasan, Pratul and Tao, Michael 
   and Ng, Ren and Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE International 
   Conference on Computer Vision (ICCV)},
   year={2015}
}
      
shading

Depth from Shading, Defocus, and Correspondence using Light-field Angular Coherence
Michael Tao, Pratul Srinivasan, Jitendra Malik, Szymon Rusinkiewicz, Ravi Ramamoorthi
Conference on Computer Vision and Pattern Recognition (CVPR), 2015

paper | abstract | bibtex | code (72MB)

Using shading information is essential to improve shape estimation from light field cameras. We develop an improved technique for local shape estimation from defocus and correspondence cues, and show how shading can be used to further refine the depth. We show that the angular pixels have angular coherence, which exhibits three properties: photoconsistency, depth consistency, and shading consistency.

@inproceedings{tao2015shading,
   title={Depth from Shading, Defocus, and 
   Correspondence Using Light-Field Angular Coherence},
   author={Tao, Michael W and Srinivasan, Pratul P 
   and Malik, Jitendra and Rusinkiewicz, Szymon 
   and Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE Conference on 
   Computer Vision and Pattern Recognition (CVPR)},
   year={2015}
}
      
lfres

A Light Transport Framework for Lenslet Light Field Cameras
Chia-Kai Liang, Ravi Ramamoorthi
ACM Transactions on Graphics (TOG), 2015

paper | abstract | bibtex

It is often stated that there is a fundamental tradeoff between spatial and angular resolution of lenslet light field cameras, but there has been limited understanding of this tradeoff theoretically or numerically. In this paper, we develop a light transport framework for understanding the fundamental limits of light field camera resolution.

@article{liang2015light,
  title={A light transport framework for lenslet light field cameras},
  author={Liang, Chia-Kai and Ramamoorthi, Ravi},
  journal={ACM Transactions on Graphics (TOG)},
  volume={34},
  number={2},
  pages={16},
  year={2015}
}
      
glossy_pami

Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras
Michael Tao, Jong-Chyi Su, Ting-Chun Wang, Jitendra Malik, Ravi Ramamoorthi
Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2015

paper | abstract | bibtex
code (5.2MB) | dataset (1.1GB)

Light-field cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive single-shot capture. However, current light-field depth estimation methods are designed for Lambertian objects and fail or degrade for glossy or specular surfaces. In this paper, we present a novel theory of the relationship between light-field data and reflectance from the dichromatic model.

@article{tao2015specular,
title={Depth Estimation and Specular Removal for 
   Glossy Surfaces Using Point and Line Consistency 
   with Light-Field Cameras},
   author={Tao, Michael and Su, Jong-Chyi and 
   Wang, Ting-Chun and Malik, Jitendra 
   and Ramamoorthi, Ravi},
   journal={IEEE Transactions on Pattern 
   Analysis and Machine Intelligence (TPAMI)},
   year={2015},
}
      

2014


glossy

Depth estimation for glossy surfaces with light-field cameras
Michael Tao, Ting-Chun Wang, Jitendra Malik, Ravi Ramamoorthi
ECCV Workshop on Light Fields for Computer Vision (L4CV), 2014

paper | abstract | bibtex
open source decoder for Lytro Illum (44MB)

Light-field cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive single-shot capture. In this paper, we develop an iterative approach to use the benefits of light-field data to estimate and remove the specular component, improving the depth estimation. The approach enables light-field data depth estimation to support both specular and diffuse scenes.

@inproceedings{tao2014glossy,
   title={Depth estimation for glossy surfaces with 
   light-field cameras},
   author={Tao, Michael W and Wang, Ting-Chun 
   and Malik, Jitendra and Ramamoorthi, Ravi},
   booktitle={Proceedings of the IEEE European 
   Conference on Computer Vision Workshops (ECCVW)},
   year={2014},
}
      

2013


depth

Depth from Combining Defocus and Correspondence Using Light-Field Cameras
Michael Tao, Sunil Hadap, Jitendra Malik, Ravi Ramamoorthi
International Conference on Computer Vision (ICCV), 2013

paper | abstract | bibtex | supp | video (38MB)
code (8.8MB) | dataset (83MB)

Light-field cameras have recently become available to the consumer market. An array of micro-lenses captures enough information that one can refocus images after acquisition, as well as shift one's viewpoint within the sub-apertures of the main lens, effectively obtaining multiple views. Thus, depth cues from both defocus and correspondence are available simultaneously in a single capture, and we show how to exploit both by analyzing the EPI.

@inproceedings{tao2013depth,
   author={Tao, Michael and Hadap, Sunil
   and Malik, Jitendra and Ramamoorthi, Ravi},
   title={Depth from combining defocus and 
   correspondence using light-field cameras},	
   booktitle={Proceedings of the IEEE International 
   Conference on Computer Vision (ICCV)},
   year={2013},
}
      
depth

External Mask Based Depth and Light Field Camera
Dikpal Reddy, Jiamin Bai, Ravi Ramamoorthi
ICCV Workshop on Consumer Depth Cameras for Vision, 2013

paper | abstract | bibtex | video (97MB)

We present a method to convert a digital single-lens reflex (DSLR) camera into a high-resolution consumer depth and light-field camera by affixing an external aperture mask to the main lens. Compared to the existing consumer depth and light field cameras, our camera is easy to construct with minimal additional costs, and our design is camera and lens agnostic. The main advantage of our design is the ease of switching between an SLR camera and a native resolution depth/light field camera. We also do not need to modify the internals of the camera or the lens.

@inproceedings{reddy2013external,
   author={Reddy, Deepti and Bai, Jie and Ramamoorthi, Ravi},
   title={External mask based depth and light field camera},	
   booktitle={Proceedings of the IEEE International 
   Conference on Computer Vision (ICCV) Workshops},
   year={2013},
}