Ravi Ramamoorthi Ronald L. Graham Chair of Computer Science
Professor CSE Department ; affiliate in ECE 
Office: 4118 EBU3B Phone: 8588221483 Fax: 8585347029 ravir@cs.ucsd.edu URL: http://www.cs.ucsd.edu/~ravir CV Publications 
Teaching 
CSE 167  Computer Graphics  Winter 2017  
CSE 274  Topics in Computer Graphics  Fall 2015 (High Quality RealTime Rendering)  Winter 2015 (CSE 291) 
CSE 190  Advanced Computer Graphics  Winter 2016  Spring 2015 
Selected Awards 
2017 IEEE Fellow  CSE Article  
2016 edX Prize for Exceptional Online Teaching and Learning (inaugural finalist)  edX Blog  UCSD art.  Prize Certificate  
2016 Ronald L. Graham (Endowed) Chair of Computer Science  CSE Article  Announcement Card  
2011 Okawa Foundation Research Grant  Award Certificate  Website  
2008 PECASE (Presidential Early Career Awards for Scientists and Engineers)  Award Certificate  Award Photo  Group Photo with President Bush  White House Press Release 
2007 SIGGRAPH Significant New Researcher Award for Computer Graphics  Video  Citation  Press Release 
Selected New Research: 201416 
Selected New Papers in 2016: SIG, SIGASIA, CVPR, ECCV (See pub. list for others) 



Specular MicroStructure (SIGGRAPH 16) 
View Synthesis Light Fields (SIGASIA 16) 
TwoShot BRDF NearField (SIGASIA 16) 
Downsampling Scattering (SIGASIA 16) 
SVBRDFInvariant LF Depth (CVPR 16) 
Linear Depth from Polarization (ECCV 16) 

Selected New Papers in 2015: TOG, SIGASIA, CVPR, ICCV (See pub. list for others) 



Accurate Fur (SIGASIA 15) 
Minimal BRDF Sampling (SIGASIA 15) 
LF Depth from Shading (CVPR 15) 
Fast 4D Shear Filter (TOG 15) 
OcclusionAware LF Depth (ICCV 15) 
Underwater Photo. Stereo (ICCV 15) 

New Papers in 2014: TOG, SIGGRAPH, SciVis (See pub. list for others) 



City Forensics IEEE SciVis 14 
Lenslet Light Field Theory (TOG 15) 
HighOrder Similarity (SIG 14) 
Rendering Glints Normal Maps (SIG 14) 
Factored AxisAligned Filtering (SIG 14) 
Discrete Stochastic Microfacet (SIG 14) 
Research Summary 
My research group develops the theoretical foundations, mathematical representations and computational models for the visual appearance of objects, digitally recreating or rendering the complexity of natural appearance. Our research program cuts across computer graphics, computer vision and signal processing with applications in sparse reconstruction and frequency analysis, Monte Carlo importance sampling, interactive photorealistic rendering, acquisition and representation of datadriven appearance, volumetric scattering, animation, image and video editing, lightfield cameras, physicsbased vision and lightinginsensitive recognition. This work has led to more than 100 publications, including more than 50 SIGGRAPH or TOG papers, and has been recognized in 2005 by a Sloan Fellowship and an NSF CAREER award, and in 2007 with an ONR Young Investigator Award and the ACM SIGGRAPH Significant New Researcher Award. More recently, I received a Presidential Early Career Award in a White House ceremony in Dec 2008, and an Okawa Foundation Award in 2011. In Spring 2016, I was appointed the inaugural holder of the Ronald L. Graham Chair of Computer Science. Most recently, I was named a finalist for the inaugural edX prize for exceptional contributions in online teaching and learning, and elevated to IEEE fellow for contributions to foundations of computer graphics and computer vision. A full CV is available.
While my focus has been primarily on academic publication, many of my papers have been refined and later widely implemented in products and commercial applications. For example, my work on spherical harmonic lighting and irradiance environment maps is now widely included in games (such as the Halo series), and is increasingly adopted in movie production (being a critical component of the rendering pipeline in Avatar in 2010, and now an integral part of RenderMan 16, since mid2011). These ideas are also being used by Adobe for relighting. My research on importance sampling has inspired a sampling and imagebased lighting pipeline that is becoming standard for production rendering (also included in RenderMan 16) and is used for example on the newly released Pixar movie, Monsters University (my recent papers discussing production use methods are presented at EGSR 2012 and the inaugural JCGT paper). Models for volumetric scattering have been used in demos by NVIDIA, and elsewhere in industry. I also participated in developing the first electronic field guide ; a subsequent iPhone app developed by Prof. Belhumeur and colleagues is now widely used by the public for visual species identification.
A somewhat out of date summary of research projects is available (the listing of papers below is up to date). I am still interested in many of the same areas, as well as a broader range of topics. My current funding and research interests fall in four main directions: (1) Signal Processing and Sparse Reconstruction of Visual Appearance, with implications across Rendering, Imaging and Animation (see position paper ); (2) A Digital Visual Appearance Pipeline for complex visually rich materials; (3) PhysicsBased Computer Vision with realistic reflectance, illumination and light transport, and more generally problems at the visiongraphics interface, including image and video editing and manipulation; (4) Light Field Cameras and RGBD data for depth and reflectance recovery, and higherlevel vision/graphics applications. Please see our Light Field Website. I am also interested in collaborations in and outside the field that leverage expertise in these areas.
Overview Talks and Videos 
Here is a selection of recent invited talks that give an overview of research.
Here is a selection of recent SIGGRAPH and other videos:
Publications 
Antialiasing Complex Global Illumination Effects in Pathspace
TOG 2016 (to appear) We present the first method to efficiently predict antialiasing footprints to prefilter color, normal, and displacementmapped appearance in the context of multibounce global illumination. We derive Fourier spectra for radiance and importance functions that allow us to compute spatialangular filtering footprints at path vertices. Paper: PDF 

LearningBased View Synthesis for Light Field Cameras
SIGGRAPH Asia 2016 we propose a novel learningbased approach to synthesize new views from a sparse set of input views for light field cameras. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. Paper: PDF Video: MPEG Dataset: Project Page 

Minimal BRDF Sampling for TwoShot NearField Reflectance Acquisition
SIGGRAPH Asia 2016 We develop a method to acquire the BRDF of a homogeneous flat sample from only two images, taken by a nearfield perspective camera, and lit by a directional light source. We develop a mathematical framework to estimate error from a given set of measurements, including the use of multiple measurements in an image simultaneously, as needed for acquisition from nearfield setups. Paper: PDF Supplementary: PDF Comparison Video: QT 

Downsampling Scattering Parameters for Rendering Anisotropic Media
SIGGRAPH Asia 2016 Volumetric microappearance models have provided remarkably highquality renderings, but are highly data intensive and usually require tens of gigabytes in storage. We introduce a joint optimization of singlescattering albedos and phase functions to accurately downsample heterogeneous and anisotropic media. Paper: PDF Video: MPEG 

Photometric Stereo in a Scattering Medium
PAMI 2016 Photometric stereo is widely used for 3D reconstruction. However, its use in scattering media such as water, biologicaltissue and fog has been limited until now, because of forward scattered light from both the source and object, as well as light scatteredback from the medium (backscatter). Here we make three contributions to address the key modes of light propagation, under thecommon single scattering assumption for dilute media. Paper: PDF 

Linear Depth Estimation from an Uncalibrated, Monocular Polarisation Image
ECCV 2016 We present a method for estimating surface height directly from a single polarisation image simply by solving a large, sparse system of linear equations. To do so, we show how to express polarisation constraints as equations that are linear in the unknown depth. Our method is applicable to objects with uniform albedo exhibiting diffuse and specular reflectance. We believe that our method is the first monocular, passive shapefromx technique that enables wellposed depth estimation with only a single, uncalibrated illumination condition. Paper: PDF 

A 4D LightField Dataset and CNN Architectures for Material Recognition
ECCV 2016 We introduce a new lightfield dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D lightfield. Our dataset contains 12 material categories, each with 100 images taken with a Lytro Illum. Since recognition networks have not been trained on 4D images before, we propose and compare several novel CNN architectures to train on lightfield images. In our experiments, the best performing CNN architecture achieves a 7% boost compared with 2D image classification. Paper: PDF HTML Comparison dataset (2D thumbnail) Full Dataset (16GB) 

Sparse Sampling for ImageBased SVBRDF Acquisition
Material Appearance Modeling Workshop, 2016 We acquire the datadriven spatiallyvarying (SV)BRDF of a flat sample from only a small number of images (typically 20). We generalize the homogenous BRDF acquisition work of Nielsen et al., who derived an optimal minmal set of lighting/view directions. We demonstrate our method on SVBRDF measurements of new flat materials, showing thatfull datadriven SVBRDF acquisition is now possible from a sparse set of only about 20 lightview pairs. Paper: PDF 

PositionNormal Distributions for Efficient Rendering of Specular Microstructure
SIGGRAPH 2016 Specular BRDF rendering traditionally approximates surface microstructure using a smooth normal distribution, but this ignores glinty effects, easily observable in the real world. We treat a specular surface as a fourdimensional positionnormal distribution, and fit this distribution using millions of 4D Gaussians, which we call elements. This leads to closedform solutions to the required BRDF evaluation and sampling queries, enabling the first practical solution to rendering specular microstructure. Paper: PDF Video: MP4 Press: UCSD PhysOrg Digital Trends Eureka Alert Tech Crunch 

Shape Estimation from Shading, Defocus, and Correspondence Using LightField
Angular Coherence
PAMI 2016 We show that combining all three sources of information: defocus, correspondence, and shading, outperforms stateoftheart lightfield depth estimation algorithms in multiple scenarios. Paper: PDF 

SVBRDFInvariant Shape and Reflectance Estimation from LightField Cameras
CVPR 2016 Lightfield cameras have recently emerged as a powerful tool for oneshot passive 3D shape capture. However, obtaining the shape of glossy objects like metals, plastics or ceramics remains challenging, since standard Lambertian cues like photoconsistency cannot be easily applied. In this paper, we derive a spatiallyvarying (SV)BRDFinvariant theory for recovering 3D shape and reflectance from lightfield cameras. Paper: PDF 

Depth from SemiCalibrated Stereo and Defocus
CVPR 2016 In this work, we propose a multicamera system where we combine a main highquality camera with two lowres auxiliary cameras. Our goal is, given the lowres depth map from the auxiliary cameras, generate a depth map from the viewpoint of the main camera. Ours is a semicalibrated system, where the auxiliary stereo cameras are calibrated, but the main camera has an interchangeable lens, and is not calibrated beforehand. Paper: PDF 

Depth Estimation with Occlusion Modeling Using Lightfield Cameras
PAMI 2016 In this paper, an occlusionaware depth estimation algorithm is developed; the method also enables identification of occlusion edges,which may be useful in other applications. It can be shown that although photoconsistency is not preserved for pixels at occlusions, it still holds in approximately half the viewpoints. Moreover, the line separating the two view regions (occluded object vs. occluder) has the same orientation as that of the occlusion edge in the spatial domain. By ensuring photoconsistency in only the occluded view region, depth estimation can be improved. Paper: PDF 

Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects
ACM Trans. Graphics Dec 2015 We present a new approach for fast sheared filtering on the GPU. Our algorithm factors the 4D sheared filter into four 1D filters. We derive complexity bounds for our method, showing that the perpixel complexity is reduced from O(n^2 l^2) to O(nl), where n is the linear filter width (filter size is O(n^2)) and l is the (usually very small) number of samples for each dimension of the light or lens per pixel (spp is l2). We thus reduce sheared filtering overhead dramatically. Paper: PDF Video: MPEG 

PhysicallyAccurate Fur Reflectance: Modeling, Measurement and Rendering
SIGGRAPH Asia 2015 In this paper, we develop a physicallyaccurate reflectance model for fur fibers. Based on anatomical literature and measurements, we develop a double cylinder model for the reflectance of a single fur fiber, where an outer cylinder represents the biological observation of a cortex covered by multiple cuticle layers, and an inner cylinder represents the scattering interior structure known as the medulla. Paper: PDF MS thesis of Chiwei Tseng Data 

Anisotropic Gaussian Mutations for Metropolis Light Transport through
HessianHamiltonian Dynamics
SIGGRAPH Asia 2015 We present a Markov Chain Monte Carlo(MCMC) rendering algorithm that extends Metropolis Light Transport by automatically and explicitly adapting to the local shape of the integrand, thereby increasing the acceptance rate. Our algorithm characterizes the local behavior of throughput in path space using its gradient as well as its Hessian. In particular, the Hessian is able to capture the strong anisotropy of the integrand. Paper: PDF 

On Optimal, Minimal BRDF Sampling for Reflectance Acquisition
SIGGRAPH Asia 2015 In this paper, we address the problem of reconstructing a measured BRDF from a limited number of samples. We present a novel mapping of the BRDF space, allowing for extraction of descriptive principal components from measured databases, such as the MERL BRDF database. We optimize for the best sampling directions, and explicitly provide the optimal set of incident and outgoing directions in the Rusinkiewicz parameterizationfor n = 1; 2; 5; 10; 20 samples. Based on the principal components, we describe a method for accurately reconstructing BRDF data from these limited sets of samples. Paper: PDF 

Photometric Stereo in a Scattering Medium
ICCV 2015 Photometric stereo is widely used for 3D reconstruction. However, its use in scattering media such as water, biological tissue and fog has been limited until now, because of forward scattered light from both the source and object, as well as light scattered back from the medium (backscatter). Here we make three contributions to address the key modes of light propagation, under the common single scattering assumption for dilute media. Paper: PDF 

Occlusionaware Depth Estimation Using Lightfield Cameras
ICCV 2015 In this paper, we develop a depth estimation algorithm for light field cameras that treats occlusion explicitly; the method also enables identification of occlusion edges, which may be useful in other applications. We show that, although pixels at occlusions do not preserve photoconsistency in general, they are still consistent in approximately half the viewpoints. Paper: PDF 

Oriented LightField Windows for Scene Flow
ICCV 2015 For Lambertian surfaces focused to the correct depth, the 2D distribution of angular rays from a pixel remains consistent. We build on this idea to develop an oriented 4D lightfield window that accounts for shearing(depth), translation (matching), and windowing. Our main application is to scene flow, a generalization of optical flow to the 3D vector field describing the motion of each point in the scene. Paper: PDF 

Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with LightField Cameras
PAMI 2015 (to appear) Lightfield cameras have now become available in both consumer and industrial applications, and recent papers havedemonstrated practical algorithms for depth recovery from a passive singleshot capture. However, current lightfield depth estimationmethods are designed for Lambertian objects and fail or degrade for glossy or specular surfaces. In this paper, wepresent a novel theory of the relationship between lightfield data and reflectance from the dichromatic model. Paper: PDF 

Depth from Shading, Defocus, and Correspondence Using LightField Angular Coherence
CVPR 2015 Using shading information is essential to improve shape estimation from light field cameras. We develop an improved technique for local shape estimation from defocus and correspondence cues, and show how shading can be used to further refine the depth. We show that the angular pixels have angular coherence, which exhibits three properties: photoconsistency, depth consistency, and shading consistency. Paper: PDF 

Probabilistic Connections for Bidirectional Path Tracing
Computer Graphics Forum (EGSR) 2015 Bidirectional path tracing (BDPT) with Multiple Importance Sampling is one of the most versatile unbiased rendering algorithms today. BDPT repeatedly generates subpaths from the eye and the lights, which are connected for each pixel and then discarded. Unfortunately, many such bidirectional connections turn out to have low contribution to the solution. Our key observation is that we can importance sample connections to an eye subpath by considering multiple light subpaths at once and creating connections probabilistically. Paper: PDF 

Filtering Environment Illumination for Interactive PhysicallyBased Rendering in Mixed Reality
EGSR 2015 We propose accurate filtering of a noisy MonteCarlo image using Fourier analysis. Our novel analysis extends previous works by showing that the shape of illumination spectra is not always a line or wedge, as in previous approximations, but rather an ellipsoid. Our primary contribution is an axisaligned filtering scheme that preserves the frequency content of the illumination.We also propose a novel application of our technique to mixed reality scenes, in which virtual objects are inserted into a real video stream so as to become indistinguishable from the real objects. Paper: PDF Video: MP4 Supplementary: PDF 

Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering
EUROGRAPHICS 2015 In this paper we survey recent advances in adaptive sampling and reconstruction. We distinguish between a priori methods that analyze the light transport equations and derive sampling rates and reconstruction filters from this analysis, and a posteriori methods that apply statistical techniques to sets of samples. Paper: PDF 

A Light Transport Framework for Lenslet Light Field Cameras
ACM Transactions on Graphics (Apr 2015). It is often stated that there is a fundamental tradeoff between spatial and angular resolution of lenslet light field cameras, but there has been limited understanding of this tradeoff theoretically or numerically. In this paper, we develop a light transport framework for understanding the fundamental limits of light field camera resolution. Paper: PDF Supplementary Images: PDF 

City Forensics: Using Visual Elements to Predict NonVisual City Attributes
IEEE TVCG [SciVis 2014]. Honorable Mention for Best Paper Award We present a method for automatically identifying and validating predictive relationships between the visual appearance of a city and its nonvisual attributes (e.g. crime statistics, housing prices, population density etc.). We also test human performance for predicting theft based on streetlevel images and show that our predictor outperforms this baseline with 33% higher accuracy on average. Paper: PDF 

HighOrder Similarity Relations in Radiative Transfer
SIGGRAPH 2014. Radiative transfer equations (RTEs) with different scattering parameters can lead to identical solution radiance fields. Similarity theory studies this effect by introducing a hierarchy of equivalence relations called similarity relations. Unfortunately, given a set of scattering parameters, it remains unclear how to find altered ones satisfying these relations, significantly limiting the theory's practical value. This paper presents a complete exposition of similarity theory, which provides fundamental insights into the structure of the RTE's parameter space. To utilize the theory in its general highorder form, we introduce a new approach to solve for the altered parameters including the absorption and scattering coefficients as well as a fully tabulated phase function. Paper: PDF Video (MPEG) 

Rendering Glints on HighResolution NormalMapped Specular Surfaces
SIGGRAPH 2014. Complex specular surfaces under sharp point lighting show a fascinating glinty appearance, but rendering it is an unsolved problem. Using Monte Carlo pixel sampling for this purpose is impractical: the energy is concentrated in tiny highlights that take up a minuscule fraction of the pixel. We instead compute an accurate solution using a completely different deterministic approach. Paper: PDF Video (MPEG) 

Factored AxisAligned Filtering for Rendering Multiple Distribution Effects
SIGGRAPH 2014. We propose an approach to adaptively sample and filter for simultaneously rendering primary (defocus blur) and secondary (soft shadows and indirect illumination) distribution effects, based on a multidimensional frequency analysis of the direct and indirect illumination light fields, and factoring texture and irradiance. Paper: PDF Video (MPEG) 

Discrete Stochastic Microfacet Models
SIGGRAPH 2014. This paper investigates rendering glittery surfaces, ones which exhibitshifting random patterns of glints as the surface or viewermoves. It applies both to dramatically glittery surfaces that containmirrorlike flakes and also to rough surfaces that exhibit more subtlesmall scale glitter, without which most glossy surfaces appeartoo smooth in closeup. Inthis paper we present a stochastic model for the effects of randomsubpixel structures that generates glitter and spatial noise that behavecorrectly under different illumination conditions and viewingdistances, while also being temporally coherent so that they lookright in motion. Paper: PDF Video (MPEG) 

Depth Estimation for Glossy Surfaces with LightField Cameras
ECCV 14 Workshop Light Fields Computer Vision. Lightfield cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive singleshot capture. In this paper, we develop an iterative approach to use the benefits of lightfield data to estimate and remove the specular component, improving the depth estimation. The approach enables lightfield data depth estimation to support both specular and diffuse scenes. Paper: PDF 

UserAssisted Video Stabilization
EGSR 2014. We present a userassisted video stabilization algorithm that is able to stabilize challenging videos. First, we cluster tracks and visualize them on the warped video. The user ensures that appropriate tracks are selected by clicking on track clusters to include or exclude them. Second, the user can directly specify how regions in the output video should look by drawing quadrilaterals to select and deform parts of the frame. Paper: PDF Video (MPEG) 

Depth from Combining Defocus and Correspondence Using LightField Cameras
ICCV 2013. Lightfield cameras have recently become available to the consumer market. An array of microlenses captures enough information that one can refocus images after acquisition, as well as shift one's viewpoint within the subapertures of the main lens, effectively obtaining multiple views. Thus, depth cues from both defocus and correspondence are available simultaneously in a single capture, and we show how to exploit both by analyzing the EPI. Paper: PDF Video (MPEG) 

External mask based depth and light field camera
ICCV 13 Workshop Consumer Depth Cameras for Vision. We present a method to convert a digital singlelens reflex (DSLR) camera into a highresolution consumer depth and lightfield camera by affixing an external aperture mask to the main lens. Compared to the existing consumer depth and light field cameras, our camera is easy to construct with minimal additional costs, and our design is camera and lens agnostic. The main advantage of our design is the ease of switching between an SLR camera and a native resolution depth/light field camera. We also do not need to modify the internals of the camera or the lens. Paper: PDF Video (MPEG) 

AxisAligned Filtering for Interactive PhysicallyBased Diffuse Indirect
Lighting
SIGGRAPH 2013. We introduce an algorithm for interactive rendering of physicallybased global illumination, based on a novel frequency analysis of indirect lighting. Our method combines adaptive sampling byMonte Carlo ray or path tracing, using a standard GPUaccelerated raytracer, with realtime reconstruction of the resulting noisy images. Paper: PDF Video (MPEG) 

Modular Flux Transfer: Efficient Rendering of HighResolution Volumes with
Repeated Structures
SIGGRAPH 2013. Common volumetric materials (fabrics, finished wood, synthesized solid textures) are structured, with repeated patterns approximated by tiling a small number of exemplar blocks. In this paper, we introduce a precomputationbased rendering approach for such volumetric media with repeated structures based on a modular transfer formulation. We model each exemplar block as a voxel grid and precompute voxeltovoxel, patchtopatch, and patchtovoxel flux transfer matrices. Paper: PDF Video (MPEG) 

What Object Motion Reveals About Shape with Unknown BRDF and Lighting
CVPR 2013. We present a theory that addresses the problem of determining shape from the (small or differential) motion of an object with unknown isotropic reflectance, under arbitrary unknown distant illumination, for both orthographic and perpsective projection. Our theory imposes fundamental limits on the hardness of surface reconstruction, independent of the method involved. Under orthographic projection, we prove that three differential motions suffice to yield an invariant that relates shape to image derivatives, regardless of BRDF and illumination. Under perspective projection, we show that four differential motions suffice to yield depth and a linear constraint on the surface gradient. Paper: PDF 

Automatic Cinemagraph Portraits
EGSR 2013. Cinemagraphs are a popular new type of visual media that lie inbetween photos and video; some parts of the frame are animated and loop seamlessly, while other parts of the frame remain completely still. Cinemagraphs are especially effective for portraits because they capture the nuances of our dynamic facial expressions. We present a completely automatic algorithm for generating portrait cinemagraphs from a short video captured with a handheld camera. Paper: PDF Video (MPEG) 

Interactive Albedo Editing in PathTraced Volumetric Materials
TOG 2013 (April Cover Image). In this paper, we develop an editing algorithm that enables a material designer to set the local (singlescattering) albedo coefficients interactively, and see an immediate update of the emergent appearance in the image. We also extend the technique to editing the overall mean free path of the material. This is a difficult problem, since the function from materials to pixel values is neither linear nor loworder polynomial. Paper: PDF Video (MPEG) 

Gloss Perception in Painterly and Cartoon Rendering
TOG 2013. We describe the first study of material perception in stylized images (specifically painting and cartoon) and use nonphotorealistic rendering algorithms to evaluate how such stylization alters the perception of gloss. This mapping allows users of NPR algorithms to predict, and correct for, the perception of gloss in their images. Paper: PDF 

Sharpening Out of Focus Images using HighFrequency Transfer
EuroGraphics 2013. We propose a new method to sharpen outoffocus images, that uses a similar but different assisting sharp image provided by the user (such as multiple images of the same subject in different positions captured using a burst of photographs). We demonstrate sharpened results on outoffocus images in macro, sports, portrait and wildlife photography. Paper: PDF Video (MPEG) Supplementary HTML 

On Differential Photometric Reconstruction for Unknown, Isotropic BRDFs
PAMI 2013. This paper presents a comprehensive theory of photometric surface reconstruction from image derivatives, in the presence of a general, unknown isotropic BRDF. We derive precise topological classes up to which the surface may be determined and specify exact priors for a full geometric reconstruction, for both shape from shading and photometric stereo. Paper: PDF 

Compressive Structured Light for Recovering Inhomogeneous Participating
Media PAMI 2013. We propose a new method named compressive structured light for recovering inhomogeneous participating media. Whereas conventional structured light methods emit coded light patterns onto the surface of an opaque object to establish correspondence for triangulation, compressive structured light projects patterns into a volume of participating medium to produce images which are integral measurements of the volume density along the line of sight. Paper: PDF 

AxisAligned Filtering for Interactive Sampled Soft Shadows
Siggraph Asia 2012. We develop a simple and efficient method for soft shadows from planar area light sources, based on explicit occlusion calculation by raytracing, followed by adaptive imagespace filtering. Since the method is based on Monte Carlo sampling, it is accurate. Since the filtering is in imagespace, it adds minimal overhead and can be performed at realtime frame rates. We obtain interactive speeds, using the Optix GPU raytracing framework. Our technical approach derives from recent work on frequency analysis and sheared pixellight filtering for offline soft shadows. While sample counts can be reduced dramatically, the sheared filtering step is slow, adding minutes of overhead. We develop the theoretical analysis to instead consider axisaligned filtering, deriving the sampling rates and filter sizes. Paper: PDF Video (MPEG) Source Code 

FrequencySpace Decomposition and Acquisition of Light Transport under
Spatially Varying Illumination
ECCV 2012. We show that, under spatially varying illumination, the light transport of diffuse scenes can be decomposed into direct, nearrange (subsurface scattering and local interreflections) and far range transports (diffuse interreflections). We show that these three component transports are redundant either in the spatial or the frequency domain and can be separated using appropriate illumination patterns, achieving a theoretical lower bound. Paper: PDF 

A Theory of Monte Carlo Visibility Sampling
ACM TOG Aug 2012. We develop a comprehensive theoretical analysis of different sampling patterns for Monte Carlo visibility. In particular, we show the benefits of uniform jitter sampling over stratified in some cases, and demonstrate that it produces the lowest variance for linear lights. Surprisingly, the best pattern depends on the shape of the light source for area lights, with uniform jitter preferred for circular lights and stratified for square lights. Paper: PDF Video: MPEG4 Talk: PPT 

Selectively DeAnimating Video
SIGGRAPH 2012. We present a semiautomated technique for selectively deanimating video to remove the largescale motions of one or more objects so that other motions are easier to see. Our technique enables a number of applications such as clearer motion visualization, simpler creation of artistic cinemagraphs (photos that include looping motions in some regions), and new ways to edit appearance and complicated motion paths in video by manipulating a deanimated representation. Paper: PDF Video: MPEG4 Teaser Video: MPEG4 

Analytic Tangent Irradiance Environment Maps for Anisotropic Surfaces
EGSR 2012. We extend spherical harmonic irradiance maps to anisotropic surfaces, replacing Lambertian reflectance with the diffuse term of the popular KajiyaKay model. We show that the terms decay even more rapidly than for Lambertian reflectance. Existing code for irradiance environment maps can be trivially adapted for realtime rendering with tangent irradiance maps. We also demonstrate an application to offline rendering of the diffuse component of fibers, using our formula as a control variate for Monte Carlo sampling. Paper: PDF 

Importance Sampling of Reflection from Hair Fibers
JCGT 2012 (Inaugural Article). Hair and fur are increasingly important visual features in production rendering, and physicallybased light scattering models are now commonly used. In this paper, we enable efficient Monte Carlo rendering of specular reflections from hair fibers. We describe a simple and practical importance sampling strategy for the reflection term in the Marschner hair model. Our method has been widely used in production for more than a year, and complete pseudocode is provided. Paper: PDF 

RealTime Rendering of Rough Refraction
IEEE TVCG Feb 2012. We present an algorithm to render objects made of transparent materials with rough surfaces in realtime, under distant illumination. Rough surfaces cause wide scattering as light enters and exits objects, which significantly complicates the rendering of such materials. We approximate the Bidirectional Transmittance Distribution Function (BTDF), using spherical Gaussians. We also propose two extensions, to support spatiallyvarying roughness and local lighting on thin objects. Paper: PDF Youtube Video (from I3D 2011) 

From the Rendering Equation to Stratified Light Transport Inversion International Journal of Computer Vision, 2012 In this work, we explore a theoretical analysis of inverse light transport, relating it to its forward counterpart, expressed in the form of the rendering equation. We show the existence of an inverse Neumann series, that zeroes out the corresponding physical bounces of light, which we refer to as stratified light transport inversion. Our practical application is to radiometric compensation, where we seek to project patterns onto realworld surfaces, undoing the effects of global illumination. Paper: PDF 

Practical Filtering for Efficient RayTraced Directional Occlusion
SIGGRAPH Asia 2011 Ambient occlusion and directional (spherical harmonic) occlusion have become a staple of production rendering, but are expensive to compute. We give a frequency analysis of shadow light fields using distant illumination with a general BRDF and normal mapping, allowing us to share ray information even among complex receivers. We also present a new rotationallyinvariant filter that easily handles samples spread over a large angular domain. Our method can deliver 4x speed up for scenes that are computationally bound by ray tracing costs. Paper: PDF Supplementary Animations AVI 

Sparse Reconstruction of Visual Appearance for Computer Graphics and Vision SPIE Keynote, Wavelets and Sparsity XIV 2011 A broad range of problems in computer graphics rendering, appearance acquisition for graphics and vision, and imaging, involve sampling, reconstruction, and integration of highdimensional (4D8D) signals. We argue that dramatically sparser sampling and reconstruction of these signals is possible, before the full dataset is acquired or simulated. Our key idea is to exploit the structure of the data that often lies in lowerfrequency, sparse, or lowdimensional spaces. Paper: PDF 

What an Image Reveals About Material Reflectance
ICCV 2011 We derive precise conditions under which material reflectance properties may be estimated from a single image of a homogeneous curved surface (canonically a sphere), lit by a directional source. Based on the observation that light is reflected along certain (a priori unknown) preferred directions such as the halfangle, we propose a semiparametric BRDF abstraction that lies between purely parametric and purely datadriven models. While it is wellknown that fitting multilobe BRDFs may be illposed under certain conditions, prior to this work, precise results for the wellposedness of BRDF estimation had remained elusive. Paper: PDF 

On the Duality of Forward and Inverse Light Transport
PAMI 2011, ECCV 2010 Inverse light transport seeks to undo global illumination effects, such as interreections, that pervade images of most scenes. This paper presents the theoretical and computational foundations for inverse light transport as a dual of forward rendering. We demonstrate two practical applications, namely, separation of individual bounces of the light transport and fast projector radiometric compensation to display images free of global illumination artifacts in realworld environments. Paper: PAMI ECCV Tech Report Video 

DataDriven Elastic Models for Cloth: Modeling and Measurement SIGGRAPH 2011 Cloth often has complicated nonlinear, anisotropic elastic behavior due to its woven pattern and fiber properties. However, most current cloth simulation techniques simply use linear and isotropic elastic models with manually selected stiffness parameters. Such simple simulations do not allow differentiating the behavior of distinct cloth materials such as silk or denim, and they cannot model most materials with fidelity to their realworld counterparts. In this paper, we present a datadriven technique to more realistically animate cloth. These measurements can be used in most cloth simulation systems to create natural and realistic clothing wrinkles and shapes, for a range of different materials. Paper: PDF Video 

Illumination Decomposition for Material Recoloring with Consistent Interreflections SIGGRAPH 2011 Changing the color of an object is a basic image editing operation, but a high quality result must also preserve natural shading. A common approach is to first compute reflectance and illumination intrinsic images. Reflectances can then be edited independently, and recomposed with the illumination. However, manipulating only the reflectance color does not account for diffuse interreflections, and can result in inconsistent shading in the edited image. We propose an approach for further decomposing illumination into direct lighting, and indirect diffuse illumination from each material. Paper: PDF 

Frequency Analysis and Sheared Filtering for Shadow Light Fields of Complex Occluders TOG 2011 Monte Carlo ray tracing of soft shadows produced by area lighting and intricate geometries, such as the shadows through plant leaves or arrays of blockers, is a critical challenge. This article develops an efficient diffuse soft shadow technique for mid to far occluders that relies on a new 4D cache and sheared reconstruction filter. Our analysis subsumes convolution soft shadows for parallel planes as a special case. Paper: PDF Supplemental Animations 

Optimizing Environment Maps for Material Depiction
EGSR 2011 We present an automated system for optimizing and synthesizing environment maps that enhance the appearance of materials in a scene. We first identify a set of lighting design principles for material depiction. Each principle specifies the distinctive visual features of a material and describes how environment maps can emphasize those features. We express these principles as linear or quadratic image quality metrics, and present a general optimization framework to solve for the environment map that maximizes these metrics. We accelerate metric evaluation using an approach dual to precomputed radiance transfer (PRT). Paper: PDF Video of Quality Metric 

A Theory of Differential Photometric Stereo for Unknown Isotropic BRDFs
CVPR 2011 We present a comprehensive theory of photometric surface reconstruction from image derivatives. For unknown isotropic BRDFs, we show that two measurements of spatial and temporal image derivatives, under unknown light sources on a circle, suffice to determine the surface. This result is the culmination of a series of fundamental observations. Our theoretical results are illustrated with several examples on synthetic and real data. Paper: PDF Tech Report 

RealTime Rough Refraction
I3D 2011 Best Paper Award We present an algorithm to render objects of transparent materials with rough surfaces in realtime, under distant illumination. Rough surfaces cause wide scattering as light enters and exits objects, which significantly complicates the rendering of such materials. We approximate the Bidirectional Transmittance Distribution Function (BTDF), using spherical Gaussians, suitable for realtime estimation of environment lighting using preconvolution. Paper: PDF Youtube Video 

MultiResolution Isotropic Strain Limiting
SIGGRAPH Asia 2010. In this paper we describe a fast strainlimiting method that allows stiff, incompliant materials to be simulated efficiently. Unlike prior approaches, which act on springs or individual strain components, this method acts on the strain tensors in a coordinateinvariant fashion allowing isotropic behavior. For triangulated surfaces in threedimensional space, we also describe a complementary edgeanglelimiting method to limit outofplane bending. To accelerate convergence, we also propose a novel multiresolution algorithm that enforces fitted limits at each level of a nonconforming hierarchy. Paper: PDF Video 

ExampleBased Wrinkle Synthesis for Clothing Animation
SIGGRAPH 2010. This paper describes a method for animating the appearance of clothing, such as pants or a shirt, that fits closely to a figure's body. Based on the observation that the wrinkles in closefitting clothing behave in a predominantly kinematic fashion, we have developed an examplebased wrinkle synthesis technique. Our method drives wrinkle generation from the pose of the figure's kinematic skeleton. This approach allows high quality clothing wrinkles to be combined with a coarse cloth simulation that computes the global and dynamic aspects of the clothing motion. Further, the combined system runs at interactive rates, making it suitable for applications where highresolution offline simulations would not be a viable option. Paper: PDF Video Papers Trailer 

Sparsely Precomputing the Light Transport Matrix for RealTime Rendering
EGSR 2010. Precomputationbased methods have enabled realtime rendering with natural illumination, allfrequency shadows, and global illumination. However, a major bottleneck is the precomputation time, that can take hours to days. While the final realtime data structures are typically heavily compressed with clustered principal component analysis and/or wavelets, a full light transport matrix still needs to be precomputed for a synthetic scene, often by exhaustive sampling and raytracing. In this paper, we show that the precomputation can be made much more efficient by adaptive and sparse sampling of light transport. We demonstrate sparse sampling and precomputation 5x faster than previous methods. Paper: PDF (if no EG DL subscription see TR) Tech Report Video 

Adaptive Wavelet Rendering
SIGGRAPH Asia 2009. Effects such as depth of field, area lighting, antialiasing and global illumination require evaluating a complex highdimensional integral at each pixel of an image. We develop a new adaptive rendering algorithm that greatly reduces the number of samples needed for Monte Carlo integration. Our method renders directly into an imagespace wavelet basis. Moreover, the method introduces minimal overhead, and can be efficiently included in an optimized raytracing system. Paper: PDF (44M) 

Removing Image Artifacts Due to Dirty Camera Lenses and Thin Occluders
SIGGRAPH Asia 2009. There are often physical layers between the scene and the imaging system. For example, the lenses of consumer digital cameras often accumulate various types of contaminants over time (e.g., fingerprints, dust, dirt). Also, photographs are often taken through a layer of thin occluders (e.g., fences, meshes, window shutters, curtains, tree branches) which partially obstruct the scene. We show that both effects can be described by a single image formation model, and removed from digital photographs. Paper: PDF 

PrecomputationBased Rendering
Foundations and Trends in Computer Graphics and Vision 3(4), 281369. Precomputationbased relighting and radiance transfer has a long history with a spurt of renewed interest, including adoption in commercial video games, due to recent mathematical developments and hardware advances. In this survey, we describe the mathematical foundations, history, current research and future directions for precomputationbased rendering. Paper: PDF 

Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur
SIGGRAPH 09. Motion blur is crucial for highquality rendering, but is also very expensive. Our first contribution is a frequency analysis of motionblurred scenes, including moving objects, specular reflections, and shadows. We show that motion induces a shear in the frequency domain, and that the spectrum of moving scenes is usually contained in a wedge. This allows us to compute adaptive spacetime sampling rates, to accelerate rendering. Our second contribution is a novel sheared reconstruction filter that is aligned to the firstorder direction of motion and enables even lower sampling rates. Paper: PDF 

Moving Gradients: A PathBased Method for Plausible Image Interpolation
SIGGRAPH 09. We describe a method for plausible interpolation of images, with a wide range of applications like temporal upsampling for smooth playback of lower frame rate video, smooth view interpolation, and animation of still images. We develop a novel pathbased framework, greater flexibility via transition points, new ways to handle visibility, and Poisson reconstruction to produce smooth interpolations. Paper: PDF Video (QT 55M) 

An Empirical BSSRDF Model
SIGGRAPH 09. Current scattering models are tuned to the two extremes of thin media and single scattering, or highly scattering materials modeled using the diffusion approximation. The vast intermediate range of materials has no efficient approximation. In this work, we simulate the full space of BSSRDFs, analogous to work on measured BRDF databases. We show new types of scattering behavior, fitting an analytic model and tabulating its parameters. This allows new efficient rendering and reflectance models for a variety of BSSRDFs. Paper: PDF 

Affine Double and Triple Product Wavelet Integrals for Rendering
ACM Transactions on Graphics 28(2), Article 14, pages 117, Apr 2009. Many problems in computer graphics involve integrations of products of functions. Double and tripleproduct integrals are commonly used in applications such as allfrequency relighting or importance sampling, but are limited to distant illumination. In contrast, nearfield lighting from planar area lights involves an affine transform of the source radiance at different points in space. Our main contribution is a novel affine double and tripleproduct integral theory. Paper: PDF Video (AVI 16M) 

Compressive Light Transport Sensing
ACM Transactions on Graphics 28(1), Article 3, pages 118, Jan 2009. In this article we propose a new framework for capturing light transport data of a real scene, based on the recently developed theory of compressive sensing for sparse signals. We develop a novel hierarchical decoding algorithm that improves reconstruction quality by exploiting interpixel coherency relations. Additionally, we design new nonadaptive illumination patterns that minimize measurement noise. Paper: PDF 

Multiscale Texture Synthesis
SIGGRAPH 08, Article 51, pages 18. The appearance of many textures changes dramatically with scale; imagine zooming into the planet from outer space to see large scale continent and ocean features, then smaller cities, forests, and finally people and trees. By using an exemplar graph with a few small singlescale exemplars and modifying a standard parallel synthesis method, we develop the first multiscale texture synthesis algorithm. Paper: PDF Video (175M) 

Light Field Transfer: Global Illumination between Real and Synthetic
Objects
SIGGRAPH 08, Article 57, pages 16. By using a light field interface between real and synthetic scenes, we can composite real and virtual objects. Moreover, we can directly simulate multiple bounces of global illumination between them. Our method is suited even for dynamic scenes, and does not require geometric properties or complex imagebased appearance capture of the real objects. Paper: PDF Video (55M) 

A Precomputed Polynomial Representation for Interactive BRDF Editing with
Global Illumination ACM Transactions on Graphics
27(2), Article 13, pages 113. Presented at SIGGRAPH 2008. We develop a mathematical framework and practical algorithms to edit BRDFs with global illumination in a complex scene. A key challenge is that light transport for multiple bounces is nonlinear in the scene BRDFs. We address this by developing a new bilinear representation of the reflection operator, deriving a polynomial multibounce tensor precomputed framework, and reducing the complexity of further bounces. Paper: PDF Video (7M) 

A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering
Human Skin SIGGRAPH Asia 2008. We introduce a layered, heterogeneous spectral reflectance model for human skin. The model captures the interscattering of light among layers, each of which may have an independent set of spatiallyvarying absorption and scattering parameters. To obtain parameters for our model, we use a novel acquisition method that begins with multispectral photographs. We create complex skin visual effects such as veins, tattoos, rashes, and freckles. Paper: PDF (8M) 

Compressive Structured Light for Recovering Inhomogeneous Participating
Media ECCV 2008. Recovering dynamic inhomogeneous participating media is a significant challenge in vision and graphics. We introduce a new framework of compressive structured light, where patterns are emitted to obtain a line integral of the volume density at each camera pixel. The framework of compressive sensing is then used to recover the density from a sparse set of patterns. Paper: PDF Video (25M) 

Searching the World's Herbaria: Visual Identification of Plant Species
ECCV 2008. This paper describes our electronic field guide project: a collaboration of researchers in computer vision, mobile computing and botany (the Smithsonian Institution). We have developed a handheld prototype and recognition algorithms that enable users to take the picture of a leaf and identify the species in the field. The field guide works for Plummer's Island, the woody plants in the DC area, and the trees of NYC's Central Park. Subsequent to this paper, Prof. Belhumeur and collaborators developed and released LeafSnap which is a free iPhone App for visual plant species identification. Paper: PDF Earlier Taxon 2006 paper (PDF) Project Website LeafSnap 

An Analysis of the BRDF InOut Factorization for ViewDependent Relighting
EuroGraphics Symposium on Rendering 2008. Interactive rendering with dynamic lighting and changing view is a long standing problem and many recent PRT methods seek to address this by a factorization of the BRDF into incident and outgoing angles. In this paper, we analyze this factorization theoretically using spherical harmonics, and derive the number of terms needed based on the BRDF. One result is that a very large number of terms (10s to 100s) are needed for specular materials. Paper: PDF Video (18M) 

Large Ray Packets for RealTime Whitted Ray Tracing
IEEE Symposium on Interactive Ray Tracing 2008. RealTime Ray Tracing going beyond primary rays and hard shadows, to reflections and refractions, is a longstanding challenge. In this work, we evaluate and develop new algorithms for traversal and frustum culling with large ray packets to get speedups of 3x6x, enabling realtime Whitted ray tracing on commodity hardware. Paper: PDF 


A First Order Analysis of Lighting, Shading, and Shadows
ACM Transactions on Graphics, article 2, pages 121, Jan 2007. We derive a complete first order or gradient theory of lighting, reflection and shadows, taking both spatial and angular variation of the light field into account. The gradient is by definition a sum of terms, allowing us to consider the relative weight of spatial and angular lighting variation, geometric curvature and bump mapping. Moreover, we derive analytic formulas for the gradients in soft shadow or penumbra regions, demonstrating applications to gradientbased interpolation and sampling. Paper: PDF 

A Theory of Locally Low Dimensional Light Transport
SIGGRAPH 07, article 62, pages 19. We develop a theory of locally low dimensional light transport, to analytically derive the dimensionality of light transport for a local patch. We analyze the eigenvalues for canonical configurations using Szego's eigenvalue theorem. We show mathematically that for symmetric patches of area A, the number of basis functions for glossy reflections increases linearly with A, while for simple cast shadows, it often increases as sqrt(A). There are practical applications to CPCA and other PRT algorithms. Paper: PDF Video (30M) 

Frequency Domain Normal Map Filtering
SIGGRAPH 07, article 28, pages 111. Filtering is critical for representing imagebased detail, such as textures or normal maps, across a variety of scales. While mipmapping textures is commonplace, accurate normal map filtering remains a challenging problem because of nonlinearities in shadingwe cannot simply average nearby surface normals. In this paper, we show analytically that normal map filtering can be formalized as a spherical convolution of the normal distribution function (NDF) and the BRDF, for a large class of common BRDFs such as Lambertian, microfacet and factored measurements. Our practical algorithms leverage a significant body of previous work that has studied lightingBRDF convolution. We show how spherical harmonics can be used to filter the NDF for Lambertian and lowfrequency specular BRDFs, while spherical von MisesFisher distributions can be used for highfrequency materials. Paper: PDF Video (103M) Very Cool Trailer (MOV 54M) 

A Theory of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency
ECCV 06, vol IV, pp 4155, PAMI 30(2), pages 197213, Feb 2008. We develop new mathematical results based on the spherical harmonic convolution framework for reflection. We derive novel identities, which are the angular frequency domain analogs to common spatial domain invariants such as reflectance ratios. These lead to more general transfer algorithms for inverse rendering, and a novel framework for checking the consistency of images, to detect tampering. Paper: PDF (PAMI 08) PDF (ECCV 06) 

Dirty Glass: Rendering Contamination on Transparent Surfaces
EuroGraphics Symposium on Rendering, 2007. Realworld transparent objects are seldom clean: Their surfaces have a variety of contaminants such as dust, dirt, and lipids. These contaminations produce a number of complex volumetric scattering effects that must be taken into account when creating realistic renderings. We construct an analytical model for optically thin contaminants, measure the spatially varying thicknesses for a number of glass panes of dust, dirt and lipids, and demonstrate renderings with a variety of volumetric scattering effects. Paper: PDF Video 

A RealTime Beam Tracer with Application to Exact Soft Shadows
EuroGraphics Symposium on Rendering, 2007. Beam tracing is one solution to efficiently calculate accurate soft shadows from area light sources. In this paper, we adapt many of the methods for accelerated ray tracing to develop a realtime beam tracer, that is as fast as the best ray tracers for primary rays, and up to 30 times faster for difficult secondary rays, as needed in soft shadows. Moreover, we obtain reference quality exact shadows, without stochastic noise. Paper: PDF Video 

TimeVarying BRDFs
IEEE Transactions on Visualization and Computer Graphics
13, 3 pages 595609, 2007. The properties of virtually all realworld materials change with time, causing their BRDFs to be timevarying. In this work, we address the acquisition, analysis, modeling and rendering of a wide range of timevarying BRDFs, including the drying of various types of paints (watercolor, spray, and oil), the drying of wet rough surfaces (cement, plaster, and fabrics), the accumulation of dusts (household and joint compound) on surfaces, and the melting of materials (chocolate). Analytic BRDF functions are fit to these measurements and the model parameters variations with time are analyzed. Each category exhibits interesting and sometimes nonintuitive parameter trends. These parameter trends are then used to develop analytic timevarying BRDF (TVBRDF) models. Paper: PDF Video (49MB) 

ViewpointCoded Structured Light
CVPR 2007. We introduce a theoretical framework and practical algorithms for replacing timecoded structured light patterns with viewpoint codes, in the form of additional camera locations. Current structured light methods typically use log(N) light patterns, encoded over time, to unambiguously reconstruct N unique depths. We demonstrate that each additional camera location may replace one frame in a temporal binary code. Paper: PDF 

4D Compression and Relighting with HighResolution Light Transport Matrices
ACM Symposium on Interactive 3D graphics, 2007, pages 8188. We use a 4D wavelet transform for relighting with allfrequency illumination. A key observation is that a standard 4D wavelet transform can actually inflate portions of the light transport matrix. Therefore, we present an adaptive 4D wavelet transform that terminates at a level that avoids inflation and maximizes sparsity in the matrix data. Finally, we present an algorithm for fast relighting from adaptively compressed transport matrices. Paper: PDF Video 
Inverse Shade Trees for NonParametric Material Representation and
Editing
SIGGRAPH 06, pages 735745. We develop an inverse shade tree framework of hierarchical matrix factorizations to provide intuitive, editable representations of highdimensional measured reflectance datasets of spatiallyvarying appearance. We introduce a new alternating constrained least squares framework for these decompositions, that preserves the key features of linearity, positivity, sparsity and domainspecific constraints. The SVBRDF is decomposed onto 1D curves and 2D maps, that are easily edited. Paper: PDF Video (24M) 

RealTime BRDF Editing in Complex Lighting
SIGGRAPH 06, pages 945954. In this project, we develop the theory and algorithms to for the first time allow users to edit measured and analytic BRDFs in real time to design materials in their final placement in a scene with complex natural illumination and cast shadows. The system can take as input a variety of analytic and datadriven reflectance models, including the curvebased BRDFs obtained from the inverse shade tree factorization. Paper: PDF (20M) Video (59M) 

TimeVarying Surface Appearance:
Acquisition, Modeling and Rendering
SIGGRAPH 06, pages 762771. We conduct the first comprehensive study of timevarying surface appearance, including acquisition of the first database of timevarying processes like burning, drying and decay. We then develop a nonlinear spacetime appearance factorization (STAF) that allows easy editing or manipulation such as control, transfer and texture synthesis. We demonstrate a variety of novel timevarying rendering applications using the STAF model. Paper: PDF Video QT (64M) Video AVI (46M) 

A Compact Factored Representation of Heterogeneous Subsurface Scattering
SIGGRAPH 06, pages 746753. Heterogeneous subsurface scattering in translucent materials is one of the most beautiful but complex effects. We acquire spatial BSSRDF datasets using a projector, and develop a novel nonlinear factorization that separates a homogeneous kernel, and heterogeneous discontinuities. This enables rendering of complex spatiallyvarying translucent materials. Paper: PDF (11M) 

Acquiring Scattering Properties of Participating Media by Dilution
SIGGRAPH 06, pages 10031012. We present a simple device and technique for robustly estimating the properties of a broad class of participating media that can be either (a) diluted in water such as juices or beverages, (b) dissolved in water such as powders and sugar/salt crystals, or (c) suspended in water, such as impurities. By diluting in water, we can measure robustly in the single scattering regime. Paper: PDF 


Reflectance Sharing: ImageBased Rendering from a Sparse Set of Images
PAMI Aug 06, pages 12871302 ,
EGSR 05, pages 253264 We develop the theoretical framework and practical results for imagebased rendering of spatiallyvarying reflectance from a very small number of images. In doing so, we trade off some spatial variation of the reflectance for an increased number of angular samples. The upcoming PAMI paper also includes a novel Fourier analysis of spatial and angular coherence. Paper: PDF Video (83M) EGSR 05 (PDF) 

Exploiting Temporal Coherence for Incremental AllFrequency Relighting
EGSR 06. ,
Current PRT methods exploit spatial coherence of the lighting (such as with wavelets) and of light transport (such as with CPCA). We consider a significant, yet unexplored form of coherence, temporal coherence of the lighting from frame to frame. We achieve speedups of 3x4x over conventional PRT with minimal implementation effort, and can trivially be added to almost any existing PRT algorithm. Paper: PDF 

Efficient Shadows from Sampled Environment Maps
JGT 06 11(1):1336 There are a number of recent methods to importance sample environment maps. However, these techniques do not exploit the coherence in visibility between nearby rays. We investigate a number of alternatives and develop a simple technique that can speed up the rendering of scenes lit by natural illumination by an order of magnitude with essentially no loss in accuracy. Paper: PDF 

Modeling Illumination Variation with Spherical Harmonics
Book chapter in Face Processing: Advanced Modeling Methods (pages
385424, 2006) The appearance of objects including human faces can vary dramatically with the lighting. We present results that use spherical harmonic illumination basis functions to understand this variation for face modeling and recognition, as well as a number of other applications in graphics and vision. Paper: PDF 

A Practical Analytic Single Scattering Model for Real Time
Rendering Siggraph 05, pages 10401049. We present a physicallybased model that allows for realtime rendering of a variety of scattering effects like glows around light sources, the effects of scattering on surface shading, and the appearance with complex lighting and BRDFs. The model is based on an analytic integration of the single scattering equations, and can be implemented with simple fragment programs on modern graphics hardware. Paper: PDF Video (74M) 

Efficiently Combining Positions and Normals for Precise 3D Geometry
Siggraph 05, pages 536543. We show how depth and normal information, such as from a depth scanner and from photometric stereo, can be efficiently combined to remove the distortions and noise in both, producing very high quality meshes for computer graphics. Paper: PDF 

Adaptive Numerical Cumulative Distribution Functions for Efficient
Importance Sampling
EGSR 05, pages 1120 Importance sampling highdimensional functions like lighting and BRDFs is increasingly important, but a direct tabular representation has storage cost exponential in the number of dimensions. By placing samples nonuniformly, we show that we can develop compact CDFs that enable new applications like sampling from oriented environment maps and multiple importance sampling. Paper: PDF 

A SignalProcessing Framework for Reflection
ACM Transactions on Graphics (volume 23(4),
Oct 2004, pages 10041042) We present a signalprocessing framework for analyzing the reflected light field from a homogeneous convex curved surface under distant illumination. This generalizes many of our previous results, showing a unified framework for 2D, 3D lambertian, 3D isotropic and 3D anisotropic cases. Paper: PDF 
Triple Product Wavelet Integrals for AllFrequency Relighting
Siggraph 04, pages 475485 We propose a new mathematical and computational analysis of precomputed light transport. We use factored forms, separately precomputing the effects of visibility and material properties. Rendering then requires computing triple product integrals at each vertex, involving the lighting, visibility and BRDF. Our main contribution is a general analysis of these triple products likely to have broad applicability in computer graphics and numerical analysis. Paper: PDF (5M) Video (17M) 

Efficient BRDF Importance Sampling Using a Factored Representation
Siggraph 04, pages 494503 We introduce a Monte Carlo Importance sampling technique for general analytic and measured BRDFs based on a new BRDF factorization. PDF (8M) 


A Fourier Theory for Cast Shadows ECCV 04, pages I 146162 ; PAMI Feb 05, pages 288295 We show that cast shadows can be mathematically analyzed for many simple configurations, resulting in a standard convolution formula that can be derived analytically in 2D and analyzed numerically in 3D. The results help explain many effects of lighting variability in 3D textures and suggest new bases for that purpose. Paper: ECCV 04 , PAMI 05 
Practical Rendering of Multiple Scattering Effects
in Participating Media
EGSR 04 Volumetric light transport effects are significant for many materials like skin, smoke, clouds, snow or water. In particular, one must consider the multiple scattering of light within the volume. We develop a general framework for incorporating analytic point spread functions based on beam spreading, while considering multiple scattering in inhomogeneous media. PDF (2M) 


Using Specularities for Recognition ICCV 03, pages 15121519 We present the first method for using specularities as a positive feature for lightinginsensitive recognition. The method is applied to very difficult objects like shiny crockery and wine glasses. Paper: PDF 

Spacetime Stereo: A Unifying Framework for Depth from
Triangulation CVPR 03, II359II366 ; PAMI Feb 05, pages 296302 We propose a common framework, spacetime stereo, which unifies many previous depth from triangulation methods like stereo, laser scanning, and coded structured light. As a practical example, we discuss a new temporal stereo technique for improved shape estimation in static scenes under variable illumination. Paper: CVPR 03 , PAMI 05 
AllFrequency Shadows Using NonLinear Wavelet Lighting Approximation
Siggraph 03, pages 376381 We present a method, based on precomputed light transport, for realtime rendering of objects under allfrequency, timevarying illumination represented as a highresolution environment map. For accurate rendering, using nonlinear wavelets is an order of magnitude faster than using linear spherical harmonics, the current best technique. PDF (1M) Video (42MB) 

Structured Importance Sampling of Environment Maps
Siggraph 03, pages 605612 We introduce structured importance sampling, a new technique for efficiently rendering scenes illuminated by distant natural illumination given in an environment map. PDF Video 


Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object PAMI Oct 2002,
pp 13221333.
We explain for the first time some classic empirical results on lighting variability, and take a first step toward analyzing many classic vision problems under complex lighting. Full Paper: PDF (.8M) 
Frequency Space Environment Map Rendering:
Siggraph 02, pages 517526 We present a new method for realtime rendering of objects with complex isotropic BRDFs under distant natural illumination, as specified by an environment map. Our approach is based on spherical frequency space analysis. Full Paper: gzipped PS (4.2M) PDF (3.3M) 

Analysis of Planar Light Fields From Homogeneous Convex Curved Surfaces Under Distant Illumination
Proceedings of Human Vision and Electronic Imaging VI (part of Photonics West, 2001), pages 185198
This relatively simple to read paper is the first on the reflection is
convolution idea underlying my PhD thesis, and considers the 2D case using
only Fourier transforms.  

On the relationship between Radiance and Irradiance: Determining the illumination from images of a convex Lambertian object
Journal of the Optical Society of America (JOSA A) Oct 2001, pages 24482459
This paper considers the 3D Lambertian case using spherical harmonics
and derives an analytic formula for the irradiance in terms of the
radiance, including the 9 parameter Lambertian BRDF approximation.
One practical application is interactive rendering with An Efficient
Representation for Irradiance Environment Maps.

A SignalProcessing Framework for Inverse Rendering: Siggraph 01, pages 117128
This paper is the most mathematical so far and derives the theory for
the general 3D case with arbitrary isotropic BRDFs.
It also applies the
results to the practical problem of inverse rendering under
complex illumination.  
An Efficient Representation for Irradiance Environment Maps:
Siggraph 01, pages 497500 We consider the rendering of diffuse objects under distant illumination, as specified by an environment map. Using an analytic expression for the irradiance in terms of spherical harmonic coefficients of the lighting, we show that one needs to compute and use only 9 coefficients, corresponding to the lowestfrequency modes of the illumination, in order to achieve average errors of only 1%. Full Paper: gzipped PS (3.4M) PDF (1M) Talk: PPT (1.8M) Video 

Efficient ImageBased Methods for
Rendering Soft Shadows:
Siggraph 00, pages 375384 We present two efficient imagebased approaches for computation and display of highquality soft shadows from area light sources. Our methods are related to shadow maps and provide the associated benefits. Full Paper: gzipped PS (4M) PDF (1.7M) Talk: PPT (1.8M) 

Creating Generative Models from Range
Images Siggraph 99, pages 195204 We have explored the creation of highlevel parametric models from lowlevel range data. Our modelbased approach is relatively insensitive to noise and missing data and is fairly robust. Full Paper: PS (2.5M) PDF (1.5M) 

Fast
Construction of Accurate Quaternion Splines:
Siggraph 97, pages 287292. Dynamic Splines with Constraints for Animation: Caltech CSTR9703. We have explored the use of improved numerical approaches for optimization to automatically create animation from keyframes. The numerical tools developed include adaptive refinement based on the EulerLagrange error functional. We have applied this approach to quaternion splines, greatly speeding up a numerical method to construct the optimal rotational curve. Full Paper: Sig 97 PDF Tech Report 
Teaching (at Berkeley) 
CS 283  Advanced Computer Graphics  Spring 2013  Fall 2010  Fall 2009 
CS 184  Computer Graphics  Fall 2012  Spring 2012  Spring 2010 
Teaching (at Columbia) 
COMS 4160  Computer Graphics  Fall 2008  Spring 2008  Fall 2006  Fall 2005  Fall 2004 
COMS 4162  Advanced Computer Graphics  Spring 2006  Spring 2005  
COMS 6160  Topics in Computer Graphics  Visual Appearance (Spr 2007)  RealTime Rendering (Fall 2004)  Appearance Models (6998 Fall 2002) 
Students, Alumni and Collaborators 
Lytro Pictures of student and alumni dinner at SIGGRAPH 2014 in Vancouver (courtesy Ren Ng). And standard 2D photograph.
Current Postdoc: Nima Khademi Kalantari
Current PhD Students: LingQi Yan, TingChun Wang, Weilun Sun, Pratul Srinivasan, Zak Murez, Matteo Mannino, Zexiang Xu, Lifan Wu, Sai Bi, Jiyang Yu, Alexandr Kuznetsov
Current MS Students: Jingwen Wang
Alumni: Muhammad Riaz (MS Aug 2016, now at Apple), Michael Tao (PhD Aug 2015, now at Apple), Soham Uday Mehta (PhD May 2015, now at Light), ChiWei Tseng (MS thesis Jun 2015, now at Dreamworks), JongChyi Su (MS Jun 2015, now PhD student at UMass), Krishna Mullia (MS Jun 2015, now at Dreamworks), Jiamin Bai (PhD Sep 2014, now at Light), Eno Toeppe (Postdoc 20132014, now at Magic Leap), Dikpal Reddy (Postdoc 20112013, now at NVIDIA), Brandon Wang (MS,BS 20112013, now at Pixar), Milos Hasan (Postdoc 20102012, now at AutoDesk), Kevin Egan (PhD Aug 2011, now at DE Shaw), Charles Han (PhD May 2011, now at Google), Manmohan Chandraker (Postdoc 20092011, now at UCSD!), Huamin Wang (Postdoc 20092011, now at Ohio State), Adrien Bousseau (Postdoc 20092010, now at INRIA), Craig Donner (Postdoc 20072009, now at Google), Jinwei Gu (PhD May 2010, now at Sarnoff), FuChung Huang (MS May 2010, now at NVIDIA after PhD), Dhruv Mahajan (PhD Aug 2009, now at Microsoft Research), Ryan Overbeck (PhD Aug 2009, now at Google), Bo Sun (PhD Aug 2008, now at Intu Financial), Aner BenArtzi (PhD May 2007, now at Sony), Jason Lawrence (Princeton) (PhD June 2006, now faculty at UVA), Simon Premoze (postdoc 20032005, now at ILM) , Sebastian Enrique (MS May 2005, now at Electronic Arts), Kalyan Sunkavalli (MS May 2006, now at Adobe after PhD at Harvard), Nandan Dixit (MS Dec 2006, now at Google), Diego Nehab (Princeton) (PhD Jun 2007, now at MSR > IMPA), YuTing Tseng (MS Dec 2008, now at Google).
Background 
I joined the Computer Science and
Engineering Department at UC San Diego, starting July 2014.
I was on the faculty of the Electrical Engineering and Computer Science Department at UC Berkeley
from January 2009 to June 2014.
Since the fall of 2002 (until Dec 2008), I was on the faculty of the Columbia Computer Science
Department.
Earlier (19982002), I completed my Ph.D in the
Stanford Computer Science
Department , working in the Computer Graphics Laboratory.
Earlier (19941998), I was an undergraduate at the California Institute of Technology,
getting a BS, MS in Computer Science and an MS in Physics.
A full CV is also available.
Personal 