Complex Illumination, Materials and Shadows in Computer Vision


Natural illumination, realistic specular reflections, and complex cast shadows are key features in determining the appearance of objects. However, they have largely been ignored in computer vision algorithms. Indeed, most algorithms assume Lambertian objects lit by a single point source, without any explicit notion of visibility. Our goal is to derive the theoretical foundations and mathematical equations and computational techniques to allow much more robust computer vision algorithms that can work in outdoor lighting conditions, with difficult specular objects, taking shadows and interreflections into account.

Primary Participants

Ravi Ramamoorthi
Tian-Tsong Ng (A*STAR)
Manmohan Chandraker (NEC)
Dikpal Reddy
Jiamin Bai
Michael Tao
Dhruv Mahajan
Ronen Basri (Weizmann Institute)
Margarita Osadchy (NEC)
David Jacobs (Maryland)


The theoretical results for the Lambertian case , and especially the 9 parameter model have been widely used to handle complex illumination in many areas of vision, including lighting-insensitive recognition, structure from motion, and photometric stereo. Recently, we have linked the convolution results with principal component analysis, deriving more compact subspaces and explaining some classic previous empirical results. Our results enable complex illumination on Lambertian matte surfaces to be handled with essentially the same complexity as point light sources. In more recent work, we have also developed a compact model of specularities, combining that with the Lambertian results to use specularities for the first time as a positive source of information in lighting-insensitive recognition. We have recently also looked at cast shadows, especially in natural 3D textures like moss, sponge or gravel, showing that in many canonical cases, a convolution result can be derived and used to explain some observations regarding lighting variability. A good overview of spherical harmonic lighting and the theoretical basis behind our methods is given in a recent book chapter. We have also derived a new class of frequency domain invariants or spherical harmonic identities . They can be used for direct transfer-based relighting, without formal inverse rendering, and to check the consistency of images to detect tampering or splicing. Most recently, we have developed a new characterization of inverse light transport, a new differential theory of surface reconstruction for general reflectance, and formal conditions for reflectance and light transport estimation.

Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object PAMI Oct 2002, pp 1322-1333.
We explain for the first time some classic empirical results on lighting variability, and take a first step toward analyzing many classic vision problems under complex lighting.

Full Paper:     PDF (.8M)

Using Specularities for Recognition ICCV 03, pages 1512-1519
We present the first method for using specularities as a positive feature for lighting-insensitive recognition. The method is applied to very difficult objects like shiny crockery and wine glasses.

Paper:     PDF

A Fourier Theory for Cast Shadows ECCV 04, pages I 146-162 ; PAMI Feb 05, pages 288-295
We show that cast shadows can be mathematically analyzed for many simple configurations, resulting in a standard convolution formula that can be derived analytically in 2D and analyzed numerically in 3D. The results help explain many effects of lighting variability in 3D textures and suggest new bases for that purpose.

Paper:     ECCV 04 ,     PAMI 05

Modeling Illumination Variation with Spherical Harmonics Book chapter in Face Processing: Advanced Modeling Methods (pages 385-424, 2005)
The appearance of objects including human faces can vary dramatically with the lighting. We present results that use spherical harmonic illumination basis functions to understand this variation for face modeling and recognition, as well as a number of other applications in graphics and vision.

Paper:     PDF

A Theory of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency ECCV 06, vol IV, pp 41-55, PAMI 30(2), pages 197-213, Feb 2008.
We develop new mathematical results based on the spherical harmonic convolution framework for reflection, deriving novel identities, which are the angular frequency domain analogs to common spatial domain invariants such as reflectance ratios.

Paper:     PDF (PAMI 08)     PDF (ECCV 06)
On the Duality of Forward and Inverse Light Transport PAMI 2011, ECCV 2010
Inverse light transport seeks to undo global illumination effects, such as interreections, that pervade images of most scenes. This paper presents the theoretical and computational foundations for inverse light transport as a dual of forward rendering. We demonstrate two practical applications, namely, separation of individual bounces of the light transport and fast projector radiometric compensation to display images free of global illumination artifacts in real-world environments.

Paper:     PAMI    ECCV     Tech Report     Video
What an Image Reveals About Material Reflectance ICCV 2011
We derive precise conditions under which material reflectance properties may be estimated from a single image of a homogeneous curved surface (canonically a sphere), lit by a directional source. Based on the observation that light is reflected along certain (a priori unknown) preferred directions such as the half-angle, we propose a semiparametric BRDF abstraction that lies between purely parametric and purely data-driven models. While it is well-known that fitting multi-lobe BRDFs may be ill-posed under certain conditions, prior to this work, precise results for the well-posedness of BRDF estimation had remained elusive.

Paper:     PDF
From the Rendering Equation to Stratified Light Transport Inversion International Journal of Computer Vision, 2012
In this work, we explore a theoretical analysis of inverse light transport, relating it to its forward counterpart, expressed in the form of the rendering equation. We show the existence of an inverse Neumann series, that zeroes out the corresponding physical bounces of light, which we refer to as stratified light transport inversion. Our practical application is to radiometric compensation, where we seek to project patterns onto real-world surfaces, undoing the effects of global illumination.

Paper:     PDF
Frequency-Space Decomposition and Acquisition of Light Transport under Spatially Varying Illumination ECCV 2012.
We show that, under spatially varying illumination, the light transport of diffuse scenes can be decomposed into direct, near-range (subsurface scattering and local inter-reflections) and far range transports (diffuse inter-reflections). We show that these three component transports are redundant either in the spatial or the frequency domain and can be separated using appropriate illumination patterns, achieving a theoretical lower bound.

Paper:     PDF     Talk: PPT
Sharpening Out of Focus Images using High-Frequency Transfer EuroGraphics 2013.
We propose a new method to sharpen out-of-focus images, that uses a similar but different assisting sharp image provided by the user (such as multiple images of the same subject in different positions captured using a burst of photographs). We demonstrate sharpened results on out-of-focus images in macro, sports, portrait and wildlife photography.

Paper:     PDF    Video (MPEG)    Supplementary HTML

Last modified: Sat Feb 9 10:13:25 PST 2013