The goal of this assignment is to implement an algorithm that reconstructs a surface using the concept of photometric stereo. Additionally, you will implement the specular removal technique of Mallick et al., which enables photometric stereo reconstruction of certain non-Lambertian materials. You can assume a Lambertian reflectance function once specularties are removed, but the albedo is unknown and nonconstant in the images. Your program will take in multiple images as input along with the light source direction (and color when necessary) for each image.

Part 1

Implement the photometric stereo technique described in Forsyth and Ponce 5.4 and the lecture notes. Your program should have two parts:

1. Read in the images and corresponding light source directions, and estimate the surface normals and albedo map.
2. Reconstruct the depth map from the normals.

Try this out on the synthetic dataset with three subsets of images:

1. im1.png, im2.png, im4.png (Easiest)
2. im1.png, im2.png, im3.png (This triplet is likely to be problematic, and if so, explain why)
3. all four images (Most accurate)

What to include in your report

You should show in your report:

1. Estimated albedo map
2. Estimated surface normals by either showing
• A needle map (you will need to subsample the image to get a needle map which can be displayed)
• Three images showing three components of surface normal
3. A wireframe of a depth map (you can use surf() in matlab).

Part 2

Implement the specularity removal technique described in Beyond Lambert: Reconstructing Specular Surfaces Using Color (by Mallick, Zickler, Kriegman, and Belhumeur; CVPR 2005). Your program should input an RGB image and light source color and output the corresponding SUV image. Try this out first with the specular sphere images and then with the pear and squash images.

What to include in your report

You should show in your report the following for each specular sphere image:

1. The original image (in RGB colorspace)
2. The recovered S channel of the image
3. The recovered diffuse part of the image - Use G = sqrt(U^2 + V^2) to represent the diffuse part

Part 3

Combine parts 1 and 2 by running your photometric stereo code on the diffuse components of the specular sphere, pear and squash images. For comparison, run your photometric stereo code on the original images (converted to grayscale) as well. You should notice eroneous "bumps" in the resulting reconstructions - the result of violating the Lambertian assumption.

What to include in your report

For each image set (specular sphere, pear, and squash) you should show in your report:

1. The recovered diffuse images {G = sqrt(U^2+V^2)}
2. Estimated albedo map (original and diffuse images)
3. Estimated surface normals (original and diffuse images) by either showing
• A needle map
• Three images showing three components of surface normal
4. A wireframe (original and diffuse images) of a depth map (you can use surf() in matlab).

Part 4 (Optional, Extra Credit)

Implement the multi-channel photometric stereo algorithm of Barsky and Petrou (Colour photometric stereo: Simultaneous reconstruction of local gradient and colour of rough textured surfaces., ICCV 2001). Show results as in part 3 using both the RGB colorspace and UV colorspace on at least one image set.

What to Hand In

You should hand in:

1. A report with the items described above, any other figures, and comments/issues that you find relevant.
2. Hardcopy of your code.
3. Email me a tar or zip file with the code and report.

Data

Synthetic Images

The following four test images will serve as input, and they can be downloaded here. The first row shows the image, the second row is the file name, and the third row is the light source direction (not normalized) used to generate the image. Note that in all four images, the strength of the light sources was a constant 1.0.

im1.png	im2.png	im3.png	im4.png
[0 0 -1]	[0 0.2 -1]	[0 -0.2 -1]	[0.2 0 -1]

Specular Sphere Images

These images of a specular sphere (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Pear Images

These images of a pear with specularities (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Squash Images

These images of a squash with specularities (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Additional Resources

Integration Code

While a naive integration routine will probably work ok for synthetic data, it might have problems with real images. Here we provide a matlab implementation of the Horn integration technique (summarized here by Satya Mallick). You still need to write your own integration code, but if your algorithm doesn't handle real images well, use this routine instead.