The goal of this assignment is to implement a couple of different algorithms that reconstruct a surface using the concept of photometric stereo. Additionally, you will implement the specular removal technique of Mallick et al., which enables photometric stereo reconstruction of certain non-Lambertian materials. You can assume a Lambertian reflectance function once specularties are removed, but the albedo is unknown and nonconstant in the images. Your program will take in multiple images as input along with the light source direction (and color when necessary) for each image. You will also implement a second example-based photometric stereo algorithm which is based on simultaneously imaging two objects of the same material, one of which has known structure.

Part 1

Implement the photometric stereo technique described in Forsyth and Ponce 5.4 and the lecture notes. Your program should have two parts:

1. Read in the images and corresponding light source directions, and estimate the surface normals and albedo map
2. Reconstruct the depth map from the normals. You can first try the naive scanline-based shape by integration method described in the book. If this does not work well on real images, you can use the matlab implementation of the Horn integration technique at the bottom of this page.

Try this out on the synthetic dataset with three subsets of images:

1. im1.png, im2.png, im4.png (Easiest)
2. im1.png, im2.png, im3.png (This triplet is likely to be problematic, and if so, explain why)
3. all four images (Most accurate)

What to include in your report

You should show in your report:

1. Estimated albedo map
2. Estimated surface normals by either showing
• A needle map (you will need to subsample the image to get a needle map which can be displayed. You can use the matlab functions meshgrid() and quiver3())
• Three images showing three components of surface normal
3. A wireframe of a depth map (you can use surf() in matlab).

Part 2

Implement the specularity removal technique described in Beyond Lambert: Reconstructing Specular Surfaces Using Color (by Mallick, Zickler, Kriegman, and Belhumeur; CVPR 2005). Your program should input an RGB image and light source color and output the corresponding SUV image. Try this out first with the specular sphere images and then with the pear and squash images.

What to include in your report

You should show in your report the following for each specular sphere, pear, and squash image:

1. The original image (in RGB colorspace)
2. The recovered S channel of the image
3. The recovered diffuse part of the image - Use G = sqrt(U^2 + V^2) to represent the diffuse part

Part 3

Combine parts 1 and 2 by running your photometric stereo code on the diffuse components of the specular sphere, pear and squash images. For comparison, run your photometric stereo code on the original images (converted to grayscale) as well. You should notice eroneous "bumps" in the resulting reconstructions - the result of violating the Lambertian assumption.

What to include in your report

For each image set (specular sphere, pear, and squash) you should show in your report:

1. The recovered diffuse images {G = sqrt(U^2+V^2)}
2. Estimated albedo map (original and diffuse images)
3. Estimated surface normals (original and diffuse images) by either showing
• A needle map
• Three images showing three components of surface normal
4. A wireframe (original and diffuse images) of a depth map (you can use surf() in matlab).

Part 4

Implement the photometric stereo algorithm described in Section 2 of Shape and Materials by Example: A Photometric Stereo Approach (by Hertzmann and Seitz). Use the algorithm to reconstruct the 3D shape of the bottle using the set of corresponding bottle and sphere images. You may find it useful to use an algorithm to speedup the nearest-neighbor search, such as a kd-tree. What are the differences in assumptions made (e.g., with respect to light source, BRDF, etc.) for this algorithm compared to the algorithm from Part 3?

What to include in your report

1. Estimated surface normals of the bottle by either showing
• A needle map
• Three images showing three components of surface normal
2. A wireframe of a depth map (you can use surf() in matlab).

What to Hand In

You should hand in:

1. A report with the items described above, any other figures, and comments/issues that you find relevant.
2. Hardcopy of your code.
3. Email me a tar or zip file with the code and report.

Data

New Synthetic Images

The following synthetic images should be used for Part 1: download here. The file contains the following data:

1. im1, im2, im3, im4 ... grayscale images.
2. l1, l2, l3 , l4 ... light source directions

It is assumed that images are indexed as im1(y,x), the light source vector l1 is a normalized vector from the optical center to the light source, and the strength of the light source is a constant 1.0.

Old Synthetic Images

These are the old synthetic images. It is preferred that you report results for Part 1 on the new synthetic images; however, it is ok to use the old synthetic images if you have already used them in your experiments. They can be downloaded here. The first row shows the image, the second row is the file name, and the third row is the light source direction (not normalized) used to generate the image. Note that in all four images, the strength of the light sources was a constant 1.0.

im1.png	im2.png	im3.png	im4.png
[0 0 -1]	[0 0.2 -1]	[0 -0.2 -1]	[0.2 0 -1]

Images For Debugging

The following files sample1 and sample2 can be used if desired for debugging purposes (since the Synthetic Images have a less intuitive 3D structure). Each mat file contains

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions

Specular Sphere Images

These images of a specular sphere (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Pear Images

These images of a pear with specularities (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Squash Images

These images of a squash with specularities (download here) are available in a matlab .mat file (graciously provided by Satya Mallick). This mat file contains,

1. im1, im2, im3, im4 ... color images.
2. l1, l2, l3 , l4 ... light source directions
3. c (when required) color of light source.

Bottle Images

These images of bottle and sphere pairs taken under the same lighting conditions (download here)

1. bottle1.png, bottle2.png ... color images.
2. sphere1.png, sphere2.png ... color images. Sphere and bottle images of the same number were taken under identical lighting conditions
3. bottlemask.png ... binary image of the bottle's segmentation mask.
4. spheremask.png ... binary image of the sphere's segmentation mask

Additional Resources

Integration Code

While a naive integration routine will probably work ok for synthetic data, it might have problems with real images. Here we provide a matlab implementation of the Horn integration technique (summarized here by Satya Mallick). You still need to write your own integration code, but if your algorithm doesn't handle real images well, use this routine instead.