Sampling and Reconstruction of Visual Appearance

CSE 274 Topics in Computer Graphics, Winter 2018, Prof. Ravi Ramamoorthi Time and Place: Tu/Th 9:30-10:50, EBU3B 2154

Overview

There are a number of topics in computer graphics which today require sampling and reconstruction of high-dimensional visual appearance datasets, such as high-quality real-time rendering with precomputed light transport, Monte Carlo rendering, light transport acquisition, and light field imaging. Perhaps the most significant recent developments have arisen in the area of Monte Carlo image synthesis via path tracing and related methods. Indeed, one of the perennial goals of computer graphics is creating high quality images which are indistinguishable from photographs: a goal referred to as photorealism.

In essence, one is sampling a high-dimensional function (8D for image coordinates, soft shadows on an area light, global illumination from incident directions, and depth of field on a lens), then reconstructing or denoising it to produce the final image. The industry has recently moved to physically-based path tracing as its preferred solution, with a low sample count, followed by advanced reconstruction and denoising methods. These advances have reduced the computation time by one to two orders of magnitude, enabling production rendering with physically-based global illumination, and this approach is beginning to be considered for real-time rendering applications. This course surveys these recent developments over the past 5-10 years, in many of which UCSD faculty have played a prominent role. (Note that no background on Monte Carlo rendering is required, since most algorithms take the actual rendering process as a black box).

Beyond traditional rendering, sampling and reconstruction of visual appearance is critical for many other applications, such as acquisition of light transport from real-world scenes, light field cameras that capture both spatial and angular distributions of light, and real-time methods based on precomputed light transport. All of these problems involve 4D-8D functions, which are prohibitive to sample by brute force, and where coherence or sparsity needs to be exploited to enable sparse sampling and sophisticated reconstrcution. The course surveys recent advances, within the last decade, in all of these areas.

This body of work is built on mathematical techniques, some of which are recent developments, including multi-dimensional Fourier analysis of light transport spectra, compressive sensing for sparse datasets, and deep learning for feature-based image denoising. While no prior background on these areas is required, the course will provide a brief high-level introduction, to the extent these topics are needed for the practical applications we consider.

Please note that CSE 274 is a topics course, covering current topics in computer graphics. Course content may change every year or two years. This course material is completely new, and you are allowed to enrol and get full credit even if you have taken CSE 274 previously with me or other faculty. Below are a few example images produced using some of the algorithms and systems we will be discussing.

Prerequisites

The course is targeted towards MS and PhD students with a knowledge of and interest in computer graphics (at the level of CSE 167 or equivalent at another university; CSE 168 is not required but would be helpful). We welcome all PhD students working in graphics, vision, and robotics. We are willing to consider students who do not completely fulfil the pre-requisites in exceptional cases, if they have a strong interest in the material. Undergraduates who have taken CSE 167 (and/or other advanced courses such as CSE 163/168) are also welcome, space permitting.

Relationship to Other Courses

The course is an integral part of the vision and graphics track for undergraduate, MS and PhD students (The instructor can help in getting any relevant permissions for it to count for credit towards the track as needed). It builds on the undergraduate graphics classes CSE 167 [taught by Profs. Schulze/Ochoa in the fall/spring] and CSE 163/168/169. In particular, CSE 168 discusses traditional rendering methods (advanced graduate version of which is CSE 272/274, both taught by Prof. Jensen in the spring), while this course essentially treats the rendering method as a black box (prior knowledge not required), and explores efficient sampling and reconstruction. If you like this course, I would highly recommend also registering for my course CSE 163 in the spring, as well as CSE 168/272 in the spring. You may also be interested in CSE 291 taught concurrently this quarter by Prof. Hao Su. Note however, CSE 168 is not required as a pre-requisite (some exposure to graphics, in the form of CSE 167 or the equivalent elsewhere is required).

Course Format and Requirements

The course will consist of lectures on the relevant topics by the instructor, student presentations of papers covering current research in the area, and student projects. A syllabus/schedule is noted below. The grading will be 30% for paper presentations, 60% for the project, and 10% for class participation. Students are expected to come to class regularly and participate in discussions, since this is an advanced graduate course.

In general, roughly (depends on the number of students in the course), 1 or 2 paper presentations will be required (if the number of students in the course exceeds the number of presentations, some may be done jointly by a group of two students). A project is not required for students taking the course S/U or P/NP; we leave it to your discretion if you want to register for 2 or 4 units in this case. An S/U registration is a good option for PhD students and others to read papers on this exciting topic and learn about the area without committing too much effort into a course project, and we encourage you to sign up. Auditors, who simply want to sit in on the course are also welcome; however, we prefer if you sign up for the course pass/fail instead [this just involves doing one or two paper presentations].

Students taking the course for a letter grade are required to do a project [this may be in groups of 2], give a presentation in class regarding their results, and also submit a final written report. Wide flexibility is available with respect to project topics, provided they relate loosely to the subject matter of the course. We expect that most projects will implement one or more of the algorithms or papers discussed in the course, showing results on sampling and reconstruction of visual appearance. We welcome suggestions from the students on alternative project ideas. The best projects will go beyond the published work in some way, such as trying out an alternative or better approach or trying to develop some variant or more general version of the technique. However, this is not essential; in general, students who fulfil all course requirements including a well-executed project will easily receive an A in the course.

As a potentially easier alternative to the project, we will also accept a well-written summary or tutorial, covering 3 or 4 papers. The best summaries will point out links between the papers not noticed by the original authors and suggest improvements or directions for future research. However, this option is recommended only as a last resort and will generally receive a slightly lower score; we prefer that you do a good project (which may involve understanding a few papers in any case).

Lectures and Office Hours

The lectures will be in EBU3B (CSE Building) 2154 from 9:30-10:50 on Tuesdays and Thursdays. Students are expected to come to the lectures and participate in discussion. Office hours will be immediately after class from 11-12 in the Professor's office, EBU3B 4118. You may also e-mail for another time to meet if that is not convenient.

Teaching Assistant

We have a teaching assistant for the course, Jiyang Yu. His e-mail is jiy173@eng.ucsd.edu . His office hours will be in 4150 (CS bldg) Mondays and Wednesdays from 11-noon, or you can e-mail him to schedule an appointment.

Topics

Topics to be covered include

Resources

Outline

The (tentative) course schedule is as follows. In general, you will likely benefit from doing the reading (i.e. the papers assigned for a particular date) before class; it will at least make for more lively discussion.

Jan 9: Jan 11: Jan 16: Jan 18: Jan 23: Jan 25: Jan 30: Feb 1: Feb 6: Feb 8: Feb 13: Feb 15: Feb 20: Feb 22: Feb 27: Mar 1: Mar 6: Mar 8: Mar 13: Mar 15:
Ravi Ramamoorthi
Last modified: Tue Oct 31 17:49:35 PDT 2017