UCSD CSE 291: Differentiable Programming (Winter 2024)
Course Description
In this course, we will study an emerging field called
differentiable programming
, which is an interdisciplinary field that combines machine learning, programming languages, numerical computing, signal processing, robotics, computer vision and computer graphics. Recent work on deep learning has revealed the power of derivative-based optimization. Instead of computing gradients of typical feedforward deep models, differentiable programming attempts to differentiate general
programs
and use them as our machine learning models. This leads to several challenges:
- How do we design machine learning models that are composed of general programs that leverage domain knowledge, instead of the typical neural network layers?
- How do we correctly and efficiently derive the derivatives of general numerical programs that can have control flow, discontinuities, and higher-order functions?
- How do we define/derive the derivatives of programs that have certain semantic meanings, such as sorting, satisfiability solvers, root solvers, differential equation solvers, or 3D rendering?
- How do we build differentiable programming languages and systems, so that we can guarantee the correctness for rapid exploration, and efficiently compile them to modern hardware architectures?
- How do we leverage these differentiable programs for real tasks in machine learning and inverse problems?
The first half of the course will be focusing on building a C-like SIMD-based differentiable programming language. You will step-by-step modify a compiler of a domain-specific language
loma (written in Python) and add support for automatic differentiation, while making sure they handle (limited) side effects, control flow, and parallelism (we will generate GPU code!). The second half of the course and the final project will be focusing on applying the programming language for different kinds of tasks, including differentiable simulation, image and signal processing, 3D rendering, etc.
Note that this course is different from, though related to,
the seminar course CSE 290 on differentiable programming.
Required Knowledge
Python and machine learning. Ideally, you should have basic compiler and parallel programming knowledge, but we do not assume you know.
Logistics
Lectures: Monday/Wednesday/Friday 10:00am-10:50pm at WLH 2112 (From 4/15, we will move to MANDE B-104)
Instructor office hour: Monday 11am-noon at CSE 4116.
TA office hour: Friday 2pm-3pm at CSE B275.
We will do most of the online discussions on
Piazza. All the annoucements will also be made through Piazza, so make sure you sign up.
Grading
There will be 3 programming homeworks (20% each) and 1 final project (40%).
Late penalty: score * clamp(1 - (seconds passed after midnight of the deadline day) / 86400, 0, 1) (no late submission for the final project)
We will use the time on Canvas to determine how many seconds have passed.
Homeworks and Projects
The homeworks involve quite a bit of programming and can be tough for the inexperienced. Start early and ask questions!
Many of them will be based on the loma compiler.
Final Project (40%): proposal due 5/20, check point due 5/29, final due 6/13.
Collaboration policy: for the homeworks, you need to do it yourself (you are free to discuss between peers). For the final project, you can have a team maximum of 2 people.
Readings (optional)
Schedule (very tentative)
4/1/2024 (Mon): Why differentiable programming? [slides]
4/3/2024 (Wed): Introduction to the loma language and compiler. [live demo]
4/5/2024 (Fri): Definition of derivatives. Total derivatives. Forward mode. Dual numbers. [slides]
4/8/2024 (Mon): Implementing forward mode automatic differentiation in loma. [live demo]
4/10/2024 (Wed): Reverse mode automatic differentiation. Computational graph. Transposition of derivative operators. Cheap gradient principle. Backpropagation as a special case of reverse mode. [slides]
4/12/2024 (Fri): Reverse mode (continued). Implementing reverse mode automatic differentiation in loma. [live demo]
4/15/2024 (Mon): Implementing reverse mode automatic differentiation in loma. (HW1 due) [live demo]
4/17/2024 (Wed): Discuss my HW1 implementation. Tracing vs source-to-source transformation. [slides]
4/19/2024 (Fri): Handling loops and function calls. [slides]
4/22/2024 (Mon): Checkpointing. [slides]
4/24/2024 (Wed): Parallel automatic differentiation. [slides]
4/26/2024 (Fri): Higher-order derivatives. The Jacobian accumulation problem. Beyond forward/reverse modes. Randomized automatic differentiation. [slides]
4/29/2024 (Mon): Sparse Jacobian and Hessian. (HW2 due). [slides]
5/1/2024 (Wed): Discuss my HW2 implementation. Implementing loops and function calls differentiation in loma.
5/3/2024 (Fri): Implementing loops and function calls differentiation in loma.
5/6/2024 (Mon): Deep learning systems (PyTorch, Tensorflow, Jax). [slides]
5/8/2024 (Wed): Functional automatic differentiation. [slides]
5/10/2024 (Fri): More AD systems: CppAD, Adept, Enzyme. [slides]
5/13/2024 (Mon): More AD systems: Enzyme, Halide, SLANG.D. (HW3 due)
5/15/2024 (Wed): Slang.D.
5/17/2024 (Fri): Discuss my HW3 implementation. Reversible checkpointing. [slides]
5/20/2024 (Mon): Implicit differentiation and differentiable optimization. (final project proposal due) [slides]
5/22/2024 (Wed): Differentiating sorting. [slides]
5/24/2024 (Fri): Differentiating stochastic programs (e.g., score vs pathwise estimators), and integrals with parametric discontinuities. Smoothing and relaxation. [slides]
5/27/2024 (Mon): Memorial day. No lecture.
5/29/2024 (Wed): Differentiating stochastic programs (e.g., score vs pathwise estimators), and integrals with parametric discontinuities. Smoothing and relaxation. (final project checkpoint due)
5/31/2024 (Fri): Variational inference and integrals with parametric discontinuities. Smoothing and relaxation.
6/3/2024 (Mon): Optimization vs Sampling. Hamiltonian Monte Carlo. [slides]
6/5/2024 (Wed): Pitfall of automatic differentiation. [slides]
6/7/2024 (Fri): Discussion: how do we make AD available everywhere?
6/13/2024 (Thu): Final project due.