In this course, we will study an emerging field called **loma** (written in Python) and add support for automatic differentiation, while making sure they handle (limited) side effects, control flow, and parallelism (we will generate GPU code!). The second half of the course and the final project will be focusing on applying the programming language for different kinds of tasks, including differentiable simulation, image and signal processing, 3D rendering, etc.
Note that this course is different from, though related to, the seminar course CSE 290 on differentiable programming.

differentiable programming, which is an interdisciplinary field that combines machine learning, programming languages, numerical computing, signal processing, robotics, computer vision and computer graphics. Recent work on deep learning has revealed the power of derivative-based optimization. Instead of computing gradients of typical feedforward deep models, differentiable programming attempts to differentiate general

programsand use them as our machine learning models. This leads to several challenges:

- How do we design machine learning models that are composed of general programs that leverage domain knowledge, instead of the typical neural network layers?
- How do we correctly and efficiently derive the derivatives of general numerical programs that can have control flow, discontinuities, and higher-order functions?
- How do we define/derive the derivatives of programs that have certain semantic meanings, such as sorting, satisfiability solvers, root solvers, differential equation solvers, or 3D rendering?
- How do we build differentiable programming languages and systems, so that we can guarantee the correctness for rapid exploration, and efficiently compile them to modern hardware architectures?
- How do we leverage these differentiable programs for real tasks in machine learning and inverse problems?

Python and machine learning. Ideally, you should have basic compiler and parallel programming knowledge, but we do not assume you know.

We will do most of the online discussions on Piazza. All the annoucements will also be made through Piazza, so make sure you sign up.

There will be 3 programming homeworks (20% each) and 1 final project (40%).

Late penalty: score * clamp(1 - (seconds passed after midnight of the deadline day) / 86400, 0, 1) (no late submission for the final project)

We will use the time on Canvas to determine how many seconds have passed.

The book Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation is the bible of this field.

The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation is also a great book focusing on how to differentiate C code.

The Elements of Differentiable Programming is a recent comprehensive introduction to the topic.