Recruitment

Who am I?

My name is Tzu-Mao Li. I am an assistant professor in UC San Diego's Computer Science & Engineering department. Check out my webpage if you haven't visited it yet.

For prospective students/postdocs

I am not actively recruiting these days. However, the following tips should still be useful for general graduate school admission.

If you want to work with me as a graduate student, please apply through the UCSD CSE department website, and select me as one of your potential advisors in the application form (otherwise, there is little chance I will notice your application). If you are interested in a postdoc position, please send me an email (tzli@ucsd.edu). You don't have to send me an email if you apply to UCSD CSE as a graduate student, unless you have some specific questions. I am happy to answer questions regarding the application and research.

I would like to see the following in your application materials:

What are your research interests? Why?
How does your background support your interests?
How do you see your research field advancing? What are the interesting problems you want to solve? What are your plans?
How do your interests align with mine? What are your thoughts on solving the problems I list below?

Be concrete and clear. Give examples. Answer the "Why" and "What" of the questions. Ideally, you want to show a non-trivial, mature view of research fields instead of just throwing buzzwords. Ideally, you want to have a concrete proposal of a research direction, but it is fine if it is vague (you don't want me to steal your idea anyway). I do not expect you to answer these questions perfectly. I probably can't answer them perfectly myself either. Try your best.

I do not expect students to have prior experience with computer graphics or programming languages (it is a plus if you have some). However, make sure you are enthusiastic about at least a subset of them (for example, the topics I list below). I do expect students to have solid mathematics skills and/or software engineering skills. Ideally, the students should have some expertise that I don't have, so we can learn from each other.

My expectations of students

First, read the Graduate Study Survival Guide from Toshiya Hachisuka (a UCSD alumni!). I agree with pretty much all of his points (the only exception is group meeting is a waste of time).
When doing research, you are the leader and the owner of the project. You should: 1) Assess how feasible it is as a research project based on your own interests (the most important), the overall trend and culture in the field, your own ability, and physical constraints. 2) Decide when to cut loss and avoid sunk-cost fallacy. 3) Actively learn the relevant skills yourself - talk to relevant people, read the relevant books, DuckDuckGo the relevant keywords, digging old related work, etc. 4) Actively search for potential research ideas, and if low-hanging fruits appear, talk to me to see if it is a reasonable idea. 5) Tell me about your plans in meetings. Note that graduate study is very different from undergrad study: there is basically no structure (and that's why I like it) and no one is going to tell you what to do. You are expected to actively seek for tasks to do. All we are going to provide are advices, and you don't have to listen to them 100% of the time. Use your own judgement, and be responsible for your projects. That being said, do listen to the advices and think about them. Ideally take notes if you are forgetful.
What do I get in the process? My main motivation to stay in the academia is to learn from a diverse set of people for a diverse set of topics, and stay on the frontier of human knowledge for the topics I am interested in. As an advisor, I expect to learn from my students. You should be prepared to teach me things I don't know, explain things in detail, spoon-feed me the information. You can tell me to read relevant information (and I will if I am interested), but tell me why it is relevant and how it will help your research. In general, I want to learn things from you through the process of collaborating with you on research projects, instead of teaching you every single technical detail.
What do I have to offer in the process? By freeing myself from many of the technical details of the project, I am capable of thinking in much higher level, linking different ideas together, and assessing the broader impact of the project: Is it even worth doing this in the first place? Are we asking the right research questions? What will be the consequence if we are successful? I can provide inspirations for you to come up with your own research ideas from these higher-level links. I am also likely to be much more familiar with the academic culture than you, so I can explain what other people in the community think to you too. I will also teach you how to communicate with your fellow academics: communicating the idea well is actually 80% of the research (even though the rest 20% may take you 80% of the time). In the process, I will also question you about all aspects of your research: Is the project really as promising as you think? Is the project really as boring as you think? Why should people care about this project? Sometimes we can brainstorm the answers together (so feel free to say you don't know, but only after you think hard about it), but be reminded that you are responsible for your own projects.
Don't be afraid of making mistakes. It's very natural to feel stupid when doing research since research is about studying unknowns. It is thus tempting to avoid mistakes by doing less. However, such decision is unproductive because you achieve much less by doing less. When it comes to research, making mistakes is better than doing nothing. As a corollary, you should avoid analysis paralysis. A lot of the insights can be gained by diving into a seemingly impossible task even if you are not successful immediately.
Be open-minded. Don't ignore your peer's research just because it's irrelevant to you (I ignored NLP for a long time and look where that has taken me). It's always related. Be curious. Ask people about their research. If they couldn't explain it well to you, it's their fault, not because you are dumb. Ask questions so that they realize they did not explain it clear enough. Be persistent and ask until you have a rough idea. Similarly, do the same when you explain your research to other people. Try as hard as possible to relate people's research to yours - sometimes it will lead to unexpected research ideas. Besides, graphics is already a very small field, I don't see the point of further dividing ourselves inside graphics.
Don't get bogged down by details. Try to understand the high-level points. You should always be able to reproduce the low-level details if given enough time. Use the power of abstraction.
Be critical and have your own opinions. When you read or hear other people's research: Think about whether they are solving an important problem. Think about whether you would have done things differently. Think about whether they have achieved their goals as they have advertised. Think about the next steps. Think about the next steps of the next steps. Think about the next steps of the next steps of the next steps.
Be transparent. Actively communicate your thoughts, ideas, progress, and issues. The more you share, the more we have chance to figure something out together.
Attend talks and ask questions. When you ask questions, don't get bogged down by the exact details people do things. Relate their research to your own and ask questions that can help your own research. Remember - it's always related.
Talk to people. Explain your research and ideas to as many people as possible. Explain it to different levels of peers: your family who know nothing about computer science, undergrads, grad students but not in graphics, grad students in graphics but don't know much about your subfields, grad students in graphics and know your subfield, professors in graphics, professors in other fields.
Take notes of your research and document the important points and decisions you have made. This is not just for your collaborators but is very important for you. It reminds you why you are here right now and where you are going. Always track back to the higher-level points for every small decision you make. Revisit your decisions every once in a while.
Different people are suited for different research styles. Find your own style - you don't need to mimic me or other people.
Grad school is not a competition. It's a process of acquiring knowledge and communicating them. You are not ranked when you graduate by the number of papers you published. It's your own journey and it is irrelevant how other people did their own ones. The important things to ask are: "How much have you learned during the process?", "How much have you taught other people?", and "Have you had fun?". Of course, finding a job after you graduate is a different aspect. However, as a CS student graduated from UCSD, you shouldn't worry about this too much unless you're aiming for something like a faculty position at Stanford.
Our collaborators may have very different opinions compared to my points above. That's fine. Remember - you are the leader of the project and you should decide how to proceed.

What's my research?

Check out my job talk slides for a general overview.

Check out my webpage for publications and notes.

I work on the interactions between computer graphics, vision, programming systems, and machine learning. My main research direction is something I call "differentiable visual computing," where we go beyond neural networks and backpropagate through computer graphics programs. In graphics, we explicitly model how the world behaves (often through physics), instead of relying on generic neural networks to learn everything from scratch. Learning from data by backpropagating through graphics programs allows us to have more control and a better understanding of our programs' behavior, enables high-performance, and makes debugging/verification easier. The applications include: helping self-driving cars to make better decisions, training robots to interact with the environment using physical information, creating more realistic virtual realities, designing buildings and rooms to have better lighting, designing 3D physical objects with desired appearance and functionalities, reconstructing 3D structures of cells from microscope images, and allowing movie artists to produce better film shots.

Differentiating general graphics programs correctly and efficiently is much more complicated than differentiating a convolution layer, due to the general computation patterns and the mathematics involved. I derive algorithms that differentiate graphics programs while taking discontinuities into account. I design compilers that explore trade-offs of derivative computation and generate correct and efficient code. I look at applications where these differentiable graphics programs can be useful.

Many graphics researchers are trying to replace graphics pipelines with deep learning components. While this is a cool direction and is in my research scope, one of my main directions is to introduce differentiable graphics programs to deep learning pipelines. I claim that this is the way that will lead us to controllable, interpretable, robust, and efficient models. I am also highly interested in non-data-driven settings, where we try to figure out latent variables solely based on our knowledge of the physical process.

Following are some of my current research thrusts:

Differentiable rendering: How do we backpropagate through the rendering equation, so that we can infer 3D information from 2D observations? How do we handle discontinuities, and how do we make the differentiation as efficient as possible? How do we differentiate rendering with arbitrary geometry and material representations, while modeling physical phenomena such as occlusion, surface, and volumetric scattering, dispersion, and diffraction? Ultimately, we want to build a differentiable renderer that can generate and differentiate noise-free million-pixel-images with billions of varied primitives within seconds, while accurately modeling optics. We want to use the renderer for artificial intelligence agents to inference 3D information, for reconstructing detailed 3D models for virtual/augmented reality, for designing and fabricating real-world 3D objects with desired optical properties, for designing imaging systems, and for analyzing biomedical data using inverse optics.

Differentiable image processing: How do we design image processing pipelines that can learn from data, can process images with tens or hundreds of megapixels, in real-time, on mobile devices? I claim that instead of stacking more convolution layers and increasing overparametrization, we should generalize the programming model we use for designing image processing algorithms. Instead of the high arithmetic intensity deep learning layers, we want to take building blocks from more traditional image processing algorithms, parametrize them, and differentiate through them to learn the parameters. To achieve this, we need better compilers that can fuse the low arithmetic intensity computation, and can differentiate array code.

related work of mine: [Li2018b] [Adams2019] [Li2020] [Bernstein2020] [Ma2022]

Differentiable physics simulation: How do we backpropagate through ODE/PDE solvers to make inference based on the dynamics? This problem is studied in the optimal control and the sensitivity analysis literature (and this is also what inspired the neural ODE work). However, differentiation of discontinuities and boundary conditions in ODEs/PDEs is not very well-understood. Furthermore, how do we efficiently map these computations to modern hardware? How do we reduce the memory usage when backpropagating an iterative solver? Answering these questions will enable us to train robot controllers orders of magnitude more efficiently, design 3D objects with physical constraints, or even have more elaborated epidemiology models.

related work of mine: [Hu2019] [Hu2020] [Bangaru2021]

Domain-specific compilers for differentiable computer graphics: How do we design compilers that take high-level computer graphics code (rendering, image processing, simulation, geometry processing), and automatically output high-performance low-level code along with the derivative code? How do we certify the correctness? Just like how deep learning frameworks democratize machine learning, I want to build an easy-to-use programmable differentiable graphics system that makes building differentiable graphics pipelines as simple as training an MNIST classifier in PyTorch, while generating reliable code.

Accelerating physically-based rendering: Physically-based rendering is known to be time-consuming, due to its need to compute multi-dimensional integrals using Monte Carlo sampling. How do we make it faster? I believe there are two keys towards the ultimate rendering algorithm: 1) re-using Monte Carlo samples through statistical analysis. 2) replacing heuristics with data-driven components.

related work of mine: [Li2012] [Wu2015] [Li2015] [Anderson2017] [Gharbi2019] [Wang2021]

Beyond differentiation: Can we derive or automatically find *something* that approximates a function locally, and better inform our inference algorithms, when comparing to derivatives? For example, Fourier analysis or wavelet analysis gives us strictly more information than derivatives (differentiation is a linear ramp in the frequency domain). How can we use them in optimization algorithms? Can we find something that scales better with dimensionality compared to Fourier analysis? Can we develop systems that help us find these quantities given a program?

related work of mine: Nothing here yet. Hopefully your work will be listed here!