Who am I?
My name is Tzu-Mao Li. I am an assistant professor in UC San Diego's Computer Science & Engineering department. Check out my webpage if you haven't visited it yet.
For prospective students/postdocs
I am not actively recruiting these days. However, the following tips should still be useful for general graduate school admission.
If you want to work with me as a graduate student, please apply through the UCSD CSE department
, and select me as one of your potential advisors in the application form
(otherwise, there is
little chance I will notice your application). If you are interested in a postdoc position, please
send me an email (firstname.lastname@example.org). You don't have to send me an email if you apply to UCSD CSE as a graduate student,
unless you have some specific questions. I am happy to answer questions regarding the application and research.
I would like to see the following in your application materials
- What are your research interests? Why?
- How does your background support your interests?
- How do you see your research field advancing? What are the interesting problems you want to solve? What are your plans?
- How do your interests align with mine? What are your thoughts on solving the problems I list below?
Be concrete and clear. Give examples. Answer the "Why" and "What" of the questions.
Ideally, you want to show a non-trivial, mature view of research fields instead of just throwing buzzwords.
Ideally, you want to have a concrete proposal of a research direction, but it is fine if it is vague (you don't want me
to steal your idea anyway).
I do not expect you to answer these questions perfectly. I probably can't answer them perfectly myself either. Try your best.
I do not expect students to have prior experience with computer graphics or programming languages (it is a plus if
you have some). However, make sure you are enthusiastic about at least a subset of them (for example, the topics I
list below). I do expect students to have solid mathematics skills and/or software engineering skills. Ideally, the
students should have some expertise that I don't have, so we can learn from each other.
My expectations of students
- First, read the Graduate Study Survival Guide from Toshiya Hachisuka (a UCSD alumni!). I agree with pretty much all of his points (the only exception is
group meeting is a waste of time).
- When doing research, you are the leader and the owner of the project.
You should: 1) Assess how feasible it is as a research project based on your own interests (the most important), the overall trend and culture in the field, your own ability, and physical constraints.
2) Decide when to cut loss and avoid sunk-cost fallacy.
3) Actively learn the relevant skills yourself - talk to relevant people, read the relevant books, DuckDuckGo the relevant keywords, digging old related work, etc.
4) Actively search for potential research ideas, and if low-hanging fruits appear, talk to me to see if it is a reasonable idea.
5) Tell me about your plans in meetings.
Note that graduate study is very different from undergrad study: there is basically no structure (and that's why I like it) and no one is going to tell you what to do.
You are expected to actively seek for tasks to do.
All we are going to provide are advices, and you don't have to listen to them 100% of the time.
Use your own judgement, and be responsible for your projects. That being said, do listen to the advices and think about them.
Ideally take notes if you are forgetful.
- What do I get in the process?
My main motivation to stay in the academia is to learn from a diverse set of people for a diverse set of topics, and stay on the frontier of human knowledge for the topics I am interested in.
As an advisor, I expect to learn from my students. You should be prepared to teach me things I don't know, explain things in detail, spoon-feed me the information.
You can tell me to read relevant information (and I will if I am interested), but tell me why it is relevant and how it will help your research.
In general, I want to learn things from you through the process of collaborating with you on research projects, instead of teaching you every single technical detail.
- What do I have to offer in the process?
By freeing myself from many of the technical details of the project, I am capable of thinking in much higher level, linking different ideas together, and assessing the broader impact of the project:
Is it even worth doing this in the first place?
Are we asking the right research questions?
What will be the consequence if we are successful?
I can provide inspirations for you to come up with your own research ideas from these higher-level links.
I am also likely to be much more familiar with the academic culture than you, so I can explain what other people in the community think to you too.
I will also teach you how to communicate with your fellow academics: communicating the idea well is actually 80% of the research (even though the rest 20% may take you 80% of the time).
In the process, I will also question you about all aspects of your research:
Is the project really as promising as you think?
Is the project really as boring as you think?
Why should people care about this project?
Sometimes we can brainstorm the answers together (so feel free to say you don't know, but only after you think hard about it), but be reminded that you are responsible for your own projects.
- Don't be afraid of making mistakes. It's very natural to feel stupid when doing research since research is about studying unknowns. It is thus tempting to avoid mistakes by doing less. However, such decision is unproductive because you achieve much less by doing less. When it comes to research, making mistakes is better than doing nothing. As a corollary, you should avoid analysis paralysis. A lot of the insights can be gained by diving into a seemingly impossible task even if you are not successful immediately.
- Be open-minded.
Don't ignore your peer's research just because it's
irrelevant to you (I ignored NLP for a long time and look where that has taken me).
It's always related. Be curious. Ask people about their research.
If they couldn't explain it well to you, it's their fault, not because you are dumb.
Ask questions so that they realize they did not explain it clear enough.
Be persistent and ask until you have a rough idea.
Similarly, do the same when you explain your research to other people.
Try as hard as possible to relate people's research to yours - sometimes it will lead to unexpected research ideas.
Besides, graphics is already a very small field, I don't see the point of further dividing ourselves inside graphics.
- Don't get bogged down by details.
Try to understand the high-level points.
You should always be able to reproduce the low-level details if given enough time. Use the power of abstraction.
- Be critical and have your own opinions.
When you read or hear other people's research:
Think about whether they are solving an important problem.
Think about whether you would have done things differently.
Think about whether they have achieved their goals as they have advertised.
Think about the next steps.
Think about the next steps of the next steps.
Think about the next steps of the next steps of the next steps.
- Be transparent. Actively communicate your thoughts, ideas, progress, and issues. The more you share, the more we have chance to figure something out together.
- Attend talks and ask questions. When you ask questions, don't get bogged down by the exact details people do things. Relate their research to your own and ask questions that can help your own research. Remember - it's always related.
- Talk to people. Explain your research and ideas to as many people as possible. Explain it to different levels of peers: your family who know nothing about computer science, undergrads, grad students but not in graphics, grad students in graphics but don't know much about your subfields, grad students in graphics and know your subfield, professors in graphics, professors in other fields.
- Take notes of your research and document the important points and decisions you have made. This is not just for your collaborators but is very important for you. It reminds you why you are here right now and where you are going. Always track back to the higher-level points for every small decision you make. Revisit your decisions every once in a while.
- Different people are suited for different research styles. Find your own style - you don't need to mimic me or other people.
- Grad school is not a competition. It's a process of acquiring knowledge and communicating them. You are not ranked when you graduate by the number of papers you published. It's your own journey and it is irrelevant how other people did their own ones.
The important things to ask are:
"How much have you learned during the process?",
"How much have you taught other people?", and
"Have you had fun?".
Of course, finding a job after you graduate is a different aspect.
However, as a CS student graduated from UCSD,
you shouldn't worry about this too much unless you're aiming for something like a faculty position at Stanford.
- Our collaborators may have very different opinions compared to my points above. That's fine. Remember - you are the leader of the project and you should decide how to proceed.
What's my research?
Check out my webpage
for publications and notes.
I work on the interactions between computer graphics, vision, programming systems, and machine learning.
My main research direction is something I call "differentiable visual computing,"
where we go beyond neural networks and backpropagate through computer graphics programs.
In graphics, we explicitly model how the world behaves (often through physics), instead of relying on generic
neural networks to learn everything from scratch. Learning from data by backpropagating through graphics programs
allows us to have more control and a better understanding of
our programs' behavior, enables high-performance, and makes debugging/verification easier.
The applications include: helping self-driving cars to make better decisions, training robots to interact with
the environment using physical information, creating more realistic virtual realities, designing buildings and
rooms to have better lighting, designing 3D physical objects with desired appearance and functionalities,
reconstructing 3D structures of cells from microscope images, and allowing movie artists to produce better film shots.
Differentiating general graphics programs correctly and efficiently is much more complicated than differentiating a convolution layer,
due to the general computation patterns and the mathematics involved. I derive algorithms that
differentiate graphics programs while taking discontinuities into account. I design compilers that
explore trade-offs of derivative computation and generate correct and efficient code. I look at applications
where these differentiable graphics programs can be useful.
Many graphics researchers are trying to replace graphics pipelines with deep learning components. While this is
a cool direction and is in my research scope, one of my main directions is to introduce differentiable
graphics programs to deep learning pipelines. I claim that this is the way that will lead us to
controllable, interpretable, robust, and efficient models. I am also highly interested in non-data-driven settings,
where we try to figure out latent variables solely based on our knowledge of the physical process.
Following are some of my current research thrusts:
How do we backpropagate through the rendering equation
, so that we can infer 3D
information from 2D observations? How do we handle discontinuities, and how do we make the differentiation as
efficient as possible? How do we differentiate rendering with arbitrary geometry and material representations, while
modeling physical phenomena such as occlusion, surface, and volumetric scattering, dispersion, and diffraction? Ultimately, we want to build a differentiable renderer that can generate and differentiate noise-free
million-pixel-images with billions of varied primitives within seconds, while accurately modeling optics. We want to
use the renderer for artificial intelligence agents to inference 3D information, for reconstructing detailed 3D models
for virtual/augmented reality, for designing and fabricating real-world 3D objects with desired optical properties, for
designing imaging systems, and for analyzing biomedical data using inverse optics.
Differentiable image processing: How do we design image processing pipelines that can learn from data,
can process images with tens or hundreds of megapixels, in real-time, on mobile devices? I claim that instead of
stacking more convolution layers and increasing overparametrization, we should generalize the programming model we use
for designing image processing algorithms. Instead of the high arithmetic intensity deep learning layers, we want to
take building blocks from more traditional image processing algorithms, parametrize them, and differentiate through them
to learn the parameters. To achieve this, we need better compilers that can fuse the low arithmetic intensity
computation, and can differentiate array code.
Differentiable physics simulation:
How do we backpropagate through ODE/PDE solvers to make
inference based on the dynamics? This problem is studied in the optimal control
and the sensitivity
(and this is also what inspired
However, differentiation of
discontinuities and boundary
conditions in ODEs/PDEs is not very
well-understood. Furthermore, how do we
efficiently map these computations to modern
hardware? How do we reduce the memory usage when
backpropagating an iterative solver? Answering
these questions will enable us to train robot
controllers orders of magnitude more efficiently, design 3D objects with physical constraints, or even have more elaborated epidemiology models
Domain-specific compilers for differentiable computer graphics: How do we design compilers that take
high-level computer graphics code (rendering, image processing, simulation, geometry processing), and automatically
output high-performance low-level code along with the derivative code? How do we certify the correctness? Just like how deep learning frameworks
democratize machine learning, I want to build an easy-to-use programmable differentiable graphics system that makes building
differentiable graphics pipelines as simple as training an MNIST classifier in PyTorch, while generating reliable code.
Accelerating physically-based rendering: Physically-based rendering is known to be time-consuming, due to its need to compute multi-dimensional integrals using Monte Carlo sampling. How do we make it faster? I believe there are two keys towards the ultimate rendering algorithm: 1) re-using Monte Carlo samples through statistical analysis. 2) replacing heuristics with data-driven components.
Beyond differentiation: Can we derive or automatically find *something* that approximates a function locally, and better inform
our inference algorithms, when comparing to derivatives? For example, Fourier analysis or wavelet analysis gives us
strictly more information than derivatives (differentiation is a linear ramp in the frequency domain). How can we use them in optimization algorithms?
Can we find something that scales better with dimensionality compared to Fourier analysis? Can we develop systems that
help us find these quantities given a program?
related work of mine: Nothing here yet. Hopefully your work will be listed here!