Tongzhou Mu 木同舟
 |
|
 |
I am looking for full-time jobs in industry. Please feel free to contact me!
About me
I am a Ph.D. candidate at the University of California San Diego, and I am fortunate to be advised by Prof. Hao Su. Prior to starting as a Ph.D. student, I earned my B.Eng. and M.Sc. degrees in computer science from Zhejiang University, and UC San Diego, respectively. I previously interned at 1X Technologies, NVIDIA Robotics Lab, Amazon Alexa AI, Wormpex AI, Intel AI, and Microsoft Research Asia.
I study embodied AI from a data-centric perspective. My previous research can be summarized as “modeling the world for AI agents” (see this figure for a brief overview). More specifically, my focus areas include Foundation Models for Robotics, Scaling Up Robotic Data Collection, as well as Reinforcement Learning and Imitation Learning.
I am the major developer of the 1st version of ManiSkill benchmark, and the lead organizer of the ManiSkill Challenge 2021 and the Generalizable Policy Learning in the Physical World Workshop. I also actively contribute to the development of the subsequent generations of ManiSkill, e.g., ManiSkill 2 and ManiSkill 3.
Research interests
My research interests include:
Foundation Model for Robotics
Task Planning with Multimodal Large Language Models (LLMs)
Grounding Internet-Scale Knowledge in the Physical World
Scaling Up Robotic Data Collection
Interfacing Human Operators and Robots using Foundation Models
Creating Tasks, Assets, and Demonstrations in Simulation
Reinforcement Learning / Imitation Learning
Learning from Demonstrations (Offline and Online)
Motion Synthesis with Generative Models (especially Diffusion Models)
Publications & Preprints
Papers sorted by years. Representative papers are highlighted.
* indicates equal contribution.
2024
 |
Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model |
 |
When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning? |
 |
ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI |
 |
Unleashing the Creative Mind: Language Model As Hierarchical Policy For Improved Exploration on Challenging Problem Solving |
 |
AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent |
 |
DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks |
2023
 |
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline |
 |
Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization |
 |
Boosting Reinforcement Learning and Planning with Demonstrations: A Survey |
 |
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills |
 |
Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation |
2022
 |
Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning |
2021
 |
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations |
 |
Arena: A Scalable and Configurable Benchmark for Policy Learning |
2020
 |
Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals |
 |
State Alignment-based Imitation Learning |
Before 2019
 |
Transfer Value or Policy? A Value-centric Framework Towards Transferrable Continuous Reinforcement Learning |
 |
Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization |
 |
Adaptive Variance Reducing for Stochastic Gradient Descent |
Talks
Modeling the Physical World for Data-Centric Robotics
Invited Talk at Stanford Vision and Learning Lab, Jul 2024
Modeling the Physical World for Embodied AI: Environments and Algorithms
Invited Talk at Microsoft Research Asia, Apr 2023
Learning Neural Structrued Policy via Policy Refactorization
Ph.D. Thesis Proposal at UC San Diego, Mar 2023
Boosting Reinforcement Learning and Planning with Demonstrations
Ph.D. Research Exam at UC San Diego, Jan 2023
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
Pre-training Robot Learning Workshop at CoRL 2022, Dec 2022
Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Invited Talk at UC Berkeley Robot Learning Lab, Nov 2021
Invited Talk at Stanford Vision and Learning Lab, Dec 2021
Task-driven Entity Abstraction from Visual Observations
Invited Talk at Qualcomm AI Lab, Mar 2020
Professional Services
Workshop Organizer: ICLR 2022 Workshop “Generalizable Policy Learning in the Physical World” (Lead organizer)
Tutorial Organizer: CVPR 2022 Tutorial “Building and Working in Environments for Embodied AI”
Challenge Organizer:
SAPIEN ManiSkill Challenge 2021 (Lead organizer)
Conference Reviewer:
Machine Learning: ICLR, NeurIPS, ICML, AAAI, AISTATS
Robotics: CoRL, ICRA, IROS
Computer Vision: ICCV, CVPR
Journal Reviewer: RA-L, TMLR
Teaching
Co-Instructor: CSE 276F Machine Learning for Robotics at UC San Diego, Spring 2024
Guest Lecturer: CSE 291-A00 Machine Learning for Robotics at UC San Diego, Winter 2023
Consultant Volunteer: CSE 291-J00 Deep Learning Lab (Computer Vision) at UC San Diego, Fall 2020
Teaching Assistant: CSE 152 Introduction to Computer Vision at UC San Diego, Fall 2018
Awards
ACM-ICPC (International Collegiate Programming Contest) Asia Regional Contest Gold Medal, 2015
China Computer Federation Elite Collegiate Award (top 108 in China), 2016
Award of Excellence for Stars of Tomorrow Internship Program, Microsoft Research Asia, 2017
Misc
I love powerlifting, tennis, golfing, and surfing in my free time.