Home / Research

3D Deep Learning and Shape Analysis

Machine learning makes 3D vision great.

Understanding 3D parts of a shape are essential for many perception and robotic tasks. Part-level understanding and part mobility is extremely useful for 3D interactions such as in Virtual and Augmented Reality.
Semantic segmentation is useful to understand the scene and objects contained in it. I believe semantic meaning reflects the concept and knowledge of how human processes the world.
3D capture involves transferring other data into 3D shapes such as facial and pose information. Conversion from 2D to 3D is also a novel way to bring artist's original painting into life.

Large Scale 3D Dataset: ShapeNet

Dataset speaks for itself.

I'm in the ShapeNet team and contributing to the dataset. It is a large-scale 3D synthetic dataset with rich annotations. It drives most of my current 3D deep learning projects. It is also being actively enriched and improved by us in 3D Shape Understanding Lab at UCSD.

Deep Learning Assisted Graphical Rendering

Machine learning makes 3D graphics great.

Traditional Monte-Carlo ray tracing needs too many rays to approximate real light transportation in order to generate well-looking image. Some scenes are even harder to trace the lights. Deep learning can be applied to restore the rendered image.
Reinforcement Learning can be applied on guiding light paths in each step of recursion, enabling smart ray selection as well as improving efficiency of rendering especially in complicated environments.

Visual Perception with Real Sensory Data

Process sensory data on cars and smartphone cameras.

Mobile cameras are pervasive and serve as the main source of visual perception. Cameras and light sensors can be used to sensing and communicate with environmental lightings. I also design LEDs that can interact with cameras.
LIDAR, Radar, and depth cameras are creating various point clouds that can be used for object recognition and scene understanding for autonomous cars. Fusing these sensory data can provide powerful perception ability.

Neural Network Acceleration for Visual Application

Inventing Binary Neural Computation on FPGAs.

BNNs have been studied extensively since they run dramatically faster at lower memory and power consumption, especially on visual applications. We develop robust algorithms to make it work as good as real-valued deep neural networks.
Logical gated arrays are natually designed for bit operations, which is a perfectly fit to BNNs. With hardware-software co-optimization, we can make real time inference fast than ever.

(Previous) Human Touch Interaction

Immersive interaction brings magic into our lives.

RFID enables seamless interaction between human and computer, and we focus on designing integrated RFID tag to sense fine-grained human touch to support Internet of Things (IoT) applications.

(Previous) Mobile Healthcare

Take care of more people with technology.

Wrist-band PPG signals are widely used to monitor heart beat activity. But such signals are corrupted with extensive motion, and we propose algorithms to recover the weak signal to allow seamless monitoring.
Electronic health record is convenient for both doctors and patients. A new tool, called SMART PCM, is created to provide and visualize genomic information at the point of care. Privacy in EHR is also considered.

(Previous) Algorithm Design on Network Graph Topology

Better heuristics on NP-hard graph problems.

Algorithms on large networks can be costly due to graphical nature. We focus on designing optimization based on graph structure to enable flexible routing and spectrum allocation in backbone networks. We also work on fast network restoration after graph links are damaged.
Subgraphs are managed by different companies and providers. It is interesting to see if one can gain more by analyzing network topology and traffic maps based on imperfect information.