NSF SCALE MoDL: Adaptivity of Deep Neural Networks
Principal Investigator
Yu-Xiang Wang, UCSB (2021 - 2024), UCSD (2024 -)
In collaboration with:
Yingbin Liang (Ohio State University)
Simon Du (University of Washington)
  
Project Summary
Products

Funded by NSF NSF DMS 2134214.

This material is based upon work supported by the National Science Foundation under Grant No. 2134214. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Project Summary

The overarching theme of the project is to systematically expand our understanding of how deep neural networks work and why or when they are better than classical methods through the lens of adaptivity. Specifically, we investigate (1) Local adaptivity: how deep neural networks adapt to varying degrees of smoothness and low-dimensional structures in each local region; (2) Relational adaptivity: the adaptivity of self-attention mechanisms in learning relational structures among different parts of a single data point (sequence, images, graphs); (3) Task adaptivity: the the power of meta-learning in adaptively identifying shared structures among multiple tasks and adapting to unseen downstream tasks with very limited data. These research directions cover some of the most successful neural network architectures and will result in new theoretical insight with profound implications in both the algorithms and applications of deep neural network models.

News

  1. Invited talk on "Neural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large Learning Rate"
    PI Wang spoke at Simons Institute program "Modern Paradigms in Generalization"
  2. Paper "Stable Minima Cannot Overfit in Univariate ReLU Networks" is selected for an Spotlight Award at NeurIPS'24
    Among the top 1.0% of all submissions.
  3. Paper "Neural Collapse Meets Differential Privacy" is selected for an Oral Presentation at ICML 2024
    Among the top 0.3% of all submissions.
  4. Kaiqi Zhang defended his PhD on May 2023
    Congratulations to Kaiqi.
  5. Chong Liu defended his PhD on Jun 2023
    Congratulations to Chong!
  6. Invited talk on "Deep Learning meet Nonparametric Regression"
    PI Wang spoke at UCSB Statistics / GeorgiaTech / USC / MBZUAI SlowDNN workshop / NUS [slides]
  7. Invited talk on "On the Adaptivity of Deep Neural Networks"
    PI Wang spoke at "Theory of Emergent Intelligence" Workshop hosted by the Institute of Advanced Studies at HKUST.

Research Results

  1. Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
    Dan Qiao, Kaiqi Zhang, Esha Singh, Daniel Soudry, Yu-Xiang Wang.
    NeurIPS 2024. [arxiv] (*Spotlight)
  2. Neural Collapse meets Differential Privacy: Curious behaviors of NoisyGD with Near-Perfect Representation Learning
    Chendi Wang, Yuqing Zhu, Weijie Su, Yu-Xiang Wang
    ICML 2024. [arxiv] (*Oral Presentation)
  3. Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
    Zixuan Zhang, Kaiqi Zhang, Minshuo Chen, Yuma Takeda, Mendi Wang, Tuo Zhao, Yu-Xiang Wang.
    NeurIPS 2024 [arxiv]
  4. Global Optimization with Parametric Function Approximation
    Chong Liu, Yu-Xiang Wang.
    ICML 2023. [arxiv]
  5. Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?
    Kaiqi Zhang, Yu-Xiang Wang.
    ICLR 2023. [arxiv]

Research Highlights