Analysis of over-parameterized neural networks has drawn significant attention in recent years. It was shown that such systems behave like convex systems under various settings, such as for two-level neural networks, and when learning is only restricted locally in the so-called neural tangent kernel space around specialized initializations. However, there are no theoretical techniques that can analyze fully trained deep neural networks. In this talk, we show under suitable representations, overparameterized deep neural networks are inherently convex, and when optimized, the system can learn effective features suitable for the underlying learning task under mild conditions. This new analysis is consistent with empirical observations that deep neural networks are capable of learning efficient feature representations. Moreover, for Residual Network (Res-Net) architecture, we construct a non-linear dynamics called neural feature flow to capture the evolution of an over-parameterized DNN trained by Gradient Descent. It is shown that when the neural feature flow process converges, it reaches a global minimal solution under suitable conditions.

Reference: 

[1] Cong Fang, Jason Lee, Pengkun Yang, Tong Zhang. "Modeling from features: a mean-field framework for over-parameterized deep neural networks." Conference on Learning Theory. PMLR, 2021.

[2] Cong Fang, Yihong Gu, Weizhong Zhang, Tong Zhang. "Convex formulation of overparameterized deep neural networks." IEEE Transactions on Information Theory 68.8 (2022): 5340-5352.

Cong Fang is now an Assistant Professor with Peking University, Beijing, China. He received the Ph.D. degree from Peking University in 2019. He was a Post-Doctoral Researcher with Princeton University in 2020 and the University of Pennsylvania in 2021. His research interests include machine learning and optimization.