[Distinguished Lecture] On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Size -

[Distinguished Lecture] On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Size

Time: 2022-04-23 Published By: Wenqiong Li

Speaker(s): Yaxiang Yuan(Chinese Academy of Sciences)

Time: 10:00-11:00 April 25, 2022

Venue: Room 77201, Jingchunyuan 78, BICMR

We investigate the stochastic gradient descent (SGD) method where the step size is given by a banded region instead of a fixed formula. The optimal convergence rate under mild conditions and large initial step size is proved. Our analysis provides comparable theoretical error bounds for SGD associated with a variety of step sizes. In addition, the convergence rates for some existing step size strategies, e.g., triangular policy and cosine-wave, can be revealed by our analytical framework under the boundary constraints. The bandwidth-based step size provides efficient and flexible step size selection in optimization. We also propose a 1/t up-down policy and give several non-monotonic step sizes. Numerical experiments demonstrate the efficiency and significant potential of the bandwidth-based step-size in many applications.

joint work with Xiaoyu Wang.

袁亚湘，中国科学院院士，中科院数学与系统科学研究院研究员。现任全国政协常委、中国科协副主席、国际工业与应用数学联合会主席、发展中国家科学院院士、巴西科学院通讯院士，美国数学学会首届会士、美国工业与应用数学会会士。袁亚湘院士在非线性优化计算方法，信赖域方法、拟牛顿方法、共轭梯度法等领域做出了突出贡献，2014年被国际数学联盟邀请在韩国举办的国际数学家大会上做45分钟报告。袁亚湘院士曾获“中国科学院青年科学家奖”一等奖、首届“冯康科学计算奖”、第三届“中国青年科学家奖”、“国家杰出青年基金”、“中国十大杰出青年”、科技部“国家重点实验室计划先进个人奖”、“北京市科学技术一等奖”、中国科协“全国优秀科技工作者”和“国家自然科学奖”二等奖、中国数学会“陈省身数学奖”以及发展中国家科学院（原第三世界科学院）数学奖、中国工业与应用数学学会“苏步青奖”、“何梁何利科技进步奖”以及美国工业与应用数学学会“杰出贡献奖”。