摘要
为改进Adam算法存在的全局收敛性较差的问题,提出一个带有角系数的AngleAdam算法.该算法利用两次连续梯度间的角度信息自适应控制步长,一定程度改进了Adam算法全局收敛性较差的问题,提升了优化能力.采用在线学习框架,对算法从遗憾界的角度进行收敛性分析,证明了AngleAdam具有次线性的遗憾.基于构造的三个非凸函数和深度神经网络模型,对AngleAdam算法的优化能力进行实验,实验结果表明该算法可得到较好的优化结果.
To improve the poor global convergence of Adam algorithm,an AngleAdam algorithm with angular coefficients was proposed.The algorithm used the angle information between two continuous gradients to adaptively control the step size,which improved the problem of poor global convergence of Adam algorithm to a certain extent,and improved the optimization ability.By using the online learning framework,the convergence of the algorithm was analyzed from the perspective of regret bound,proving that the AngleAdam had sublinear regret.Based on the constructed three non-convex functions and the depth neural network model,the optimization ability of the AngleAdam algorithm was tested.Experimental results show that the algorithm can obtain better optimization results.
作者
姜志侠
宋佳帅
刘宇宁
JIANG Zhixia;SONG Jiashuai;LIU Yuning(School of Mathematics and Statistics,Changchun University of Science and Technology,Changchun 130022,China;CEC GienTech Technology Co.Ltd.,Beijing 100192,China)
出处
《华中科技大学学报（自然科学版）》
EI
CAS
CSCD
北大核心
2023年第5期137-143,共7页
Journal of Huazhong University of Science and Technology(Natural Science Edition)
基金
吉林省自然科学基金资助项目(YDZJ202201ZYTS519)
国家自然科学基金资助项目(11426045).
关键词
机器学习
梯度下降类算法
Adam算法
全局收敛性
遗憾界
角度信息
machine learning
gradient descent algorithm
Adam algorithm
global convergence
regret bound
angle information
