摘要
联邦学习致力于在保证用户数据隐私安全的同时,通过多用户共建的方式提升机器学习模型的泛化性能.为此,在用户利用本地数据训练模型后,参数服务器需要聚合多个用户的模型参数并使用户基于聚合后参数继续本地训练.指数滑动平均是一种被广泛使用的参数聚合更新方法.然而当用户本地训练速度相差较大时,指数滑动平均方法无法消除由此造成的聚合参数偏差,从而显著影响模型整体训练效率.针对上述问题,本文提出了一种基于权重摘要和更新版本感知的异步联邦学习聚合更新方法,通过合理控制不同训练速度用户提交的参数在聚合参数中所占比例,以及主动更新落后用户使用的聚合参数,从而有效解决本地训练速度差异对聚合参数造成的负面影响.实验结果表明,相较于指数滑动平均策略,本文提出的参数聚合更新方法在MNIST、CIFAR-10数据集上均能显著提升训练效率.
Federated learning enables multiple users to create a generalized machine-learning model in collaboration while ensuring the privacy and security of user data. In federated learning,users train their own models with local data. The server then collects models from users,aggregate them and return the results to users for further training. One of the frequently used aggregation strategies is the exponential moving average strategy. However,this strategy cannot deal with the model bias caused bydifferent training speeds of various users,which may lead to the inefficient overall training speed. To address this problem,this paper proposes a weight-profile and version-aware aggregation strategy. The proposed strategy can alleviate the impact of different local training speeds on aggregated model by controlling the weights of models from users with different training speeds in the aggregated model,and updating models of stragglers with the newest aggregated model actively. Experimental results show that the proposed strategy can obviously improve the overall training speed on datasets of MNIST and CIFAR-10 when they are compared with the exponential moving average strategy.
作者
陈瑞锋
谢在鹏
朱晓瑞
屈志昊
CHEN Rui-feng;XIE Zai-peng;ZHU Xiao-rui;QU Zhi-hao(Collegeof Computer and Information,Hohai University,Nanjing 211100,China)
出处
《小型微型计算机系统》
CSCD
北大核心
2021年第12期2473-2478,共6页
Journal of Chinese Computer Systems
基金
国家重点研发课题项目(2016YFC0402710)资助
国家自然科学基金重点项目(61832005)资助。
关键词
联邦学习
分布式机器学习
聚合更新算法
人工神经网络
federated learning
distributed machine learning
aggregation update algorithm
artificial neural networks
