摘要
深度确定性策略梯度算法(DDPG)是一种用途广泛的深度强化学习方法,但它往往会受到梯度估计不稳定的影响.最近的一些方法(如近端策略优化算法PPO)只是限制在较低速度下进行策略更新以保持稳定性.在本文中,我们在一个优势演员评论家算法(A2C)架构下对问题进行建模.我们首先分析了A2C中简化解析解的运算,其中策略更新的不稳定性主要归因于两个因素:动作估计的方差和累积奖励的方差.为了解决这个问题,我们提出了一种新的基于最佳基线的近端策略优化算法称为Fast-PPO.具体来说,我们混合了最优基线既考虑了动作估计的优势,又考虑了累积奖励的估计.实验结果表明,我们的方法不仅保证动作估计可以在正确的方向上更快地收敛,而且也保证了在较低的方差下了累积奖励的收敛速度.
Deep deterministic policy gradient(DDPG)is a useful deep reinforcement learning approach,but it tends to suffer from instability of gradient estimates.Recent methods such as PPO just limit the policy update under lower speed to keep stability by chance.In this paper,we model the problem under an advantage actor-critic(A2C)architecture.We first analyze the operation of the simplified analytic solution in A2C,where the instability of the policy update mainly be attribute to two factors:the variance of action estimate and the variance of accumulative rewards.To solve it,we propose a new method of PPO with the optimal baseline called FastPPO.In detail,our hybrid optimal baseline considers both the advantage of action estimate and the estimate of accumulative rew ard.The experimental results demonstrated that our method guarantees that not only the action estimate can be converged faster in the right direction,but also the accumulative reward can be under lower variance.
作者
肖竹
谢宁
陈佳黎
刘保生
姜峰
杨郭镳
XIAO Zhu;XIE Ning;CHEN Jia-li;LIU Bao-sheng;JIANG Feng;YANG Guo-biao(Center for Future Media,School of Computer Science and Engineering University of Electronic Science and Technology of China,Chengdu 611731,China)
出处
《小型微型计算机系统》
CSCD
北大核心
2020年第7期1351-1356,共6页
Journal of Chinese Computer Systems
基金
国家自然科学基金项目(61602088)资助
中央高校基本科研业务费基础研究项目(Y03019023601008011)资助。
关键词
深度强化学习
策略梯度算法
PPO
最优基线
deep reinforcement learning
policy gradient
Proximal Policy Optimization
optimal baseline
