摘要
赛道形状与路面材质变化对自动驾驶赛车的行为决策带来了严峻挑战。为应对道路间的动力学差异,本文提出一种基于持续强化学习(CRL)的高速赛车决策算法。该算法将不同道路看作独立任务。算法的第1训练阶段负责提取描述不同任务上赛车动力学的低维特征,从而计算出任务间的相似性关系。算法的第2训练阶段负责为策略学习过程提供2个持续强化学习约束:其一是权重正则化约束,策略网络中对于旧任务重要的权重将在新任务学习期间被限制更新,其限制力度由任务相似性自适应调节;其二是奖励函数约束,鼓励在新任务学习期间策略的旧任务性能不下降。设计不同任务排序下的赛车实验和持续强化学习评价指标以评估算法性能。实验结果表明,所提算法能在既不存储旧任务数据也不扩展策略网络的条件下获得比基准方法更出色的驾驶性能。
The variety of road shapes and materials presents a serious decision-making challenge for high-speed autonomous racing.To address the issue of dynamics gap between various roads,a decision-making algorithm based on continual reinforcement learning(CRL)is proposed.These roads are considered as different tasks.The first training stage of the algorithm extracts low-dimension task features that can characterize the vehicle dynamics on different roads.These features are used to compute the task similarity.The second training stage of the algorithm provides two CRL constraints for policy learning.One is the weight regularization constraint,which restricts the updates of policy weights that are important for old tasks.This restriction is adaptively regulated by task similarity.The other is the reward constraint,which encourages no performance degradation on old tasks while the policy is learning a new task.Racing experiments with different task sequences and CRL metrics are set to evaluate the algorithm.The results show that the proposed algorithm outperforms baselines without storing old tasks’data or expanding policy network size.
作者
牛京玉
胡瑜
李玮
韩银和
NIU Jingyu;HU Yu;LI Wei;HAN Yinhe(Research Center for Intelligent Computing Systems,Institute of Computing Technology,Chinese Academy of Sciences,Beijing 100190;University of Chinese Academy of Sciences,Beijing 100049)
出处
《高技术通讯》
CAS
北大核心
2024年第1期1-14,共14页
Chinese High Technology Letters
基金
国家自然科学基金(62176250,62003323)
中国科学院计算技术研究所计算机体系结构国家重点实验室创新项目(CARCH5203,CARCH5406)资助。
关键词
强化学习(RL)
持续学习
行为决策
自动驾驶赛车
动力学特征提取
reinforcement learning(RL)
continual learning
decision making
autonomous racing
dynamics feature extraction
