摘要
近年来,随着大数据时代进入人类的生活之后,人们的生活中出现很多无法识别的文本、语义等其他数据,这些数据的量十分庞大,语义也错综复杂,这使得分类任务更加困难。如何让计算机对这些信息进行准确的分类,已成为当前研究的重要任务。在此过程中,中文新闻文本分类成为这个领域的一个分支,这对国家舆论的控制、用户日常行为了解、用户未来言行的预判都有着至关重要的作用。针对新闻文本分类模型参数量多和训练时间过长的不足,在最大限度保留模型性能的情况下压缩训练时间,力求二者折中,故提出基于BERT-CNN的知识蒸馏。根据模型压缩的技术特点,将BERT作为教师模型,CNN作为学生模型,先将BERT进行预训练后再让学生模型泛化教师模型的能力。实验结果表明,在模型性能损失约2.09%的情况下,模型参数量压缩约为原来的1/82,且时间缩短约为原来的1/670。
In recent years,after the era of big data has entered human life,many unrecognizable text,semantic and other data have appeared in people’s lives,which are very large in volume and intricate in semantics,which makes the classification task more difficult.How to make computers classify this information accurately has become an important task of current research.In this process,Chinese news text classification has become a branch in this field,which has a crucial role in the control of national public opinion,the understanding of users’ daily behavior,and the prediction of users’ future speech and behavior.In view of the shortage of news text classification models with large number of parameters and long training time,the BERT-CNN based knowledge distillation is proposed to compress the training time while maximizing the model performance and striving for a compromise between the two.According to the technical characteristics of model compression,BERT is used as the teacher model and CNN is used as the student model,and BERT is pre-trained first before allowing the student model to generalize the capability of the teacher model.The experimental results show that the model parametric number compression is about 1/82 and the time reduction is about 1/670 with the model performance loss of about 2.09%.
作者
叶榕
邵剑飞
张小为
邵建龙
Ye Rong;Shao Jianfei;Zhang Xiaowei;Shao Jianlong(School of Information Engineering and Automation,Kunming University of Science and Technology,Kunming 650500,China)
出处
《电子技术应用》
2023年第1期8-13,共6页
Application of Electronic Technique
基金
国家自然科学基金项目(61732005)。
