摘要
使用预训练语言模型的微调方法在以文本分类为代表的许多自然语言处理任务中取得了良好的效果,尤其以基于Transformer框架的BERT模型为典型代表。然而,BERT直接使用[CLS]对应的向量作为文本表征,没有从全局和局部考虑文本的特征,从而限制了模型的分类性能。因此,本文提出一种引入池化操作的文本分类模型,使用平均池化、最大池化以及K-MaxPooling等池化方法从BERT输出矩阵中提取文本的表征向量。实验结果表明,与原始的BERT模型相比,本文提出的引入池化操作的文本分类模型具有更好的性能,在实验的所有文本分类任务中,其准确率和F1-Score值均优于BERT模型。
The fine-tuning method using the pre-trained language model has achieved good results in many natural language processing tasks represented by text classification,BERT model based on the Transformer framework as a typical representative especially.However,BERT uses the vector corresponding to [CLS]as the text representation directly,and does not consider the local features and global features of texts,which limits the classification performance of the model.Therefore,this paper proposes a text classification model that introduces a pooling operation,and uses pooling methods such as average pooling,maximum pooling,and K-MaxPooling to extract the representation vector of texts from the output matrix of BERT.The experimental results show that compared with the original BERT model,the text classification model with pooling operation proposed in this paper has better performance.In all text classification tasks in the experiment,its accuracy and F1-Score value are better than BERT model.
作者
张军
邱龙龙
ZHNAG Jun;QIU Long-long(School of Software,East China University of Technology,Nanchang 330013,China;School of Information Engineering,East China University of Technology,Nanchang 330013,China)
出处
《计算机与现代化》
2022年第6期1-7,共7页
Computer and Modernization
基金
国家自然科学基金资助项目(62162002,61662002,61972293,61902189)
江西省自然科学基金资助项目(20212BAB202002)
江苏省自然科学基金资助项目(BK20180821)。
