摘要
针对现有基于Seq2Seq框架的语法纠错模型对句子语义信息利用不充分的问题,提出一种基于预训练语言模型与自注意力机制的英语语法纠错模型。通过预训练语言模型与卷积神经网络结构进行编码,获取句子的局部语义信息;通过自注意力机制来提升单词语义信息表征的准确性;采用SRU网络将预测单词分布与注意力得分分布进行融合,并结合束搜索策略生成最终结果;运用集成学习进一步提升语法纠错模型性能。在CoNLL-2014和JFLEG数据集上的实验结果表明,相较于其他基线语法纠错方法,文中提出的方法能够取得更好的纠错效果。
The existing grammatical error correction models based on Seq2Seq framework cannot make full use of semantic information of English sentences.Therefore,this paper proposes an English grammatical error correction model based on pre-training language model and self-attention mechanism.Firstly,the local semantic information of the English sentences is obtained by encoding of the pre-training language model and the convolutional neural network.Secondly,self-attention mechanism is used to improve the accuracy of word semantic information representation.Thirdly,the SRU network is used to fuse the predicted word distribution with the attention score distribution,and combined with the beam search strategy to generate the final result.Finally,ensemble learning is used to further improve the performance of the proposed model.Experiment results on CoNLL-2014 and JFLEG data sets show that the proposed method can achieve better grammatical error correction performance than several baseline models.
作者
郝琛
HAO Chen(School of International Studies,Wenzhou Business College,Wenzhou 325035,Zhejiang Province,China)
出处
《信息技术》
2023年第12期147-155,161,共10页
Information Technology
关键词
预训练模型
自注意力
语法纠错
束搜索
集成学习
pre-training model
self-attention
grammatical error correction
beam search
ensemble learning
