摘要
图卷积网络近年来受到大量关注,同时自注意机制作为Transformer结构及众多预训练模型的核心之一也得到广泛运用。该文从原理上分析发现,自注意机制可视为图卷积网络的一种泛化形式,其以所有输入样本为节点,构建有向全连接图进行卷积,且节点间连边权重可学。在多个文本分类数据集上的对比实验一致显示,使用自注意机制的模型较使用图卷积网络的对照模型分类效果更佳,甚至超过了目前图卷积网络用于文本分类任务的最先进水平,并且随着数据规模的增大,两者分类效果的差距也随之扩大。这些证据表明,自注意力机制更具表达能力,在文本分类任务上能够相对图卷积网络带来分类效果的提升。
Graph Convolutional Networks has drawn much attention recently,and the self-attention mechanism has been widely applied as the core of the Transformer and many pre-trained models.We disclose that the self-attention mechanism can be seen as a generalization of Graph Convolutional Networks,in that it takes all input samples as nodes and then constructs a directed fully connected graph with learnable edge weights for convolution.Experiments show that the self-attention mechanism achieves better text classification accuracy than many state-of-the-art Graph Convolutional Networks.Meanwhile,the performance gap of classification widens as the data size increases.These show that the self-attention mechanism is more expressive,and may surpass Graph Convolutional Networks with potential performance improvements on the task of text classification.
作者
蒋浩泉
张儒清
郭嘉丰
范意兴
程学旗
JIANG Haoquan;ZHANG Ruqing;GUO Jiafeng;FAN Yixing;CHENG Xueqi(Key Laboratory of Network Data Science and Technology,Institute of Computing Technology,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100049,China)
出处
《中文信息学报》
CSCD
北大核心
2021年第12期84-93,共10页
Journal of Chinese Information Processing
基金
北京智源人工智能研究院项目(BAAI2019ZD0306)
国家自然科学基金(62006218,61902381,61773362,61872338)
中国科学院青年创新促进项目(20144310,2016102,2021100)
国家重点研发计划(2016QY02D0405)
联想-中科院联合实验室青年科学家项目
王宽诚教育基金会项目
重庆市基础科学与前沿技术研究专项项目(重点)(cstc2017jcjyBX0059)
关键词
图卷积网络
自注意机制
文本分类
graph convolutional networks
self-attention mechanism
text classification
