摘要
图像的自然语言描述(image captioning)是一个融合计算机视觉、自然语言处理和机器学习的跨领域课题。它作为多模态处理的关键技术,近年来取得了显著成果。当前研究大多针对图像生成英文摘要,而对于中文摘要的生成方法研究较少。该文提出了一种基于多模态神经网络的图像中文摘要生成方法。该方法由编码器和解码器组成,编码器基于卷积神经网络,包括单标签视觉特征提取网络和多标签关键词特征预测网络,解码器基于长短时记忆网络,由多模态摘要生成网络构成。在解码过程中,该文针对长短时记忆网络的特点提出了四种多模态摘要生成方法 CNIC-X、CNIC-H、CNIC-C和CNIC-HC。在中文摘要数据集Flickr8k-CN上实验,结果表明该文提出的方法优于现有的中文摘要生成模型。
Image captioning is a cross-domain task which connects computer vision,natural language processing and machine learning.As a key technology of multimodal processing,it has made remarkable progress in the recent years.Research on image caption generation has typically focused on generating a caption in English for an image,but generating Chinese caption is lack of research.In this paper,we propose a method generating Chinese image caption based on multimodal neural network.This method belongs to the family of encoder-decoder.Encoder based on convolutional neural network,consists of single-label visual feature extraction network and multi-label keyword prediction network.Decoder based on long short-term memory,consists of multimodal caption generation network.During the process of decoding,we propose four multimodal caption generation methods：CNIC-X,CNIC-H,CNICC and CNIC-HC.Experimental results on Chinese multimodal dataset Flickr8 k-CN show that the proposed method outperforms state-of-the-art Chinese image captioning methods.
出处
《中文信息学报》
CSCD
北大核心
2017年第6期162-171,共10页
Journal of Chinese Information Processing
基金
国家自然科学基金(61772505)
青海省自然科学基金(2016-ZJ-Y04
2016-ZJ-740)
关键词
图像中文摘要
多模态处理
神经网络
Chinese image captioning
multimodal processing
neural network
