摘要
介绍一种视频多模态情感识别方法。一个视频通常通过文本、声音和视觉图像等多模态信息来表达同一种情感主题,而如何将同一个视频中不同异构数据之间的信息融合并最大程度地利用是目前需要重点攻克的难题。通过互信息最大化的方法,高效融合视频中的文本、声音与视觉图像等多模态异构数据,尽可能多地消除模态之间的差异,最终实现对视频的情感进行识别分析。在公开的MOSEI多模态数据集上进行实验,实验结果显示MAE值达55.4。相比之前的一些模型,本模型效果更优,且实验模型构造不繁琐,为后面相关的研究打下良好的基础。
A method for multimodal sentiment recognition in video is introduced in this paper.A video usually expresses the same sentiment theme through multimodal information such as text,sound,and visual images,andfusingthe information between different modalities and make full use of them is the current key problems that need to be overcome.This paper uses the method of maximizing mutual information to efficiently fuse multimodal heterogeneous data such as text,sound and visual images in videos to eliminate as many differences between modalities as possible,and finally realize the recognition and analysis of video sentiment.Experiments are carried out on the public MOSEI multimodal dataset,and the results show that the MAE value reaches 55.4.Compared with conventional models,the effect of this model is better,and the construction of the experimental model is not cumbersome,which can provide reference for related research.
作者
陈诗汉
马洪江
王婷
何松泽
CHEN Shihan;MA Hongjiang;WANG Ting;HE Songze(College of Computer,Chengdu University of Information Technology,Chengdu 610200,China)
出处
《成都信息工程大学学报》
2022年第6期656-661,共6页
Journal of Chengdu University of Information Technology
基金
四川省科技厅重点研发资助项目(2021YFG0031、2022YFG0375)
四川省科技服务业示范资助项目(2021GFW130)
关键词
多模态融合
视频情感分析
互信息最大化
multimodal fusion
video sentiment analysis
mutual information maximization
