Please wait a minute...
Advanced Search
数据分析与知识发现  2021, Vol. 5 Issue (6): 103-114     https://doi.org/10.11925/infotech.2096-3467.2020.1159
  研究论文 本期目录 | 过刊浏览 | 高级检索 |
基于多层语义融合的图文信息情感分类研究*
谢豪,毛进(),李纲
武汉大学信息资源研究中心 武汉 430072
Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion
Xie Hao,Mao Jin(),Li Gang
Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China
全文: PDF (3270 KB)   HTML ( 22
输出: BibTeX | EndNote (RIS)      
摘要 

【目的】 对海量不同模态的社交媒体数据进行有效的情感分析,更好地了解公众的情感和意见倾向。【方法】 为充分挖掘图文之间的关联性和互补性,提出一种基于多层语义融合的社交媒体图文信息情感分类模型,首先通过文本-图像语义关联模型、图像-文本语义关联模型、多模态语义深度关联融合模型三个子模型挖掘图文之间的双向多层次语义关联,进而使用加权策略对三个子模型的情感分类得分进行决策级融合得到最终情感分类结果。【结果】 在真实图文数据集上的实验结果表明,与最优基线模型相比,所提模型在各项评估指标均能达到最优,其中准确率提高了1.0百分点,F1值提高了1.2百分点。【局限】 实验仅在一个数据集上进行,没有对模型的鲁棒性和可扩展性做进一步验证。【结论】 所提模型在情感分类任务上能够更加充分地挖掘社交媒体图文信息之间的关联性和互补性。

服务
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章
谢豪
毛进
李纲
关键词 图文融合注意力机制多模态情感分类社交媒体    
Abstract

[Objective] This paper conducts sentiment analysis of images and text on social media data, aiming to better understand the public's emotions and opinion tendencies. [Methods] To fully explore the correlation and complementarity between images and text, this paper proposes an image-text sentiment classification model in social media based on multi-layer semantic fusion. There are three sub-models in our study: text-image semantic association model, image-text semantic association model, and multimodal semantic deep association fusion model. We used these sub-models to explore the bidirectional and multi-level semantic associations between images and text. Then, we obtained the final classification results using a weighting strategy on the sentiment classification scores generated by the three sub-models. [Results] We examined our model with real image-text data sets and found it achieved the best performance in all evaluation metrics. The accuracy and F1 values of our model were 1.0% and 1.2% better than those of the optimal baseline model. [Limitations] We only evaluated the model’s performance with one single dataset. More research is needed to examine the robustness and scalability of the model. [Conclusions] In the sentiment classification task, the proposed model could more effectively explore the correlation and complementarity between image and text information on social media.

Key wordsImage-Text Fusion    Attention Mechanism    Multi-Modality    Sentiment Classification    Social Media
收稿日期: 2020-11-24      出版日期: 2021-03-10
ZTFLH:  G350  
基金资助:*国家自然科学基金重大项目(71790612);国家自然科学基金创新研究群体项目(71921002)
通讯作者: 毛进     E-mail: danveno@163.com
引用本文:   
谢豪,毛进,李纲. 基于多层语义融合的图文信息情感分类研究*[J]. 数据分析与知识发现, 2021, 5(6): 103-114.
Xie Hao,Mao Jin,Li Gang. Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion. Data Analysis and Knowledge Discovery, 2021, 5(6): 103-114.
链接本文:  
https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/10.11925/infotech.2096-3467.2020.1159      或      https://manu44.magtech.com.cn/Jwk_infotech_wk3/CN/Y2021/V5/I6/103
Fig.1  图文对数据示例
Fig.2  基于多层语义融合的图文情感分类整体框架
Fig.3  文本-图像语义关联模型框架
Fig.4  图像-文本语义关联模型框架
Fig.5  多模态语义深度关联融合模型框架
模态 参数名称 参数值
文本 GloVe维度 100
文本最大长度
LSTM神经元
50
1 024
图片 图像尺寸 224×224
ResNet输出层 conv4_block6_out
其他 Dropout
优化函数
学习率
批量规模
损失函数
0.5
RMSprop
1×10-5
20
binary_crossentropy
Table 1  模型重要参数设置
方法 算法 Accuracy Recall Precision F1
基准方法 STM 0.768 0.780 0.749 0.764
SIM 0.810 0.772 0.823 0.797
Early Fusion
Late Fusion
DMAF
0.840
0.830
0.856
0.813
0.817
0.801
0.848
0.828
0.889
0.831
0.823
0.843
本文方法 TISAM 0.812 0.763 0.833 0.797
ITSAM 0.832 0.755 0.880 0.813
MSDAFM
MSDAFM-L
0.852
0.866
0.817
0.817
0.868
0.895
0.843
0.855
Table 2  算法性能对比
Fig.6  不同αβ值时模型准确率
Fig.7  不同αβ值时模型F1值
[1] 喻涛, 罗可. 结合产品特征的评论情感分类模型[J]. 计算机工程与应用, 2019,55(16):108-114.
[1] (Yu Tao, Luo Ke. Commentary Sentiment Classification Model Combining Product Features[J]. Computer Engineering and Applications, 2019,55(16):108-114.)
[2] Pandeya Y R, Lee J. Deep Learning-Based Late Fusion of Multimodal Information for Emotion Classification of Music Video[J]. Multimedia Tools and Applications, 2020,80(38):1-19.
[3] Jia Y X, Chen Z Y, Yu S W. Reader Emotion Classification of News Headlines[C]// Proceedings of 2009 International Conference on Natural Language Processing and Knowledge Engineering. IEEE, 2009. DOI: 10.1109/NLPKE.2009.5313762.
[4] Winster S G, Kumar M N. Automatic Classification of Emotions in News Articles Through Ensemble Decision Tree Classification Techniques[J]. Journal of Ambient Intelligence and Humanized Computing, 2020. DOI: 10.1007/s12652-020-02373-5.
[5] Turney P D. Thumbs up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews[C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002: 417-424.
[6] Nasukawa T, Yi J. Sentiment Analysis: Capturing Favorability Using Natural Language Processing[C]// Proceedings of the 2nd International Conference on Knowledge Capture. 2003: 70-77.
[7] Mullen T, Collier N. Sentiment Analysis Using Support Vector Machines with Diverse Information Sources[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 412-418.
[8] Xu F, Pan Z, Xia R. E-Commerce Product Review Sentiment Classification Based on a Naïve Bayes Continuous Learning Framework[J]. Information Processing & Management, 2020,57(5):102221.
doi: 10.1016/j.ipm.2020.102221
[9] Xie X, Ge S L, Hu F P, et al. An Improved Algorithm for Sentiment Analysis Based on Maximum Entropy[J]. Soft Computing, 2019,23(2):599-611.
doi: 10.1007/s00500-017-2904-0
[10] Maas A, Daly R E, Pham P T, et al. Learning Word Vectors for Sentiment Analysis[C]// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011: 142-150.
[11] 张璞, 李逍, 刘畅. 基于情感词汇与机器学习的方面级情感分类[J]. 计算机工程与设计, 2020,41(1):128-133.
[11] (Zhang Pu, Li Xiao, Liu Chang. Aspect Level Sentiment Classification Based on Sentiment Words and Machine Learning[J]. Computer Engineering and Design, 2020,41(1):128-133.)
[12] Chen J, Yan S, Wong K C. Verbal Aggression Detection on Twitter Comments: Convolutional Neural Network for Short-Text Sentiment Analysis[J]. Neural Computing and Applications, 2020,32(15):10809-10818.
doi: 10.1007/s00521-018-3442-0
[13] Long F, Zhou K, Ou W H. Sentiment Analysis of Text Based on Bidirectional LSTM with Multi-Head Attention[J]. IEEE Access, 2019,7:141960-141969.
doi: 10.1109/ACCESS.2019.2942614
[14] Wang M, Ning Z H, Li T, et al. Information Geometry Enhanced Fuzzy Deep Belief Networks for Sentiment Classification[J]. International Journal of Machine Learning and Cybernetics, 2019,10(11):3031-3042.
doi: 10.1007/s13042-018-00920-3
[15] Li M G, Li W R, Wang F, et al. Applying BERT to Analyze Investor Sentiment in Stock Market[J]. Neural Computing and Applications, 2020. DOI: 10.1007/s00521-020-05411-7.
[16] Li B, Feng S H, Xiong W H, et al. Scaring or Pleasing: Exploit Emotional Impact of an Image[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 1365-1366.
[17] Vonikakis V, Winkler S. Emotion-Based Sequence of Family Photos[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 1371-1372.
[18] Borth D, Ji R R, Chen T, et al. Large-Scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs[C]// Proceedings of the 21st ACM International Conference on Multimedia. 2013: 223-232.
[19] Xu C, Cetintas S, Lee K C, et al. Visual Sentiment Prediction with Deep Convolutional Neural Networks [OL]. arXiv Preprint, arXiv:1411.5731.
[20] Song K K, Yao T, Ling Q, et al. Boosting Image Sentiment Analysis with Visual Attention[J]. Neurocomputing, 2018,312:218-228.
doi: 10.1016/j.neucom.2018.05.104
[21] Rao T R, Li X X, Zhang H M, et al. Multi-Level Region-Based Convolutional Neural Network for Image Emotion Classification[J]. Neurocomputing, 2019,333:429-439.
doi: 10.1016/j.neucom.2018.12.053
[22] 范涛, 吴鹏, 曹琪. 基于深度学习的多模态融合网民情感识别研究[J]. 信息资源管理学报, 2020,10(1):39-48.
[22] (Fan Tao, Wu Peng, Cao Qi. The Research of Sentiment Recognition of Online Users Based on DNNs Multimodal Fusion[J]. Journal of Information Resources Management, 2020,10(1):39-48.)
[23] Poria S, Chaturvedi I, Cambria E, et al. Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis[C]// Proceedings of 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016: 439-448.
[24] Cao D L, Ji R R, Lin D Z, et al. A Cross-Media Public Sentiment Analysis System for Microblog[J]. Multimedia Systems, 2016,22(4):479-486.
doi: 10.1007/s00530-014-0407-8
[25] 缪裕青, 汪俊宏, 刘同来, 等. 图文融合的微博情感分析方法[J]. 计算机工程与设计, 2019,40(4):1099-1105.
[25] (Miao Yuqing, Wang Junhong, Liu Tonglai, et al. Joint Visual-Textual Approach for Microblog Sentiment Analysis[J]. Computer Engineering and Design, 2019,40(4):1099-1105.)
[26] 凌海彬, 缪裕青, 张万桢, 等. 多特征融合的图文微博情感分析[J]. 计算机应用研究, 2020,37(7):1935-1939, 1951.
[26] (Ling Haibin, Miao Yuqing, Zhang Wanzhen, et al. Multimedia Sentiment Analysis on Microblog Based on Multi-Feature Fusion[J]. Application Research of Computers, 2020,37(7):1935-1939, 1951.)
[27] Zhao Z Y, Zhu H Y, Xue Z H, et al. An Image-Text Consistency Driven Multimodal Sentiment Analysis Approach for Social Media[J]. Information Processing & Management, 2019,56(6):102097.
doi: 10.1016/j.ipm.2019.102097
[28] Truong Q T, Lauw H W. VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019: 305-312.
[29] Huang F R, Zhang X M, Zhao Z H, et al. Image-Text Sentiment Analysis via Deep Multimodal Attentive Fusion[J]. Knowledge-Based Systems, 2019,167:26-37.
doi: 10.1016/j.knosys.2019.01.019
[30] You Q Z, Luo J B, Jin H L, et al. Cross-Modality Consistent Regression for Joint Visual-Textual Sentiment Analysis of Social Multimedia[C]// Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 2016: 13-22.
[31] You Q Z, Jin H L, Luo J B. Visual Sentiment Analysis by Attending on Local Image Regions[C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017: 231-237.
[32] You Q Z, Cao L L, Jin H L, et al. Robust Visual-Textual Sentiment Analysis: When Attention Meets Tree-Structured Recursive Neural Networks[C]// Proceedings of the 24th ACM International Conference on Multimedia. 2016: 1008-1017.
[33] Zadeh A, Chen M H, Poria S, et al. Tensor Fusion Network for Multimodal Sentiment Analysis[OL]. arXiv Preprint, arXiv:1707.07250.
[34] Ramos J. Using TF-IDF to Determine Word Relevance in Document Queries[C]// Proceedings of the 1st International Conference on Machine Learning. 2003,242:133-142.
[35] Ojala T, Pietikainen M, Maenpaa T. Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002,24(7):971-987.
doi: 10.1109/TPAMI.2002.1017623
[36] Lowe D G. Distinctive Image Features from Scale-Invariant Keypoints[J]. International Journal of Computer Vision, 2004,60(2):91-110.
doi: 10.1023/B:VISI.0000029664.99615.94
[37] Dalal N, Triggs B. Histograms of Oriented Gradients for Human Detection[C]// Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). IEEE, 2005: 886-893.
[38] Bay H, Tuytelaars T, van Gool L. SURF: Speeded up Robust Features[C]// Proceedings of the 9th European Conference on Computer Vision. 2006: 404-417.
[39] Baltrušaitis T, Ahuja C, Morency L P. Multimodal Machine Learning: A Survey and Taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018,41(2):423-443.
doi: 10.1109/TPAMI.2018.2798607
[1] 范涛,王昊,吴鹏. 基于图卷积神经网络和依存句法分析的网民负面情感分析研究*[J]. 数据分析与知识发现, 2021, 5(9): 97-106.
[2] 杨晗迅, 周德群, 马静, 罗永聪. 基于不确定性损失函数和任务层级注意力机制的多任务谣言检测研究*[J]. 数据分析与知识发现, 2021, 5(7): 101-110.
[3] 尹鹏博,潘伟民,张海军,陈德刚. 基于BERT-BiGA模型的标题党新闻识别研究*[J]. 数据分析与知识发现, 2021, 5(6): 126-134.
[4] 马莹雪,赵吉昌. 自然灾害期间微博平台的舆情特征及演变*——以台风和暴雨数据为例[J]. 数据分析与知识发现, 2021, 5(6): 66-79.
[5] 余本功,朱晓洁,张子薇. 基于多层次特征提取的胶囊网络文本分类研究*[J]. 数据分析与知识发现, 2021, 5(6): 93-102.
[6] 韩普,张展鹏,张明淘,顾亮. 基于多特征融合的中文疾病名称归一化研究*[J]. 数据分析与知识发现, 2021, 5(5): 83-94.
[7] 张国标,李洁. 融合多模态内容语义一致性的社交媒体虚假新闻检测*[J]. 数据分析与知识发现, 2021, 5(5): 21-29.
[8] 段建勇,魏晓鹏,王昊. 基于多角度共同匹配的多项选择机器阅读理解模型 *[J]. 数据分析与知识发现, 2021, 5(4): 134-141.
[9] 王雨竹,谢珺,陈波,续欣莹. 基于跨模态上下文感知注意力的多模态情感分析 *[J]. 数据分析与知识发现, 2021, 5(4): 49-59.
[10] 李菲菲,吴璠,王中卿. 基于生成式对抗网络和评论专业类型的情感分类研究 *[J]. 数据分析与知识发现, 2021, 5(4): 72-79.
[11] 蒋翠清,王香香,王钊. 基于消费者关注度的汽车销量预测方法研究*[J]. 数据分析与知识发现, 2021, 5(1): 128-139.
[12] 尹浩然,曹金璇,曹鲁喆,王国栋. 扩充语义维度的BiGRU-AM突发事件要素识别研究*[J]. 数据分析与知识发现, 2020, 4(9): 91-99.
[13] 黄露,周恩国,李岱峰. 融合特定任务信息注意力机制的文本表示学习模型*[J]. 数据分析与知识发现, 2020, 4(9): 111-122.
[14] 刘倩, 李晨亮. 基于社交媒体的话题演变研究综述*[J]. 数据分析与知识发现, 2020, 4(8): 1-14.
[15] 李纲, 管为栋, 马亚雪, 毛进. 学术论文的社交媒体可见性预测研究*[J]. 数据分析与知识发现, 2020, 4(8): 63-74.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
版权所有 © 2015 《数据分析与知识发现》编辑部
地址:北京市海淀区中关村北四环西路33号 邮编:100190
电话/传真:(010)82626611-6626,82624938
E-mail:jishu@mail.las.ac.cn