[Objective] This paper conducts sentiment analysis of images and text on social media data, aiming to better understand the public's emotions and opinion tendencies. [Methods] To fully explore the correlation and complementarity between images and text, this paper proposes an image-text sentiment classification model in social media based on multi-layer semantic fusion. There are three sub-models in our study: text-image semantic association model, image-text semantic association model, and multimodal semantic deep association fusion model. We used these sub-models to explore the bidirectional and multi-level semantic associations between images and text. Then, we obtained the final classification results using a weighting strategy on the sentiment classification scores generated by the three sub-models. [Results] We examined our model with real image-text data sets and found it achieved the best performance in all evaluation metrics. The accuracy and F1 values of our model were 1.0% and 1.2% better than those of the optimal baseline model. [Limitations] We only evaluated the model’s performance with one single dataset. More research is needed to examine the robustness and scalability of the model. [Conclusions] In the sentiment classification task, the proposed model could more effectively explore the correlation and complementarity between image and text information on social media.
(Yu Tao, Luo Ke. Commentary Sentiment Classification Model Combining Product Features[J]. Computer Engineering and Applications, 2019,55(16):108-114.)
[2]
Pandeya Y R, Lee J. Deep Learning-Based Late Fusion of Multimodal Information for Emotion Classification of Music Video[J]. Multimedia Tools and Applications, 2020,80(38):1-19.
[3]
Jia Y X, Chen Z Y, Yu S W. Reader Emotion Classification of News Headlines[C]// Proceedings of 2009 International Conference on Natural Language Processing and Knowledge Engineering. IEEE, 2009. DOI: 10.1109/NLPKE.2009.5313762.
[4]
Winster S G, Kumar M N. Automatic Classification of Emotions in News Articles Through Ensemble Decision Tree Classification Techniques[J]. Journal of Ambient Intelligence and Humanized Computing, 2020. DOI: 10.1007/s12652-020-02373-5.
[5]
Turney P D. Thumbs up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews[C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002: 417-424.
[6]
Nasukawa T, Yi J. Sentiment Analysis: Capturing Favorability Using Natural Language Processing[C]// Proceedings of the 2nd International Conference on Knowledge Capture. 2003: 70-77.
[7]
Mullen T, Collier N. Sentiment Analysis Using Support Vector Machines with Diverse Information Sources[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004: 412-418.
[8]
Xu F, Pan Z, Xia R. E-Commerce Product Review Sentiment Classification Based on a Naïve Bayes Continuous Learning Framework[J]. Information Processing & Management, 2020,57(5):102221.
doi: 10.1016/j.ipm.2020.102221
[9]
Xie X, Ge S L, Hu F P, et al. An Improved Algorithm for Sentiment Analysis Based on Maximum Entropy[J]. Soft Computing, 2019,23(2):599-611.
doi: 10.1007/s00500-017-2904-0
[10]
Maas A, Daly R E, Pham P T, et al. Learning Word Vectors for Sentiment Analysis[C]// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011: 142-150.
(Zhang Pu, Li Xiao, Liu Chang. Aspect Level Sentiment Classification Based on Sentiment Words and Machine Learning[J]. Computer Engineering and Design, 2020,41(1):128-133.)
[12]
Chen J, Yan S, Wong K C. Verbal Aggression Detection on Twitter Comments: Convolutional Neural Network for Short-Text Sentiment Analysis[J]. Neural Computing and Applications, 2020,32(15):10809-10818.
doi: 10.1007/s00521-018-3442-0
[13]
Long F, Zhou K, Ou W H. Sentiment Analysis of Text Based on Bidirectional LSTM with Multi-Head Attention[J]. IEEE Access, 2019,7:141960-141969.
doi: 10.1109/ACCESS.2019.2942614
[14]
Wang M, Ning Z H, Li T, et al. Information Geometry Enhanced Fuzzy Deep Belief Networks for Sentiment Classification[J]. International Journal of Machine Learning and Cybernetics, 2019,10(11):3031-3042.
doi: 10.1007/s13042-018-00920-3
[15]
Li M G, Li W R, Wang F, et al. Applying BERT to Analyze Investor Sentiment in Stock Market[J]. Neural Computing and Applications, 2020. DOI: 10.1007/s00521-020-05411-7.
[16]
Li B, Feng S H, Xiong W H, et al. Scaring or Pleasing: Exploit Emotional Impact of an Image[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 1365-1366.
[17]
Vonikakis V, Winkler S. Emotion-Based Sequence of Family Photos[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 1371-1372.
[18]
Borth D, Ji R R, Chen T, et al. Large-Scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs[C]// Proceedings of the 21st ACM International Conference on Multimedia. 2013: 223-232.
[19]
Xu C, Cetintas S, Lee K C, et al. Visual Sentiment Prediction with Deep Convolutional Neural Networks [OL]. arXiv Preprint, arXiv:1411.5731.
[20]
Song K K, Yao T, Ling Q, et al. Boosting Image Sentiment Analysis with Visual Attention[J]. Neurocomputing, 2018,312:218-228.
doi: 10.1016/j.neucom.2018.05.104
[21]
Rao T R, Li X X, Zhang H M, et al. Multi-Level Region-Based Convolutional Neural Network for Image Emotion Classification[J]. Neurocomputing, 2019,333:429-439.
doi: 10.1016/j.neucom.2018.12.053
(Fan Tao, Wu Peng, Cao Qi. The Research of Sentiment Recognition of Online Users Based on DNNs Multimodal Fusion[J]. Journal of Information Resources Management, 2020,10(1):39-48.)
[23]
Poria S, Chaturvedi I, Cambria E, et al. Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis[C]// Proceedings of 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016: 439-448.
[24]
Cao D L, Ji R R, Lin D Z, et al. A Cross-Media Public Sentiment Analysis System for Microblog[J]. Multimedia Systems, 2016,22(4):479-486.
doi: 10.1007/s00530-014-0407-8
(Miao Yuqing, Wang Junhong, Liu Tonglai, et al. Joint Visual-Textual Approach for Microblog Sentiment Analysis[J]. Computer Engineering and Design, 2019,40(4):1099-1105.)
(Ling Haibin, Miao Yuqing, Zhang Wanzhen, et al. Multimedia Sentiment Analysis on Microblog Based on Multi-Feature Fusion[J]. Application Research of Computers, 2020,37(7):1935-1939, 1951.)
[27]
Zhao Z Y, Zhu H Y, Xue Z H, et al. An Image-Text Consistency Driven Multimodal Sentiment Analysis Approach for Social Media[J]. Information Processing & Management, 2019,56(6):102097.
doi: 10.1016/j.ipm.2019.102097
[28]
Truong Q T, Lauw H W. VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019: 305-312.
[29]
Huang F R, Zhang X M, Zhao Z H, et al. Image-Text Sentiment Analysis via Deep Multimodal Attentive Fusion[J]. Knowledge-Based Systems, 2019,167:26-37.
doi: 10.1016/j.knosys.2019.01.019
[30]
You Q Z, Luo J B, Jin H L, et al. Cross-Modality Consistent Regression for Joint Visual-Textual Sentiment Analysis of Social Multimedia[C]// Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 2016: 13-22.
[31]
You Q Z, Jin H L, Luo J B. Visual Sentiment Analysis by Attending on Local Image Regions[C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017: 231-237.
[32]
You Q Z, Cao L L, Jin H L, et al. Robust Visual-Textual Sentiment Analysis: When Attention Meets Tree-Structured Recursive Neural Networks[C]// Proceedings of the 24th ACM International Conference on Multimedia. 2016: 1008-1017.
[33]
Zadeh A, Chen M H, Poria S, et al. Tensor Fusion Network for Multimodal Sentiment Analysis[OL]. arXiv Preprint, arXiv:1707.07250.
[34]
Ramos J. Using TF-IDF to Determine Word Relevance in Document Queries[C]// Proceedings of the 1st International Conference on Machine Learning. 2003,242:133-142.
[35]
Ojala T, Pietikainen M, Maenpaa T. Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002,24(7):971-987.
doi: 10.1109/TPAMI.2002.1017623
[36]
Lowe D G. Distinctive Image Features from Scale-Invariant Keypoints[J]. International Journal of Computer Vision, 2004,60(2):91-110.
doi: 10.1023/B:VISI.0000029664.99615.94
[37]
Dalal N, Triggs B. Histograms of Oriented Gradients for Human Detection[C]// Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). IEEE, 2005: 886-893.
[38]
Bay H, Tuytelaars T, van Gool L. SURF: Speeded up Robust Features[C]// Proceedings of the 9th European Conference on Computer Vision. 2006: 404-417.
[39]
Baltrušaitis T, Ahuja C, Morency L P. Multimodal Machine Learning: A Survey and Taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018,41(2):423-443.
doi: 10.1109/TPAMI.2018.2798607