Research on Recognition Model and Recognition Effect of Network Public Opinion Visual Emotion under Multi-Dimensional Attention Mechanism
Wang Xiwei1,2,3,Wang Qiuyue4(),Cai Hongtian5
1School of Business and Management, Jilin University, Changchun 130022, China 2Research Center for Big Data Management, Jilin University, Changchun 130022, China 3Cyberspace Governance Research Center, Jilin University, Changchun 130022, China 4School of Economics, Jilin University, Changchun 130022, China 5School of Software, Jilin University, Changchun 130022, China
[Objective] To fill the current deficiency in visual emotional analysis research, a ResNet34-based improved emotion analysis model was constructed to analyze and improve the accuracy of image emotion classification. [Methods] Firstly, a visual emotion recognition model was established based on the ResNet34. Then, by integrating the CBAM module and Non-Local module, emotion features were learned and represented. Finally, the above model was used to classify and recognize emotional features, and compared with VGG16 and ResNet50 models. [Results] The recognition effect of the constructed model was verified through experiments, and the research results showed that the accuracy, precision, recall, and F1 score of the model reached 84.42%, 84.10%, 83.70%, and 83.80% respectively. Compared with the baseline models of the VGG16 and ResNet50, the accuracy of the proposed model was improved by 4.17% and 3.44%, and the F1 score was improved by 4.20% and 3.30%. [Limitations] The scale of the test dataset is relatively small, the effectiveness of annotation was not calculated using metrics such as the Pearson correlation coefficient, and a comparison was not made with visual-based emotion classification algorithms. [Conclusions] From the perspective of visual emotional analysis, optimizing the emotion recognition model supplements the analysis mode of emotional computation, providing support for the extraction and analysis of emotional features in public opinion information.
王晰巍, 王秋月, 蔡宏天. 多维度注意力机制下网络舆情视觉情感识别模型及识别效果研究*[J]. 数据分析与知识发现, 2024, 8(3): 156-167.
Wang Xiwei, Wang Qiuyue, Cai Hongtian. Research on Recognition Model and Recognition Effect of Network Public Opinion Visual Emotion under Multi-Dimensional Attention Mechanism. Data Analysis and Knowledge Discovery, 2024, 8(3): 156-167.
(Wu Peng, Qiang Shaohua, Gao Qingning. Modelling Internet Users’ Negative Emotion Based on SOAR Model[J]. Chinese Journal of Management Science, 2018, 26(3): 126-138.)
(Kong Jingyuan, Teng Guangqing, Wang Siming, et al. Impact of Party’s Responses on Netizens’ Emotions in Public Opinion[J]. Library and Information Service, 2020, 64(18): 89-96.)
doi: 10.13266/j.issn.0252-3116.2020.18.010
[3]
Huang F R, Zhang X M, Zhao Z H, et al. Image-Text Sentiment Analysis via Deep Multimodal Attentive Fusion[J]. Knowledge-Based Systems, 2019, 167: 26-37.
doi: 10.1016/j.knosys.2019.01.019
[4]
You Q Z, Luo J B, Jin H L, et al. Cross-Modality Consistent Regression for Joint Visual-Textual Sentiment Analysis of Social Multimedia[C]// Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 2016: 13-22.
[5]
Picard R W. Affective Computing[M]. Cambridge: MIT Press, 2000.
[6]
Tan X W, Cai Y, Xu J Y, et al. Improving Aspect-Based Sentiment Analysis via Aligning Aspect Embedding[J]. Neurocomputing, 2020, 383: 336-347.
doi: 10.1016/j.neucom.2019.12.035
[7]
Hu M Q, Liu B. Mining and Summarizing Customer Reviews[C]// Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2004: 168-177.
[8]
Plutchik R. The Nature of Emotions: Human Emotions Have Deep Evolutionary Roots, a Fact That May Explain Their Complexity and Provide Tools for Clinical Practice[J]. American Scientist, 2001, 89(4): 344-350.
doi: 10.1511/2001.28.344
(Zhang Yazhou, Rong Lu, Song Dawei, et al. A Survey on Multimodal Sentiment Analysis[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(5): 426-438.)
doi: 10.16451/j.cnki.issn1003-6059.202005005
[10]
Lazarus R S. Emotion and Adaptation[J]. Basic Books, 1991.DOI:10.1097/00005053-199303000-00014.
[11]
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2. 2013: 3111-3119.
(Zhu Xiaoxia, Song Jiaxin, Meng Jianfang. Analysis of Online Public Opinion Information Based on the Dynamic Theme-Emotion Evolution Model[J]. Information Science, 2019, 37(7): 72-78.)
(Li Tingting, Ji Donghong. Sentiment Analysis of Micro-blog Based on SVM and CRF Using Various Combinations of Features[J]. Application Research of Computers, 2015, 32 (4): 978-981.)
[14]
Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1746-1751.
[15]
Ma Y K, Peng H Y, Cambria E. Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018: 5876-5883.
[16]
Lu X, Suryanarayan P, Adams R B, et al. On Shape and the Computability of Emotions[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 229-238.
(Fan Tao, Wu Peng, Wang Hao, et al. Sentiment Analysis of Online Users Based on Multimodal Co-Attention[J]. Journal of the China Society for Scientific and Technical Information, 2021, 40(6): 656-665.)
[18]
Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1800-1807.
(Hu Yamin, Wu Xiaoyan, Chen Fang. Review of Technology Term Recognition Studies Based on Machine Learning[J]. Data Analysis and Knowledge Discovery, 2022, 6(2/3): 7-17.)
(Ni Liang, Wu Peng, Zhou Xueqing. Topic Detecting on Multimodal News Data Based on Deep Learning[J]. Data Analysis and Knowledge Discovery, DOI: 10.11925/infotech.2096-3467.2023.0021.)
(Wei Jianxiang, Lu Qian, Han Pu, et al. Event Detection Model Based on Semantic Information Fusion[J]. Data Analysis and Knowledge Discovery, 2023, 7(12): 64-74.)
[22]
Pang B, Lee L. Opinion Mining and Sentiment Analysis[J]. Foundations and Trends in Information Retrieval, 2008, 2(1-2): 1-135.
doi: 10.1561/1500000011
[23]
Nasukawa T, Yi J. Sentiment Analysis: Capturing Favorability Using Natural Language Processing[C]// Proceedings of the 2nd International Conference on Knowledge Capture. 2003: 70-77.
[24]
Sasikala D, Sukumaran S. A Survey on Lexicon and Machine Learning Based Classification Methods for Sentimental Analysis[J]. International Journal of Research and Analytical Reviews, 2019, 6(2): 256-259.
[25]
Prabha M I, Srikanth G U. Survey of Sentiment Analysis Using Deep Learning Techniques[C]// Proceedings of the 2019 1st International Conference on Innovations in Information and Communication Technology. 2019: 1-9.
[26]
Tedmori S, Awajan A. Sentiment Analysis Main Tasks and Applications: A Survey[J]. Journal of Information Processing Systems, 2019, 15(3): 500-519.
[27]
Yu Y H, Lin H F, Meng J N, et al. Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks[J]. Algorithms, 2016, 9(2): 41.
doi: 10.3390/a9020041
(Fan Tao, Wang Hao, Lin Kerou, et al. A Probein to Netizen Sentiment Analysis in Vision-based Internet Public Opinion Events[J]. Information and Documentation Services, 2022, 43(4): 83-91.)
(Huang Zhaopei, Zhang Fengyuan, Zhao Jinming, et al. A Survey of Transfer Learning Problems in Emotion Recognition[J]. Journal of Signal Processing, 2023, 39(4): 588-615.)
[30]
Yang J F, Sun M, Sun X X. Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network[C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017: 224-230.
[31]
Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large- Scale Image Recognition[OL]. arXiv Preprint, arXiv: 1409.1556.
[32]
Yanulevskaya V, Uijlings J, Bruni E, et al. In the Eye of the Beholder: Employing Statistical Analysis and Eye Tracking for Analyzing Abstract Paintings[C]// Proceedings of the 20th ACM International Conference on Multimedia. 2012: 349-358.
[33]
Yan Q M, Sun Y B, Fan S J, et al. Polarity-Aware Attention Network for Image Sentiment Analysis[J]. Multimedia Systems, 2023, 29(1): 389-399.
doi: 10.1007/s00530-022-00935-5
(Teng Lei, Yan Xin, Xu Guangyi, et al. Fine-Grained Sentiment Analysis Based on Capsule Network[J]. Journal of Chinese Computer Systems, 2020, 41(12): 2550-2556.)
[35]
Farag H H, Said L A A, Rizk M R M, et al. Hyperparameters Optimization for ResNet and Xception in the Purpose of Diagnosing COVID-19[J]. Journal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology, 2021(2):41.
[36]
Katsurai M, Satoh S. Image Sentiment Analysis Using Latent Correlations among Visual, Textual, and Sentiment Views[C]//Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. 2016: 2837-2841.
[37]
Woo S, Park J, Lee J Y, et al. CBAM: Convolutional Block Attention Module[C] // Proceedings of the European Conference on Computer Vision. 2018: 3-19.
[38]
He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[39]
Meena G, Mohbey K K, Kumar S, et al. Image-Based Sentiment Analysis Using InceptionV3 Transfer Learning Approach[J]. SN Computer Science, 2023, 4: 242.
doi: 10.1007/s42979-023-01695-3
[40]
Liu P F, Chang S C, Huang X H, et al. Contextualized Non-local Neural Networks for Sequence Learning[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conference and 9th AAAI Symposium on Educational Advances in Artificial Intelligence. 2019: 6762-6769.
[41]
Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. 2017: 618-626.
