Data Analysis and Knowledge Discovery  2021, Vol. 5 Issue (6): 103-114    DOI: 10.11925/infotech.2096-3467.2020.1159
 Current Issue | Archive | Adv Search |
Sentiment Classification of Image-Text Information with Multi-Layer Semantic Fusion
Xie Hao,Mao Jin(),Li Gang
Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China
 Download: PDF (3270 KB)   HTML ( 18 )  Export: BibTeX | EndNote (RIS)
Abstract

[Objective] This paper conducts sentiment analysis of images and text on social media data, aiming to better understand the public's emotions and opinion tendencies. [Methods] To fully explore the correlation and complementarity between images and text, this paper proposes an image-text sentiment classification model in social media based on multi-layer semantic fusion. There are three sub-models in our study: text-image semantic association model, image-text semantic association model, and multimodal semantic deep association fusion model. We used these sub-models to explore the bidirectional and multi-level semantic associations between images and text. Then, we obtained the final classification results using a weighting strategy on the sentiment classification scores generated by the three sub-models. [Results] We examined our model with real image-text data sets and found it achieved the best performance in all evaluation metrics. The accuracy and F1 values of our model were 1.0% and 1.2% better than those of the optimal baseline model. [Limitations] We only evaluated the model’s performance with one single dataset. More research is needed to examine the robustness and scalability of the model. [Conclusions] In the sentiment classification task, the proposed model could more effectively explore the correlation and complementarity between image and text information on social media.

Received: 24 November 2020      Published: 10 March 2021
 ZTFLH: G350
Fund:National Natural Science Foundation of China(71790612);National Natural Science Foundation of China(71921002)
Corresponding Authors: Mao Jin     E-mail: danveno@163.com
 Examples of Image-Text Data The Framework of Image-Text Sentiment Classification Based on Multi-layer Semantic Fusion The Framework of Text-Image Semantic Association Model The Framework of Image-Text Semantic Association Model The Framework of Multimodal Semantic Deep Association Fusion Model Setting of Important Parameters Algorithm Performance α and $β$"> Accuracy at Different Values of $α$ and $β$ α and $β$"> F1 Value at Different Values of $α$ and $β$