|
|
|
| Microblog Sentiment Analysis with Multi-Head Self-Attention Pooling and Multi-Granularity Feature Interaction Fusion |
Yan Shangyi,Wang Jingya( ),Liu Xiaowen,Cui Yumeng,Tao Zhizhong,Zhang Xiaofan |
| School of Information and Cyber Security, People’s Public Security University of China, Beijing 100038, China |
|
|
|
|
Abstract [Objective] This paper tries to efficiently and accurately extract sentiment information from Weibo texts and improve sentiment analysis performance. [Methods] First, we used WoBERT Plus and ALBERT to dynamically encode the character and word-level texts. Then, we extracted key local features with convolution operation. Next, we utilized cross-channel feature fusion and multi-head self-attention pooling operation to extract global semantic information and filter out critical data. Finally, we fused character-level and word-level semantic information using a multi-granularity feature interaction fusion operation and generated the classification results with the Softmax function. [Results] This model’s accuracy and F1 value were 98.51% and 98.53% on the weibo_senti_100k dataset and 80.11% and 75.62% on the SMP2020-EWECT dataset, respectively. Its performance was better than the advanced sentiment analysis models on each dataset. [Limitations] Our model does not include multimodal information such as video, image, and audio for sentiment classification. [Conclusions] The proposed model could effectively accomplish sentiment analysis of Weibo texts.
|
|
Received: 29 April 2022
Published: 07 June 2023
|
|
|
| Fund:National Social Science Fund of China(20AZD114);CCF-Green Alliance Technology “Kun Peng” Research Fund Project(CCF-NSFOCUS 2020011);Public Safety Behavioral Sciences Laboratory Open Subject Fund Program of PPSUC(2020SYS08) |
|
Corresponding Authors:
Wang Jingya，E-mail：wangjingya@ppsuc.edu.cn
|
| [1] |
何跃, 朱灿. 基于微博的意见领袖网情感特征分析——以 “非法疫苗”事件为例[J]. 数据分析与知识发现, 2017, 1(9): 65-73.
|
| [1] |
(He Yue, Zhu Can. Sentiment Analysis of Weibo Opinion Leaders——Case Study of “Illegal Vaccine” Event[J]. Data Analysis and Knowledge Discovery, 2017, 1(9): 65-73.)
|
| [2] |
罗双玲, 夏昊翔, 王延章. 微博社会网络及传播研究评述[J]. 情报学报, 2015, 34(12): 1304-1313.
|
| [2] |
(Luo Shuangling, Xia Haoxiang, Wang Yanzhang. Review on Research of Social Networks of Micro-Blogging and Its Propagation Dynamics[J]. Journal of the China Society for Scientific and Technical Information, 2015, 34(12): 1304-1313.)
|
| [3] |
沈彬, 严馨, 周丽华, 等. 基于ERNIE和双重注意力机制的微博情感分析[J]. 云南大学学报(自然科学版), 2022, 44(3): 480-489.
|
| [3] |
(Shen Bin, Yan Xin, Zhou Lihua, et al. Microblog Sentiment Analysis with ERNIE and Dual Attention Mechanism[J]. Journal of Yunnan University: Natural Sciences Edition, 2022, 44(3): 480-489.)
|
| [4] |
Alharbi N M, Alghamdi N S, Alkhammash E H, et al. Evaluation of Sentiment Analysis via Word Embedding and RNN Variants for Amazon Online Reviews[J]. Mathematical Problems in Engineering, 2021. https://doi.org/10.1155/2021/5536560.
|
| [5] |
Lee G T, Kim C O, Song M. Semisupervised Sentiment Analysis Method for Online Text Reviews[J]. Journal of Information Science, 2021, 47(3): 387-403.
doi: 10.1177/0165551520910032
|
| [6] |
曾子明, 杨倩雯. 基于LDA和AdaBoost多特征组合的微博情感分析[J]. 数据分析与知识发现, 2018, 2(8): 51-59.
|
| [6] |
(Zeng Ziming, Yang Qianwen. Sentiment Analysis for Micro-blogs with LDA and AdaBoost[J]. Data Analysis and Knowledge Discovery, 2018, 2(8): 51-59.)
|
| [7] |
Jin W Z, Zhu H, Yang G C. An Efficient Character-Level and Word-Level Feature Fusion Method for Chinese Text Classification[J]. Journal of Physics: Conference Series, 2019, 1229(1): 012057.
doi: 10.1088/1742-6596/1229/1/012057
|
| [8] |
陈欣, 杨小兵, 姚雨虹. 字词融合的双通道混合神经网络情感分析模型[J]. 小型微型计算机系统, 2021, 42(2): 279-284.
|
| [8] |
(Chen Xin, Yang Xiaobing, Yao Yuhong. Two-Channel Mixed Neural Network Sentiment Analysis Model Based on Character and Word Fusion[J]. Journal of Chinese Computer Systems, 2021, 42(2): 279-284.)
|
| [9] |
韩普, 张伟, 张展鹏, 等. 基于特征融合和多通道的突发公共卫生事件微博情感分析[J]. 数据分析与知识发现, 2021, 5(11): 68-79.
|
| [9] |
(Han Pu, Zhang Wei, Zhang Zhanpeng, et al. Sentiment Analysis of Weibo Posts on Public Health Emergency with Feature Fusion and Multi-channel[J]. Data Analysis and Knowledge Discovery, 2021, 5(11): 68-79.)
|
| [10] |
Yan W, Zhou L F, Qian Z Y, et al. Sentiment Analysis of Student Texts Using the CNN-BiGRU-AT Model[J]. Scientific Programming, 2021, 2021: 8405623.
|
| [11] |
You H F, Yu L, Tian S W, et al. MC-Net: Multiple Max-pooling Integration Module and Cross Multi-scale Deconvolution Network[J]. Knowledge-Based Systems, 2021, 231: 107456.
doi: 10.1016/j.knosys.2021.107456
|
| [12] |
杨冰, 徐丹, 张豪远, 等. 基于改进的DenseNet-BC对少数民族服饰的识别[J]. 浙江大学学报(理学版), 2021, 48(6): 676-683.
|
| [12] |
(Yang Bing, Xu Dan, Zhang Haoyuan, et al. Minority Clothing Recognition Based on Improved DenseNet-BC[J]. Journal of Zhejiang University (Science Edition), 2021, 48(6): 676-683.)
|
| [13] |
Tong X, Wang J Y, Jiao K N, et al. Robustness Detection Method of Chinese Spam Based on the Features of Joint Characters-Words[C]// Proceedings of International Conference on Computer Engineering and Networks. 2020: 845-851.
|
| [14] |
Chen W T, Fan C X, Wu Y X, et al. A Chinese Character-Level and Word-Level Complementary Text Classification Method[C]// Proceedings of 2020 International Conference on Technologies and Applications of Artificial Intelligence. 2020: 187-192.
|
| [15] |
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. 2013: 3111-3119.
|
| [16] |
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019: 4171-4186,
|
| [17] |
Peters M E, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations[C]// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long Papers). 2018: 2227-2237.
|
| [18] |
Radford A, Narasimhan K, Salimans T, et al. Improving Language Understanding by Generative Pre-training[EB/OL]. [2022-04-20]. https://www.cs.ubc.ca/-amuham01/LI-NG530/papers/radford2018improving.pdf.
|
| [19] |
Lan Z Z, Chen M D, Goodman S, et al. ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations[OL]. arXiv Preprint, arXiv: 1909.11942.
|
| [20] |
Liu Y H, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[OL]. arXiv Preprint, arXiv: 1907.11692.
|
| [21] |
Cui Y M, Che W X, Liu T, et al. Pre-training with Whole Word Masking for Chinese BERT[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514.
doi: 10.1109/TASLP.2021.3124365
|
| [22] |
Joshi M, Chen D Q, Liu Y H, et al. SpanBERT: Improving Pre-Training by Representing and Predicting Spans[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 64-77.
doi: 10.1162/tacl_a_00300
|
| [23] |
苏剑林. 提速不掉点: 基于词颗粒度的中文WoBERT[EB/OL]. [2022-04-18]. https://kexue.fm/archives/7758.
|
| [23] |
(Su Jianlin. Speed up Without Losing Points: Chinese WoBERT Based on Word Granularity[EB/OL]. [2022-04-18]. https://kexue.fm/archives/7758. )
|
| [24] |
ZhuiyiTechnology. 以词为基本单位的中文BERT[EB/OL].[2022-04-18]. https://github.com/ZhuiyiTechnology/WoBERT.
|
| [24] |
(ZhuiyiTechnology. Chinese BERT with Word as Basic Unit[EB/OL].[2022-04-18]. https://github.com/ZhuiyiTechnology/WoBERT. )
|
| [25] |
张仰森, 王胜, 魏文杰, 等. 融合语义信息与问题关键信息的多阶段注意力答案选取模型[J]. 计算机学报, 2021, 44(3): 491-507.
|
| [25] |
(Zhang Yangsen, Wang Sheng, Wei Wenjie, et al. An Answer Selection Model Based on Multi-Stage Attention Mechanism with Combination of Semantic Information and Key Information of the Question[J]. Chinese Journal of Computers, 2021, 44(3): 491-507.)
|
| [26] |
Sangeetha K, Prabha D. Sentiment Analysis of Student Feedback Using Multi-head Attention Fusion Model of Word and Context Embedding for LSTM[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12(3): 4117-4126.
doi: 10.1007/s12652-020-01791-9
|
| [27] |
India M, Safari P, Hernando J. Self Multi-head Attention for Speaker Recognition[OL]. arXiv Preprint, arXiv: 1906.09890.
|
| [28] |
Fang Y, Gao J, Huang C, et al. Self Multi-head Attention-Based Convolutional Neural Networks for Fake News Detection[J]. PLoS One, 2019, 14(9): e0222713.
|
| [29] |
Yao L, Mao C S, Luo Y. Graph Convolutional Networks for Text Classification[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019, 33: 7370-7377.
|
| [30] |
郭樊容, 黄孝喜, 王荣波, 等. 基于Transformer和图卷积神经网络的隐喻识别[J]. 数据分析与知识发现, 2022, 6(4): 120-129.
|
| [30] |
(Guo Fanrong, Huang Xiaoxi, Wang Rongbo, et al. Identifying Metaphor with Transformer and Graph Convolutional Network[J]. Data Analysis and Knowledge Discovery, 2022, 6(4): 120-129.)
|
| [31] |
蒋浩泉, 张儒清, 郭嘉丰, 等. 图卷积网络与自注意机制在文本分类任务上的对比分析[J]. 中文信息学报, 2021, 35(12): 84-93.
|
| [31] |
(Jiang Haoquan, Zhang Ruqing, Guo Jiafeng, et al. A Comparative Study of Graph Convolutional Networks and Self-Attention Mechanism on Text Classification[J]. Journal of Chinese Information Processing, 2021, 35(12): 84-93.)
|
| [32] |
SophonPlus. ChineseNlpCorpus[EB/OL]. [2022-04-17]. https://github.com/SophonPlus/ChineseNlpCorpus/blob/master/datasets/weibo_senti_100k/intro.ipynb.
|
| [33] |
BrownSweater. BERT_SMP2020-EWECT[EB/OL]. [2022-04-18]. https://github.com/BrownSweater/BERT_SMP2020-EWECT/tree/main/data/raw.
|
| [34] |
Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1746-1751.
|
| [35] |
Cho K, van Merrienboer B, Gulcehre C, et al. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 1724-1734.
|
| [36] |
Zhou P, Shi W, Tian J, et al. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2:Short Papers). 2016: 207-212.
|
| [37] |
Lai S W, Xu L H, Liu K, et al. Recurrent Convolutional Neural Networks for Text Classification[C]// Proceedings of the 29th AAAI Conference on Artificial Intelligence. 2015: 2267-2273.
|
| [38] |
杜琳, 曹东, 林树元, 等. 基于BERT与Bi-LSTM融合注意力机制的中医病历文本的提取与自动分类[J]. 计算机科学, 2020, 47(S2): 416-420.
|
| [38] |
(Du Lin, Cao Dong, Lin Shuyuan, et al. Extraction and Automatic Classification of TCM Medical Records Based on Attention Mechanism of BERT and Bi-LSTM[J]. Computer Science, 2020, 47(S2): 416-420.)
|
| [39] |
赵宏, 傅兆阳, 赵凡. 基于BERT和层次化Attention的微博情感分析研究[J]. 计算机工程与应用, 2022, 58(5): 156-162.
doi: 10.3778/j.issn.1002-8331.2107-0448
|
| [39] |
(Zhao Hong, Fu Zhaoyang, Zhao Fan. Microblog Sentiment Analysis Based on BERT and Hierarchical Attention[J]. Computer Engineering and Applications, 2022, 58(5): 156-162.)
doi: 10.3778/j.issn.1002-8331.2107-0448
|
| [40] |
Wang B K, Shan D H, Fan A W, et al. A Sentiment Classification Method of Web Social Media Based on Multidimensional and Multilevel Modeling[J]. IEEE Transactions on Industrial Informatics, 2022, 18(2): 1240-1249.
doi: 10.1109/TII.2021.3085663
|
| [41] |
Mu Z B, Zheng S, Wang Q M. ACL-RoBERTa-CNN Text Classification Model Combined with Contrastive Learning[C]// Proceedings of 2021 International Conference on Big Data Engineering and Education. 2021: 193-197.
|
| [42] |
Qiu H, Fan C D, Yao J, et al. Chinese Microblog Sentiment Detection Based on CNN-BiGRU and Multihead Attention Mechanism[J]. Scientific Programming, 2020, 2020: 8865983.
|
| [43] |
Tang F H, Nongpong K. Chinese Sentiment Analysis Based on Lightweight Character-Level BERT[C]// Proceedings of 2021 13th International Conference on Knowledge and Smart Technology. 2021: 27-32.
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
