[Objective] Once the width of the convolution kernel is the same as the dimension of the word vector, the convolution layer will have too many parameters. The sparse connection of convolution operation, the spatial invariance, and the channel specificity of convolution are not suitable for text tasks. This paper will address these issues. [Methods] We proposed a sentiment analysis model for texts based on patch attention mechanism and Involution. The model first transformed the single-word vector after word segmentation and transformed the one-dimensional word vector into n×n word matrix blocks. Then, we spliced the word matrix blocks of multiple words in the sentence into a sentence matrix. Third, the patch attention mechanism layer enhanced the sentence matrix’s context relevance and position order information of text features. Fourth, we used the involution with spatial specificity and channel invariance to extract the sentence matrix features. Finally, we used the full connection layer for text sentiment classification. [Results] We examined the proposed model with three public data sets waimai_10k, IMDB, and Tweet. Its classification precision reached 88.47%, 86.22%, and 94.42%, respectively, which were 6.47%, 7.72%, 9.35% and 1.07%, 1.01%, 0.59% higher than Bi-LSTM model in word vector convolution network and recurrent neural network. [Limitations] The classification accuracy of this model on large datasets is not as high as on small and medium-sized datasets. [Conclusions] The proposed model solves the problems of excessive parameters, sparse connection of convolution operation, spatial invariance, and channel specificity of convolution, which yield better performance than the traditional convolution models.
林哲, 陈平华. 基于块注意力机制和Involution的文本情感分析模型*[J]. 数据分析与知识发现, 2023, 7(11): 37-45.
Lin Zhe, Chen Pinghua. Analyzing Text Sentiments Based on Patch Attention and Involution. Data Analysis and Knowledge Discovery, 2023, 7(11): 37-45.
(Wang Ting, Yang Wenzhong. Review of Text Sentiment Analysis Methods[J]. Computer Engineering and Applications, 2021, 57(12): 11-24.)
doi: 10.3778/j.issn.1002-8331.2101-0022
(Yang Shuxin, Zhang Nan. Text Sentiment Analysis Based on Sentiment Lexicon and Context Language Model[J]. Journal of Computer Applications, 2021, 41(10): 2829-2834.)
doi: 10.11772/j.issn.1001-9081.2020121900
(Ouyang Jihong, Liu Yanhui, Li Ximing, et al. Multi-Grain Sentiment/Topic Model Based on LDA[J]. Acta Electronica Sinica, 2015, 43(9): 1875-1880.)
doi: 10.3969/j.issn.0372-2112.2015.09.029
[4]
Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014:1746-1751.
(Xing Changzheng, Li Shan. Deep Learning Method for Text Sentiment Analysis[J]. Computer Applications and Software, 2018, 35(8): 102-106.)
[7]
Basnet A, Timalsina A K. Improving Nepali News Recommendation Using Classification Based on LSTM Recurrent Neural Networks[C]// Proceedings of the IEEE 3rd International Conference on Computing, Communication and Security. 2018: 138-142.
(Teng Jinbao, Kong Weiwei, Tian Qiaoxin, et al. Multi-Channel Attention Mechanism Text Classification Model Based on CNN and LSTM[J]. Computer Engineering and Applications, 2021, 57(23): 154-162.)
doi: 10.3778/j.issn.1002-8331.2104-0212
(Wang Youwei, Zhu Chen, Zhu Jianming, et al. User Interest Dictionary and LSTM Based Method for Personalized Emotion Classification[J]. Computer Science, 2021, 48(S2): 251-257.)
[10]
Lai S W, Xu L H, Liu K, et al. Recurrent Convolutional Neural Networks for Text Classification[C]// Proceedings of the 29th AAAI Conference on Artificial Intelligence. 2015: 2267-2273.
(Zhu Ye, Chen Shiping. Commentary Text Sentiment Analysis Combining Convolution Neural Network and Attention[J]. Journal of Chinese Computer Systems, 2020, 41(3): 551-557.)
(Wu Di, Jiang Liting, Wang Lulu, et al. Research on Classification of Tourist Questions Combined with Multi-Head Attention Mechanism[J]. Computer Engineering and Applications, 2022, 58(3): 165-171.)
doi: 10.3778/j.issn.1002-8331.2008-0151
[13]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
[14]
Mnih V, Heess N, Graves A, et al. Recurrent Models of Visual Attention[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2. 2014: 2204-2212.
[15]
Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[OL]. arXiv Preprint, arXiv: 1409.0473.
(Cheng Shuyu, Guo Zeying, Liu Wei, et al. Research on Multi-Granularity Sentence Interaction Natural Language Inference Based on Attention Mechanism[J]. Journal of Chinese Computer Systems, 2019, 40(6): 1215-1220.)
[17]
Li D, Hu J, Wang C H, et al. Involution: Inverting the Inherence of Convolution for Visual Recognition[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 12316-12325.
(Lin Zhijie, Zheng Qiulan, Liang Yong, et al. Medical Image Segmentation Model Based on Involution U-Net[J]. Computer Engineering, 2022, 48(8): 180-186.)
doi: 10.19678/j.issn.1000-3428.0062023
[20]
Mikolov T, Chen K, Corrado G, et al. Efficient Estimation of Word Representations in Vector Space[OL]. arXiv Preprint, arXiv: 1301.3781.
[21]
Sharfuddin A A, Tihami M N, Islam M S. A Deep Recurrent Neural Network with BiLSTM Model for Sentiment Classification[C]// Proceedings of the International Conference on Bangla Speech and Language Processing. 2018: 1-4.