1School of Management, Hefei University of Technology, Hefei 230009, China 2Key Laboratory of Process Optimization & Intelligent Decision-making (Hefei University of Technology),Ministry of Education, Hefei 230009, China
[Objective] This paper proposes a BERT-based Attention Gated Convolution Neural Network model (BAGCNN), aiming to improve the traditional aspect-level sentiment analysis algorithm. [Methods] First, the pre-trained BERT model generated feature representation for the texts and aspect words. Then, we introduced the Multi Head Self-attention Mechanism to solve the problem of long-distance dependence of aspect words. Finally, we selectively extracted the multi-level context features paralleling the aspect words with the Gated Convolution Neural Network. [Results] Compared to the benchmark model, the accuracy of our new model was improved by 4.24, 4.01 and 3.89 percentage points on restaurant, laptop and twitter datasets. The size of the downstream parallel structure of the model was also reduced by 1.27 MB. [Limitations] The proposed model did not work well with data sets having significantly different text length. [Conclusions] The new BAGCNN model could effectively remove the context information irrelevant to the aspect words.
1. Even when the chef is not in the house, the food and service are right on target.
Aspect
chef(O)
food(P)
service(P)
BERT-SPC
×
√
√
BERT-AEN
×
×
√
TD-BERT
×
√
√
Our(BAGCNN)
√
√
√
2. Food was average and creme brulee was awful - the sugar was charred, not caramelized and smelled of kerosene.
Aspect
Food(O)
brulee(N)
sugar(N)
BERT-SPC
×
√
√
BERT-AEN
×
√
√
TD-BERT
×
√
√
Our(BAGCNN)
×
√
√
3. The food is uniformly exceptional, with a very capable kitchen which will proudly whip up whatever you feel like eating, whether it 's on the menu or not.
(Zhao Mingqing, Wu Shengqiang. Research on Stock Market Weighted Prediction Method Based on Micro-blog Sentiment Analysis[J]. Data Analysis and Knowledge Discovery, 2019, 3(2): 43-51.)
(Li Tiejun, Yan Duanwu, Yang Xiongfei. Recommending Microblogs Based on Emotion-Weighted Association Rules[J]. Data Analysis and Knowledge Discovery, 2020, 4(4): 27-33.)
(Zeng Ziming, Wan Pinyu. Sentiment Analysis of Public Safety Events in Micro-blog Based on Double-layered Attention and Bi-LSTM[J]. Information Science, 2019, 37(6): 23-29.)
[4]
Kiritchenko S, Zhu X, Cherry C, et al. Nrc-canada-2014: Detecting Aspects and Sentiment in Customer Reviews [C]//Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). 2014: 437-442.
[5]
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality [C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013: 3111-3119.
[6]
Pennington J, Socher R, Manning C D. GloVe: Global Vectors for Word Representation [C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1532-1543.
[7]
Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding [C]//Proceeding of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019: 4171-4186.
[8]
Peters M E, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations [C]//Proceeding of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2018: 2227-2237.
[9]
Hoang M, Bihorac O A, Rouces J. Aspect-Based Sentiment Analysis Using BERT [C]//Proceedings of the 22nd Nordic Conference on Computational Linguistics. 2019: 187-196.
[10]
Gao Z, Feng A, Song X, et al. Target-dependent Sentiment Classification with BERT[J]. IEEE Access, 2019, 7: 154290-154299.
doi: 10.1109/Access.6287639
[11]
Xu Q, Zhu L, Dai T, et al. Aspect-Based Sentiment Classification with Multi-Attention Network[J]. Neurocomputing, 2020, 388: 135-143.
doi: 10.1016/j.neucom.2020.01.024
[12]
Hu M, Zhao S, Zhang L, et al. CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 4601-4610.
[13]
Song Y, Wang J, Jiang T, et al. Targeted Sentiment Classification with Attentional Encoder Network [C]//Proceedings of the 28th International Conference on Artificial Neural Networks. Springer, Cham, 2019: 93-103.
[14]
Zhao F, Wu Z, Dai X. Attention Transfer Network for Aspect-level Sentiment Classification [C]//Proceedings of the 28th International Conference on Computational Linguistics. 2020: 811-821.
[15]
Zhang C, Li Q, Song D. Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 4568-4578.
[16]
Chen C, Teng Z, Zhang Y. Inducing Target-Specific Latent Structures for Aspect Sentiment Classification [C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 5596-5607.
[17]
Wang K, Shen W, Yang Y, et al. Relational Graph Attention Network for Aspect-based Sentiment Analysis [C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 3229-3238.
[18]
Kipf T N, Welling M. Semi-Supervised Classification with Graph Convolutional Networks[OL]. arXiv Preprint, arXiv: 1609.02907.
[19]
Veličković P, Cucurull G, Casanova A, et al. Graph Attention Networks[OL]. arXiv Preprint, arXiv: 1710.10903.
[20]
Xue W, Li T. Aspect Based Sentiment Analysis with Gated Convolutional Networks [C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 2514-2523.
[21]
Phan M H, Ogunbona P O. Modelling Context and Syntactical Features for Aspect-Based Sentiment Analysis [C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 3211-3220.
[22]
Zhang S, Xu X, Pang Y, et al. Multi-layer Attention Based CNN for Target-Dependent Sentiment Classification[J]. Neural Processing Letters, 2020, 51(3): 2089-2103.
doi: 10.1007/s11063-019-10017-9
(Cao Weidong, Li Jiaqi, Wang Huaichao. Analysis of Targeted Sentiment by the Attention Gated Convolutional Network Model[J]. Journal of Xidian University, 2019, 46(6): 30-36.)
[24]
Liu N, Shen B. Aspect-based Sentiment Analysis with Gated Alternate Neural Network[J]. Knowledge-Based Systems, 2020, 188: 105010.
doi: 10.1016/j.knosys.2019.105010
[25]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
[26]
Dong L, Wei F, Tan C, et al. Adaptive Recursive Neural Network for Target-Dependent Twitter Sentiment Classification [C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. 2014: 49-54.
[27]
Huang B, Ou Y, Carley K M. Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks [C]//Proceedings of the 11th International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation. Springer, Cham, 2018: 197-206.
[28]
Ma D, Li S, Zhang X, et al. Interactive Attention Networks for Aspect-Level Sentiment Classification [C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017: 4068-4074.
[29]
Jiang N, Tian F, Li J, et al. MAN: Mutual Attention Neural Networks Model for Aspect-Level Sentiment Classification in SIoT[J]. IEEE Internet of Things Journal, 2020, 7(4): 2901-2913.
doi: 10.1109/JIoT.6488907
[30]
Wang Y, Huang M, Zhu X, et al. Attention-based LSTM for Aspect-Level Sentiment Classification [C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016: 606-615.