Classifying Customer Complaints Based on Multi-head Co-attention Mechanism
Wang jinzheng1,Yang Ying1,2(),Yu Bengong1,2
1School of Management, Hefei University of Technology, Hefei 230009, China 2Key Laboratory of Process Optimization & Intelligent Decision-making of Ministry of Education, Hefei 230009, China
[Objective] This paper tries to improve the insufficient learning of the relationship between features in the traditional text classification model. [Methods] We developed a text classification model for customer complaints based on multi-head co-attention mechanism. Firstly, we used the BERT pre-training model to create text vectors. Then, we constructed the Text-CNN and BiLSTM multi-channel feature networks to extract the local and global features of the complaints. Finally, we used the collaborative attention mechanism to learn the relationship between the local and global features to classify complaints. [Results] We examined our model with a public dataset (THUCNews) and its accuracy reached 97.25%, while the accuracy on the telecom customer complaint dataset reached 86.20%. Compared with the single channel baseline model with the best performance and the multi-channel model without feature interaction, the accuracy of the proposed model on telecom customer complaint dataset was improved by 0.54% and 0.35%, respectively. [Limitations] We only examined the interaction between the two features. With the small-scale telecom customer complaint dataset, the classification of some complaint is not satisfactory. [Conclusions] Multi-channel feature extraction network can enrich text information and fully extract text features. Co-attention mechanism can effectively learn the relationship between text features, and improve the model’s classification performance.
王金政, 杨颖, 余本功. 基于多头协同注意力机制的客户投诉文本分类模型*[J]. 数据分析与知识发现, 2023, 7(1): 128-137.
Wang jinzheng, Yang Ying, Yu Bengong. Classifying Customer Complaints Based on Multi-head Co-attention Mechanism. Data Analysis and Knowledge Discovery, 2023, 7(1): 128-137.
( Liang Xinlu, Li Meijuan. Text Categorization of Complain in Telecommunication Industry and Its Applied Research[J]. Chinese Journal of Management Science, 2015, 23(S1): 188-192.)
Li Rongyan, Jin Xin, Wang Chunhui, et al. A New Algorithm of Chinese Text Classification[J]. Journal of Beijing Normal University(Natural Science), 2006(5): 501-505.)
( Yu Bengong, Chen Yangnan, Yang Ying. Classifying Short Text Complaints with nBD-SVM Model[J]. Data Analysis and Knowledge Discovery, 2019, 3(5): 77-85.)
( Han Yongpeng, Chen Cai, Su Hang, Liang Yi, et al. Hybrid Neural Network Text Classification Model with Channel Features[J]. Journal of Chinese Information Processing, 2021, 35(2): 78-88.)
( Tian Qiaoxin, Kong Weiwei, Teng Jinbao, et al. Text Sentiment Analysis Model Based on Parallel Hybrid Network and Attention Mechanism[J/OL]. Computer Engineering. [2022-05-10]. https://kns.cnki.net/kcms/detail/31.1289.tp.20211015.0640.010.html.)
( Liu Yue, Zhai Donghai, Ren Qingning. News Text Classification Based on CNLSTM Model with Attention Mechanism[J]. Computer Engineering, 2019, 45(7): 303-308,314.)
( Huang Jinjie, Lin Jiangquan, He Yongjun, et al. Chinese Short Text Classification Algorithm Based on Local Semantics and Context[J]. Computer Engineering and Applications, 2021, 57(6): 94-100.)
doi: 10.3778/j.issn.1002-8331.1912-0185
( Zhang Yu, Liu Kaifeng, Zhang quanxin, et al. A Combined-Convolutional Neural Network for Chinese News Text Classification[J]. Acta Electronica Sinica, 2021, 49(6): 1059-1067.)
doi: 10.12263/DZXB.20200134
[11]
Liu C, Xu X L. AMFF: A New Attention-Based Multi-Feature Fusion Method for Intention Recognition[J]. Knowledge-Based Systems, 2021, 233: 107525.
doi: 10.1016/j.knosys.2021.107525
[12]
Niu Z Y, Zhong G Q, Hui Y. A Review on the Attention Mechanism of Deep Learning[J]. Neurocomputing, 2021, 452: 48-62.
doi: 10.1016/j.neucom.2021.03.091
[13]
Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[C]// Proceedings of International Conference on Learning Representations. 2015.
[14]
Lu J S, Yang J W, Batra D, et al. Hierarchical Question-Image Co-Attention for Visual Question Answering[C]// Proceedings of the 30th Conference on Neural Information Processing Systems. 2016.
[15]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st Conference on Neural Information Processing Systems. 2017.
[16]
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019:4171-4186.
[17]
Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1746-1751.
[18]
Li W J, Qi F, Tang M, et al. Bidirectional LSTM with Self-attention Mechanism and Multi-channel Features for Sentiment Classification[J]. Neurocomputing, 2020, 387: 63-77.
doi: 10.1016/j.neucom.2020.01.006
[19]
He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016.
[20]
Wang R S, Li Z, Cao J, et al. Convolutional Recurrent Neural Networks for Text Classification[C]// Proceedings of 2019 International Joint Conference on Neural Networks. 2019.
