|
|
Classifying Customer Complaints Based on Multi-head Co-attention Mechanism |
Wang jinzheng1,Yang Ying1,2( ),Yu Bengong1,2 |
1School of Management, Hefei University of Technology, Hefei 230009, China 2Key Laboratory of Process Optimization & Intelligent Decision-making of Ministry of Education, Hefei 230009, China |
|
|
Abstract [Objective] This paper tries to improve the insufficient learning of the relationship between features in the traditional text classification model. [Methods] We developed a text classification model for customer complaints based on multi-head co-attention mechanism. Firstly, we used the BERT pre-training model to create text vectors. Then, we constructed the Text-CNN and BiLSTM multi-channel feature networks to extract the local and global features of the complaints. Finally, we used the collaborative attention mechanism to learn the relationship between the local and global features to classify complaints. [Results] We examined our model with a public dataset (THUCNews) and its accuracy reached 97.25%, while the accuracy on the telecom customer complaint dataset reached 86.20%. Compared with the single channel baseline model with the best performance and the multi-channel model without feature interaction, the accuracy of the proposed model on telecom customer complaint dataset was improved by 0.54% and 0.35%, respectively. [Limitations] We only examined the interaction between the two features. With the small-scale telecom customer complaint dataset, the classification of some complaint is not satisfactory. [Conclusions] Multi-channel feature extraction network can enrich text information and fully extract text features. Co-attention mechanism can effectively learn the relationship between text features, and improve the model’s classification performance.
|
Received: 25 March 2022
Published: 16 February 2023
|
|
Fund:National Natural Science Foundation of China(72071061) |
Corresponding Authors:
Yang Ying,ORCID:0000-0002-9912-3443,E-mail: yangying@hfut.edu.cn。
|
[1] |
梁昕露, 李美娟. 电信业投诉分类方法及其应用研究[J]. 中国管理科学, 2015, 23(S1): 188-192.
|
[1] |
( Liang Xinlu, Li Meijuan. Text Categorization of Complain in Telecommunication Industry and Its Applied Research[J]. Chinese Journal of Management Science, 2015, 23(S1): 188-192.)
|
[2] |
李荣艳, 金鑫, 王春辉, 等. 一种新的中文文本分类算法[J]. 北京师范大学学报(自然科学版), 2006(5): 501-505.
|
[2] |
Li Rongyan, Jin Xin, Wang Chunhui, et al. A New Algorithm of Chinese Text Classification[J]. Journal of Beijing Normal University(Natural Science), 2006(5): 501-505.)
|
[3] |
翟林, 刘亚军. 支持向量机的中文文本分类研究[J]. 计算机与数字工程, 2005(3): 21-23,45.
|
[3] |
( Zhai Lin, Liu Yajun. Research on Chinese Text Categorization Based on Support Vector Machine[J]. Computer & Digital Engineering, 2005(3): 21-23,45.)
|
[4] |
余本功, 陈杨楠, 杨颖. 基于nBD-SVM模型的投诉短文本分类[J]. 数据分析与知识发现, 2019, 3(5): 77-85.
|
[4] |
( Yu Bengong, Chen Yangnan, Yang Ying. Classifying Short Text Complaints with nBD-SVM Model[J]. Data Analysis and Knowledge Discovery, 2019, 3(5): 77-85.)
|
[5] |
韩永鹏, 陈彩, 苏航, 等. 融合通道特征的混合神经网络文本分类模型[J]. 中文信息学报, 2021, 35(2): 78-88.
|
[5] |
( Han Yongpeng, Chen Cai, Su Hang, Liang Yi, et al. Hybrid Neural Network Text Classification Model with Channel Features[J]. Journal of Chinese Information Processing, 2021, 35(2): 78-88.)
|
[6] |
田乔鑫, 孔韦韦, 滕金保, 等. 基于并行混合网络与注意力机制的文本情感分析模型[J/OL]. 计算机工程. [2022-05-10]. https://kns.cnki.net/kcms/detail/31.1289.tp.20211015.0640.010.html.
|
[6] |
( Tian Qiaoxin, Kong Weiwei, Teng Jinbao, et al. Text Sentiment Analysis Model Based on Parallel Hybrid Network and Attention Mechanism[J/OL]. Computer Engineering. [2022-05-10]. https://kns.cnki.net/kcms/detail/31.1289.tp.20211015.0640.010.html.)
|
[7] |
刘月, 翟东海, 任庆宁. 基于注意力CNLSTM模型的新闻文本分类[J]. 计算机工程, 2019, 45(7): 303-308, 314.
|
[7] |
( Liu Yue, Zhai Donghai, Ren Qingning. News Text Classification Based on CNLSTM Model with Attention Mechanism[J]. Computer Engineering, 2019, 45(7): 303-308,314.)
|
[8] |
王艳, 王胡燕, 余本功. 基于多特征融合的中文文本分类研究[J]. 数据分析与知识发现, 2021, 5(10):1-14.
|
[8] |
( Wang Yan, Wang Huyan, Yu Bengong. Chinese Text Classification with Feature Fusion[J]. Data Analysis and Knowledge Discovery, 2021, 5(10): 1-14.)
|
[9] |
黄金杰, 蔺江全, 何勇军, 等. 局部语义与上下文关系的中文短文本分类算法[J]. 计算机工程与应用, 2021, 57(6): 94-100.
doi: 10.3778/j.issn.1002-8331.1912-0185
|
[9] |
( Huang Jinjie, Lin Jiangquan, He Yongjun, et al. Chinese Short Text Classification Algorithm Based on Local Semantics and Context[J]. Computer Engineering and Applications, 2021, 57(6): 94-100.)
doi: 10.3778/j.issn.1002-8331.1912-0185
|
[10] |
张昱, 刘开峰, 张全新, 等. 基于组合-卷积神经网络的中文新闻文本分类[J]. 电子学报, 2021, 49(6): 1059-1067.
doi: 10.12263/DZXB.20200134
|
[10] |
( Zhang Yu, Liu Kaifeng, Zhang quanxin, et al. A Combined-Convolutional Neural Network for Chinese News Text Classification[J]. Acta Electronica Sinica, 2021, 49(6): 1059-1067.)
doi: 10.12263/DZXB.20200134
|
[11] |
Liu C, Xu X L. AMFF: A New Attention-Based Multi-Feature Fusion Method for Intention Recognition[J]. Knowledge-Based Systems, 2021, 233: 107525.
doi: 10.1016/j.knosys.2021.107525
|
[12] |
Niu Z Y, Zhong G Q, Hui Y. A Review on the Attention Mechanism of Deep Learning[J]. Neurocomputing, 2021, 452: 48-62.
doi: 10.1016/j.neucom.2021.03.091
|
[13] |
Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[C]// Proceedings of International Conference on Learning Representations. 2015.
|
[14] |
Lu J S, Yang J W, Batra D, et al. Hierarchical Question-Image Co-Attention for Visual Question Answering[C]// Proceedings of the 30th Conference on Neural Information Processing Systems. 2016.
|
[15] |
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st Conference on Neural Information Processing Systems. 2017.
|
[16] |
Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019:4171-4186.
|
[17] |
Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1746-1751.
|
[18] |
Li W J, Qi F, Tang M, et al. Bidirectional LSTM with Self-attention Mechanism and Multi-channel Features for Sentiment Classification[J]. Neurocomputing, 2020, 387: 63-77.
doi: 10.1016/j.neucom.2020.01.006
|
[19] |
He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016.
|
[20] |
Wang R S, Li Z, Cao J, et al. Convolutional Recurrent Neural Networks for Text Classification[C]// Proceedings of 2019 International Joint Conference on Neural Networks. 2019.
|
[21] |
张冲. 基于Attention-Based LSTM模型的文本分类技术的研究[D]. 南京: 南京大学, 2016.
|
[21] |
(Zhang Chong, Text Classification Based on Attention-Based LSTM Model[D]. Nanjing: Nanjing University, 2016.)
|
[22] |
胡朝举, 梁宁. 基于深层注意力的LSTM的特定主题情感分析[J]. 计算机应用研究, 2019, 36(4):1075-1079.
|
[22] |
( Hu Chaoju, Liang Ning. Deeper Attention-based LSTM for Aspect Sentiment Analysis[J]. Application Research of Computers, 2019, 36(4): 1075-1079.)
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
|
Shared |
|
|
|
|
|
Discussed |
|
|
|
|