Please wait a minute...
Data Analysis and Knowledge Discovery  2023, Vol. 7 Issue (1): 128-137    DOI: 10.11925/infotech.2096-3467.2022.0258
Current Issue | Archive | Adv Search |
Classifying Customer Complaints Based on Multi-head Co-attention Mechanism
Wang jinzheng1,Yang Ying1,2(),Yu Bengong1,2
1School of Management, Hefei University of Technology, Hefei 230009, China
2Key Laboratory of Process Optimization & Intelligent Decision-making of Ministry of Education, Hefei 230009, China
Download: PDF (1116 KB)   HTML ( 32 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper tries to improve the insufficient learning of the relationship between features in the traditional text classification model. [Methods] We developed a text classification model for customer complaints based on multi-head co-attention mechanism. Firstly, we used the BERT pre-training model to create text vectors. Then, we constructed the Text-CNN and BiLSTM multi-channel feature networks to extract the local and global features of the complaints. Finally, we used the collaborative attention mechanism to learn the relationship between the local and global features to classify complaints. [Results] We examined our model with a public dataset (THUCNews) and its accuracy reached 97.25%, while the accuracy on the telecom customer complaint dataset reached 86.20%. Compared with the single channel baseline model with the best performance and the multi-channel model without feature interaction, the accuracy of the proposed model on telecom customer complaint dataset was improved by 0.54% and 0.35%, respectively. [Limitations] We only examined the interaction between the two features. With the small-scale telecom customer complaint dataset, the classification of some complaint is not satisfactory. [Conclusions] Multi-channel feature extraction network can enrich text information and fully extract text features. Co-attention mechanism can effectively learn the relationship between text features, and improve the model’s classification performance.
Key words： Text Classification      Multi-head Co-attention Mechanism      Customer Complaints     
Received: 25 March 2022      Published: 16 February 2023
ZTFLH:  TP391  
Fund:National Natural Science Foundation of China(72071061)
Corresponding Authors: Yang Ying，ORCID：0000-0002-9912-3443，E-mail： yangying@hfut.edu.cn。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
jinzheng Wang
Ying Yang
Bengong Yu

Cite this article:

Wang jinzheng, Yang Ying, Yu Bengong. Classifying Customer Complaints Based on Multi-head Co-attention Mechanism. Data Analysis and Knowledge Discovery, 2023, 7(1): 128-137.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0258     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2023/V7/I1/128

Customer Complaint Text Classification Model Based on a Multi-head Co-attention Mechanism
Structure of Text-CNN
Structure of LSTM
Multi-head Co-attention Mechanism
Co-attention Mechanism
实验环境 参数配置
操作系统 Windows 10
IDE Jupyter Notebook 6.0.1
PyTorch 1.8.0
GPU RTX3060（12GB）
Python 3.8.5
Experimental Environment Parameters
Relationship Between Epoch and Loss
参数名称 参数值
词向量维度 768
句子最大长度 500
Epochs 50
优化器 SGD
损失函数 CrossEntropy
BiLSTM隐层大小 256
Text-CNN卷积核大小 3、5、7
Text-CNN卷积核通道数 256、256、256
最大池化尺寸 490
学习率 0.001
漏失率 0.5
注意力层头数 8
Parameter Setting
类别 标签 训练集 测试集 总计
第1类 0 1 441 167 1 608
第2类 1 1 666 185 1 851
第3类 2 3 065 338 3 403
第4类 3 1 396 151 1 547
总计 7 568 841 8 409
Statistics of Telecom Customer Complaint Data
模型 类别 THUCNews 电信客户投诉数据集
准确率/% F1/% 准确率/% F1/%
NLSTM 单通道 93.39[7] 93.39[7] 82.75 82.73
RCNN 单通道 95.69 95.67 83.71 83.71
LSTM+att 单通道 94.87[5] 94.85[5] 84.24 84.23
BiLSTM+att 单通道 95.05[5] 95.02[5] 85.32 85.32
BiLSTM+max-pooling 单通道 94.16 94.14 85.42 85.40
CNLSTM 单通道 96.87[7] 96.86[7] 85.66 85.60
组合-CNN 多通道 95.57[10] 95.55[10] 84.54 84.34
CFC-LSTM-multi 多通道 96.21[5] 96.20[5] 85.75 85.75
本文方法 多通道 97.25 97.24 86.20 86.20
Experimental Results of Comparison Methods
模型 THUCNews 电信客户投诉数据集
准确率（%） F1（%） 准确率（%） F1（%）
CNN-LSTM-Co 96.83 96.83 86.05 86.04
本文方法 97.25 97.24 86.20 86.20
Comparison with Different Attention Mechanism
模型 THUCNews 电信客户投诉数据集
准确率（%） F1（%） 准确率（%） F1（%）
Text-CNN 92.37 92.35 85.37 85.36
BiLSTM 94.07 94.07 85.21 85.15
Text-CNN+BiLSTM 94.63 94.62 85.85 85.82
本文方法 97.25 97.24 86.20 86.20
Comparison Results of Ablation Experiment
Confusion Matrix of Complaint Dataset
Confusion Matrix of THUCNews Dataset
[1] 梁昕露, 李美娟. 电信业投诉分类方法及其应用研究[J]. 中国管理科学, 2015, 23(S1): 188-192.
[1] ( Liang Xinlu, Li Meijuan. Text Categorization of Complain in Telecommunication Industry and Its Applied Research[J]. Chinese Journal of Management Science, 2015, 23(S1): 188-192.)
[2] 李荣艳, 金鑫, 王春辉, 等. 一种新的中文文本分类算法[J]. 北京师范大学学报(自然科学版), 2006(5): 501-505.
[2] Li Rongyan, Jin Xin, Wang Chunhui, et al. A New Algorithm of Chinese Text Classification[J]. Journal of Beijing Normal University(Natural Science), 2006(5): 501-505.)
[3] 翟林, 刘亚军. 支持向量机的中文文本分类研究[J]. 计算机与数字工程, 2005(3): 21-23,45.
[3] ( Zhai Lin, Liu Yajun. Research on Chinese Text Categorization Based on Support Vector Machine[J]. Computer & Digital Engineering, 2005(3): 21-23,45.)
[4] 余本功, 陈杨楠, 杨颖. 基于nBD-SVM模型的投诉短文本分类[J]. 数据分析与知识发现, 2019, 3(5): 77-85.
[4] ( Yu Bengong, Chen Yangnan, Yang Ying. Classifying Short Text Complaints with nBD-SVM Model[J]. Data Analysis and Knowledge Discovery, 2019, 3(5): 77-85.)
[5] 韩永鹏, 陈彩, 苏航, 等. 融合通道特征的混合神经网络文本分类模型[J]. 中文信息学报, 2021, 35(2): 78-88.
[5] ( Han Yongpeng, Chen Cai, Su Hang, Liang Yi, et al. Hybrid Neural Network Text Classification Model with Channel Features[J]. Journal of Chinese Information Processing, 2021, 35(2): 78-88.)
[6] 田乔鑫, 孔韦韦, 滕金保, 等. 基于并行混合网络与注意力机制的文本情感分析模型[J/OL]. 计算机工程. [2022-05-10]. https://kns.cnki.net/kcms/detail/31.1289.tp.20211015.0640.010.html.
[6] ( Tian Qiaoxin, Kong Weiwei, Teng Jinbao, et al. Text Sentiment Analysis Model Based on Parallel Hybrid Network and Attention Mechanism[J/OL]. Computer Engineering. [2022-05-10]. https://kns.cnki.net/kcms/detail/31.1289.tp.20211015.0640.010.html.)
[7] 刘月, 翟东海, 任庆宁. 基于注意力CNLSTM模型的新闻文本分类[J]. 计算机工程, 2019, 45(7): 303-308, 314.
[7] ( Liu Yue, Zhai Donghai, Ren Qingning. News Text Classification Based on CNLSTM Model with Attention Mechanism[J]. Computer Engineering, 2019, 45(7): 303-308,314.)
[8] 王艳, 王胡燕, 余本功. 基于多特征融合的中文文本分类研究[J]. 数据分析与知识发现, 2021, 5(10):1-14.
[8] ( Wang Yan, Wang Huyan, Yu Bengong. Chinese Text Classification with Feature Fusion[J]. Data Analysis and Knowledge Discovery, 2021, 5(10): 1-14.)
[9] 黄金杰, 蔺江全, 何勇军, 等. 局部语义与上下文关系的中文短文本分类算法[J]. 计算机工程与应用, 2021, 57(6): 94-100.
doi: 10.3778/j.issn.1002-8331.1912-0185
[9] ( Huang Jinjie, Lin Jiangquan, He Yongjun, et al. Chinese Short Text Classification Algorithm Based on Local Semantics and Context[J]. Computer Engineering and Applications, 2021, 57(6): 94-100.)
doi: 10.3778/j.issn.1002-8331.1912-0185
[10] 张昱, 刘开峰, 张全新, 等. 基于组合-卷积神经网络的中文新闻文本分类[J]. 电子学报, 2021, 49(6): 1059-1067.
doi: 10.12263/DZXB.20200134
[10] ( Zhang Yu, Liu Kaifeng, Zhang quanxin, et al. A Combined-Convolutional Neural Network for Chinese News Text Classification[J]. Acta Electronica Sinica, 2021, 49(6): 1059-1067.)
doi: 10.12263/DZXB.20200134
[11] Liu C, Xu X L. AMFF: A New Attention-Based Multi-Feature Fusion Method for Intention Recognition[J]. Knowledge-Based Systems, 2021, 233: 107525.
doi: 10.1016/j.knosys.2021.107525
[12] Niu Z Y, Zhong G Q, Hui Y. A Review on the Attention Mechanism of Deep Learning[J]. Neurocomputing, 2021, 452: 48-62.
doi: 10.1016/j.neucom.2021.03.091
[13] Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[C]// Proceedings of International Conference on Learning Representations. 2015.
[14] Lu J S, Yang J W, Batra D, et al. Hierarchical Question-Image Co-Attention for Visual Question Answering[C]// Proceedings of the 30th Conference on Neural Information Processing Systems. 2016.
[15] Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st Conference on Neural Information Processing Systems. 2017.
[16] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). 2019:4171-4186.
[17] Kim Y. Convolutional Neural Networks for Sentence Classification[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1746-1751.
[18] Li W J, Qi F, Tang M, et al. Bidirectional LSTM with Self-attention Mechanism and Multi-channel Features for Sentiment Classification[J]. Neurocomputing, 2020, 387: 63-77.
doi: 10.1016/j.neucom.2020.01.006
[19] He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016.
[20] Wang R S, Li Z, Cao J, et al. Convolutional Recurrent Neural Networks for Text Classification[C]// Proceedings of 2019 International Joint Conference on Neural Networks. 2019.
[21] 张冲. 基于Attention-Based LSTM模型的文本分类技术的研究[D]. 南京: 南京大学, 2016.
[21] (Zhang Chong, Text Classification Based on Attention-Based LSTM Model[D]. Nanjing: Nanjing University, 2016.)
[22] 胡朝举, 梁宁. 基于深层注意力的LSTM的特定主题情感分析[J]. 计算机应用研究, 2019, 36(4):1075-1079.
[22] ( Hu Chaoju, Liang Ning. Deeper Attention-based LSTM for Aspect Sentiment Analysis[J]. Application Research of Computers, 2019, 36(4): 1075-1079.)
[1] Wang Weijun, Ning Zhiyuan, Du Yi, Zhou Yuanchun. Identifying Interdisciplinary Sci-Tech Literature Based on Multi-Label Classification[J]. 数据分析与知识发现, 2023, 7(1): 102-112.
[2] Ye Han,Sun Haichun,Li Xin,Jiao Kainan. Classification Model for Long Texts with Attention Mechanism and Sentence Vector Compression[J]. 数据分析与知识发现, 2022, 6(6): 84-94.
[3] Tu Zhenchao, Ma Jing. Item Categorization Algorithm Based on Improved Text Representation[J]. 数据分析与知识发现, 2022, 6(5): 34-43.
[4] Chen Guo, Ye Chao. News Classification with Semi-Supervised and Active Learning[J]. 数据分析与知识发现, 2022, 6(4): 28-38.
[5] Xiao Yuejun, Li Honglian, Zhang Le, Lv Xueqiang, You Xindong. Classifying Chinese Patent Texts with Feature Fusion[J]. 数据分析与知识发现, 2022, 6(4): 49-59.
[6] Yang Lin, Huang Xiaoshuo, Wang Jiayang, Ding Lingling, Li Zixiao, Li Jiao. Identifying Subtypes of Clinical Trial Diseases with BERT-TextCNN[J]. 数据分析与知识发现, 2022, 6(4): 69-81.
[7] Xu Yuemei, Fan Zuwei, Cao Han. A Multi-Task Text Classification Model Based on Label Embedding of Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(2/3): 105-116.
[8] Bai Simeng,Niu Zhendong,He Hui,Shi Kaize,Yi Kun,Ma Yuanchi. Biomedical Text Classification Method Based on Hypergraph Attention Network[J]. 数据分析与知识发现, 2022, 6(11): 13-24.
[9] Huang Xuejian, Liu Yuyang, Ma Tinghuai. Classification Model for Scholarly Articles Based on Improved Graph Neural Network[J]. 数据分析与知识发现, 2022, 6(10): 93-102.
[10] Xie Xingyu, Yu Bengong. Automatic Classification of E-commerce Comments with Multi-Feature Fusion Model[J]. 数据分析与知识发现, 2022, 6(1): 101-112.
[11] Chen Jie,Ma Jing,Li Xiaofeng. Short-Text Classification Method with Text Features from Pre-trained Models[J]. 数据分析与知识发现, 2021, 5(9): 21-30.
[12] Zhou Zeyu,Wang Hao,Zhao Zibo,Li Yueyan,Zhang Xiaoqin. Construction and Application of GCN Model for Text Classification with Associated Information[J]. 数据分析与知识发现, 2021, 5(9): 31-41.
[13] Yu Bengong,Zhu Xiaojie,Zhang Ziwei. A Capsule Network Model for Text Classification with Multi-level Feature Extraction[J]. 数据分析与知识发现, 2021, 5(6): 93-102.
[14] Zhou Zhichao. Review of Automatic Citation Classification Based on Machine Learning[J]. 数据分析与知识发现, 2021, 5(12): 14-24.
[15] Wang Yan, Wang Huyan, Yu Bengong. Chinese Text Classification with Feature Fusion[J]. 数据分析与知识发现, 2021, 5(10): 1-14.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn