Question Classification Based on Bidirectional GRU with Hierarchical Attention and Multi-channel Convolution
Yu Bengong1,2,Zhu Mengdi1()
1School of Management, Hefei University of Technology, Hefei 230009, China 2Key Laboratory of Process Optimization & Intelligent Decision-making, Ministry of Education, Hefei University of Technology, Hefei 230009, China
[Objective] This paper proposes a method to extract multi-level features from the question texts, aiming to better understand their semantics and address the issues facing text classification. [Methods] First, we constructed multi-channel attention feature matrices based on the multi-feature attention mechanism at the word level. It enriched the semantic representation of the texts and fully utilized the interrogative words, properties and position features from the questions. Then, we convolved the new matrices to obtain phrase-level feature representation. Third, we rearranged the vector representation and fed data to the bidirectional GRU（Gated Recurrent Unit） to access forward and backward semantic features respectively. Finally, we applied the latent topic attention to strengthen the topic information in the bidirectional contextual features, and generated the final text vector for the classification results. [Results] The accuracy rates of proposed model with three Chinese question datasets were 93.89%, 94.47% and 94.23% respectively, which were 5.82% and 4.50% higher than those of the LSTM and CNN. [Limitations] We only examined our new model with three Chinese question corpus. [Conclusions] The proposed model fully understands the semantic features of question texts, and improves the performance of question classification.
Zhao Z, Yang Q F, Cai D, et al. Video Question Answering via Hierarchical Spatio-Temporal Attention Networks[C]// Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017: 3518-3524.
[2]
Sarrouti M, Lachkar A, Ouatik S E A, Biomedical Question Types Classification Using Syntactic and Rule Based Approach[C]// Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K). IEEE, 2015,1:265-272.
[3]
Basaj D, Rychalska B, Biecek P, et al. How Much Should You Ask? On the Question Structure in QA Systems[OL]. arXiv Preprint, arXiv: 1809. 03734.
( Zhou Yuan, Liu Huailan, Du Pengpeng, et al. Research of Text Classification Model Based on the Improved TF-IDF Feature Extraction[J]. Information Science, 2017,35(5):111-118.)
( Qiu Yunfei, Liu Cong. Intention Classification Optimization Method Based on Collaborative Training[J]. Journal of Modern Information, 2019,39(5):57-63, 73.)
[6]
Xie W, Gao D, Hao T. A Feature Extraction and Expansion-based Approach for Question Target Identification and Classification[C]// Proceedings of the China Conference on Information Retrieval. Springer, 2017: 249-260.
[7]
Hasan A M, Zakaria L Q. Question Classification Using Support Vector Machine and Pattern Matching[J]. Journal of Theoretical and Applied Information Technology, 2016,87(2):259-265.
( Ye Zhonglin, Yang Yan, Jia Zhen, et al. Short Question Classification Based on Semantic Extensions[J]. Journal of Computer Applications, 2015,35(3):792-796.)
doi: 10.11772/j.issn.1001-9081.2015.03.792
( Du Hui, Yu Xiaoming, Liu Yue, et al. CNN with Part-of-Speech and Attention Mechanism for Targeted Sentiment Classification[J]. Pattern Recognition and Artificial Intelligence, 2018,31(12):1120-1126.)
[11]
Bairaktaris A, Symeonidis S, Arampatzis A. DUTH at SemEval-2019 Task 8: Part-Of-Speech Features for Question Classification[C]// Proceedings of the 13th International Workshop on Semantic Evaluation. 2019: 1155-1159.
[12]
Kim Y. Convolutional Neural Networks for Sentence Classification[OL]. arXiv Preprint, arXiv: 1408.5882.
[13]
Xiao G Y, Mo J Q, Chow E, et al. Multi-task CNN for Classification of Chinese Legal Questions[C]// Proceedings of the 2017 IEEE 14th International Conference on e-Business Engineering. 2017: 84-90.
( Chen Ke, Liang Bin, Ke Wende, et al. Chinese Micro-blog Sentiment Analysis Based on Multi-channels Convolutional Neural Networks[J]. Journal of Computer Research and Development, 2018,55(5):945-957.)
[15]
Tan C Q, Wei F R, Zhou Q Y, et al. Context-aware Answer Sentence Selection with Hierarchical Gated Recurrent Neural Networks[J]. IEEE Transactions on Audio, Speech, and Language Processing, 2018,26(3):540-549.
[16]
Chen S, Zheng B, Hao T Y. Capsule-based Bidirectional Gated Recurrent Unit Networks for Question Target Classification[C]// Proceedings of the 24th China Conference on Information Retrieval. Springer, 2018: 67-77.
[17]
Zhou C T, Sun C L, Liu Z Y, et al. A C-LSTM Neural Network for Text Classification[OL]. arXiv Preprint, arXiv: 1511.08630.
[18]
Zhang Z Q, Robinson D, Tepper J. Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network[C]// Proceedings of the European Semantic Web Conference. Springer, 2018: 745-760.
[19]
Zhou X Q, Hu B, Chen Q, et al. Recurrent Convolutional Neural Network for Answer Selection in Community Question Answering[J]. Neurocomputing, 2018,274:8-18.
doi: 10.1016/j.neucom.2016.07.082
[20]
Mnih V, Heess N, Graves A, et al. Recurrent Models of Visual Attention[C]// Proceedings of the Conference and Workshop on Neural Information Processing Systems. 2014: 2204-2212.
[21]
Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[OL]. arXiv Preprint, arXiv: 1409.0473.
[22]
Chen Q, Hu Q M, Huang X J, et al. Enhancing Recurrent Neural Networks with Positional Attention for Question Answering[C]// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017: 993-996.
( Tao Zhiyong, Li Xiaobing, Liu Ying, et al. Classifying Short Texts with Improved-Attention Based Bidirectional Long Memory Network[J]. Data Analysis and Knowledge Discovery, 2019,3(12):21-29.)
[24]
Liu J, Yang Y H, Lv S Q, et al. Attention-based BiGRU-CNN for Chinese Question Classification[J]. Journal of Ambient Intelligence and Humanized Computing. DOI: 10.1007/s12652-019-01344-9.
pmid: 20975986
[25]
Shen Y, Deng Y, Yang M, et al. Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs[C]// Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018: 901-904.
[26]
Yang M, Tu W T, Qu Q, et al. Advanced Community Question Answering by Leveraging External Knowledge and Multi-Task Learning[J]. Knowledge-Based Systems, 2019,171:106-119.
doi: 10.1016/j.knosys.2019.02.006
[27]
Yang Z C, Yang D Y, Dyer C, et al. Hierarchical Attention Networks for Document Classification[C]// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016: 1480-1489.
[28]
Yu B G, Xu Q T, Zhang P H. Question Classification Based on MAC-LSTM[C]// Proceedings of the 2018 IEEE 3rd International Conference on Data Science in Cyberspace (DSC). IEEE, 2018: 69-75.
[29]
Tran N K, Niedereee C. Multihop Attention Networks for Question Answer Matching[C]// Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. 2018: 325-334.
( Zhu Maoran, Wang Yilei, Gao Song, et al. A Deep-learning Model Based on Attention Mechanism for Chinese Comparative Relation Detection[J]. Journal of the China Society for Scientific and Technical Information, 2019,38(6):612-621.)
( Zeng Ziming, Wan Pinyu. Sentiment Analysis of Public Safety Events in Micro-blog Based on Double-layered Attention and Bi-LSTM[J]. Information Science, 2019,37(6):23-29.)
( Li Chao, Chai Yumei, Nan Xiaofei, et al. Research on Problem Classification Method Based on Deep Learning[J]. Computer Science, 2016,43(12):115-119.)
