Please wait a minute...
Data Analysis and Knowledge Discovery  2020, Vol. 4 Issue (10): 37-46    DOI: 10.11925/infotech.2096-3467.2019.1301
Current Issue | Archive | Adv Search |
Improving Online Q&A Service with Deep Learning
Ding Heng(),Li Yingxuan
School of Information Management, Central China Normal University, Wuhan 430079, China
Download: PDF (934 KB)   HTML ( 15
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper develops a neural network model to improve the online questioning and answering services.[Methods] First, we retrieved and constructed our experimental dataset from Yahoo Answers and Yahoo! L6 platform. Then, we proposed a neural network model (CNMNN) based on semantic matching matrix,variable-size convolutional layer, and multiple layer perceptron. Finally, we compared the results our model with the MQ2QC、IBLM、DRMM and MatchPyramid methods. [Results] The proposed model was 45.0%, 38.7%, 33.4%, 34.8% and 52.9% higher than the best results on relevance metrics of nDCG@5, nDCG@10, nDCG@20, MRR and MAP. It also gained 31.5%, 23.6%, 25.5%, 38.1%, 36.9% and 30.7% improvements on diversity metrics of α-nDCG@5, α-nDCG@10, α-nDCG@20 and ERR-IA@5, ERR-IA@10 and ERR-IA@20.[Limitations] We did not include new method to further diversify the results.[Conclusions] The new CNMNN model can effectively calculate the semantic relevance between queries and natural language questions at phrase level. It also avoids the issue of feature signal compression due to hierarchical convolution operation.

Key wordsQuery Suggestion      Deep Learning      Community-based Question and Answering     
Received: 04 December 2019      Published: 09 November 2020
ZTFLH:  TP393  
Corresponding Authors: Ding Heng     E-mail: me@gmail.com

Cite this article:

Ding Heng,Li Yingxuan. Improving Online Q&A Service with Deep Learning. Data Analysis and Knowledge Discovery, 2020, 4(10): 37-46.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2019.1301     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2020/V4/I10/37

Sample of Query Suggestion for Community-based Question and Answering
Structure of CNMNN Neural Network Model
分值 判断规则
2 (C1) 用户可能会使用查询词搜索该问题
(C2) 用户可能会使用查询词的简单修改(通过同义替换修改一个词或词组)搜索该问题
1 (C3) 问题包含多个提问意图,且覆盖了查询词的提问意图
(C4) 用户可能会使用查询词的复杂修改(通过同义替换修改两个及以上的词或词组)搜索该问题
0 (C5) 查询词与自然语言问句无关
Scoring Rules of Manually Annotated Correlation
方法 nDCG@5 nDCG@10 nDCG@20 MRR MAP
MQ2QC 0.448 0.458 0.502 0.468 0.310
IBLM 0.447 0.470 0.503 0.481 0.334
DRMM 0.382 0.429 0.506 0.379 0.269
MatchPyramid 0.484 0.517 0.572 0.528 0.326
CNMNN 0.702 0.717 0.763 0.712 0.511
Comparison Results of Relevance Score
方法 nDCG@5 nDCG@10 nDCG@20 MRR MAP
CNMNN(SDF) 0.508 0.524 0.581 0.536 0.302
CNMNN(MF) 0.698 0.713 0.748 0.677 0.495
Comparison Results of Relevance Score
方法 ERR-IA α-nDCG
@5 @10 @20 @5 @10 @20
MQ2QC 0.122 0.140 0.153 0.407 0.412 0.479
IBLM 0.124 0.144 0.151 0.417 0.447 0.464
DRMM 0.079 0.109 0.136 0.427 0.440 0.468
MatchPyramid 0.014 0.125 0.139 0.475 0.484 0.538
CNMNN 0.163 0.178 0.192 0.656 0.663 0.703
Comparison Results of Diversity Score
方法 查询词:14th Amendment
MQ2QC [1] what is the significance of the 14th amendment
[2] was the 13th 14th amendments ratifed
[3] how does the 14th amendment violate states rights
IBLM [1] what things have violated the 14th amendment
[2] does anyone know what the 14th amendment is
[3] is the death penalty a violation of the 8th and 14th amendments
DRMM [1] who made the 13th amendment
[2] what is the importance of the third amendment
[3] what is the 14th amendment
MatchPyramid [1] what things have violated the 14th amendment
[2] what exactly is the 14th amendment
[3] who made the 13th amendment
CNMNN [1] what is the 14th amendment
[2] is abortion a violation of the 14th amendment to the constitution of the us
[3] what is a 14th amendment citizen
Top 3 Results of Query Suggestion Based on the Query “14th Amendment”
[1] 李亚楠, 王斌, 李锦涛. 搜索引擎查询推荐技术综述[J]. 中文信息学报, 2010,24(6):75-84.
[1] ( Li Ya’nan, Wang Bin, Li Jintao. A Survey of Query Suggestion in Search Engine[J]. Journal of Chinese Information Processing, 2010,24(6):75-84.)
[2] 孟玲玲. 基于WordNet的语义相似性度量及其在查询推荐中的应用研究[D]. 上海: 华东师范大学, 2014.
[2] ( Meng Lingling. Research on Semantic Similarity Metric Based on WordNet and Its Application in Query Suggestion[D]. Shanghai: East China Normal University, 2014.)
[3] Yang J M, Cai R, Jing F, et al. Search-based Query Suggestion[C]//Proceedings of the 17th ACM Conference on Information and Knowledge Management. 2008: 1439-1440.
[4] 季岚石. 基于搜索日志的查询推荐算法研究[D]. 长春: 吉林大学, 2013.
[4] ( Ji Lanshi. The Query Recommendation Algorithm Research Based on the Search Logs[D]. Changchun: Jilin University, 2013.)
[5] Ding H, Balog K. Generating Synthetic Data for Neural Keyword-to-Question Models[C]//Proceedings of the 4th ACM SIGIR International Conference on the Theory of Information Retrieval. 2018: 51-58.
[6] Xu J X, Croft W B. Quary Expansion Using Local and Global Document Analysis[J]. ACM SIGIR Forum, 2017,51(2):168-175.
doi: 10.1145/3130348.3130364
[7] Garigliotti D, Balog K. Generating Query Suggestions to Support Task-based Search[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2017: 1153-1156.
[8] Cao H H, Jiang D X, Pei J, et al. Context-aware Query Suggestion by Mining Click-through and Session Data[C]//Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2008: 875-883.
[9] Mei Q Z, Zhou D Y, Church K. Query Suggestion Using Hitting Time[C]//Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, 2008: 469-478.
[10] 张伟男. 社区型问答中问句检索关键技术研究[D]. 哈尔滨:哈尔滨工业大学, 2014.
[10] ( Zhang Weinan. Research on Key Techniques of Question Retrieval for Community Question Answering[D]. Harbin: Harbin Institute of Technology, 2014.)
[11] 刘欣, 席耀一, 王波, 等. WordNet和词向量相结合的句子检索方法[J]. 信息工程大学学报, 2017,18(4):486-491.
[11] ( Liu Xin, Xi Yaoyi, Wang Bo, et al. WordNet and Word Embedding Based Sentence Retrieval Method[J]. Journal of Information Engineering University, 2017,18(4):486-491.)
[12] Xue X B, Jeon J, Croft W B. Retrieval Models for Question and Answer Archives[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2008: 475-482.
[13] Zhou G, Cai L, Zhao J, et al. Phrase-based Translation Model for Question Retrieval in Community Question Answer Archives[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. 2011: 653-662.
[14] Ichikawa H, Hakoda K, Hashimoto T, et al. Efficient Sentence Retrieval Based on Syntactic Structure[C]//Proceedings of the COLING/ACL on Main Conference Poster Sessions. ACL, 2006: 399-406.
[15] Wang K, Ming Z Y, Chua T S, et al. A Syntactic Tree Matching Approach to Finding Similar Questions in Community-based QA Services[C]//Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2009: 187-194.
[16] Cai L, Zhou G Y, Liu K, et al. Learning the Latent Topics for Question Retrieval in Community QA[C]//Proceedings of the 5th International Joint Conference on Natural Language Processing. 2011: 273-281.
[17] Zhang K, Wu W, Wu H C, et al. Question Retrieval with High Quality Answers in Community Question Answering[C]//Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 2014: 371-380.
[18] Gao Y J, Chen L, Li R, et al. Mapping Queries to Questions: Towards Understanding Users’ Information Needs[C]//Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2013: 977-980.
[19] Wu H C, Wu W, Zhou M, et al. Improving Search Relevance for Short Queries in Community Question Answering[C]//Proceedings of the 7th ACM International Conference on Web Search and Data Mining. ACM, 2014: 43-52.
[20] Fan Y X, Pang L, Hou J P, et al. MatchZoo: A Toolkit for Deep Text Matching[OL]. arXiv Preprint, arXiv:1707.07270, 2017.
[21] Guo J F, Fan Y X, Ai Q Y, et al. A Deep Relevance Matching Model for Ad-hoc Retrieval[C]//Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 2016: 55-64.
[22] Pang L, Lan Y Y, Guo J F, et al. Text Matching as Image Recognition[OL]. arXiv Preprint, arXiv: 1602.06359, 2016.
[23] Glorot X, Bordes A, Bengio Y. Deep Sparse Rectifier Neural Networks[J]. Journal of Machine Learning Research, 2011,15:315-323.
[24] Aghdam H H, Heravi E J. Guide to Convolutional Neural Networks: A Practical Application to Traffic-Sign Detection and Classification[M]. Springer, 2017.
[25] Kalchbrenner N, Grefenstette E, Blunsom P, et al. A Convolutional Neural Network for Modelling Sentences[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL, 2014: 655-665.
[26] Fleiss J L, Cohen J. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability[J]. Educational and Psychological Measurement, 1973,33(3):613-619.
doi: 10.1177/001316447303300309
[27] Burges C J C. From RankNet to LambdaRank to LambdaMART: An Overview[R/OL].[2010-08-02]. https://www.microsoft.com/en-us/research/uploads/prod/2016/02/MSR-TR-2010-82.pdf.
[28] Sanderson M. Test Collection Based Evaluation of Information Retrieval Systems[J]. Foundations and Trends in Information Retrieval, 2010,4(4):247-375.
doi: 10.1561/1500000009
[29] Clarke C L A, Kolla M, Cormack G V, et al. Novelty and Diversity in Information Retrieval Evaluation[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2008: 659-666.
[30] Chapelle O, Ji S H, Liao C Y, et al. Intent-based Diversification of Web Search Results: Metrics and Algorithms[J]. Information Retrieval, 2011,14(6):572-592.
doi: 10.1007/s10791-011-9167-7
[1] Zhou Zeyu,Wang Hao,Zhao Zibo,Li Yueyan,Zhang Xiaoqin. Construction and Application of GCN Model for Text Classification with Associated Information[J]. 数据分析与知识发现, 2021, 5(9): 31-41.
[2] Zhao Danning,Mu Dongmei,Bai Sen. Automatically Extracting Structural Elements of Sci-Tech Literature Abstracts Based on Deep Learning[J]. 数据分析与知识发现, 2021, 5(7): 70-80.
[3] Xu Yuemei, Wang Zihou, Wu Zixin. Predicting Stock Trends with CNN-BiLSTM Based Multi-Feature Integration Model[J]. 数据分析与知识发现, 2021, 5(7): 126-138.
[4] Huang Mingxuan,Jiang Caoqing,Lu Shoudong. Expanding Queries Based on Word Embedding and Expansion Terms[J]. 数据分析与知识发现, 2021, 5(6): 115-125.
[5] Zhong Jiawa,Liu Wei,Wang Sili,Yang Heng. Review of Methods and Applications of Text Sentiment Analysis[J]. 数据分析与知识发现, 2021, 5(6): 1-13.
[6] Zhang Guobiao,Li Jie. Detecting Social Media Fake News with Semantic Consistency Between Multi-model Contents[J]. 数据分析与知识发现, 2021, 5(5): 21-29.
[7] Chang Chengyang,Wang Xiaodong,Zhang Shenglei. Polarity Analysis of Dynamic Political Sentiments from Tweets with Deep Learning Method[J]. 数据分析与知识发现, 2021, 5(3): 121-131.
[8] Feng Yong,Liu Yang,Xu Hongyan,Wang Rongbing,Zhang Yonggang. Recommendation Model Incorporating Neighbor Reviews for GRU Products[J]. 数据分析与知识发现, 2021, 5(3): 78-87.
[9] Hu Haotian,Ji Jinfeng,Wang Dongbo,Deng Sanhong. An Integrated Platform for Food Safety Incident Entities Based on Deep Learning[J]. 数据分析与知识发现, 2021, 5(3): 12-24.
[10] Zhang Qi,Jiang Chuan,Ji Youshu,Feng Minxuan,Li Bin,Xu Chao,Liu Liu. Unified Model for Word Segmentation and POS Tagging of Multi-Domain Pre-Qin Literature[J]. 数据分析与知识发现, 2021, 5(3): 2-11.
[11] Lv Xueqiang,Luo Yixiong,Li Jiaquan,You Xindong. Review of Studies on Detecting Chinese Patent Infringements[J]. 数据分析与知识发现, 2021, 5(3): 60-68.
[12] Cheng Bin,Shi Shuicai,Du Yuncheng,Xiao Shibin. Keyword Extraction for Journals Based on Part-of-Speech and BiLSTM-CRF Combined Model[J]. 数据分析与知识发现, 2021, 5(3): 101-108.
[13] Li Danyang, Gan Mingxin. Music Recommendation Method Based on Multi-Source Information Fusion[J]. 数据分析与知识发现, 2021, 5(2): 94-105.
[14] Yu Chuanming, Zhang Zhengang, Kong Lingge. Comparing Knowledge Graph Representation Models for Link Prediction[J]. 数据分析与知识发现, 2021, 5(11): 29-44.
[15] Han Pu, Zhang Wei, Zhang Zhanpeng, Wang Yuxin, Fang Haoyu. Sentiment Analysis of Weibo Posts on Public Health Emergency with Feature Fusion and Multi-Channel[J]. 数据分析与知识发现, 2021, 5(11): 68-79.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626,82624938   E-mail:jishu@mail.las.ac.cn