Please wait a minute...
Data Analysis and Knowledge Discovery  2020, Vol. 4 Issue (10): 37-46    DOI: 10.11925/infotech.2096-3467.2019.1301
Current Issue | Archive | Adv Search |
Improving Online Q&A Service with Deep Learning
Ding Heng(),Li Yingxuan
School of Information Management, Central China Normal University, Wuhan 430079, China
Download: PDF (934 KB)   HTML ( 7
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper develops a neural network model to improve the online questioning and answering services.[Methods] First, we retrieved and constructed our experimental dataset from Yahoo Answers and Yahoo! L6 platform. Then, we proposed a neural network model (CNMNN) based on semantic matching matrix,variable-size convolutional layer, and multiple layer perceptron. Finally, we compared the results our model with the MQ2QC、IBLM、DRMM and MatchPyramid methods. [Results] The proposed model was 45.0%, 38.7%, 33.4%, 34.8% and 52.9% higher than the best results on relevance metrics of nDCG@5, nDCG@10, nDCG@20, MRR and MAP. It also gained 31.5%, 23.6%, 25.5%, 38.1%, 36.9% and 30.7% improvements on diversity metrics of α-nDCG@5, α-nDCG@10, α-nDCG@20 and ERR-IA@5, ERR-IA@10 and ERR-IA@20.[Limitations] We did not include new method to further diversify the results.[Conclusions] The new CNMNN model can effectively calculate the semantic relevance between queries and natural language questions at phrase level. It also avoids the issue of feature signal compression due to hierarchical convolution operation.

Key wordsQuery Suggestion      Deep Learning      Community-based Question and Answering     
Received: 04 December 2019      Published: 09 November 2020
ZTFLH:  TP393  
Corresponding Authors: Ding Heng     E-mail: me@gmail.com

Cite this article:

Ding Heng,Li Yingxuan. Improving Online Q&A Service with Deep Learning. Data Analysis and Knowledge Discovery, 2020, 4(10): 37-46.

URL:

http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2019.1301     OR     http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2020/V4/I10/37

Sample of Query Suggestion for Community-based Question and Answering
Structure of CNMNN Neural Network Model
分值 判断规则
2 (C1) 用户可能会使用查询词搜索该问题
(C2) 用户可能会使用查询词的简单修改(通过同义替换修改一个词或词组)搜索该问题
1 (C3) 问题包含多个提问意图,且覆盖了查询词的提问意图
(C4) 用户可能会使用查询词的复杂修改(通过同义替换修改两个及以上的词或词组)搜索该问题
0 (C5) 查询词与自然语言问句无关
Scoring Rules of Manually Annotated Correlation
方法 nDCG@5 nDCG@10 nDCG@20 MRR MAP
MQ2QC 0.448 0.458 0.502 0.468 0.310
IBLM 0.447 0.470 0.503 0.481 0.334
DRMM 0.382 0.429 0.506 0.379 0.269
MatchPyramid 0.484 0.517 0.572 0.528 0.326
CNMNN 0.702 0.717 0.763 0.712 0.511
Comparison Results of Relevance Score
方法 nDCG@5 nDCG@10 nDCG@20 MRR MAP
CNMNN(SDF) 0.508 0.524 0.581 0.536 0.302
CNMNN(MF) 0.698 0.713 0.748 0.677 0.495
Comparison Results of Relevance Score
方法 ERR-IA α-nDCG
@5 @10 @20 @5 @10 @20
MQ2QC 0.122 0.140 0.153 0.407 0.412 0.479
IBLM 0.124 0.144 0.151 0.417 0.447 0.464
DRMM 0.079 0.109 0.136 0.427 0.440 0.468
MatchPyramid 0.014 0.125 0.139 0.475 0.484 0.538
CNMNN 0.163 0.178 0.192 0.656 0.663 0.703
Comparison Results of Diversity Score
方法 查询词:14th Amendment
MQ2QC [1] what is the significance of the 14th amendment
[2] was the 13th 14th amendments ratifed
[3] how does the 14th amendment violate states rights
IBLM [1] what things have violated the 14th amendment
[2] does anyone know what the 14th amendment is
[3] is the death penalty a violation of the 8th and 14th amendments
DRMM [1] who made the 13th amendment
[2] what is the importance of the third amendment
[3] what is the 14th amendment
MatchPyramid [1] what things have violated the 14th amendment
[2] what exactly is the 14th amendment
[3] who made the 13th amendment
CNMNN [1] what is the 14th amendment
[2] is abortion a violation of the 14th amendment to the constitution of the us
[3] what is a 14th amendment citizen
Top 3 Results of Query Suggestion Based on the Query “14th Amendment”
[1] 李亚楠, 王斌, 李锦涛. 搜索引擎查询推荐技术综述[J]. 中文信息学报, 2010,24(6):75-84.
[1] ( Li Ya’nan, Wang Bin, Li Jintao. A Survey of Query Suggestion in Search Engine[J]. Journal of Chinese Information Processing, 2010,24(6):75-84.)
[2] 孟玲玲. 基于WordNet的语义相似性度量及其在查询推荐中的应用研究[D]. 上海: 华东师范大学, 2014.
[2] ( Meng Lingling. Research on Semantic Similarity Metric Based on WordNet and Its Application in Query Suggestion[D]. Shanghai: East China Normal University, 2014.)
[3] Yang J M, Cai R, Jing F, et al. Search-based Query Suggestion[C]//Proceedings of the 17th ACM Conference on Information and Knowledge Management. 2008: 1439-1440.
[4] 季岚石. 基于搜索日志的查询推荐算法研究[D]. 长春: 吉林大学, 2013.
[4] ( Ji Lanshi. The Query Recommendation Algorithm Research Based on the Search Logs[D]. Changchun: Jilin University, 2013.)
[5] Ding H, Balog K. Generating Synthetic Data for Neural Keyword-to-Question Models[C]//Proceedings of the 4th ACM SIGIR International Conference on the Theory of Information Retrieval. 2018: 51-58.
[6] Xu J X, Croft W B. Quary Expansion Using Local and Global Document Analysis[J]. ACM SIGIR Forum, 2017,51(2):168-175.
doi: 10.1145/3130348.3130364
[7] Garigliotti D, Balog K. Generating Query Suggestions to Support Task-based Search[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2017: 1153-1156.
[8] Cao H H, Jiang D X, Pei J, et al. Context-aware Query Suggestion by Mining Click-through and Session Data[C]//Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2008: 875-883.
[9] Mei Q Z, Zhou D Y, Church K. Query Suggestion Using Hitting Time[C]//Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, 2008: 469-478.
[10] 张伟男. 社区型问答中问句检索关键技术研究[D]. 哈尔滨:哈尔滨工业大学, 2014.
[10] ( Zhang Weinan. Research on Key Techniques of Question Retrieval for Community Question Answering[D]. Harbin: Harbin Institute of Technology, 2014.)
[11] 刘欣, 席耀一, 王波, 等. WordNet和词向量相结合的句子检索方法[J]. 信息工程大学学报, 2017,18(4):486-491.
[11] ( Liu Xin, Xi Yaoyi, Wang Bo, et al. WordNet and Word Embedding Based Sentence Retrieval Method[J]. Journal of Information Engineering University, 2017,18(4):486-491.)
[12] Xue X B, Jeon J, Croft W B. Retrieval Models for Question and Answer Archives[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2008: 475-482.
[13] Zhou G, Cai L, Zhao J, et al. Phrase-based Translation Model for Question Retrieval in Community Question Answer Archives[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. 2011: 653-662.
[14] Ichikawa H, Hakoda K, Hashimoto T, et al. Efficient Sentence Retrieval Based on Syntactic Structure[C]//Proceedings of the COLING/ACL on Main Conference Poster Sessions. ACL, 2006: 399-406.
[15] Wang K, Ming Z Y, Chua T S, et al. A Syntactic Tree Matching Approach to Finding Similar Questions in Community-based QA Services[C]//Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2009: 187-194.
[16] Cai L, Zhou G Y, Liu K, et al. Learning the Latent Topics for Question Retrieval in Community QA[C]//Proceedings of the 5th International Joint Conference on Natural Language Processing. 2011: 273-281.
[17] Zhang K, Wu W, Wu H C, et al. Question Retrieval with High Quality Answers in Community Question Answering[C]//Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 2014: 371-380.
[18] Gao Y J, Chen L, Li R, et al. Mapping Queries to Questions: Towards Understanding Users’ Information Needs[C]//Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2013: 977-980.
[19] Wu H C, Wu W, Zhou M, et al. Improving Search Relevance for Short Queries in Community Question Answering[C]//Proceedings of the 7th ACM International Conference on Web Search and Data Mining. ACM, 2014: 43-52.
[20] Fan Y X, Pang L, Hou J P, et al. MatchZoo: A Toolkit for Deep Text Matching[OL]. arXiv Preprint, arXiv:1707.07270, 2017.
[21] Guo J F, Fan Y X, Ai Q Y, et al. A Deep Relevance Matching Model for Ad-hoc Retrieval[C]//Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 2016: 55-64.
[22] Pang L, Lan Y Y, Guo J F, et al. Text Matching as Image Recognition[OL]. arXiv Preprint, arXiv: 1602.06359, 2016.
[23] Glorot X, Bordes A, Bengio Y. Deep Sparse Rectifier Neural Networks[J]. Journal of Machine Learning Research, 2011,15:315-323.
[24] Aghdam H H, Heravi E J. Guide to Convolutional Neural Networks: A Practical Application to Traffic-Sign Detection and Classification[M]. Springer, 2017.
[25] Kalchbrenner N, Grefenstette E, Blunsom P, et al. A Convolutional Neural Network for Modelling Sentences[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL, 2014: 655-665.
[26] Fleiss J L, Cohen J. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability[J]. Educational and Psychological Measurement, 1973,33(3):613-619.
doi: 10.1177/001316447303300309
[27] Burges C J C. From RankNet to LambdaRank to LambdaMART: An Overview[R/OL].[2010-08-02]. https://www.microsoft.com/en-us/research/uploads/prod/2016/02/MSR-TR-2010-82.pdf.
[28] Sanderson M. Test Collection Based Evaluation of Information Retrieval Systems[J]. Foundations and Trends in Information Retrieval, 2010,4(4):247-375.
doi: 10.1561/1500000009
[29] Clarke C L A, Kolla M, Cormack G V, et al. Novelty and Diversity in Information Retrieval Evaluation[C]//Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2008: 659-666.
[30] Chapelle O, Ji S H, Liao C Y, et al. Intent-based Diversification of Web Search Results: Metrics and Algorithms[J]. Information Retrieval, 2011,14(6):572-592.
doi: 10.1007/s10791-011-9167-7
[1] Huang Lu,Zhou Enguo,Li Daifeng. Text Representation Learning Model Based on Attention Mechanism with Task-specific Information[J]. 数据分析与知识发现, 2020, 4(9): 111-122.
[2] Zhao Yang, Zhang Zhixiong, Liu Huan, Ding Liangping. Classification of Chinese Medical Literature with BERT Model[J]. 数据分析与知识发现, 2020, 4(8): 41-49.
[3] Xu Chenfei, Ye Haiying, Bao Ping. Automatic Recognition of Produce Entities from Local Chronicles with Deep Learning[J]. 数据分析与知识发现, 2020, 4(8): 86-97.
[4] Yu Chuanming, Wang Manyi, Lin Hongjun, Zhu Xingyu, Huang Tingting, An Lu. A Comparative Study of Word Representation Models Based on Deep Learning[J]. 数据分析与知识发现, 2020, 4(8): 28-40.
[5] Wang Xinyun,Wang Hao,Deng Sanhong,Zhang Baolong. Classification of Academic Papers for Periodical Selection[J]. 数据分析与知识发现, 2020, 4(7): 96-109.
[6] Jiao Qihang,Le Xiaoqiu. Generating Sentences of Contrast Relationship[J]. 数据分析与知识发现, 2020, 4(6): 43-50.
[7] Wang Mo,Cui Yunpeng,Chen Li,Li Huan. A Deep Learning-based Method of Argumentative Zoning for Research Articles[J]. 数据分析与知识发现, 2020, 4(6): 60-68.
[8] Deng Siyi,Le Xiaoqiu. Coreference Resolution Based on Dynamic Semantic Attention[J]. 数据分析与知识发现, 2020, 4(5): 46-53.
[9] Yu Chuanming,Yuan Sai,Zhu Xingyu,Lin Hongjun,Zhang Puliang,An Lu. Research on Deep Learning Based Topic Representation of Hot Events[J]. 数据分析与知识发现, 2020, 4(4): 1-14.
[10] Su Chuandong,Huang Xiaoxi,Wang Rongbo,Chen Zhiqun,Mao Junyu,Zhu Jiaying,Pan Yuhao. Identifying Chinese / English Metaphors with Word Embedding and Recurrent Neural Network[J]. 数据分析与知识发现, 2020, 4(4): 91-99.
[11] Liu Tong,Ni Weijian,Sun Yujian,Zeng Qingtian. Predicting Remaining Business Time with Deep Transfer Learning[J]. 数据分析与知识发现, 2020, 4(2/3): 134-142.
[12] Chuanming Yu,Haonan Li,Manyi Wang,Tingting Huang,Lu An. Knowledge Representation Based on Deep Learning:Network Perspective[J]. 数据分析与知识发现, 2020, 4(1): 63-75.
[13] Mengji Zhang,Wanyu Du,Nan Zheng. Predicting Stock Trends Based on News Events[J]. 数据分析与知识发现, 2019, 3(5): 11-18.
[14] Jingjing Pei,Xiaoqiu Le. Identifying Coordinate Text Blocks in Discourses[J]. 数据分析与知识发现, 2019, 3(5): 51-56.
[15] Zhixiong Zhang,Huan Liu,Liangping Ding,Pengmin Wu,Gaihong Yu. Identifying Moves of Research Abstracts with Deep Learning Methods[J]. 数据分析与知识发现, 2019, 3(12): 1-9.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626,82624938   E-mail:jishu@mail.las.ac.cn