Identifying User Satisfaction Levels and Evolution Patterns in Exploratory Search
Zhao Yiming1,2,3,Chen Zhan2,3,4,Zhang Fan2,5()
1The Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China 2School of Information Management, Wuhan University, Wuhan 430072, China 3Big Data Institute, Wuhan University, Wuhan 430072, China 4National Demonstration Center for Experimental Library and Information Science Education, Wuhan University, Wuhan 430072, China 5Center for Science, Technology & Education Assessment, Wuhan University, Wuhan 430072, China
[Objective] This paper identifies the user satisfaction levels in exploratory search and reveals the interaction and evolution between user satisfaction and reconstruction patterns of queries. [Methods] First, we retrieved the characteristics of user queries and their temporal sequences. Then, we used four supervised learning algorithms to predict user satisfaction levels. Third, we identified the interaction between user satisfaction and query reformulations. Finally, we developed new recommendation strategies for query reformulation in intelligent exploratory search assistance. [Results] We examined the proposed model with an open benchmark dataset, and the model’s prediction accuracy reached 74%, surpassing existing baseline models. There is a significant association between user satisfaction and query reformulation patterns. [Limitations] User satisfaction represents only one of the search perspectives. Future research should focus on constructing a comprehensive and unified description and classification system for users in exploratory search. [Conclusions] The proposed model further enhances the performance of the user satisfaction prediction. It provides theoretical support for intelligent search assistance strategy.
Marchionini G. Exploratory Search: From Finding to Understanding[J]. Communications of the ACM, 2006, 49(4): 41-46.
[2]
赵一鸣. 智能时代的搜索与问答服务创新研究[M]. 北京: 科学出版社, 2020.
[2]
(Zhao Yiming. Research on the Innovation of Search and Question Answering Service in the Intelligent Age[M]. Beijing: Science Press, 2020.)
[3]
Liu J Q, Sarkar S, Shah C. Identifying and Predicting the States of Complex Search Tasks[C]// Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 2020: 193-202.
[4]
Liu J Q, Yu R. State-Aware Meta-Evaluation of Evaluation Metrics in Interactive Information Retrieval[C]// Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021: 3258-3262.
(Zhao Yiming, Cheng Zong, Chen Yijin. Collaborative Analysis of Exploratory Search Path and Search Intention Conversion Path[J]. Information and Documentation Services, 2021, 42(6): 82-90.)
[6]
Lee H J, Lee J, Makara K A, et al. Does Higher Education Foster Critical and Creative Learners? An Exploration of Two Universities in South Korea and the USA[J]. Higher Education Research & Development, 2015, 34(1): 131-146.
[7]
Anderson L W, Krathwohl D R, Airasian P W, et al. A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives[M]. New York: Addison Wesley Longman, 2001.
[8]
Capra R, Arguello J. Using Trails to Support Users with Tasks of Varying Scope[C]// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019: 977-980.
[9]
Capra R, Arguello J, O’Brien H, et al. The Effects of Manipulating Task Determinability on Search Behaviors and Outcomes[C]// Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018: 445-454.
[10]
Choi B, Arguello J, Capra R, et al. OrgBox: A Knowledge Representation Tool to Support Complex Search Tasks[C]// Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. 2021: 219-228.
[11]
Crescenzi A, Capra R, Arguello J. Time Limits, Information Search and the Use of Search Assistance[C]// Proceedings of the 2017 Conference on Human Information Interaction and Retrieval. 2017: 349-352.
[12]
Kelly D. Methods for Evaluating Interactive Information Retrieval Systems with Users[J]. Foundations and Trends® in Information Retrieval, 2009, 3(1-2): 1-224.
doi: 10.1561/1500000012
[13]
Al-Maskari A, Sanderson M, Clough P. The Relationship Between IR Effectiveness Measures and User Satisfaction[C]// Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2007: 773-774.
[14]
Huffman S B, Hochster M. How Well does Result Relevance Predict Session Satisfaction?[C]// Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2007: 567-574.
[15]
Wang B, Liu J Q. Extracting the Implicit Search States from Explicit Behavioral Signals in Complex Search Tasks[J]. Proceedings of the Association for Information Science and Technology, 2021, 58(1): 854-856.
doi: 10.1002/pra2.v58.1
[16]
Kim Y, Hassan A, White R W, et al. Modeling Dwell Time to Predict Click-Level Satisfaction[C]// Proceedings of the 7th ACM International Conference on Web Search and Data Mining. 2014: 193-202.
[17]
Liu Y Q, Chen Y, Tang J H, et al. Different Users, Different Opinions: Predicting Search Satisfaction with Mouse Movement Information[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2015: 493-502.
[18]
Zhang F, Mao J X, Liu Y Q, et al. Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics[C]// Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020: 379-388.