1Wuhan Library, Chinese Academy of Sciences, Wuhan 430071, China 2Department of Library, Information and Archives Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China
[Objective] This paper proposes a method to evaluate the consistency of scholarly journal article reviewers. [Methods] We developed a consistency index based on the knowledge from reviews and bibliometric data. Then, we conducted hypothesis test to examine whether experts with higher consistency scores make a more accurate evaluation of the paper. [Results] We found high-consistency experts could identify papers with academic community recognition, which was also maintained over time. [Limitations] The proposed consistency index could not replace journal editors in selecting experts, however, it helps to make reviewer selection efficiently and effectively. [Conclusions] It is feasible to calculate the consistency index based on historical data to select reviewers of scholarly articles.
( Xu Zhiying. Research Progress of Peer Review of Scientific Articles[J]. Chinese Journal of Scientific and Technical Periodicals, 2014,25(11):1355-1359.)
[2]
Ware M. Peer Review: Benefits, Perceptions and Alternatives[M]. Publishing Research Consortium, 2008.
[3]
Kelly J, Sadeghieh T, Adeli K. Peer Review in Scientific Publications: Benefits, Critiques, & a Survival Guide[J]. The Journal of the International Federation of Clinical Chemistry and Laboratory Medicine, 2014,25(3):227-243.
( Liu Chenxia, Wei Xiuju, Wang Liu, et al. Empirical Quantitative Research on the Influencing Factors of Academic Journals[J]. Journal of Library and Information Science in Agriculture, 2019,31(7):66-73.)
[5]
Froman R D. Blinded Review Revisited[J]. Research in Nursing & Health, 2010,33(4):273-275.
doi: 10.1002/nur.20385
( He Jie, Wang Chenghong, Liu Ke. Some Ideas on Evaluating the Review Activities of Peer-reviewers[J]. Bulletin of National Natural Science Foundation of China, 2004(1):49-52.)
[7]
刘鲁宁. 科技项目同行评议体系反评估模型分析与设计[D]. 哈尔滨: 哈尔滨工业大学, 2007.
[7]
( Liu Luning. The Analysis and Design of the Meta-assessment Model of Technology Project Peer-review System[D]. Harbin: Harbin Institute of Technology, 2007.)
[8]
刘津. 基于层次分析法的同行评议专家反评估模型研究[D]. 天津: 河北工业大学, 2015.
[8]
( Liu Jin. Research on Evaluation Model of Anti Expert Peer Review based on AHP[D]. Tianjin: Hebei University of Technology, 2015.)
( Yang Sujuan. Research on Construction of Anti-Evaluation System of Technology Projects Peer Review Evaluation Experts[D]. Shenyang: Shenyang Ligong University, 2009.)
( Li Guangwen, Wu Da. Empirical Research on Peer Review Experts’ Anti-assessment[J]. Tianjin Science & Technology, 2011,38(4):92-94.)
[12]
Federico B, Francisco G, Flaminio S. The F3-index. Valuing Reviewers for Scholarly Journals[J]. Journal of Informetrics, 2019,13(1):78-86.
doi: 10.1016/j.joi.2018.11.007
pmid: WOS:000460550800006
[13]
Callier F M, Desoer C A. Linear System Theory[M]. Springer Science & Business Media, 2012.
[14]
Zou K H, Tuncali K, Silverman S G. Correlation and Simple Linear Regression[J]. Radiology, 2003,227(3):617-622.
doi: 10.1148/radiol.2273011499
( Liao Jianqiao, Wen Peng, Hu Lingfang. The Research on Evaluation Criterion Concerning Academic Manuscript in China[J]. Studies in Science of Science, 2010,28(8):1128-1134.)
[16]
Lance C E, Butts M M, Michels L C. The Sources of Four Commonly Reported Cutoff Criteria-What did They Really Say?[J]. Organizational Research Methods, 2006,9(2):202-220.
doi: 10.1177/1094428105284919
