|
|
Evaluating Consistency of Scholarly Article Reviewers |
Yue Mingliang1,Li Fushan1( ),Tang Hongbo1,Lv Xinhua1,Ma Tingcan1,2 |
1Wuhan Library, Chinese Academy of Sciences, Wuhan 430071, China 2Department of Library, Information and Archives Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China |
|
|
Abstract [Objective] This paper proposes a method to evaluate the consistency of scholarly journal article reviewers. [Methods] We developed a consistency index based on the knowledge from reviews and bibliometric data. Then, we conducted hypothesis test to examine whether experts with higher consistency scores make a more accurate evaluation of the paper. [Results] We found high-consistency experts could identify papers with academic community recognition, which was also maintained over time. [Limitations] The proposed consistency index could not replace journal editors in selecting experts, however, it helps to make reviewer selection efficiently and effectively. [Conclusions] It is feasible to calculate the consistency index based on historical data to select reviewers of scholarly articles.
|
Received: 06 August 2020
Published: 29 October 2020
|
|
Fund:Youth Innovation Team Project of the Chinese Academy of Sciences Literature and Information Center(Y8KZ491);Literature and Information Capacity Building Project of Chinese Academy of Sciences(Y9290001);National Natural Science Foundation of China(71603252) |
Corresponding Authors:
Li Fushan
E-mail: lifs@whlib.ac.cn
|
[1] |
徐志英. 科学文章同行评议研究进展[J]. 中国科技期刊研究, 2014,25(11):1355-1359.
|
[1] |
( Xu Zhiying. Research Progress of Peer Review of Scientific Articles[J]. Chinese Journal of Scientific and Technical Periodicals, 2014,25(11):1355-1359.)
|
[2] |
Ware M. Peer Review: Benefits, Perceptions and Alternatives[M]. Publishing Research Consortium, 2008.
|
[3] |
Kelly J, Sadeghieh T, Adeli K. Peer Review in Scientific Publications: Benefits, Critiques, & a Survival Guide[J]. The Journal of the International Federation of Clinical Chemistry and Laboratory Medicine, 2014,25(3):227-243.
|
[4] |
刘晨霞, 魏秀菊, 王柳, 等. 学术期刊影响力影响因素实证研究[J]. 农业图书情报, 2019,31(7):66-73.
|
[4] |
( Liu Chenxia, Wei Xiuju, Wang Liu, et al. Empirical Quantitative Research on the Influencing Factors of Academic Journals[J]. Journal of Library and Information Science in Agriculture, 2019,31(7):66-73.)
|
[5] |
Froman R D. Blinded Review Revisited[J]. Research in Nursing & Health, 2010,33(4):273-275.
doi: 10.1002/nur.20385
|
[6] |
何杰, 王成红, 刘克. 对同行评议专家评议工作进行评估的一些思考[J]. 中国科学基金, 2004 (1):49-52.
|
[6] |
( He Jie, Wang Chenghong, Liu Ke. Some Ideas on Evaluating the Review Activities of Peer-reviewers[J]. Bulletin of National Natural Science Foundation of China, 2004(1):49-52.)
|
[7] |
刘鲁宁. 科技项目同行评议体系反评估模型分析与设计[D]. 哈尔滨: 哈尔滨工业大学, 2007.
|
[7] |
( Liu Luning. The Analysis and Design of the Meta-assessment Model of Technology Project Peer-review System[D]. Harbin: Harbin Institute of Technology, 2007.)
|
[8] |
刘津. 基于层次分析法的同行评议专家反评估模型研究[D]. 天津: 河北工业大学, 2015.
|
[8] |
( Liu Jin. Research on Evaluation Model of Anti Expert Peer Review based on AHP[D]. Tianjin: Hebei University of Technology, 2015.)
|
[9] |
杨素娟. 科技项目立项同行评议评审专家反评价体系构建研究[D]. 沈阳: 沈阳理工大学, 2009.
|
[9] |
( Yang Sujuan. Research on Construction of Anti-Evaluation System of Technology Projects Peer Review Evaluation Experts[D]. Shenyang: Shenyang Ligong University, 2009.)
|
[10] |
徐洪峰, 龙军, 张昊. 基于数据预处理的专家反评估方法[J]. 计算机工程, 2012,38(6):75-77.
|
[10] |
( Xu Hongfeng, Long Jun, Zhang Hao. Anti-assessment Method for Expert Based on Data Preprocessing[J]. Computer Engineering, 2012,38(6):75-77.)
|
[11] |
李光文, 吴达. 同行评议专家反评估实证研究[J]. 天津科技, 2011,38(4):92-94.
|
[11] |
( Li Guangwen, Wu Da. Empirical Research on Peer Review Experts’ Anti-assessment[J]. Tianjin Science & Technology, 2011,38(4):92-94.)
|
[12] |
Federico B, Francisco G, Flaminio S. The F3-index. Valuing Reviewers for Scholarly Journals[J]. Journal of Informetrics, 2019,13(1):78-86.
doi: 10.1016/j.joi.2018.11.007
pmid: WOS:000460550800006
|
[13] |
Callier F M, Desoer C A. Linear System Theory[M]. Springer Science & Business Media, 2012.
|
[14] |
Zou K H, Tuncali K, Silverman S G. Correlation and Simple Linear Regression[J]. Radiology, 2003,227(3):617-622.
doi: 10.1148/radiol.2273011499
|
[15] |
廖建桥, 文鹏, 胡凌芳. 我国学术论文评审标准研究[J]. 科学学研究, 2010,28(8):1128-1134.
|
[15] |
( Liao Jianqiao, Wen Peng, Hu Lingfang. The Research on Evaluation Criterion Concerning Academic Manuscript in China[J]. Studies in Science of Science, 2010,28(8):1128-1134.)
|
[16] |
Lance C E, Butts M M, Michels L C. The Sources of Four Commonly Reported Cutoff Criteria-What did They Really Say?[J]. Organizational Research Methods, 2006,9(2):202-220.
doi: 10.1177/1094428105284919
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
|
Shared |
|
|
|
|
|
Discussed |
|
|
|
|