Please wait a minute...
Data Analysis and Knowledge Discovery  2021, Vol. 5 Issue (4): 115-122    DOI: 10.11925/infotech.2096-3467.2020.0770
Current Issue | Archive | Adv Search |
Evaluating Consistency of Scholarly Article Reviewers
Yue Mingliang1,Li Fushan1(),Tang Hongbo1,Lv Xinhua1,Ma Tingcan1,2
1Wuhan Library, Chinese Academy of Sciences, Wuhan 430071, China
2Department of Library, Information and Archives Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China
Download: PDF (871 KB)   HTML ( 8 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper proposes a method to evaluate the consistency of scholarly journal article reviewers. [Methods] We developed a consistency index based on the knowledge from reviews and bibliometric data. Then, we conducted hypothesis test to examine whether experts with higher consistency scores make a more accurate evaluation of the paper. [Results] We found high-consistency experts could identify papers with academic community recognition, which was also maintained over time. [Limitations] The proposed consistency index could not replace journal editors in selecting experts, however, it helps to make reviewer selection efficiently and effectively. [Conclusions] It is feasible to calculate the consistency index based on historical data to select reviewers of scholarly articles.
Key words： Research Paper Review      Expert      Consistency Evaluation      Hypothesis Test     
Received: 06 August 2020      Published: 29 October 2020
ZTFLH:  分类号: G350  
Fund:Youth Innovation Team Project of the Chinese Academy of Sciences Literature and Information Center(Y8KZ491);Literature and Information Capacity Building Project of Chinese Academy of Sciences(Y9290001);National Natural Science Foundation of China(71603252)
Corresponding Authors: Li Fushan     E-mail: lifs@whlib.ac.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Mingliang Yue
Fushan Li
Hongbo Tang
Xinhua Lv
Tingcan Ma

Cite this article:

Yue Mingliang,Li Fushan,Tang Hongbo,Lv Xinhua,Ma Tingcan. Evaluating Consistency of Scholarly Article Reviewers. Data Analysis and Knowledge Discovery, 2021, 5(4): 115-122.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2020.0770     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2021/V5/I4/115

Distribution of Reviewed Papers of 177 Effective Experts
Distribution of Number of Reviewers
Distribution of Publication Years
Distribution of Citation Times
Distribution of Consistency Index of 170 Experts
Recognition Ability of Experts with High/Low Consistency Index
比较项 0σ 0.1σ 0.2σ 0.3σ 0.4σ 0.5σ
低 高 低 高 低 高 低 高 低 高 低 高
样本量 38 26 45 19 48 16 51 13 54 10 59 5
均值 0.84 1.93 0.890 2.20 0.91 2.40 0.96 2.55 1.00 2.82 1.10 3.43
p值 0.00
Test Statistics of Different Consistency Indices
比较项 0σ 0.1σ 0.2σ 0.3σ 0.4σ 0.5σ
低 高 低 高 低 高 低 高 低 高 低 高
样本量 32 23 35 20 43 12 45 10 47 8 49 6
均值 0.84 1.79 0.88 1.88 0.92 2.39 0.97 2.45 0.99 2.75 1.00 3.00
p值 0.00
Test Statistics of Different Consistency Indices（m=2, n=8）
Recognition Ability Calculated from Historical Data and Verification Data
n 8 10 12
组别 前 后 前 后 前 后
样本量 12 7 8 7 6 6
均值 1.608 2.212 1.709 2.212 1.656 2.210
p值 0.482 0.613 1.000
Test Statistics of Historical Data and Verification Data
[1] 徐志英. 科学文章同行评议研究进展[J]. 中国科技期刊研究, 2014,25(11):1355-1359.
[1] ( Xu Zhiying. Research Progress of Peer Review of Scientific Articles[J]. Chinese Journal of Scientific and Technical Periodicals, 2014,25(11):1355-1359.)
[2] Ware M. Peer Review: Benefits, Perceptions and Alternatives[M]. Publishing Research Consortium, 2008.
[3] Kelly J, Sadeghieh T, Adeli K. Peer Review in Scientific Publications: Benefits, Critiques, & a Survival Guide[J]. The Journal of the International Federation of Clinical Chemistry and Laboratory Medicine, 2014,25(3):227-243.
[4] 刘晨霞, 魏秀菊, 王柳, 等. 学术期刊影响力影响因素实证研究[J]. 农业图书情报, 2019,31(7):66-73.
[4] ( Liu Chenxia, Wei Xiuju, Wang Liu, et al. Empirical Quantitative Research on the Influencing Factors of Academic Journals[J]. Journal of Library and Information Science in Agriculture, 2019,31(7):66-73.)
[5] Froman R D. Blinded Review Revisited[J]. Research in Nursing & Health, 2010,33(4):273-275.
doi: 10.1002/nur.20385
[6] 何杰, 王成红, 刘克. 对同行评议专家评议工作进行评估的一些思考[J]. 中国科学基金, 2004 (1):49-52.
[6] ( He Jie, Wang Chenghong, Liu Ke. Some Ideas on Evaluating the Review Activities of Peer-reviewers[J]. Bulletin of National Natural Science Foundation of China, 2004(1):49-52.)
[7] 刘鲁宁. 科技项目同行评议体系反评估模型分析与设计[D]. 哈尔滨: 哈尔滨工业大学, 2007.
[7] ( Liu Luning. The Analysis and Design of the Meta-assessment Model of Technology Project Peer-review System[D]. Harbin: Harbin Institute of Technology, 2007.)
[8] 刘津. 基于层次分析法的同行评议专家反评估模型研究[D]. 天津: 河北工业大学, 2015.
[8] ( Liu Jin. Research on Evaluation Model of Anti Expert Peer Review based on AHP[D]. Tianjin: Hebei University of Technology, 2015.)
[9] 杨素娟. 科技项目立项同行评议评审专家反评价体系构建研究[D]. 沈阳: 沈阳理工大学, 2009.
[9] ( Yang Sujuan. Research on Construction of Anti-Evaluation System of Technology Projects Peer Review Evaluation Experts[D]. Shenyang: Shenyang Ligong University, 2009.)
[10] 徐洪峰, 龙军, 张昊. 基于数据预处理的专家反评估方法[J]. 计算机工程, 2012,38(6):75-77.
[10] ( Xu Hongfeng, Long Jun, Zhang Hao. Anti-assessment Method for Expert Based on Data Preprocessing[J]. Computer Engineering, 2012,38(6):75-77.)
[11] 李光文, 吴达. 同行评议专家反评估实证研究[J]. 天津科技, 2011,38(4):92-94.
[11] ( Li Guangwen, Wu Da. Empirical Research on Peer Review Experts’ Anti-assessment[J]. Tianjin Science & Technology, 2011,38(4):92-94.)
[12] Federico B, Francisco G, Flaminio S. The F3-index. Valuing Reviewers for Scholarly Journals[J]. Journal of Informetrics, 2019,13(1):78-86.
doi: 10.1016/j.joi.2018.11.007 pmid: WOS:000460550800006
[13] Callier F M, Desoer C A. Linear System Theory[M]. Springer Science & Business Media, 2012.
[14] Zou K H, Tuncali K, Silverman S G. Correlation and Simple Linear Regression[J]. Radiology, 2003,227(3):617-622.
doi: 10.1148/radiol.2273011499
[15] 廖建桥, 文鹏, 胡凌芳. 我国学术论文评审标准研究[J]. 科学学研究, 2010,28(8):1128-1134.
[15] ( Liao Jianqiao, Wen Peng, Hu Lingfang. The Research on Evaluation Criterion Concerning Academic Manuscript in China[J]. Studies in Science of Science, 2010,28(8):1128-1134.)
[16] Lance C E, Butts M M, Michels L C. The Sources of Four Commonly Reported Cutoff Criteria-What did They Really Say?[J]. Organizational Research Methods, 2006,9(2):202-220.
doi: 10.1177/1094428105284919
[1] Pan Youneng,Ni Xiuli. Recommending Online Medical Experts with Labeled-LDA Model[J]. 数据分析与知识发现, 2020, 4(4): 34-43.
[2] Pengcheng Xu,Qiang Bi. Identifying Domain Experts Based on Knowledge Super-Network[J]. 数据分析与知识发现, 2019, 3(11): 89-98.
[3] Ye Guanghui,Xia Lixin. Review of Expert Retrieval and Expert Ranking Studies[J]. 数据分析与知识发现, 2017, 1(2): 1-10.
[4] Tan Xueqing,Zhang Lei,Huang Cuicui,Luo Lin. A Collaborative Filtering and Recommendation Algorithm Using Trust of Domain-Experts and Similarity[J]. 现代图书情报技术, 2016, 32(7-8): 101-109.
[5] Ma Tianyi,Zhang Pengzhu,Feng Haoyin. Knowledge Requirement Model for Online Outsourcing Tasks[J]. 现代图书情报技术, 2016, 32(3): 74-81.
[6] Li Gang, Ye Guanghui, Zhang Yan. Feature Recognition of Niche Expert——Empirical Analysis Based on MetaFilter Dataset[J]. 现代图书情报技术, 2015, 31(6): 71-77.
[7] Li Gang, Ye Guanghui. Research on Credibility Evaluation Mechanism of Experts Retrieval Under User's Control[J]. 现代图书情报技术, 2014, 30(7): 107-113.
[8] Li Zhenqing, Liu Jianyi, Wang Cong, Wu Xu. Research and Implementation of Peer-review Experts Selection System[J]. 现代图书情报技术, 2012, 28(5): 81-86.
[9] Zhang Xiaojuan, Lu Wei, Cheng Qikai. Application of PLSA on Expertise Identifying in the Field of Library and Information Science[J]. 现代图书情报技术, 2012, 28(2): 76-81.
[10] Dong Hui Xu Lei. Knowledge Representation in History Field Expert System Application Based on Ontology[J]. 现代图书情报技术, 2010, 26(7/8): 72-78.
[11] Zhao Hongbin,Lu Wei. The Study of Expert Research Field Automatic Recognition[J]. 现代图书情报技术, 2010, 26(2): 63-67.
[12] Sun Sufen,Luo Changshou,Wei Qingfeng. Design and Implementation of Automatic Question-answering System About Agricultural Operative Technology Based on Web[J]. 现代图书情报技术, 2009, 25(7-8): 70-74.
[13] Lu Wei,Zhao Haozhen. Enterprise Expert Search Based on Combing Document Weight[J]. 现代图书情报技术, 2008, 24(7): 38-42.
[14] Frédéric MARTIN. Dynamic Management of Digital Rights for Long-term Preservation:the Expert System Approach[J]. 现代图书情报技术, 2008, 24(1): 75-82.
[15] Song Yunlong . Study on Ontology-based Reference on the Web[J]. 现代图书情报技术, 2006, 1(3): 27-30.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn