Please wait a minute...
Data Analysis and Knowledge Discovery  2024, Vol. 8 Issue (5): 127-138    DOI: 10.11925/infotech.2096-3467.2023.0242
Current Issue | Archive | Adv Search |
Evaluating Innovation Quality of Academic Papers——Case Study of Pluripotent Stem Cells
Wang Xuefeng(),Yu Huiyan,Zheng Sijia,Lei Ming
School of Management, Beijing Institute of Technology, Beijing 100081, China
Download: PDF (1193 KB)   HTML ( 7 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This study constructs an evaluation model for academic paper innovation quality. It explores a new method combining quantitative and qualitative approaches and promotes the progressive innovation of scientific research. [Methods] Balancing the innovative novelty and impact characteristics, we utilized the Doc2Vec algorithm to convert unstructured textual content into a vector space model. Then, we used cosine similarity to measure text content’s similarity. Simultaneously, we constructed a calculation method for the innovation impact index using the local citation network of the paper under evaluation. Third, we mapped the novelty and impact measurements onto a two-dimensional scatter plot. Finally, we constructed a model for evaluating the innovation quality of academic papers based on regional division. [Results] Empirical results on pluripotent stem cell technology showed that the proposed method is consistent with the F1000 recommendation results and can partly compensate for the deficiencies in the current evaluation of the innovation quality of academic papers. [Limitations] We only discussed the impacts of academic papers’ novelty and innovation. There are many other factors influencing the quality of academic paper innovation. [Conclusions] Our new model can provide quantitative data support for qualitative peer review and represents a beneficial exploration of quantitative evaluation of the innovation quality of academic papers.
Key words： Innovation Quality      Novelty Index      Disruption Index      Doc2Vec      Pluripotent Stem Cell     
Received: 23 March 2023      Published: 12 September 2023
ZTFLH:  G353  
Fund:National Natural Science Foundation of China(72074020)
Corresponding Authors: Wang Xuefeng，ORCID：0000-0002-4857-6944，E-mail：wxf5122@bit.edu.cn。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Xuefeng Wang
Huiyan Yu
Sijia Zheng
Ming Lei

Cite this article:

Wang Xuefeng, Yu Huiyan, Zheng Sijia, Lei Ming. Evaluating Innovation Quality of Academic Papers——Case Study of Pluripotent Stem Cells. Data Analysis and Knowledge Discovery, 2024, 8(5): 127-138.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2023.0242     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2024/V8/I5/127

Research Framework
参数 说明
dm 模型训练算法，dm=1，为分布式内存模型；dm=0，为分布式词袋模型（DBOW）
Vector_size 向量的维度
Window 用于预测的上下文词与预测的词之间的距离
Min_count 忽略总频数小于该参数的所有词
Workers 训练模型的并行线程数
Epochs 迭代次数
Training Parameter
Local Citation Network of Academic Papers to be Evaluated
Two-dimensional Scatter Diagram of Academic Papers in Novelty Index and Innovation Impact Index
排名 论文DOI 最大相似度 新颖性指数 被引
频次
1 10.1111/vde.12214 0.51 0.49 12
2 10.1117/1.NPh.2.3.031204 0.52 0.48 18
3 10.1371/journal.pone.0126259 0.52 0.48 72
4 10.1093/hmg/ddv345 0.52 0.48 15
5 10.1167/iovs.15-17477 0.53 0.47 10
6 10.1093/schbul/sbu032 0.53 0.47 32
7 10.1016/j.toxicon.2015.04.015 0.53 0.47 15
8 10.1161/JAHA.115.002146 0.54 0.47 60
9 10.1371/journal.pone.0116892 0.54 0.46 30
10 10.1167/iovs.15-17251 0.54 0.46 24
Top10 Academic Papers in Novelty Index
排名 论文DOI 创新影响
指数		 被引
频次		 参考文献
数量
1 10.1074/jbc.M114.601401 0.99 27 84
2 10.1101/cshperspect.a017152 0.99 18 113
3 10.1007/s12265-015-9606-8 0.99 23 121
4 10.1007/s13277-015-3152-5 0.99 42 33
5 10.1371/journal.pone.0132566 0.99 37 76
6 10.1016/j.molcel.2015.04.012 0.99 21 23
7 10.2174/1389450116666150113121054 0.99 153 80
8 10.1016/j.jvs.2014.04.067 0.99 11 42
9 10.1159/000430374 0.99 11 35
10 10.1016/j.cmet.2015.07.015 0.99 30 51
Top10 Academic Papers in Innovation Impact Index
Innovative Quality Evaluation for Academic Papers on Pluripotent Stem Cells
序号 待评价论文DOI 新颖性指数 创新影响指数 F1000因子 领域相对排名
1 10.1016/j.stem.2015.07.009 0.41 0.97 9.5 Top 2% in Developmental Biology
2 10.1016/j.stem.2015.01.003 0.39 0.97 5.1 Top 2% in Cell Biology
3 10.1038/nature14217 0.40 0.96 9.5 Top 2% in Developmental Biology
4 10.1073/pnas.1418845112 0.45 0.96 9.3 Top 5% in Development Biology
5 10.1038/labinvest.2014.104 0.39 0.96 4.7 Top 5% in Development Biology
6 10.1038/ng.3380 0.42 0.96 9.4 Top 1% in Genomics & Genetics
7 10.1038/nbt.3392 0.39 0.96 14.8 Top 1% in Developmental Biology
8 10.1038/ncomms8307 0.41 0.95 9.5 Top 1% in Genomics & Genetics
9 10.1001/jama.2015.0894 0.40 0.95 14.3 Top 1% in Genomics & Genetics
10 10.1038/nature14465 0.39 0.95 14.6 Top 1% in Cell Biology
11 10.1371/journal.pone.0116892 0.46 0.95 4.7 Top 5% in Development Biology
12 10.7554/eLife.09571 0.39 0.95 9.4 Top 2% in Developmental Biology
13 10.1038/ncb3200 0.42 0.94 9.4 Top 2% in Developmental Biology
14 10.1038/mp.2014.141 0.39 0.94 9.5 Top 1% in Genomics & Genetics
15 10.1038/ncomms9687 0.39 0.94 4.8 Top 2% in Cell Biology
16 10.1016/j.devcel.2015.06.021 0.40 0.94 9.5 Top 2% in Developmental Biology
17 10.1038/srep08883 0.39 0.94 9.9 Top 2% in Developmental Biology
18 10.1016/j.cell.2015.06.034 0.41 0.93 10.5 Top 0.5% in Neuroscience
19 10.1242/dev.108266 0.46 0.93 9.4 Top 2% in Developmental Biology
20 10.1083/jcb.201405110 0.40 0.93 4.9 Top 2% in Cell Biology
Evaluation Results of High Innovation Quality Academic Papers and F1000 Recommendation Information
序号 待评价论文DOI 新颖性指数 创新影响指数 F1000因子 发表时间 评价时间
1 10.1038/nature15695 0.42 0.90 57.1 2015.10.22 2015.12.01；2016.10.06；2019.02.20
2 10.1016/j.cell.2015.03.025 0.39 0.85 38.1 2015.08.09 2016.06.14；2017.3.20
3 10.1038/nature14215 0.41 0.85 18.8 2015.05.14 2015.08.27；2015.09.07
4 10.1016/j.stem.2015.07.006 0.39 0.83 19.1 2015.08.06 2015.08.16；2015.09.01
5 10.1016/j.cell.2014.12.013 0.39 0.93 5.6 2015.06.15 2015.12.10
6 10.1038/ncomms8711 0.39 0.92 9.4 2015.06.10 2018.07.23
7 10.1016/j.stem.2014.10.020 0.45 0.91 9.3 2015.07.08 2015.08.11
8 10.1038/NMETH.3312 0.42 0.91 10.7 2015.08.12 2017.05.17
9 10.1038/nature14425 0.39 0.90 18.8 2015.07.04 2017.08.02
10 10.1038/nn.4004 0.39 0.87 18.6 2015.04.27 2015.06.11
11 10.1038/nature14973 0.39 0.82 19.8 2015.09.03 2015.09.09
Evaluation Results of Academic Papers to be Evaluated and F1000 Recommendation Information
[1] 刘萌, 赵蔚. 科研项目绩效评价中存在的问题与对策研究[J]. 农业科技管理, 2018, 37(6): 76-78.
[1] (Liu Meng, Zhao Wei. Research on Problems and Countermeasures in Performance Evaluation of Scientific Research Project[J]. Management of Agricultural Science and Technology, 2018, 37(6): 76-78.)
[2] 国务院办公厅. 国务院办公厅印发《关于完善科技成果评价机制的指导意见》[EB/OL]. (2021-08-02). [2022-04-06]. http://www.gov.cn/xinwen/2021-08/02/content_5629039.htm.
[2] (The State Council. Guiding Opinions of the General Office of the State Council on Improving the Evaluation Mechanism of Scientific and Technological Achievements[EB/OL]. (2021-08-02). [2022-04-06]. http://www.gov.cn/xinwen/2021-08/02/content_5629039.htm.)
[3] 李贺, 杜杏叶. 基于知识元的学术论文内容创新性智能化评价研究[J]. 图书情报工作, 2020, 64(1): 93-104.
doi: 10.13266/j.issn.0252-3116.2020.01.012
[3] (Li He, Du Xingye. Research on Intelligent Evaluation for the Content Innovation of Academic Papers[J]. Library and Information Service, 2020, 64(1): 93-104.)
doi: 10.13266/j.issn.0252-3116.2020.01.012
[4] 钱玲飞, 贺婉莹, 杨建林. 论文学术创新力特征指标体系研究[J]. 情报科学, 2021, 39(1): 56-64.
[4] (Qian Lingfei, He Wanying, Yang Jianlin. The Characteristic Index System of Academic Innovation Ability[J]. Information Science, 2021, 39(1): 56-64.)
[5] 周露阳. 论审评学术论文创新因素的指标体系[J]. 编辑学报, 2006, 18(1): 68-70.
[5] (Zhou Luyang. Index System for Identifying Innovation Factors in Academic Papers[J]. Acta Editologica, 2006, 18(1): 68-70.)
[6] 钱佳佳, 罗卓然, 陆伟. 基于问题-方法组合的科技论文新颖性度量与创新类型识别[J]. 图书情报工作, 2021, 65(14): 82-89.
doi: 10.13266/j.issn.0252-3116.2021.14.010
[6] (Qian Jiajia, Luo Zhuoran, Lu Wei. Novelty Measurement and Innovation Type Identification of Scientific Literature Based on Question-Method Combination[J]. Library and Information Service, 2021, 65(14): 82-89.)
doi: 10.13266/j.issn.0252-3116.2021.14.010
[7] 逯万辉, 苏金燕, 余倩. 学术成果主题新颖性与学术引用的相关关系研究[J]. 情报资料工作, 2018(6): 68-73.
[7] (Lu Wanhui, Su Jinyan, Yu Qian. Research on Correlation of Academic Achievement Theme Novelty and Academic Citation[J]. Information and Documentation Services, 2018(6): 68-73.)
[8] 罗卓然, 王玉琦, 钱佳佳, 等. 学术论文创新性评价研究综述[J]. 情报学报, 2021, 40(7): 780-790.
[8] (Luo Zhuoran, Wang Yuqi, Qian Jiajia, et al. Research Review on Innovation Evaluation of Academic Papers[J]. Journal of the China Society for Scientific and Technical Information, 2021, 40(7): 780-790.)
[9] Amabile T M. The Social Psychology of Creativity: A Componential Conceptualization[J]. Journal of Personality and Social Psychology, 1983, 45(2): 357-376.
[10] 谢珍, 马建霞, 胡文静. 面向代表作评价的学术论文创新性测度方法[J]. 情报理论与实践, 2022, 45(7): 81-88.
doi: 10.16353/j.cnki.1000-7490.2022.07.012
[10] (Xie Zhen, Ma Jianxia, Hu Wenjing. The Innovation Measurement Method for Academic Papers Oriented to the Evaluation of Representative Works[J]. Information Studies: Theory & Application, 2022, 45(7): 81-88.)
doi: 10.16353/j.cnki.1000-7490.2022.07.012
[11] Shibayama S, Wang J. Measuring Originality in Science[J]. Scientometrics, 2020, 122(1): 409-427.
doi: 10.1007/s11192-019-03263-0
[12] Lee F. Recombinant Uncertainty in Technological Search[J]. Management Science, 2001, 47(1): 117-132.
[13] Uzzi B, Mukherjee S, Stringer M, et al. Atypical Combinations and Scientific Impact[J]. Science, 2013, 342(6157): 468-472.
doi: 10.1126/science.1240474 pmid: 24159044
[14] He Y J, Luo J X. Novelty, Conventionality, and Value of Invention[M]// Gero J. Design Computing and Cognition’16. Cham: Springer, 2017: 23-38.
[15] Wang J, Veugelers R, Stephan P. Bias Against Novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators[J]. Research Policy, 2017, 46(8): 1416-1436.
[16] 沈阳. 一种基于关键词的创新度评价方法[J]. 情报理论与实践, 2007, 30(1): 125-127.
[16] (Shen Yang. A Keyword-Based Innovation Evaluation Method[J]. Information Studies: Theory & Application, 2007, 30(1): 125-127.)
[17] 杨建林, 钱玲飞. 基于关键词对逆文档频率的主题新颖度度量方法[J]. 情报理论与实践, 2013, 36(3): 99-102.
[17] (Yang Jianlin, Qian Lingfei. Theme Novelty Measurement Based on Inverse Document Frequency of Keyword Pairs[J]. Information Studies: Theory & Application, 2013, 36(3): 99-102.)
[18] 杨京, 王芳, 白如江. 一种基于研究主题对比的单篇学术论文创新力评价方法[J]. 图书情报工作, 2018, 62(17): 75-83.
doi: 10.13266/j.issn.0252-3116.2018.17.010
[18] (Yang Jing, Wang Fang, Bai Rujiang. A Method to Evaluate Academic Papers’ Innovation Based on the Research Theme Comparing[J]. Library and Information Service, 2018, 62(17): 75-83.)
doi: 10.13266/j.issn.0252-3116.2018.17.010
[19] 任海英, 王德营, 王菲菲. 主题词组合新颖性与论文学术影响力的关系研究[J]. 图书情报工作, 2017, 61(9): 87-93.
doi: 10.13266/j.issn.0252-3116.2017.09.011
[19] (Ren Haiying, Wang Deying, Wang Feifei. Relationship Between Novelty of Key-Term Combinations and Papers’ Scientific Impact[J]. Library and Information Service, 2017, 61(9): 87-93.)
doi: 10.13266/j.issn.0252-3116.2017.09.011
[20] Tsai F S, Zhang Y. D2S: Document-to-Sentence Framework for Novelty Detection[J]. Knowledge and Information Systems, 2011, 29(2): 419-433.
[21] 逯万辉, 谭宗颖. 学术成果主题新颖性测度方法研究——基于Doc2Vec和HMM算法[J]. 数据分析与知识发现, 2018, 2(3): 22-29.
[21] (Lu Wanhui, Tan Zongying. Measuring Novelty of Scholarly Articles[J]. Data Analysis and Knowledge Discovery, 2018, 2(3): 22-29.)
[22] Catalini C, Lacetera N, Oettl A. The Incidence and Role of Negative Citations in Science[J]. PNAS, 2015, 112(45): 13823-13826.
doi: 10.1073/pnas.1502280112 pmid: 26504239
[23] Hemmat Esfe M, Wongwises S, Asadi A, et al. Mandatory and Self-Citation; Types, Reasons, Their Benefits and Disadvantages[J]. Science and Engineering Ethics, 2015, 21(6): 1581-1585.
doi: 10.1007/s11948-014-9598-9 pmid: 25398506
[24] 韩毅, 夏慧, 童迎. 引用结构视角下单篇学术文献影响力测度的ID指数[J]. 情报学报, 2013, 32(9): 968-975.
[24] (Han Yi, Xia Hui, Tong Ying. ID Index to Measure the Impact of Single Academic Papers in the Context of Citation Structure[J]. Journal of the China Society for Scientific and Technical Information, 2013, 32(9): 968-975.)
[25] Funk R J, Owen-Smith J. A Dynamic Network Measure of Technological Change[J]. Management Science, 2017, 63(3): 791-817.
[26] Wu L F, Wang D S, Evans J A. Large Teams Develop and Small Teams Disrupt Science and Technology[J]. Nature, 2019, 566(7744): 378-382.
[27] 陈靖元. 基于词语权重分析的中文文本相似检测技术研究[D]. 郑州: 郑州大学, 2021.
[27] (Chen Jingyuan. Research on Chinese Text Similarity Detection Technology Based on Word Weight Analysis[D]. Zhengzhou: Zhengzhou University, 2021.)
[28] Zhang Y, Tsai F S. Chinese Novelty Mining[C]// Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. 2009: 1561-1570.
[29] 郭丽娜, 李星琛, 左文革, 等. 颠覆性研究文献计量识别方法述评[J]. 数字图书馆论坛, 2020(3): 17-24.
[29] (Guo Lina, Li Xingchen, Zuo Wenge, et al. A Review of Disruptive Work and Its Bibliometric Identification Methods[J]. Digital Library Forum, 2020(3): 17-24.)
[30] 秦丽颖, 张瑞, 任晓琳, 等. 三维培养人多能干细胞的研究与进展[J]. 中国组织工程研究, 2019, 23(17): 2753-2761.
[30] (Qin Liying, Zhang Rui, Ren Xiaolin, et al. Three-Dimensional Culture of Human Pluripotent Stem Cells[J]. Chinese Journal of Tissue Engineering Research, 2019, 23(17): 2753-2761.)
[31] 姬小利, 王敏, 李玲玲, 等. 诱导多能干细胞研究的文献计量学分析及启示[J]. 中华医学科研管理杂志, 2017, 30(6): 459-463.
[31] (Ji Xiaoli, Wang Min, Li Lingling, et al. Bibliometrics Analysis of Literature Based on Induced Pluripotent Stem Cells and Its Implication[J]. Chinese Journal of Medical Science Research Management, 2017, 30(6): 459-463.)
[32] 何林, 白晋伟, 苏伟. 基于Scopus数据分析多能干细胞研究领域的文章、作者及高影响力机构[J]. 中国组织工程研究, 2020, 24(7): 1144-1148.
[32] (He Lin, Bai Jinwei, Su Wei. A Bibliometric Analysis on Articles, Authors and High-Impact Institutions in the Field of Pluripotent Stem Cells Based on Data from Scopus[J]. Chinese Journal of Tissue Engineering Research, 2020, 24(7): 1144-1148.)
[33] Lin C L, Ho Y S. A Bibliometric Analysis of Publications on Pluripotent Stem Cell Research[J]. Cell Journal, 2015, 17(1): 59-70.
pmid: 25870835
[34] Bornmann L, Tekles A. Disruption Index Depends on Length of Citation Window[J]. Información y Comunicación Biomédica, DOI: https://doi.org/10.3145/epi.2019.mar.07.
[35] Bornmann L, Leydesdorff L. The Validation of (Advanced) Bibliometric Indicators Through Peer Assessments: A Comparative Study Using Data from InCites and F1000[J]. Journal of Informetrics, 2013, 7(2): 286-291.
[36] Faculty Opinions[EB/OL]. [2022-04-06]. https://facultyopinions.com/search/articles.
[37] Bornmann L. Interrater Reliability and Convergent Validity of F1000Prime Peer Review[J]. Journal of the Association for Information Science and Technology, 2015, 66(12): 2415-2426.
[38] Waltman L, Costas R. F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison with Citations[J]. Journal of the Association for Information Science and Technology, 2014, 65(3): 433-445.
[1] Jia Xiaoting,Wang Mingyang,Cao Yu. Automatic Abstracting of Chinese Document with Doc2Vec and Improved Clustering Algorithm[J]. 数据分析与知识发现, 2018, 2(2): 86-95.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn