|
|
|
| ChatGPT-Based Scientific Paper Entity Recognition: Performance Measurement and Availability Research |
Zhang Yingyi1,Zhang Chengzhi2( ),Zhou Yi1,Chen Bikun1 |
1School of Sociology, Soochow University, Suzhou 215123, China
2School of Economics & Management, Nanjing University of Science and Technology, Nanjing 210094, China |
|
|
|
|
Abstract [Objective] This paper aims to use a large language model for entity recognition tasks of academic papers. [Methods] We utilized ChatGPT, a large language model, as an entity recognition tool, a pseudo-label generation tool, and a training set generation tool. Then, we analyzed ChatGPT’s performance, price, and time for the tasks. [Results] The F1 of the ChatGPT-based method in all three perspectives is higher than that of the neural network baseline model trained with a small dataset. For example, the F1 from the perspective of entity recognition was 21.4% higher than the model trained by manually annotating 10 abstracts. The ChatGPT-based methods had stable performance on academic paper datasets in different disciplines. [Limitations] We only examined the new method with English academic paper abstract datasets. More research is needed to examine it with the Chinese datasets. [Conclusions] ChatGPT can identify entities from academic paper abstracts with little manually annotated data. The recognition results need to be further filtered to be applied to downstream tasks.
|
|
Received: 19 May 2023
Published: 12 September 2023
|
|
|
| Fund:The National Natural Science Foundation of China(72074113);The Youth Cross Research Team Project of Social Sciences of Soochow University |
|
Corresponding Authors:
Zhang Chengzhi，ORCID：0000-0001-9522-2914，E-mail： zhangcz@njust.edu.cn。
|
| [1] |
Heffernan K, Teufel S. Identifying Problems and Solutions in Scientific Text[J]. Scientometrics, 2018, 116(2): 1367-1382.
doi: 10.1007/s11192-018-2718-6
pmid: 30147202
|
| [2] |
Kovačević A, Konjović Z, Milosavljević B, et al. Mining Methodologies from NLP Publications: A Case Study in Automatic Terminology Recognition[J]. Computer Speech & Language, 2012, 26(2): 105-126.
|
| [3] |
Luo Z R, Lu W, He J G, et al. Combination of Research Questions and Methods: A New Measurement of Scientific Novelty[J]. Journal of Informetrics, 2022, 16(2): Article No. 101282.
|
| [4] |
马费成, 张帅. 我国图书情报领域新兴交叉学科发展探析[J]. 中国图书馆学报, 2023, 49(2): 4-14.
|
| [4] |
(Ma Feicheng, Zhang Shuai. The Development of Emerging Interdisciplines in Library and Information Science in China[J]. Journal of Library Science in China, 2023, 49(2): 4-14.)
|
| [5] |
Wadden D, Wennberg U, Luan Y, et al. Entity, Relation, and Event Extraction with Contextualized Span Representations[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. ACL, 2019: 5784-5789.
|
| [6] |
Zhong Z X, Chen D Q. A Frustratingly Easy Approach for Entity and Relation Extraction[C]// Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2021: 50-61.
|
| [7] |
Zhang H H, Ren F L. BERTatDE at SemEval-2020 Task 6: Extracting Term-Definition Pairs in Free Text Using Pre-Trained Model[C]// Proceedings of the 14th Workshop on Semantic Evaluation. 2020: 690-696.
|
| [8] |
Ding N, Chen Y L, Han X, et al. Prompt-Learning for Fine-Grained Entity Typing[OL]. arXiv Preprint, arXiv: 2108.10604.
|
| [9] |
Kan Z G, Feng L H, Yin Z Y, et al. A Unified Generative Framework Based on Prompt Learning for Various Information Extraction Tasks[OL]. arXiv Preprint, arXiv: 2209.11570.
|
| [10] |
李鸿鹏, 马博, 杨雅婷, 等. 基于槽位语义增强提示学习的篇章级事件抽取方法[J]. 计算机工程, 2023, 49(9):23-31.
doi: 10.19678/j.issn.1000-3428.0066170
|
| [10] |
(Li Hongpeng, Ma Bo, Yang Yating, et al. Document-Level Event Extraction Based on Slot Semantic Enhanced Prompt Learning[J]. Computer Engineering, 2023, 49(9):23-31.)
doi: 10.19678/j.issn.1000-3428.0066170
|
| [11] |
张华平, 李林翰, 李春锦. ChatGPT中文性能测评与风险应对[J]. 数据分析与知识发现, 2023, 7(3): 16-25.
|
| [11] |
(Zhang Huaping, Li Linhan, Li Chunjin. ChatGPT Performance Evaluation on Chinese Language and Risk Measures[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 16-25.)
|
| [12] |
Ma Y B, Cao Y X, Hong Y, et al. Large Language Model is Not a Good Few-Shot Information Extractor, But a Good Reranker for Hard Samples![OL]. arXiv Preprint, arXiv: 2303.08559.
|
| [13] |
Polak M P, Morgan D. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering[OL]. arXiv Preprint, arXiv: 2303.05352.
|
| [14] |
Das A, Du X Y, Wang B, et al. Automatic Error Analysis for Document-Level Information Extraction[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. ACL, 2022: 3960-3975.
|
| [15] |
Ding B S, Qin C W, Liu L L, et al. Is GPT-3 a Good Data Annotator?[OL]. arXiv Preprint, arXiv: 2212.10450.
|
| [16] |
Wei X, Cui X Y, Cheng N, et al. Zero-Shot Information Extraction via Chatting with ChatGPT[OL]. arXiv Preprint, arXiv: 2302.10205.
|
| [17] |
Agrawal M, Hegselmann S, Lang H, et al. Large Language Models are Few-Shot Clinical Information Extractors[C]// Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 1998-2022.
|
| [18] |
Gutiérrez B J, McNeal N, Washington C, et al. Thinking About GPT-3 In-Context Learning for Biomedical IE? Think Again[OL]. arXiv Preprint, arXiv: 2203.08410.
|
| [19] |
Luan Y, He L H, Ostendorf M, et al. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. ACL, 2018: 3219-3232.
|
| [20] |
Brack A, D’Souza J, Hoppe A, et al. Domain-Independent Extraction of Scientific Concepts from Research Articles[C]// Proceedings of the 42nd European Conference on Information Retrieval Research. Springer, 2020: 251-266.
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
