1School of Sociology, Soochow University, Suzhou 215123, China 2School of Economics & Management, Nanjing University of Science and Technology, Nanjing 210094, China
[Objective] This paper aims to use a large language model for entity recognition tasks of academic papers. [Methods] We utilized ChatGPT, a large language model, as an entity recognition tool, a pseudo-label generation tool, and a training set generation tool. Then, we analyzed ChatGPT’s performance, price, and time for the tasks. [Results] The F1 of the ChatGPT-based method in all three perspectives is higher than that of the neural network baseline model trained with a small dataset. For example, the F1 from the perspective of entity recognition was 21.4% higher than the model trained by manually annotating 10 abstracts. The ChatGPT-based methods had stable performance on academic paper datasets in different disciplines. [Limitations] We only examined the new method with English academic paper abstract datasets. More research is needed to examine it with the Chinese datasets. [Conclusions] ChatGPT can identify entities from academic paper abstracts with little manually annotated data. The recognition results need to be further filtered to be applied to downstream tasks.
Heffernan K, Teufel S. Identifying Problems and Solutions in Scientific Text[J]. Scientometrics, 2018, 116(2): 1367-1382.
doi: 10.1007/s11192-018-2718-6
pmid: 30147202
[2]
Kovačević A, Konjović Z, Milosavljević B, et al. Mining Methodologies from NLP Publications: A Case Study in Automatic Terminology Recognition[J]. Computer Speech & Language, 2012, 26(2): 105-126.
[3]
Luo Z R, Lu W, He J G, et al. Combination of Research Questions and Methods: A New Measurement of Scientific Novelty[J]. Journal of Informetrics, 2022, 16(2): Article No. 101282.
(Ma Feicheng, Zhang Shuai. The Development of Emerging Interdisciplines in Library and Information Science in China[J]. Journal of Library Science in China, 2023, 49(2): 4-14.)
[5]
Wadden D, Wennberg U, Luan Y, et al. Entity, Relation, and Event Extraction with Contextualized Span Representations[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. ACL, 2019: 5784-5789.
[6]
Zhong Z X, Chen D Q. A Frustratingly Easy Approach for Entity and Relation Extraction[C]// Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2021: 50-61.
[7]
Zhang H H, Ren F L. BERTatDE at SemEval-2020 Task 6: Extracting Term-Definition Pairs in Free Text Using Pre-Trained Model[C]// Proceedings of the 14th Workshop on Semantic Evaluation. 2020: 690-696.
[8]
Ding N, Chen Y L, Han X, et al. Prompt-Learning for Fine-Grained Entity Typing[OL]. arXiv Preprint, arXiv: 2108.10604.
[9]
Kan Z G, Feng L H, Yin Z Y, et al. A Unified Generative Framework Based on Prompt Learning for Various Information Extraction Tasks[OL]. arXiv Preprint, arXiv: 2209.11570.
(Li Hongpeng, Ma Bo, Yang Yating, et al. Document-Level Event Extraction Based on Slot Semantic Enhanced Prompt Learning[J]. Computer Engineering, 2023, 49(9):23-31.)
doi: 10.19678/j.issn.1000-3428.0066170
(Zhang Huaping, Li Linhan, Li Chunjin. ChatGPT Performance Evaluation on Chinese Language and Risk Measures[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 16-25.)
[12]
Ma Y B, Cao Y X, Hong Y, et al. Large Language Model is Not a Good Few-Shot Information Extractor, But a Good Reranker for Hard Samples![OL]. arXiv Preprint, arXiv: 2303.08559.
[13]
Polak M P, Morgan D. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering[OL]. arXiv Preprint, arXiv: 2303.05352.
[14]
Das A, Du X Y, Wang B, et al. Automatic Error Analysis for Document-Level Information Extraction[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. ACL, 2022: 3960-3975.
[15]
Ding B S, Qin C W, Liu L L, et al. Is GPT-3 a Good Data Annotator?[OL]. arXiv Preprint, arXiv: 2212.10450.
[16]
Wei X, Cui X Y, Cheng N, et al. Zero-Shot Information Extraction via Chatting with ChatGPT[OL]. arXiv Preprint, arXiv: 2302.10205.
[17]
Agrawal M, Hegselmann S, Lang H, et al. Large Language Models are Few-Shot Clinical Information Extractors[C]// Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 1998-2022.
[18]
Gutiérrez B J, McNeal N, Washington C, et al. Thinking About GPT-3 In-Context Learning for Biomedical IE? Think Again[OL]. arXiv Preprint, arXiv: 2203.08410.
[19]
Luan Y, He L H, Ostendorf M, et al. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. ACL, 2018: 3219-3232.
[20]
Brack A, D’Souza J, Hoppe A, et al. Domain-Independent Extraction of Scientific Concepts from Research Articles[C]// Proceedings of the 42nd European Conference on Information Retrieval Research. Springer, 2020: 251-266.
