[Objective] This study aims at developing a new argumentative zoning method based on deep learning language representation model to achieve better performance. [Methods] We adopted a pre-trained deep learning language representation model BERT, and improved model input with sentence position feature to conduct transfer learning on training data from biochemistry journals. The learned sentence representations were then fed into neural network classifier to achieve argumentative zoning classification. [Results] The experiment indicated that for the eleven-class task, the method achieved significant improvement for most classes. The accuracy reached 81.3%, improved by 29.7% compared to the best performance from previous studies. For the seven core classes, the model achieved an accuracy of 85.5%. [Limitations] Due to limitation on experiment environment, our refined model was trained based on pre-trained parameters, which could limit the potential for classification performance. [Conclusions] The proposed method showed significant improvement compared to shallow machine learning schema or original BERT model, and was able to avoid tedious work of feature engineering. The method is independent of language, hence also suitable for research articles in Chinese language.
王末,崔运鹏,陈丽,李欢. 基于深度学习的学术论文语步结构分类方法研究*[J]. 数据分析与知识发现, 2020, 4(6): 60-68.
Wang Mo,Cui Yunpeng,Chen Li,Li Huan. A Deep Learning-based Method of Argumentative Zoning for Research Articles. Data Analysis and Knowledge Discovery, 2020, 4(6): 60-68.
Liakata M, Saha S, Dobnik S, et al. Automatic Recognition of Conceptualization Zones in Scientific Articles and Two Life Science Applications[J]. Bioinformatics, 2012,28(7):991-1000.
Teufel S, Moens M. Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status[J]. Computational Linguistics, 2002,28(4):409-445.
( Wang Lifei, Liu Xia. Constructing a Model for the Automatic Identification of Move Structure in English Research Article Abstracts[J]. Technology Enhanced Foreign Language Education, 2017(2):45-50, 64.)
Guo Y, Korhonen A, Poibeau T. A Weakly-Supervised Approach to Argumentative Zoning of Scientific Documents [C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2011: 273-283.
( Meng Yu, Wu Xingquan. Writing Paradigm of Plasma Physics SCI Journal Articles from the Perspective of Move Analysis Theory[J]. Journal of University of Shanghai for Science and Technology(Social Science) , 2018,40(3):201-206.)
Teufel S, Carletta J, Moens M. An Annotation Scheme for Discourse-Level Argumentation in Research Articles [C]//Proceedings of the 9th Conference on European Chapter of the Association for Computational Linguistics. 1999: 110-117.
( Xu Fang. A Survey on English Academic Paper Genre Studies[J]. Journal of Southeast University (Philosphy and Social Science), 2013,15(5):128-133.)
Nasar Z, Jaffry S W, Malik M K. Information Extraction from Scientific Articles: A Survey[J]. Scientometrics, 2018,117(3):1931-1990.
Gupta S, Manning C D. Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers [C]//Proceedings of the 5th International Joint Conference on Natural Language Processing. 2011: 1-9.
Houngbo H, Mercer R E. Method Mention Extraction from Scientific Research Papers [C]//Proceedings of COLING 2012. 2012:1211-1222.
Ruch P, Boyer C, Chichester C, et al. Using Argumentation to Extract Key Sentences from Biomedical Abstracts[J]. International Journal of Medical Informatics, 2007,76(3):195-200.
Lakhanpal S, Gupta A, Agrawal R. Towards Extracting Domains from Research Publications [C]// Proceedings of MAICS 2015. 2015:117-120.
Lin J, Karakos D, Demner-Fushman D, et al. Generative Content Models for Structural Analysis of Medical Abstracts [C]//Proceedings of the HLT-NAACL BioNLP Workshop on Linking Natural Language and Biology. 2006: 65-72.
Wu J C, Chang Y C, Liou H C, et al. Computational Analysis of Move Structures in Academic Abstracts [C]//Proceedings of the COLING/ACL on Interactive Presentation Sessions. 2006: 41-44.
Hirohata K, Okazaki N, Ananiadou S, et al. Identifying Sections in Scientific Abstracts Using Conditional Random Fields [C]//Proceedings of the 3rd International Joint Conference on Natural Language Processing: Volume-I. 2008: 381-388.
Lin S, Ng J P, Pradhan S, et al. Extracting Formulaic and Free Text Clinical Research Articles Metadata Using Conditional Random Fields [C]//Proceedings of the NAACL HLT 2010 2nd Louhi Workshop on Text and Data Mining of Health Documents. 2010: 90-95.
Ronzano F, Saggion H. Dr. Inventor Framework: Extracting Structured Information from Scientific Publications [C]//Proceedings of the International Conference on Discovery Science. 2015: 209-220.
Anthony L, Lashkia G V. Mover: A Machine Learning Tool to Assist in the Reading and Writing of Technical Papers[J]. IEEE Transactions on Professional Communication, 2003,46(3):185-193.
Guo Y, Korhonen A, Liakata M, et al. Identifying the Information Structure of Scientific Abstracts: An Investigation of Three Different Schemes [C]//Proceedings of the 2010 Workshop on Biomedical Natural Language Processing. 2010: 99-107.
Dayrell C, Candido Jr A, Lima G, et al. Rhetorical Move Detection in English Abstracts: Multi-label Sentence Classifiers and Their Annotated Corpora [C]//Proceedings of the 8th International Conference on Language Resources and Evaluation. 2012: 1604-1609.
Liu H. Automatic Argumentative-Zoning Using Word2vec [OL]. arXiv Preprint, arXiv: 1703. 10152.
Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality [C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013: 3111-3119.
Pennington J, Socher R, Manning C. Glove: Global Vectors for Word Representation [C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1532-1543.
Devlin J, Chang M-W, Lee K, et al. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding[OL]. arXiv Preprint, arXiv: 1810. 04805.
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[OL]. arXiv Preprint, arXiv: 1706. 03762.
Beltagy I, Lo K, Cohan A. SciBERT: A Pretrained Language Model for Scientific Text [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 3606-3611.