【目的】 梳理深度学习模型在术语识别中的研究现状与面临挑战。【文献范围】 在中国知网和Web of Science中,分别以主题=“术语识别”+“术语抽取”、主题=“(extract terms OR term recognition OR technology detection OR relation classification) AND deep learning AND ner”作为检索式进行检索,共筛选73篇文献进行述评。【方法】 对基于深度学习的术语识别一般框架、模型的选择及各模型的优缺点、未来发展趋势进行综述。【结果】 基于深度学习的术语识别方法可划分为使用单一神经网络模型、复合神经网络模型和结合深度学习模型的术语识别三大类。从方法使用来看,以BiLSTM-CRF为核心及延伸的模型是术语识别的主流方法;BERT及BERT的优化模型是近年来的研究热点;在特定领域倾向于使用多任务模型代替神经网络模型;迁移学习以及主动学习的应用成为新的研究方向。【局限】 仅对已有研究的不同模型及训练结果进行结构化分析,缺少对不同模型在同一数据集上的训练效果对比,待未来进一步研究。【结论】 基于深度学习的术语识别未来可在术语标注模式、融合术语的多维特征、小数据集或零数据集的术语识别技术、跨领域模型泛化、结果可解释性和完善评价方法等方面深入研究。
[Objective] This paper reviews the current developments and challenges facing term recognition studies based on deep learning. [Coverage] We searched the 中国知网 and the Web of Science using queries of 主题=“术语识别”+“术语抽取”, and subject = “(extract terms OR term recognition OR technology detection OR relation classification) AND deep learning AND ner”. A total of 73 articles were retrieved. [Methods] We reviewed these studies on the general framework of deep learning-based term recognition, model selection, advantages and disadvantages of various models, and future development trends. [Results] Deep learning-based term recognition methods can be categorized into three major types: single neural network models, composite neural network models, and models combining deep learning. BiLSTM-CRF models are the mainstream method for term recognition, with BERT and its optimized models being recent research hotspots. In specific domains, multi-task models are preferred over neural network models, and the application of transfer learning and active learning has become a new research direction. [Limitations] We only conducted a structured analysis of different models and training results of existing studies, lacking a comparison of training effects of different models on the same dataset, requiring further research in the future. [Conclusion] Future research in deep learning-based term recognition should focus on term annotation patterns, integrating multidimensional features of terms, term recognition techniques for small or zero datasets, cross-domain model generalization, interpretability of results, and improvement of evaluation methods.
阮光册, 钟静涵, 张祎笛. 基于深度学习的术语识别研究综述[J]. 数据分析与知识发现, 2024, 8(4): 64-75.
Ruan Guangce, Zhong Jinghan, Zhang Yidi. Review of Term Recognition Studies Based on Deep Learning. Data Analysis and Knowledge Discovery, 2024, 8(4): 64-75.
Zhang Z Q, Gao J, Ciravegna F. SemRe-Rank: Improving Automatic Term Extraction By Incorporating Semantic Relatedness With Personalised PageRank[J]. ACM Transactions on Knowledge Discovery from Data, 2018, 12(5): 57.
[2]
Zadeh B Q, Handschuh S. Evaluation of Technology Term Recognition with Random Indexing[C]// Proceedings of the 9th International Conference on Language Resources and Evaluation. 2014: 4027-4032.
(Liu Jianhua, Zhang Zhixiong, Xu Jian, et al. Automatic Term Recognition——An Important Method for Text Mining on Scientific Literature[J]. New Technology of Library and Information Service, 2008, 24(8): 12-17.)
(Ding Junjun, Zheng Yanning, Hua Bolin. Extraction of Academic Concept Attribute Based on Rules[J]. Information Studies: Theory & Application, 2011, 34(12): 10-14, 33.)
(Liu Shengqi, Zhu Donghua. Automatic Term Extraction Based on TValue[J]. Journal of the China Society for Scientific and Technical Information, 2013, 32(11): 1164-1173.)
(Li Lishuang, Wang Yiwen, Huang Degen. Term Extraction Based on Information Entropy and Word Frequency Distribution Variety[J]. Journal of Chinese Information Processing, 2015, 29(1): 82-87.)
[7]
Lossio-Ventura J A, Jonquet C, Roche M, et al. Biomedical Terminology Extraction: A New Combination of Statistical and Web Mining Approaches[C]// Proceedings of JADT 2014: 12es Journées Internationales d'Analyse statistique des Données Textuelles. 2014: 421-432.
[8]
Frantzi K, Ananiadou S, Mima H. Automatic Recognition of Multi-word Terms: The C-Value/Nc-Value Method[J]. International Journal on Digital Libraries, 2000, 3(2): 115-130.
[9]
Astrakhantsev N. Automatic Term Acquisition from Domain-Specific Text Collection by Using Wikipedia[J]. Proceedings of the Institute for System Programming of RAS, 2014, 26(4): 7-20.
[10]
Lossio-Ventura J A, Jonquet C, Roche M, et al. Yet Another Ranking Function for Automatic Multiword Term Extraction[C]// Proceedings of the 9th International Conference on Natural Language Processing. 2014: 52-64.
(Pang Beibei, Gou Juanqiong, Zhang Lei, et al. Research on Polysemy Recognition Based on Topic Analysis and Usage Context Fusion[J]. Information Studies: Theory & Application, 2022, 45(1): 177-186.)
doi: 10.16353/j.cnki.1000-7490.2022.01.023
[12]
Nugumanova A, Akhmed-Zaki D, Mansurova M, et al. NMF-Based Approach to Automatic Term Extraction[J]. Expert Systems with Applications, 2022, 199: 117179.
(Zhang Hua, Ye Na, Zhou Qiaoli, et al. Classification Strategy Based Term Recognition Systems Combination[J]. Journal of Chinese Computer Systems, 2015, 36(2): 385-390.)
(Qu Peng, Wang Huilin. Term Extraction in Patent Information Services[J]. Information Science, 2015, 33(9): 66-71.)
[15]
Wang R, Liu W, McDonald C. Featureless Domain-Specific Term Extraction with Minimal Labelled Data[C]// Proceedings of the Australasian Language Technology Association Workshop 2016. 2016: 103-112.
[16]
Terryn A R, Hoste V, Lefever E. Tagging Terms in Text: A Supervised Sequential Labelling Approach to Automatic Term Extraction[J]. Terminology, 2022, 28(1): 157-189.
(Hu Yamin, Wu Xiaoyan, Chen Fang. Review of Technology Term Recognition Studies Based on Machine Learning[J]. Data Analysis and Knowledge Discovery, 2022, 6(2/3): 7-17.)
(Zhu Qingsong, Leng Fuhai. Existing Problems and Developing Trends of Automatic Term Recognition[J]. Library and Information Service, 2012, 56(18): 104-109.)
[21]
Li J, Sun A, Han J, et al. A Survey on Deep Learning for Named Entity Recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 34(1): 50-70.
[22]
Song B S, Li F, Liu Y S, et al. Deep Learning Methods for Biomedical Named Entity Recognition: A Survey and Qualitative Comparison[J]. Briefings in Bioinformatics, 2021, 22(6): bbab282.
(Zhang Wei, Wang Hao, Deng Sanhong, et al. Sentiment Term Extraction and Application of Chinese Ancient Poetry Text for Digital Humanities[J]. Journal of Library Science in China, 2021, 47(4): 113-131.)
[24]
Li L S, Jin L K, Huang D G. Exploring Recurrent Neural Networks to Detect Named Entities from Biomedical Text[C]// Proceedings of Chinese National Conference on Chinese Computational Linguistics, International Symposium on Natural Language Processing Based on Naturally Annotated Big Data. 2015: 279-290.
[25]
Khosla K, Jones R, Bowman N. Featureless Deep Learning Methods for Automated Key-Term Extraction[EB/OL]. [2023-02-08]. https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/custom/15848334.pdf.
[26]
Kucza M, Niehues J, Zenkel T, et al. Term Extraction via Neural Sequence Labeling a Comparative Evaluation of Strategies Using Recurrent Neural Networks[C]// Proceedings of the Interspeech 2018. 2018: 2072-2076.
(Zhao Hong, Wang Fang. A Deep Learning Model and Self-Training Algorithm for Theoretical Terms Extraction[J]. Journal of the China Society for Scientific and Technical Information, 2018, 37(9): 923-938.)
(Xiao Lianjie, Meng Tao, Wang Wei, et al. Entity Recognition of Intelligence Method Based on Deep Learning: Taking Area of Security Intelligence for Example[J]. Data Analysis and Knowledge Discovery, 2019, 3(10): 20-28.)
[29]
Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
[30]
Venugopalan M, Gupta D. An Enhanced Guided LDA Model Augmented with BERT Based Semantic Strength for Aspect Term Extraction in Sentiment Analysis[J]. Knowledge-Based Systems, 2022, 246: 108668.
[31]
Gao Y Z, Yuan Y. Feature-Less End-to-End Nested Term Extraction[C]// Proceedings of CCF International Conference on Natural Language Processing and Chinese Computing. 2019: 607-616.
[32]
Huang Z H, Xu W, Yu K. Bidirectional LSTM-CRF Models for Sequence Tagging[OL]. arXiv Preprint, arXiv: 1508.01991.
[33]
Lample G, Ballesteros M, Subramanian S, et al. Neural Architectures for Named Entity Recognition[C]// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2016: 260-270.
[34]
Luo H S, Li T R, Liu B, et al. Improving Aspect Term Extraction with Bidirectional Dependency Tree Representation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 27(7): 1201-1212.
[35]
Lyu X R, Lyu X Q, Sun F S, et al. Patent Domain Terminology Extraction Based on Multi-feature Fusion and BiLSTM-CRF Model[C]// Proceedings of FSDM 2018. 2018: 495-500.
(Wang Hao, Deng Sanhong, Su Xinning, et al. A Study on Chinese Terminology Recognition of Theory and Method from Information Science: Based on Deep Learning[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(8): 817-828.)
(Xu Fei, Ye Wenhao, Song Yinghua. Part-of-Speech Automated Annotation of Food Safety Events Based on BiLSTM-CRF[J]. Journal of the China Society for Scientific and Technical Information, 2018, 37(12): 1204-1211.)
[38]
Li P L, Yuan Z M, Tu W B, et al. Medical Knowledge Extraction and Analysis from Electronic Medical Records Using Deep Learning[J]. Chinese Medical Sciences Journal, 2019, 34 (2): 133-139.
(Feng Luanluan, Li Junhui, Li Peifeng, et al. Technology and Terminology Detection Oriented National Defense Science[J]. Computer Science, 2019, 46(12): 231-236.)
doi: 10.11896/jsjkx.190300069
[40]
Phongwattana T, Chan J H. Development of Biomedical Corpus Enlargement Platform Using BERT for Bio-entity Recognition[C]// Proceedings of International Conference on Neural Information Processing. 2019: 454-463.
(Liu Liu, Qin Tianyun, Wang Dongbo. Automatic Extraction of Traditional Music Terms of Intangible Cultural Heritage[J]. Data Analysis and Knowledge Discovery, 2020, 4(12): 68-75.)
[42]
Liu W J, Zhou P, Zhao Z, et al. K-BERT: Enabling Language Representation with Knowledge Graph[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020: 2901-2908.
(Hou Weitao, Ji Donghong. Research on Clinic Event Recognition Based Bi-LSTM[J]. Application Research of Computers, 2018, 35(7): 1974-1977.)
[44]
Joshi M, Chen D Q, Liu Y H, et al. SpanBERT: Improving Pre-training by Representing and Predicting Spans[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 64-77.
(Li Minghao, Liu Zhong, Yao Yaozhe. LSTM-CRF Based Symptom Term Recognition on Traditional Chinese Medical Case[J]. Journal of Computer Applications, 2018, 38(S2): 42-46.)
[46]
Desimone R. Neural Mechanisms of Selective Visual Attention[J]. Annual Review of Neuroscience, 1995, 18: 193-222.
pmid: 7605061
(Wu Jun, Cheng Yao, Hao Han, et al. Automatic Extraction of Chinese Terminology Based on BERT Embedding and BiLSTM-CRF Model[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(4): 409-418.)
(Ren Qiutong, Wang Hao, Xiong Xin, et al. Extracting Drama Terms with GCN Long-Distance Constrain[J]. Data Analysis and Knowledge Discovery, 2021, 5(12): 123-136.)
[49]
Han X W, Xu L Z, Qiao F. CNN-BiLSTM-CRF Model for Term Extraction in Chinese Corpus[C]// Proceedings of International Conference on Web Information Systems and Applications. 2018: 267-274.
[50]
Peng L F, Gao D, Bai Y J. A Study on Standardization of Security Evaluation Information for Chemical Processes Based on Deep Learning[J]. Processes, 2021, 9(5): 832.
(Ma Jianhong, Zhang Yamei, Yao Shuang, et al. Terminology Extraction for New Energy Vehicle Based on BiLSTM_Attention_CRF Model[J]. Application Research of Computers, 2019, 36(5): 1385-1389, 1395.)
[52]
李焕. 基于深度学习与主动学习的中医术语识别研究[D]. 北京: 北京工业大学, 2019.
[52]
(Li Huan. Research of Chinese Medicine Terminology Recognition Based on Deep Learning and Active Learning[D]. Beijing: Beijing University of Technology, 2019.)
[53]
Zhang X H, Zhang Y Y, Zhang Q, et al. Extracting Comprehensive Clinical Information for Breast Cancer Using Deep Learning Methods[J]. International Journal of Medical Informatics, 2019, 132: 103985.
(Zhai Yujia, Tian Jingwen, Zhao Yue. Algorithm Term Extraction and Innovation Evolution Path Construction Based on BERT-BiLSTM-CRF Model[J]. Information Science, 2022, 40(4): 71-78.)
(Li Zhenzhen, Zhong Yongheng, Wang Hui, et al. Research on Domain Term Extraction Method Based on Deep Learning and Statistical Information[J]. Frontiers of Data & Computing, 2022, 4(2): 87-98.)
(Xiong Xin, Wang Hao, Deng Sanhong. A Study on Term Extraction Model with Transfer Learning for Knowledge Graph of Local Chronicles[J]. Information Studies: Theory & Application, 2021, 44(4): 176-184.)
(Zheng Guangmin, Yi Tianyuan, Tang Dongxin, et al. The Chinese Ethnic Medicine Knowledge Extraction Based on BERT-BiLSTM-CRF Model[J]. Journal of Wuhan University(Natural Science Edition), 2021, 67(5): 393-402.)
(Zhang Zefeng, Mao Cunli, Yu Zhengtao, et al. Sensitive Judicial Public Opinion Information Recognition with the Domain Terminology Dictionary[J]. Journal of Chinese Information Processing, 2022, 36(9): 76-83, 92.)
[59]
Caruana R. Multitask Learning[J]. Machine Learning, 1997, 28(1): 41-75.
[60]
Li T, Hu Y, Ju A, et al. Adversarial Active Learning for Named Entity Recognition in Cybersecurity[J]. Computers, Materials & Continua, 2021, 66(1):407-420.
[61]
Crichton G, Pyysalo S, Chiu B, et al. A Neural Network Multi-Task Learning Approach to Biomedical Named Entity Recognition[J]. BMC Bioinformatics, 2017, 18(1): 368.
doi: 10.1186/s12859-017-1776-8
pmid: 28810903
[62]
Hong S K, Lee J G. DTranNER: Biomedical Named Entity Recognition with Deep Learning-Based Label-Label Transition Model[J]. BMC Bioinformatics, 2020, 21: 53.
doi: 10.1186/s12859-020-3393-1
pmid: 32046638
[63]
Greenberg N, Bansal T, Verga P, et al. Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 2824-2829.
[64]
Peng Y F, Chen Q Y, Lu Z Y. An Empirical Study of Multi-task Learning on BERT for Biomedical Text Mining[C]// Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing. 2020: 205-214.
[65]
Zhuang F Z, Luo P, Xiong H, et al. Cross-Domain Learning from Multiple Sources: A Consensus Regularization Perspective[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(12): 1664-1678.
[66]
Giorgi J M, Bader G D. Transfer Learning for Biomedical Named Entity Recognition with Neural Networks[J]. Bioinformatics, 2018, 34(23): 4087-4094.
doi: 10.1093/bioinformatics/bty449
pmid: 29868832
[67]
le Guillarme N, Thuiller W. TaxoNERD: Deep Neural Models for the Recognition of Taxonomic Entities in the Ecological and Evolutionary Literature[J]. Methods in Ecology and Evolution, 2022, 13(3): 625-641.
(Liu Yufei, Yin Li, Zhang Kai, et al. Deep Transfer Learning for Technical Term Extraction—A Case Study in Computer Numerical Control System[J]. Journal of Intelligence, 2019, 38(10): 168-175.)
(Liu Kang, Qian Xu, Wang Ziqiang. Survey on Active Learning Algorithms[J]. Computer Engineering and Applications, 2012, 48(34): 1-4, 22.)
[70]
Liang G Q, Leung C W K. Improving Model Generalization: A Chinese Named Entity Recognition Case Study[C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing(Volume 2:Short Papers). 2021: 992-997.
(Zhang Han, Guo Yuanbo, Li Tao. Domain Named Entity Recognition Combining GAN and BiLSTM-Attention-CRF[J]. Journal of Computer Research and Development, 2019, 56(9): 1851-1858.)
(Wang Kunfeng, Gou Chao, Duan Yanjie, et al. Generative Adversarial Networks: The State of the Art and Beyond[J]. Acta Automatica Sinica, 2017, 43(3): 321-332.)
[73]
Fan F L, Xiong J J, Li M Z, et al. On Interpretability of Artificial Neural Networks: A Survey[J]. IEEE Transactions on Radiation and Plasma Medical Sciences, 2021, 5(6): 741-760.