Please wait a minute...
Data Analysis and Knowledge Discovery  2023, Vol. 7 Issue (9): 136-145    DOI: 10.11925/infotech.2096-3467.2022.0812
Current Issue | Archive | Adv Search |
Recognizing Chinese Medical Literature Entities Based on Multi-Task and Transfer Learning
Han Pu1,2(),Gu Liang1,Ye Dongyu1,Chen Wenqi1
1School of Management, Nanjing University of Posts & Telecommunications, Nanjing 210003, China
2Jiangsu Provincial Key Laboratory of Data Engineering and Knowledge Service, Nanjing 210023, China
Download: PDF (1241 KB)   HTML ( 20 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper uses transfer learning and multi-task learning to solve the problems of cold start and boundary in Chinese medical literature entity recognition, and further improve the recognition accuracy. [Methods] Firstly, we constructed a hybrid deep learning BERT-BiLSTM-IDCNN-CRF medical literature entity recognition model. Secondly, based on transfer learning, the medical semantic features were enriched through instance, model and feature transfer. Thirdly, we constructed a coarse-grained three-classification task through multi-task learning to assist the main task in utilizing the entity boundary information effectively. Finally, we introduced the self-attention mechanism and highway network to capture global information, optimize deep network training and establish the TLMT-BBIC-HS model. [Results] The model had an F1 value of 92.98% on the Chinese diabetes medical literature dataset, which is 15.99% and 16.44% higher than the benchmark models BERT-BiLSTM-CRF and BERT-IDCNN-CRF. [Limitations] The domain suitability of this model needs to be verified. [Conclusions] The TLMT-BBIC-HS model can transfer and share medical knowledge, which is more suitable for Chinese medical Literature entity recognition. It could effectively extract medical information and construct knowledge graphs and question answering systems.
Key words： Medical Literature Entity Extraction      Multi-Task Learning      Transfer Learning      Attention Mechanism      Highway Network     
Received: 03 August 2022      Published: 21 March 2023
ZTFLH:  G350  
  TP391  
Fund:The National Social Science Fund of China(22BTQ096)
Corresponding Authors: Han Pu， ORCID：0000-0001-5867-4292，E-mail： hanpu@njupt.edu.cn。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Pu Han
Liang Gu
Dongyu Ye
Wenqi Chen

Cite this article:

Han Pu, Gu Liang, Ye Dongyu, Chen Wenqi. Recognizing Chinese Medical Literature Entities Based on Multi-Task and Transfer Learning. Data Analysis and Knowledge Discovery, 2023, 7(9): 136-145.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0812     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2023/V7/I9/136

原句子 任务 标注规则
乏力等症状消失。 中文医学文献实体识别 B-Symptom I-Symptom
O O O O O O
粗粒度三分类 B I O O O O O O
Multi-Task Annotation Method
Research Framework
Embedding Vector Training of BERT-Base-Chinese
TLMT-BBIC-HS Model for Entity Recognition of Chinese Medical Literature
Original and Annotated Documents in Dataset
模型参数 BiLSTM/IDCNN BERT 预训练
向量维度 100 768 768
隐藏层单元个数 128 768 768
Batch Size 32 16 32
Epoch 100 100 100
Clip 5 - -
学习速率 0.001 0.001 2e-5
最大序列长度 - 128 128
Dropout机制 0.5
优化器 Adam
Experimental Parameter Setting
实验内容 模型 P （%） R （%） F1 （%）
基准模型 BERT-BiLSTM-CRF 76.12 76.45 76.29
BERT-IDCNN-CRF 76.33 75.35 75.84
BERT-BiLSTM-IDCNN-CRF 77.37 77.25 77.31
迁移学习 +模型迁移 M-BBIC 82.00 81.08 81.54
+实例迁移 I-BBIC 82.14 82.20 82.17
+特征迁移 TL-BBIC 89.75 90.18 89.96
多任务 TLMT-BBIC 90.69 91.26 90.97
自注意力 TLMT-BBIC-S 91.80 91.55 91.68
Highway网络 TLMT-BBIC-HS 92.00 92.56 92.28
Comparison of Experimental Results
F1 Values for 15 Categories of Entity Recognition
模型 P（%） R（%） F1（%）
双层BiLSTM-CRF（何春辉等[26]） - - 72.89
Fusion Multi Feature-CNN-
BiLSTM-CRF（Shang等[27]）		 79.47 76.72 78.07
B-SABCN（Deng等[28]） 78.29 78.03 78.16
RoBERTa-CRF（Wang等[29]） 91.18 91.36 91.27
Research in Chinese Diabetes Literature Dataset
[1] 赵旸, 张智雄, 刘欢, 等. 基于BERT模型的中文医学文献分类研究[J]. 数据分析与知识发现, 2020, 4(8): 41-49.
[1] (Zhao Yang, Zhang Zhixiong, Liu Huan, et al. Classification of Chinese Medical Literature with BERT Model[J]. Data Analysis and Knowledge Discovery, 2020, 4(8): 41-49.)
[2] 李跃艳, 王昊, 邓三鸿, 等. 面向事件本体的医学文本语义关联化研究[J]. 情报学报, 2022, 41(5): 497-511.
[2] (Li Yueyan, Wang Hao, Deng Sanhong, et al. Research on Semantic Relevance of Medical Text Oriented to Event Ontology[J]. Journal of the China Society for Scientific and Technical Information, 2022, 41(5): 497-511.)
[3] Coden A, Savova G, Sominsky I, et al. Automatically Extracting Cancer Disease Characteristics from Pathology Reports into a Disease Knowledge Representation Model[J]. Journal of Biomedical Informatics, 2009, 42(5): 937-949.
doi: 10.1016/j.jbi.2008.12.005 pmid: 19135551
[4] Jiang M, Chen Y K, Liu M, et al. A Study of Machine-Learning-Based Approaches to Extract Clinical Entities and Their Assertions from Discharge Summaries[J]. Journal of the American Medical Informatics Association, 2011, 18(5): 601-606.
doi: 10.1136/amiajnl-2011-000163 pmid: 21508414
[5] Liu Z J, Yang M, Wang X L, et al. Entity Recognition from Clinical Texts via Recurrent Neural Network[J]. BMC Medical Informatics and Decision Making, 2017, 17(Suppl 2): Article No.67.
[6] Gajendran S, Manjula D, Sugumaran V. Character Level and Word Level Embedding with Bidirectional LSTM—Dynamic Recurrent Neural Network for Biomedical Named Entity Recognition from Literature[J]. Journal of Biomedical Informatics, 2020, 112: Article No.103609.
[7] Li X Y, Zhang H, Zhou X H. Chinese Clinical Named Entity Recognition with Variant Neural Structures Based on BERT Methods[J]. Journal of Biomedical Informatics, 2020, 107: Article No.103422.
[8] 吕江海, 杜军平, 周南, 等. 基于膨胀卷积迭代与注意力机制的实体名识别方法[J]. 计算机工程, 2021, 47(1): 58-65.
doi: 10.19678/j.issn.1000-3428.0055986
[8] (Lü Jianghai, Du Junping, Zhou Nan, et al. Entity Name Recognition Method Based on Dilated Convolutional Iterative and Attention Mechanism[J]. Computer Engineering, 2021, 47(1): 58-65.)
doi: 10.19678/j.issn.1000-3428.0055986
[9] Giorgi J M, Bader G D. Transfer Learning for Biomedical Named Entity Recognition with Neural Networks[J]. Bioinformatics, 2018, 34(23): 4087-4094.
doi: 10.1093/bioinformatics/bty449 pmid: 29868832
[10] Smetanin S, Komarov M. Deep Transfer Learning Baselines for Sentiment Analysis in Russian[J]. Information Processing & Management, 2021, 58(3): Article No.102484.
[11] Fu W L, Xue B, Gao X Y, et al. Transductive Transfer Learning Based Genetic Programming for Balanced and Unbalanced Document Classification Using Different Types of Features[J]. Applied Soft Computing, 2021, 103: Article No.107172.
[12] Mignone P, Pio G, D’Elia D, et al. Exploiting Transfer Learning for the Reconstruction of the Human Gene Regulatory Network[J]. Bioinformatics, 2020, 36(5): 1553-1561.
doi: 10.1093/bioinformatics/btz781 pmid: 31608946
[13] 熊欣, 王昊, 邓三鸿. 面向方志知识图谱的术语抽取模型迁移学习研究[J]. 情报理论与实践, 2021, 44(4): 176-184.
[13] (Xiong Xin, Wang Hao, Deng Sanhong. A Study on Term Extraction Model with Transfer Learning for Knowledge Graph of Local Chronicles[J]. Information Studies: Theory & Application, 2021, 44(4): 176-184.)
[14] 韩普, 张展鹏, 张伟. 基于多任务学习和多态语义特征的中文疾病名称归一化研究[J]. 情报学报, 2021, 40(11): 1234-1244.
[14] (Han Pu, Zhang Zhanpeng, Zhang Wei. Chinese Disease Name Normalization Based on Multi-Task Learning and Polymorphic Semantic Features[J]. Journal of the China Society for Scientific and Technical Information, 2021, 40(11): 1234-1244.)
[15] Crichton G, Pyysalo S, Chiu B, et al. A Neural Network Multi-Task Learning Approach to Biomedical Named Entity Recognition[J]. BMC Bioinformatics, 2017, 18(1): 1-14.
[16] Wu C C, Luo G, Guo C, et al. An Attention-Based Multi-Task Model for Named Entity Recognition and Intent Analysis of Chinese Online Medical Questions[J]. Journal of Biomedical Informatics, 2020, 108: Article No.103511.
[17] Aguilar G, Maharjan S, López-Monroy A P, et al. A Multi-Task Approach for Named Entity Recognition in Social Media Data[OL]. arXiv Preprint, arXiv: 1906.04135.
[18] Wang D S, Fan H J, Liu J F. Learning with Joint Cross-Document Information via Multi-Task Learning for Named Entity Recognition[J]. Information Sciences, 2021, 579: 454-467.
doi: 10.1016/j.ins.2021.08.015
[19] Srivastava R K, Greff K, Schmidhuber J. Highway Networks[OL]. arXiv Preprint, arXiv: 1505.00387.
[20] Zuo M, Zhang Y. Dataset-Aware Multi-Task Learning Approaches for Biomedical Named Entity Recognition[J]. Bioinformatics, 2020, 36(15): 4331-4338.
doi: 10.1093/bioinformatics/btaa515 pmid: 32415963
[21] Liu L Y, Shang J B, Ren X A, et al. Empower Sequence Labeling with Task-Aware Neural Language Model[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018, 32(1): 5253-5260.
[22] Narayanan S, Achan P, Rangan P V, et al. Unified Concept and Assertion Detection Using Contextual Multi-Task Learning in a Clinical Decision Support System[J]. Journal of Biomedical Informatics, 2021, 122: Article No.103898.
[23] 王东波, 刘畅, 朱子赫, 等. SikuBERT与SikuRoBERTa:面向数字人文的《四库全书》预训练模型构建及应用研究[J]. 图书馆论坛, 2022, 42(6): 31-43.
[23] (Wang Dongbo, Liu Chang, Zhu Zihe, et al. Construction and Application of Pre-Trained Models of Siku Quanshu in Orientation to Digital Humanities[J]. Library Tribune, 2022, 42(6): 31-43.)
[24] Aliyun. A Labeled Chinese Dataset for Diabetes[EB/OL]. [2022-06-28]. https://tianchi.aliyun.com/competition/entrance/231687/information.
[25] Aya Mohamed Abdelaty Elkased. 面向生物医学文献的基于BioBERT的药品相互作用抽取增强模型[D]. 哈尔滨: 哈尔滨工业大学, 2021.
[25] (Aya Mohamed Abdelaty Elkased. Enhanced Drug-Drug Interaction Extraction Model from Biomedical Text Using BioBERT[D]. Harbin: Harbin Institute of Technology, 2021.)
[26] 何春辉, 王梦贤, 何小波. 基于双层Bi-LSTM-CRF模型的糖尿病领域命名实体识别[J]. 邵阳学院学报(自然科学版), 2020, 17(1): 21-26.
[26] He Chunhui, Wang Mengxian, He Xiaobo. Named Entity Recognition in the Field of Diabetes Based on Double-layer Bi-LSTM-CRF Model[J]. Journal of Shaoyang University (Natural Science Edition), 2020, 17(1): 21-26.)
[27] Shang F J, Ran C F. An Entity Recognition Model Based on Deep Learning Fusion of Text Feature[J]. Information Processing & Management, 2022, 59(2): Article No.102841.
[28] Deng J F, Cheng L L, Wang Z W. Self-Attention-Based BiGRU and Capsule Network for Named Entity Recognition[OL]. arXiv Preprint, arXiv: 2002.00735.
[29] Wang Y, Sun Y N, Ma Z C, et al. Named Entity Recognition in Chinese Medical Literature Using Pretraining Models[J]. Scientific Programming, 2020, 2020: Article No.8812754.
[1] He Li,Yang Meihua,Liu Luyao. Detecting Events with SPO Semantic and Syntactic Information[J]. 数据分析与知识发现, 2023, 7(9): 114-124.
[2] Han Pu, Zhong Yule, Lu Haojie, Ma Shiwen. Identifying Named Entities of Adverse Drug Reaction with Adversarial Transfer Learning[J]. 数据分析与知识发现, 2023, 7(3): 131-141.
[3] Zhou Ning, Zhong Na, Jin Gaoya, Liu Bin. Chinese Text Sentiment Analysis Based on Dual Channel Attention Network with Hybrid Word Embedding[J]. 数据分析与知识发现, 2023, 7(3): 58-68.
[4] Su Mingxing, Wu Houyue, Li Jian, Huang Ju, Zhang Shunxiang. AEMIA:Extracting Commodity Attributes Based on Multi-level Interactive Attention Mechanism[J]. 数据分析与知识发现, 2023, 7(2): 108-118.
[5] Wang jinzheng, Yang Ying, Yu Bengong. Classifying Customer Complaints Based on Multi-head Co-attention Mechanism[J]. 数据分析与知识发现, 2023, 7(1): 128-137.
[6] Peng Cheng, Zhang Chunxia, Zhang Xin, Guo Jingtao, Niu Zhendong. Reasoning Model for Temporal Knowledge Graph Based on Entity Multiple Unit Coding[J]. 数据分析与知识发现, 2023, 7(1): 138-149.
[7] Zhao Ruijie, Tong Xinyu, Liu Xiaohua, Lu Yonghe. Entity Recognition and Labeling for Medical Literature Based on Neural Network[J]. 数据分析与知识发现, 2022, 6(9): 100-112.
[8] Chen Yuanyuan, Ma Jing. Detecting Multimodal Sarcasm Based on SC-Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(9): 40-51.
[9] Tang Jiao, Zhang Lisheng, Sang Chunyan. News Recommendation with Latent Topic Distribution and Long and Short-Term User Representations[J]. 数据分析与知识发现, 2022, 6(9): 52-64.
[10] Zhao Pengwu, Li Zhiyi, Lin Xiaoqi. Identifying Relationship of Chinese Characters with Attention Mechanism and Convolutional Neural Network[J]. 数据分析与知识发现, 2022, 6(8): 41-51.
[11] Zhang Ruoqi, Shen Jianfang, Chen Pinghua. Session Sequence Recommendation with GNN, Bi-GRU and Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(6): 46-54.
[12] Ye Han,Sun Haichun,Li Xin,Jiao Kainan. Classification Model for Long Texts with Attention Mechanism and Sentence Vector Compression[J]. 数据分析与知识发现, 2022, 6(6): 84-94.
[13] Zhou Zeyu, Wang Hao, Zhang Xiaoqin, Tao Fao, Ren Qiutong. Classification Model for Chinese Traditional Embroidery Based on Xception-TD[J]. 数据分析与知识发现, 2022, 6(2/3): 338-347.
[14] Guo Hangcheng, He Yanqing, Lan Tian, Wu Zhenfeng, Dong Cheng. Identifying Moves from Scientific Abstracts Based on Paragraph-BERT-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 298-307.
[15] Xu Yuemei, Fan Zuwei, Cao Han. A Multi-Task Text Classification Model Based on Label Embedding of Attention Mechanism[J]. 数据分析与知识发现, 2022, 6(2/3): 105-116.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn