Please wait a minute...
Data Analysis and Knowledge Discovery  2023, Vol. 7 Issue (3): 131-141    DOI: 10.11925/infotech.2096-3467.2022.0392
Current Issue | Archive | Adv Search |
Identifying Named Entities of Adverse Drug Reaction with Adversarial Transfer Learning
Han Pu1,2(),Zhong Yule1,Lu Haojie1,Ma Shiwen1
1School of Management, Nanjing University of Posts & Telecommunications, Nanjing 210003, China
2Jiangsu Provincial Key Laboratory of Data Engineering and Knowledge Service, Nanjing 210023, China
Download: PDF (1079 KB)   HTML ( 20 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper proposes an entity recognition model for adverse drug reactions based on adversarial transfer learning, ATL-BCA, aiming to address the problem of non-standard entity representations and insignificant boundaries in online health communities. [Methods] Firstly, we generated the external semantic feature vectors fused with the online medical domain knowledge with Word2Vec. Secondly, based on the transfer learning, we utilized the shared and private BiLSTM to extract the shared boundary information and private features for entity recognition and word segmentation tasks. Next, we used the multi-head attention mechanism to capture the overall sentence dependency and used adversarial training to filter the private information of the word segmentation task. This helped us eliminate the influence of redundant features on the entity recognition task. Finally, we predicted the label sequence results with the help of CRF constraints. [Results] We used a self-constructed social media adverse drug reaction dataset to examine the proposed model with. The F1 value of the new model reached 91.35%, which is 5.28% and 2.98% higher than Word2Vec-BiLSTM-CRF and BERT-BiLSTM-CRF. [Limitations] We only retrieved the experimental data from Sanjiu Health & Medicine Site, the scale of the constructed dataset is relatively small. [Conclusions] The ATL-BCA model fully utilizes the shared boundary information between entity recognition and word segmentation tasks. It also filters the private features of the word segmentation tasks, effectively improving the entity recognition performance of adverse drug reactions in online health communities.
Key words： Adversarial Transfer Learning      Multi-Head Attention Mechanism      Named Entity Recognition      Adverse Drug Reactions     
Received: 23 April 2022      Published: 13 April 2023
ZTFLH:  TP391  
Fund:National Social Science Fund of China(17CTQ022);Major Project of Philosophy and Social Science Research in Colleges and Universities in Jiangsu Province(2020SJZDA102);National Innovation Training Program for College Students(SZDG2021040)
Corresponding Authors: Han Pu，ORCID:0000-0001-5867-4292，E-mail:hanpu@niupt.edu.cn。   
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Pu Han
Yule Zhong
Haojie Lu
Shiwen Ma

Cite this article:

Han Pu, Zhong Yule, Lu Haojie, Ma Shiwen. Identifying Named Entities of Adverse Drug Reaction with Adversarial Transfer Learning. Data Analysis and Knowledge Discovery, 2023, 7(3): 131-141.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2022.0392     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2023/V7/I3/131

Overall Structure Diagram of ATL-BCA Model
Experimental Process of Adverse Drug Reactions Entity Recognition
示例 BIO标注
效果还行但副作用好大，吃了几天后头晕嗜睡，四肢无
力。		 效/O果/O还/O行/O但/O副/O作/O用/O好/O大/O，/O吃/O了/O几/B-Time天/I-Time后/O头/B-Symptom晕/I-Symptom嗜/I-Symptom睡/I-Symptom，/O四/B-Part肢/I-Part无/B-Symptom力/I-Symptom。/O
Examples of Adverse Drug Reactions Entities Labeling
类别 数量 比例
药品 737 17.70%
适应症 1 194 28.67%
剂量 289 6.93%
时间 464 11.14%
不良反应 1 080 25.93%
部位 401 9.63%
合计 4 165 100%
Dataset Entity Distribution
模型参数名称 说明 参数值
Character Embedding Size 字向量维度 100
Hidden Size of LSTM LSTM隐藏层维度 120
Batch_size 每批数据量的大小 64
Loss Weight Coefficient 损失权重系数 0.06
Dropout 随机断开输入神经元的比例 0.3
Projections Head数量 8
Learning Rate 学习速率 0.001
Optimizer 优化器 Adam
Parameter Settings
模型 P/% R/% F1/%
CRF 81.69 76.32 78.91
BiLSTM 84.15 79.36 81.68
BiLSTM-CRF 86.55 84.88 85.70
Experimental Results of Benchmark Model
模型 P/% R/% F1/%
Word2Vec-BiLSTM-CRF 85.97 86.17 86.07
Word2Vec-BiLSTM-CRF-Attention 89.24 87.81 88.51
BERT-BiLSTM-CRF 88.69 88.06 88.37
ATL-BCA 89.52 90.06 89.79
Experimental Results of ATL-BCA Model and Mainstream Model
外部语料 P/% R/% F1/%
Wiki 90.41 89.21 89.81
百度百科 91.36 89.81 90.58
在线医疗健康 91.18 91.52 91.35
Experimental Results of Introducing Different External Semantic Features
标注方式 P/% R/% F1/%
BIO 91.18 91.52 91.35
BIOES 90.76 91.12 90.93
BMES 89.26 91.40 90.32
Experimental Results of Different Labeling Methods
实例 吃完肚子不舒服开始拉肚子
分词 吃│完│肚子│不│舒服│开始│拉肚子
S│S│B E│S│B E│B E│B M E
BiLSTM-CRF 吃│完│肚子│不│舒服│开始│拉│肚子
O│O│B-Part I-Part│O│O O│O O│O B-Part I-Part
ATL-BCA 吃│完│肚子│不│舒服│开始│拉肚子
O│O│B-Part I-Part│O│O O│O O│B-Symptom I-Symptom I-Symptom
Example of ATL-BCA Model
[1] Nikfarjam A, Sarker A, O’Connor K, et al. Pharmacovigilance from Social Media: Mining Adverse Drug Reaction Mentions Using Sequence Labeling with Word Embedding Cluster Features[J]. Journal of the American Medical Informatics Association, 2015, 22(3): 671-681.
doi: 10.1093/jamia/ocu041 pmid: 25755127
[2] 柳鹏程, 王佳域, 陈锦敏, 等. 欧美药物警戒政策研究及对我国的启示[J]. 中国药物警戒, 2020, 17(12): 877-882.
[2] ( Liu Pengcheng, Wang Jiayu, Chen Jinmin, et al. Pharmacovigilance Laws in Europe and America and Their Implications for China[J]. Chinese Journal of Pharmacovigilance, 2020, 17(12): 877-882.)
[3] 张亚飞, 于琦, 王于心, 等. 基于药物论坛中潜在不良反应与适应症的知识发现体系构建[J]. 中华医学图书情报杂志, 2020, 29(7): 38-43.
[3] ( Zhang Yafei, Yu Qi, Wang Yuxin, et al. Establishment of Knowledge Discovery System for Potential Adverse Reactions of Drugs and Indications for Their Use in Drug Forum[J]. Chinese Journal of Medical Library and Information Science, 2020, 29(7): 38-43.)
[4] 范馨月, 崔雷. 基于文本挖掘的药物副作用知识发现研究[J]. 数据分析与知识发现, 2018, 2(3): 79-86.
[4] ( Fan Xinyue, Cui Lei. Using Text Mining to Discover Drug Side Effects: Case Study of PubMed[J]. Data Analysis and Knowledge Discovery, 2018, 2(3): 79-86.)
[5] Leaman R, Wojtulewicz L, Sullivan R, et al. Towards Internet-Age Pharmacovigilance: Extracting Adverse Drug Reactions from User Posts to Health-Related Social Networks[C]// Proceedings of the 2010 Workshop on Biomedical Natural Language Processing. ACM, 2010: 117-125.
[6] 刘浏, 王东波. 命名实体识别研究综述[J]. 情报学报, 2018, 37(3): 329-340.
[6] ( Liu Liu, Wang Dongbo. A Review on Named Entity Recognition[J]. Journal of the China Society for Scientific and Technical Information, 2018, 37(3): 329-340.)
[7] 庄福振, 罗平, 何清, 等. 迁移学习研究进展[J]. 软件学报, 2015, 26(1): 26-39.
[7] ( Zhuang Fuzhen, Luo Ping, He Qing, et al. Survey on Transfer Learning Research[J]. Journal of Software, 2015, 26(1): 26-39.)
[8] 杨飘, 董文永. 基于BERT嵌入的中文命名实体识别方法[J]. 计算机工程, 2020, 46(4): 40-45.
[8] ( Yang Piao, Dong Wenyong. Chinese Named Entity Recognition Method Based on BERT Embedding[J]. Computer Engineering, 2020, 46(4): 40-45.)
[9] Luo W C, Yang F. An Empirical Study of Automatic Chinese Word Segmentation for Spoken Language Understanding and Named Entity Recognition[C]// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. ACL, 2016: 238-248.
[10] Peng N Y, Dredze M. Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning[OL]. arXiv Preprint, arXiv: 1603.00786.
[11] Denton E, Chintala S, Szlam A, et al. Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks[OL]. arXiv Preprint, arXiv: 1506.05751.
[12] Tzeng E, Hoffman J, Saenko K, et al. Adversarial Discriminative Domain Adaptation[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2962-2971.
[13] Gurulingappa H, Mateen-Rajput A, Toldo L. Extraction of Potential Adverse Drug Events from Medical Case Reports[J]. Journal of Biomedical Semantics, 2012, 3(1): 15.
doi: 10.1186/2041-1480-3-15 pmid: 23256479
[14] Xu J, Lee H J, Ji Z C, et al. UTH_CCB System for Adverse Drug Reaction Extraction from Drug Labels at TAC-ADR 2017[C]// Proceedings of the 2017 Text Analysis Conference. 2017.
[15] Narayanan S, Mannam K, Rajan S P, et al. Evaluation of Transfer Learning for Adverse Drug Event and Medication Entity Extraction[C]// Proceedings of the 3rd Clinical Natural Language Processing Workshop. 2020: 55-64.
[16] Florez E, Precioso F, Riveill M, et al. Named Entity Recognition Using Neural Networks for Clinical Notes[C]// Proceedings of the 2018 International Workshop on Medication and Adverse Drug Event Detection. PMLR, 2018: 7-15.
[17] Kang N, Singh B, Bui C, et al. Knowledge-Based Extraction of Adverse Drug Events from Biomedical Text[J]. BMC Bioinformatics, 2014, 15: 64.
doi: 10.1186/1471-2105-15-64 pmid: 24593054
[18] Dandala B, Joopudi V, Devarakonda M. Adverse Drug Events Detection in Clinical Notes by Jointly Modeling Entities and Relations Using Neural Networks[J]. Drug Safety, 2019, 42(1): 135-146.
doi: 10.1007/s40264-018-0764-x pmid: 30649738
[19] Chen Y, Zhou C, Li T, et al. Named Entity Recognition from Chinese Adverse Drug Event Reports with Lexical Feature Based BiLSTM-CRF and Tri-Training[J]. Journal of Biomedical Informatics, 2019, 96: 103252.
doi: 10.1016/j.jbi.2019.103252
[20] Christopoulou F, Tran T T, Sahu S K, et al. Adverse Drug Events and Medication Relation Extraction in Electronic Health Records with Ensemble Deep Learning Methods[J]. Journal of the American Medical Informatics Association, 2020, 27(1): 39-46.
doi: 10.1093/jamia/ocz101 pmid: 31390003
[21] Wu H, Ji J T, Tian H M, et al. Chinese-Named Entity Recognition from Adverse Drug Event Records: Radical Embedding-Combined Dynamic Embedding-Based BERT in a Bidirectional Long Short-Term Conditional Random Field (Bi-LSTM-CRF) Model[J]. JMIR Medical Informatics, 2021, 9(12): e26407.
doi: 10.2196/26407
[22] Cocos A, Fiks A G, Masino A J. Deep Learning for Pharmacovigilance: Recurrent Neural Network Architectures for Labeling Adverse Drug Reactions in Twitter Posts[J]. Journal of the American Medical Informatics Association, 2017, 24(4): 813-821.
doi: 10.1093/jamia/ocw180 pmid: 28339747
[23] 魏巍, 郑杜. 融合统计学习和语义过滤的ADR信号抽取模型构建研究[J]. 图书情报工作, 2018, 62(5): 115-124.
doi: 10.13266/j.issn.0252-3116.2018.05.013
[23] ( Wei Wei, Zheng Du. The Study of Adverse Drug Reaction Signal Extraction Framework Based on the Integrated Statistical Learning and Semantic Filter[J]. Library and Information Service, 2018, 62(5): 115-124.)
doi: 10.13266/j.issn.0252-3116.2018.05.013
[24] 牟冬梅, 琚沅红, 戴文浩, 等. 虚拟健康社区文本数据知识发现策略与模型[J]. 图书情报工作, 2018, 62(5): 125-131.
doi: 10.13266/j.issn.0252-3116.2018.05.014
[24] ( Mu Dongmei, Ju Yuanhong, Dai Wenhao, et al. Knowledge Discovery Strategy and Model of Virtual Health Community Text Data[J]. Library and Information Service, 2018, 62(5): 125-131.)
doi: 10.13266/j.issn.0252-3116.2018.05.014
[25] Metke-Jimenez A, Karimi S. Concept Extraction to Identify Adverse Drug Reactions in Medical Forums: A Comparison of Algorithms[OL]. arXiv Preprint, arXiv: 1504.06936.
[26] Yates A, Goharian N, Frieder O. Extracting Adverse Drug Reactions from Social Media[C]// Proceedings of the 29th AAAI Conference on Artificial Intelligence. ACM, 2015: 2460-2467.
[27] Tang B Z, Hu J L, Wang X L, et al. Recognizing Continuous and Discontinuous Adverse Drug Reaction Mentions from Social Media Using LSTM-CRF[J]. Wireless Communications and Mobile Computing, 2018, 2018: 1-8.
[28] Stanovsky G, Gruhl D, Mendes P. Recognizing Mentions of Adverse Drug Reaction in Social Media Using Knowledge-Infused Recurrent Models[C]// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. 2017: 142-151.
[29] Lehmann J, Isele R, Jakob M, et al. DBpedia—a Large-Scale, Multilingual Knowledge Base Extracted from Wikipedia[J]. Semantic Web, 2015, 6(2): 167-195.
doi: 10.3233/SW-140134
[30] Asghari M, Sierra-Sosa D, Elmaghraby A S. BINER: A Low-Cost Biomedical Named Entity Recognition[J]. Information Sciences, 2022, 602: 184-200.
doi: 10.1016/j.ins.2022.04.037
[31] 朱笑笑, 杨尊琦, 刘婧. 基于Bi-LSTM和CRF的药品不良反应抽取模型构建[J]. 数据分析与知识发现, 2019, 3(2): 90-97.
[31] ( Zhu Xiaoxiao, Yang Zunqi, Liu Jing. Construction of an Adverse Drug Reaction Extraction Model Based on Bi-LSTM and CRF[J]. Data Analysis and Knowledge Discovery, 2019, 3(2): 90-97.)
[32] Ruder S, Peters M E, Swayamdipta S, et al. Transfer Learning in Natural Language Processing[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. 2019: 15-18.
[33] Yadav S, Ekbal A, Saha S, et al. A Unified Multi-Task Adversarial Learning Framework for Pharmacovigilance Mining[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 5234-5245.
[34] 薛振宇, 线岩团, 余正涛, 等. 融合词典与对抗迁移的越南语事件实体识别[J]. 计算机工程, 2022, 48(3): 107-114.
[34] ( Xue Zhenyu, Xian Yantuan, Yu Zhengtao, et al. Vietnamese Event Entity Recognition Combining Dictionary and Adversarial Transfer[J]. Computer Engineering, 2022, 48(3): 107-114.)
[35] 董哲, 邵若琦, 陈玉梁, 等. 基于BERT和对抗训练的食品领域命名实体识别[J]. 计算机科学, 2021, 48(5): 247-253.
doi: 10.11896/jsjkx.200800181
[35] ( Dong Zhe, Shao Ruoqi, Chen Yuliang, et al. Named Entity Recognition in Food Field Based on BERT and Adversarial Training[J]. Computer Science, 2021, 48(5): 247-253.)
doi: 10.11896/jsjkx.200800181
[36] Cao P F, Chen Y B, Liu K, et al. Adversarial Transfer Learning for Chinese Named Entity Recognition with Self-Attention Mechanism[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 182-192.
[37] Bekoulis G, Deleu J, Demeester T, et al. Adversarial Training for Multi-Context Joint Entity and Relation Extraction[OL]. arXiv Preprint, arXiv: 1808.06876.
[38] Zhang T, Lin H, Ren Y, et al. Identifying Adverse Drug Reaction Entities from Social Media with Adversarial Transfer Learning Model[J]. Neurocomputing, 2021, 453: 254-262.
doi: 10.1016/j.neucom.2021.05.007
[39] Kang K M, Tian S W, Yu L. Named Entity Recognition of Local Adverse Drug Reactions in Xinjiang Based on Transfer Learning[J]. Journal of Intelligent & Fuzzy Systems, 2021, 40(5): 8899-8914.
[40] Cornegruta S, Bakewell R, Withey S, et al. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks[C]// Proceedings of the 7th International Workshop on Health Text Mining and Information Analysis. 2016: 17-27.
[41] 朱张莉, 饶元, 吴渊, 等. 注意力机制在深度学习中的研究进展[J]. 中文信息学报, 2019, 33(6): 1-11.
[41] ( Zhu Zhangli, Rao Yuan, Wu Yuan, et al. Research Progress of Attention Mechanism in Deep Learning[J]. Journal of Chinese Information Processing, 2019, 33(6): 1-11.)
[42] 石磊, 王毅, 成颖, 等. 自然语言处理中的注意力机制研究综述[J]. 数据分析与知识发现, 2020, 4(5): 1-14.
[42] ( Shi Lei, Wang Yi, Cheng Ying, et al. Review of Attention Mechanism in Natural Language Processing[J]. Data Analysis and Knowledge Discovery, 2020, 4(5): 1-14.)
[43] Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM, 2017: 6000-6010.
[44] Sutton C, McCallum A. An Introduction to Conditional Random Fields for Relational Learning[A]// Koller D. Introduction to Statistical Relational Learning[M]. Cambridge: MIT Press, 2006.
[45] Carlezon W A, Béguin C, Knoll A T, et al. Kappa-Opioid Ligands in the Study and Treatment of Mood Disorders[J]. Pharmacology & Therapeutics, 2009, 123(3): 334-343.
[46] Subramanyam K K, Sangeetha S. Deep Contextualized Medical Concept Normalization in Social Media Text[J]. Procedia Computer Science, 2020, 171: 1353-1362.
doi: 10.1016/j.procs.2020.04.145
[1] Pei Wei, Sun Shuifa, Li Xiaolong, Lu Ji, Yang Liu, Wu Yirong. Medical Named Entity Recognition with Domain Knowledge[J]. 数据分析与知识发现, 2023, 7(3): 142-154.
[2] Duan Yufeng, He Guoxiu. Analysis of Neural Network Modules for Named Entity Recognition of Chinese Medical Texts[J]. 数据分析与知识发现, 2023, 7(2): 26-37.
[3] Zhao Ruijie, Tong Xinyu, Liu Xiaohua, Lu Yonghe. Entity Recognition and Labeling for Medical Literature Based on Neural Network[J]. 数据分析与知识发现, 2022, 6(9): 100-112.
[4] Hu Jiming, Qian Wei, Wen Peng, Lv Xiaoguang. Text Semantic Representation with Structure-Function and Entity Recognition: Case Study of Medical Records[J]. 数据分析与知识发现, 2022, 6(8): 110-121.
[5] Zhang Yunqiu, Wang Yang, Li Bocheng. Identifying Named Entities of Chinese Electronic Medical Records Based on RoBERTa-wwm Dynamic Fusion Model[J]. 数据分析与知识发现, 2022, 6(2/3): 242-250.
[6] Yu Chuanming, Lin Hongjun, Zhang Zhengang. Joint Extraction Model for Entities and Events with Multi-task Deep Learning[J]. 数据分析与知识发现, 2022, 6(2/3): 117-128.
[7] Zhang Fangcong, Qin Qiuli, Jiang Yong, Zhuang Runtao. Named Entity Recognition for Chinese EMR with RoBERTa-WWM-BiLSTM-CRF[J]. 数据分析与知识发现, 2022, 6(2/3): 251-262.
[8] Fan Tao, Wang Hao, Zhang Wei, Li Xiaomin. Extracting Entities from Intangible Cultural Heritage Texts Based on Machine Reading Comprehension[J]. 数据分析与知识发现, 2022, 6(12): 70-79.
[9] Xu Chenfei, Ye Haiying, Bao Ping. Automatic Recognition of Produce Entities from Local Chronicles with Deep Learning[J]. 数据分析与知识发现, 2020, 4(8): 86-97.
[10] Gao Yuan,Shi Yuanlei,Zhang Lei,Cao Tianyi,Feng Jun. Reconstructing Tour Routes Based on Travel Notes[J]. 数据分析与知识发现, 2020, 4(2/3): 165-172.
[11] Ma Jianxia,Yuan Hui,Jiang Xiang. Extracting Name Entities from Ecological Restoration Literature with Bi-LSTM+CRF[J]. 数据分析与知识发现, 2020, 4(2/3): 78-88.
[12] Liu Jingru,Song Yang,Jia Rui,Zhang Yipeng,Luo Yong,Ma Jingdong. A BiLSTM-CRF Model for Protected Health Information in Chinese[J]. 数据分析与知识发现, 2020, 4(10): 124-133.
[13] Han Huang,Hongyu Wang,Xiaoguang Wang. Automatic Recognizing Legal Terminologies with Active Learning and Conditional Random Field Model[J]. 数据分析与知识发现, 2019, 3(6): 66-74.
[14] Xiaoxiao Zhu,Zunqi Yang,Jing Liu. Construction of an Adverse Drug Reaction Extraction Model Based on Bi-LSTM and CRF[J]. 数据分析与知识发现, 2019, 3(2): 90-97.
[15] Meishan Chen,Chenxi Xia. Identifying Entities of Online Questions from Cancer Patients Based on Transfer Learning[J]. 数据分析与知识发现, 2019, 3(12): 61-69.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn